xxl Posted July 7, 2021 Author Share Posted July 7, 2021 (edited) I add BitBuster - pretty fast, nice compression and very short procedure (even without the use of ZP). BitBuster: 1774 https://www.cpcwiki.eu/forum/programming/dumb-question-15839-bitbuster-1-2/msg31362/#msg31362 In a few days I will complete the sources on the site - https://xxl.atari.pl/bitbuster-decompressor/ example of use: debitbust6502.obx Edited July 7, 2021 by xxl 4 Quote Link to comment Share on other sites More sharing options...
rensoup Posted September 16, 2021 Share Posted September 16, 2021 Having run out of disk space for PoP (using APL), I thought it was time to have another look at the available compressors: test file: 39,689 PoPCore.bin 23,972 PoPCore.bin.shrinkler (-d -p -9) 24,761 PoPCore.bin.zx0 (-f) 25,131 PoPCore.bin.apl 25,150 PoPCore.bin.deflate 25,201 popcore.bin.exomizer (-raw) 25,688 popcore.bin.packfire (-t) 26,712 PoPCore.bin.pucrunch Shrinkler is better on paper, but its decoder is twice bigger than zx0 (+-450 bytes?) and takes 3KB of temp storage. the dealbreaker: it takes 1 minute to decompress while others are done in seconds! ZX0 systematically beats APL by a few percent on every file I've tried. It decrunches very quickly as well (using Elmer's version which takes 192 bytes and a little bit of ZP). ZX0 seems to be the best packer/depacker available right now for the A8. PoP just wouldn't fit on disk without it Many thanks @xxl and @elmer for this very useful thread and converting those depackers. ( and Einar Saukas for the compressor ) 7 Quote Link to comment Share on other sites More sharing options...
xxl Posted September 17, 2021 Author Share Posted September 17, 2021 (edited) it's true, I can see a bright future for this decompressor for Atari and more and more people are seeing it. One cannot forget about the considerable speed of decopression and the ease of use. I know that the new version of DC Montezuma's uses this decompression on the fly for levels without any slowdowns, especially since it is a version with a minimalist use of ZP (https://xxl.atari.pl/zx0-decompressor/) On the last SV2020+1 one of the best coders did tests and was also surprised ? MadPascal also has a module for this decompression ... there is also a special boot loader for files packed with this compressor, it can also be used when loading packed exe programs in .CAS files ... it is worth a try Edited September 17, 2021 by xxl 8 Quote Link to comment Share on other sites More sharing options...
xxl Posted October 9, 2021 Author Share Posted October 9, 2021 the ZX0 mutation appeared, the decompressor is slower, but the efficiency of the ZX5 takes the lead anyway. only the Shrinkler is higher (of the available Atari decompressors). code translated from Z80 with minor optimizations. https://github.com/einar-saukas/ZX5 The source of the decompressor for 6502 will be available tonight. ZX5: 1532 zx5-gfx.obx 5 Quote Link to comment Share on other sites More sharing options...
xxl Posted October 9, 2021 Author Share Posted October 9, 2021 and done: http://xxl.atari.pl/zx5-decompressor/ 7 Quote Link to comment Share on other sites More sharing options...
xxl Posted October 10, 2021 Author Share Posted October 10, 2021 Einar Saukas has changed the ZX0 format and published a new version of the v2 compressor - the new compressor is not compatible with the old decompressors!!! https://github.com/einar-saukas/ZX0 below I publish the new decompressor: http://xxl.atari.pl/zx0-decompressor/ if you used ZX0 v1 be sure to upgrade!!! 3 Quote Link to comment Share on other sites More sharing options...
ggn Posted October 11, 2021 Share Posted October 11, 2021 Hello @xxl, This is an interesting thread, thanks for creating it. One small suggestion I have for improving it is to try a larger set of files than a single one, preferably with different content. Otherwise I think the results can be misleading. Case in point: For a ST game I'm working on I tried a few packers mentioned in this thread just to see which produces the best packing ratio, and here's my results so far. Test 1: 9 files of graphics (tiles) in ST interleaved bitplane format. Original combined size: 144k LSZA1: 31k LZSA2: 27k ZX5: 27k Cranker: 35k Exomizer (data mode): 29k ZX0 v1: 29k ZX0 v2: 29k ARJBETA mode 4: 28k ARJBETA mode 7: 24k Of course this is half of the story, if I break the values down on a file-by-file basis I see that on some files zx0/zx5 takes the lead while on others zx0/zx5 perform much poorer than arjbeta. But on cumulative size arjbeta mode 7 takes the lead. Unfortunately this mode is not very practical for 8 bit machines as it requires a massive 13-14k table for depacking! Test 2: The levels file Original file size: 15607 bytes LSZA1: 4147 LZSA2: 3785 ZX5: 3550 Cranker: 4345 Exomizer (data mode): 3469 Exomizer (level mode): 3486 ZX0 v1: 3627 ZX0 v2: 3627 ARJBETA mode 4: 3807 ARJBETA mode 7: 3426 I hope the above illustrates how differnt packers on different data sets can yield much different result than a single file! (As to what packer I will probably use? I'll probably go for LZSA 2, in my case it is a good compromise between depacking speeds (very fast) and pack ratio (one of the better ones)) P.S. for completion's sake, here's what arjbeta manages with your reference conan.gfx: ARJBETA mode 4: 1841 ARJBETA mode 7: 1657 As to where to get arjbeta - well, there hasn't been an official distro to my knowledge, but you could download the packer for windows from http://onorisoft.free.fr/jag/bjlpak.zip (which contains the mode 7 unpacker in 68k asm), and I can attach the mode 4 depacker here (again 68k asm) mode4new.s 1 1 Quote Link to comment Share on other sites More sharing options...
xxl Posted October 11, 2021 Author Share Posted October 11, 2021 (edited) That's right, we should always test different methods on our data. I'm saving the results - thanks. 1 hour ago, ggn said: ARJBETA mode 4: 1841 ARJBETA mode 7: 1657 in 68k world there are still PackFire and Shrinkler - both in two versions and both better than ARJBET, even lightning fast and practically no requirements PackFire Tiny.... check on your own data. please note that the results can - and indeed do - vary considerably for data for 8-bit machines - above all, the size of the input file matters. As for ZX5 and ZX0, actually in most cases ZX5 is stronger but there are examples that it is worse: RiverRaid.ROM: 8192 LZ4 : 7414 BitBuster : 6618 aPLib : 6366 ZX5 : 6334 ZX0 : 6313 PackFire Tiny: 6284 Deflate : 6192 Shrinkler NP : 6020 Landscape.xex: 30653 LZ4 : 16934 BitBuster : 14591 PackFire Tiny: 13823 aPLib : 13679 ZX0 : 13530 Deflate : 13486 ZX5 : 13459 Shrinkler NP : 12764 Edited October 11, 2021 by xxl Quote Link to comment Share on other sites More sharing options...
xxl Posted October 11, 2021 Author Share Posted October 11, 2021 2 hours ago, ggn said: ARJBETA mode 4: 1841 would you please guide me i try to reproduce your results but i get this: arjbeta.exe a -m4 conan.arj conan.gfx file size: 1944 Quote Link to comment Share on other sites More sharing options...
cRc_atari Posted October 11, 2021 Share Posted October 11, 2021 Betty's issues intro.dat (gfx+msx+code) : 9˙086 zopfli -v --deflate --i10000 : 4˙957 exomizer raw : 4˙870 bbpack : 5˙186 packfire -t : 4˙881 zx5 : 4˙805 Quote Link to comment Share on other sites More sharing options...
ggn Posted October 11, 2021 Share Posted October 11, 2021 5 hours ago, xxl said: in 68k world there are still PackFire and Shrinkler - both in two versions and both better than ARJBET, even lightning fast and practically no requirements PackFire Tiny.... check on your own data. I'll do that for sure, it's just that both packers take unreasonable amounts of time for my tastes to depack even the small files such as 4k or 8k (well, small for ST/Amiga in any case). Since my project will be loading in game, the depacking time won't be a one-time cost. Waiting for a game for a long time to change graphics data isn't very appealing to me. In any case I'll do some tests so we have an academic comparison. 3 hours ago, xxl said: would you please guide me i try to reproduce your results but i get this: arjbeta.exe a -m4 conan.arj conan.gfx file size: 1944 Certainly, and my mistake for not clarifying this in my post above. The file arjbeta generates is an actual ARJ achive, so it adds a lot of unnecessary header. This can be removed by using the "dumparj" command in the archive I linked. So, dumparj conan.arj conan.gfx conan_packed_with_arj.gfx will generate "conan_packed_with_arj.gfx" which can then be depacked using mode 4/7 routines. 2 Quote Link to comment Share on other sites More sharing options...
ggn Posted October 11, 2021 Share Posted October 11, 2021 Can't edit my post above so let's make another one for PackFire and Shrinkler. Test 1: PackFire (large model): 24k PackFire (tiny model): 42k Shrinkler: 23k Test 2: PackFire (large model): 3183 PackFire (tiny model): 3471 Shrinkler: 3188 So, PackFire seems pretty much equivalent to arjbeta mode7 for the graphics files, and for the levels file both PackFire and Shrinkler beat everything by a large margin. However, like I said in the previous post, I don't think I'll be subjecting people to slow depack speeds :). 1 Quote Link to comment Share on other sites More sharing options...
xxl Posted October 12, 2021 Author Share Posted October 12, 2021 (edited) something small: ZX7 ( https://github.com/antoniovillena/zx7mini ) http://xxl.atari.pl/zx7-decompressor/ ZX7: 1801 - pretty good zx7-gfx.obx Edited October 12, 2021 by xxl 1 Quote Link to comment Share on other sites More sharing options...
xxl Posted October 13, 2021 Author Share Posted October 13, 2021 On 10/11/2021 at 8:07 PM, ggn said: Certainly, and my mistake for not clarifying this in my post above. The file arjbeta generates is an actual ARJ achive, so it adds a lot of unnecessary header. This can be removed by using the "dumparj" command in the archive I linked. So, dumparj conan.arj conan.gfx conan_packed_with_arj.gfx will generate "conan_packed_with_arj.gfx" which can then be depacked using mode 4/7 routines. unfortunately, I am not able to properly unpack the data with this 68k procedure, I suspect I have a wrong stream of packed data. could you please attach the packed conan.gfx data? Quote Link to comment Share on other sites More sharing options...
ggn Posted October 14, 2021 Share Posted October 14, 2021 Okay, I had a go at depacking the files myself and found a small "gotcha" of dumparj: it actually adds a 4 byte header which has the original file size (in big endian). So, in 68k code, the correct way to call the depacking routines is: lea m4_pack,A1 lea m4_unpack,A0 move.l (A1)+,D0 jsr decode_m4 lea m7_unpack,A0 lea m7_pack,A1 move.l (A1)+,D0 bsr unarj So, from the packed buffer we read the unpacked size, advance by 4 bytes, then call the depack routines. Just for posterity I uploaded the packed files as well as the source code for both depackers, including a small addition by me to call the depackers and compare the bytes to the reference unpacked file. Apologies for the confusion! Goes to show I shouldn't be writing hasty posts without testing... conan.m4 conan.m7 UNPACK.S 1 Quote Link to comment Share on other sites More sharing options...
xxl Posted October 17, 2021 Author Share Posted October 17, 2021 On 10/14/2021 at 12:31 PM, ggn said: I uploaded the packed files as well as the source code for both depackers thanks to this procedure (and the mini emulator 68k on 6502 - the decompressor code on this emulator took over 1KB and after optimization to 6502 only about 200 bytes) we read the compressor data format, 6502 decompressor by Konop, Rastan, xxl in general ... the procedure is rather slow, and the compressor itself does not sin with efficiency, but I am placing it as a chronicler unarj4-gfx.obx 1 Quote Link to comment Share on other sites More sharing options...
xxl Posted October 19, 2021 Author Share Posted October 19, 2021 As you can easily see ARJ m4 wastes bits encoding the amount of data to copy from the dictionary, for example to encode length 255 it needs 14 bits. It also has a limitation for the offset to 4 KB - this is already an overkill even for 8bit machines. Source for the decompressor available here: https://xxl.atari.pl/arj4-decompressor/ 2 Quote Link to comment Share on other sites More sharing options...
Estece Posted November 1, 2021 Share Posted November 1, 2021 I will throw a lossy idea for conan.gfx image compression. DXT1 aka S3TC for atari,any one? Quote Link to comment Share on other sites More sharing options...
rensoup Posted November 1, 2021 Share Posted November 1, 2021 35 minutes ago, Estece said: I will throw a lossy idea for conan.gfx image compression. DXT1 aka S3TC for atari,any one? easy: dxt1 is 4bit per pixel so a 2bit per pixel image of 7680 bytes will "compress" down to 15360 bytes ? 2 Quote Link to comment Share on other sites More sharing options...
Estece Posted November 2, 2021 Share Posted November 2, 2021 lousy idea conan4.webp Quote Link to comment Share on other sites More sharing options...
xxl Posted November 4, 2021 Author Share Posted November 4, 2021 Unfortunately this thread will no longer be updated by me, if there is something new it will be directly on the site: https://xxl.atari.pl/ in the menu on the right are procedures to use in your own programs. 1 Quote Link to comment Share on other sites More sharing options...
Xebec Posted December 14, 2021 Share Posted December 14, 2021 Not that I'm expecting it to beat other compressors - but did anyone ever try Scrunch on this test? I remember that being the first compression utility I ever came across.. Quote Link to comment Share on other sites More sharing options...
E474 Posted January 26, 2023 Share Posted January 26, 2023 On 11/4/2021 at 6:07 PM, xxl said: Unfortunately this thread will no longer be updated by me, if there is something new it will be directly on the site: https://xxl.atari.pl/ in the menu on the right are procedures to use in your own programs. Hi @xxl, Thanks for posting the code on the linked web-site. It says copyright at the bottom of the linked web-page, am I free to use the code as I wish, or are there any restrictions, license rules, etc? Quote Link to comment Share on other sites More sharing options...
xxl Posted January 27, 2023 Author Share Posted January 27, 2023 On 1/26/2023 at 2:03 AM, E474 said: Thanks for posting the code on the linked web-site. It says copyright at the bottom of the linked web-page, am I free to use the code as I wish, or are there any restrictions, license rules, etc? for every line of code you have to drink one beer. or not if you don't want to. 1 4 Quote Link to comment Share on other sites More sharing options...
E474 Posted January 27, 2023 Share Posted January 27, 2023 Great, thanks very much. It sounds like it's covered by the Beer Some Days license. 2 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.