Jump to content
IGNORED

Most efficient compression for Atari


xxl

Recommended Posts

I add BitBuster - pretty fast, nice compression and very short procedure (even without the use of ZP).

 

BitBuster: 1774

 

https://www.cpcwiki.eu/forum/programming/dumb-question-15839-bitbuster-1-2/msg31362/#msg31362

 

In a few days I will complete the sources on the site - https://xxl.atari.pl/bitbuster-decompressor/

 

example of use:

debitbust6502.obx

Edited by xxl
  • Like 4
Link to comment
Share on other sites

  • 2 months later...


Having run out of disk space for PoP (using APL), I thought it was time to have another look at the available compressors:

 

test file: 39,689 PoPCore.bin


23,972 PoPCore.bin.shrinkler (-d -p -9)

24,761 PoPCore.bin.zx0 (-f)

25,131 PoPCore.bin.apl

25,150 PoPCore.bin.deflate

25,201 popcore.bin.exomizer (-raw)

25,688 popcore.bin.packfire (-t)

26,712 PoPCore.bin.pucrunch


Shrinkler is better on paper, but its decoder is twice bigger than zx0 (+-450 bytes?) and takes 3KB of temp storage. the dealbreaker: it takes 1 minute to decompress while others are done in seconds!

 

ZX0 systematically beats APL by a few percent on every file I've tried. It decrunches very quickly as well (using Elmer's version which takes 192 bytes and a little bit of ZP).

 

ZX0 seems to be the best packer/depacker available right now for the A8. PoP just wouldn't fit on disk without it

 

Many thanks @xxl and @elmer for this very useful thread and converting those depackers. ( and Einar Saukas for the compressor )

 

  • Like 6
Link to comment
Share on other sites

it's true, I can see a bright future for this decompressor for Atari and more and more people are seeing it. One cannot forget about the considerable speed of decopression and the ease of use.
I know that the new version of DC Montezuma's uses this decompression on the fly for levels without any slowdowns, especially since it is a version with a minimalist use of ZP :) (https://xxl.atari.pl/zx0-decompressor/)
On the last SV2020+1 one of the best coders did tests and was also surprised ?
MadPascal also has a module for this decompression ...

there is also a special boot loader for files packed with this compressor,

it can also be used when loading packed exe programs in .CAS files

... it is worth a try

Edited by xxl
  • Like 8
Link to comment
Share on other sites

  • 3 weeks later...

the ZX0 mutation appeared, the decompressor is slower, but the efficiency of the ZX5 takes the lead anyway. only the Shrinkler is higher (of the available Atari decompressors).

code translated from Z80 with minor optimizations.

https://github.com/einar-saukas/ZX5

 

The source of the decompressor for 6502 will be available tonight.

 

ZX5: 1532 

zx5-gfx.obx

  • Like 5
Link to comment
Share on other sites

Einar Saukas has changed the ZX0 format and published a new version of the v2 compressor - the new compressor is not compatible with the old decompressors!!!

 

https://github.com/einar-saukas/ZX0

 

 

below I publish the new decompressor:

 

http://xxl.atari.pl/zx0-decompressor/

 

if you used ZX0 v1 be sure to upgrade!!!

 

  • Like 3
Link to comment
Share on other sites

Hello @xxl,

 

This is an interesting thread, thanks for creating it. One small suggestion I have for improving it is to try a larger set of files than a single one, preferably with different content. Otherwise I think the results can be misleading.

 

Case in point: For a ST game I'm working on I tried a few packers mentioned in this thread just to see which produces the best packing ratio, and here's my results so far.

 

Test 1: 9 files of graphics (tiles) in ST interleaved bitplane format.

Original combined size: 144k

LSZA1: 31k

LZSA2: 27k

ZX5: 27k

Cranker: 35k

Exomizer (data mode): 29k

ZX0 v1: 29k

ZX0 v2: 29k

ARJBETA mode 4: 28k

ARJBETA mode 7: 24k

 

Of course this is half of the story, if I break the values down on a file-by-file basis I see that on some files zx0/zx5 takes the lead while on others zx0/zx5 perform much poorer than arjbeta. But on cumulative size arjbeta mode 7 takes the lead. Unfortunately this mode is not very practical for 8 bit machines as it requires a massive 13-14k table for depacking!

 

Test 2: The levels file

Original file size: 15607 bytes

LSZA1: 4147

LZSA2: 3785

ZX5: 3550

Cranker: 4345

Exomizer (data mode): 3469

Exomizer (level mode): 3486

ZX0 v1: 3627

ZX0 v2: 3627

ARJBETA mode 4: 3807

ARJBETA mode 7: 3426

 

I hope the above illustrates how differnt packers on different data sets can yield much different result than a single file!

 

(As to what packer I will probably use? I'll probably go for LZSA 2, in my case it is a good compromise between depacking speeds (very fast) and pack ratio (one of the better ones))

 

P.S. for completion's sake, here's what arjbeta manages with your reference conan.gfx:

 

ARJBETA mode 4: 1841

ARJBETA mode 7: 1657

 

As to where to get arjbeta - well, there hasn't been an official distro to my knowledge, but you could download the packer for windows from http://onorisoft.free.fr/jag/bjlpak.zip (which contains the mode 7 unpacker in 68k asm), and I can attach the mode 4 depacker here (again 68k asm)

mode4new.s

  • Like 1
  • Thanks 1
Link to comment
Share on other sites

That's right, we should always test different methods on our data.

I'm saving the results - thanks.

1 hour ago, ggn said:

ARJBETA mode 4: 1841

ARJBETA mode 7: 1657

 

in 68k world there are still PackFire and Shrinkler - both in two versions and both better than ARJBET, even lightning fast and practically no requirements PackFire Tiny.... check on your own data.

please note that the results can - and indeed do - vary considerably for data for 8-bit machines - above all, the size of the input file matters.

 

As for ZX5 and ZX0, actually in most cases ZX5 is stronger but there are examples that it is worse:

 

 

RiverRaid.ROM: 8192

LZ4          : 7414 
BitBuster    : 6618 
aPLib        : 6366
ZX5          : 6334 
ZX0          : 6313 
PackFire Tiny: 6284 
Deflate      : 6192 
Shrinkler NP : 6020 

 
Landscape.xex: 30653

LZ4          : 16934
BitBuster    : 14591 
PackFire Tiny: 13823
aPLib        : 13679
ZX0          : 13530
Deflate      : 13486
ZX5          : 13459
Shrinkler NP : 12764
Edited by xxl
Link to comment
Share on other sites

Betty's issues intro.dat (gfx+msx+code)         : 9˙086

zopfli  -v --deflate --i10000  : 4˙957   

exomizer raw                      : 4˙870
bbpack                                : 5˙186
packfire -t                            : 4˙881
zx5                                     : 4˙805

 

Link to comment
Share on other sites

5 hours ago, xxl said:

in 68k world there are still PackFire and Shrinkler - both in two versions and both better than ARJBET, even lightning fast and practically no requirements PackFire Tiny.... check on your own data.

I'll do that for sure, it's just that both packers take unreasonable amounts of time for my tastes to depack even the small files such as 4k or 8k (well, small for ST/Amiga in any case). Since my project will be loading in game, the depacking time won't be a one-time cost. Waiting for a game for a long time to change graphics data isn't very appealing to me. In any case I'll do some tests so we have an academic comparison.

 

3 hours ago, xxl said:

would you please guide me i try to reproduce your results but i get this:

 

arjbeta.exe a -m4 conan.arj conan.gfx

 

file size: 1944

Certainly, and my mistake for not clarifying this in my post above. The file arjbeta generates is an actual ARJ achive, so it adds a lot of unnecessary header. This can be removed by using the "dumparj" command in the archive I linked. So, 

dumparj conan.arj conan.gfx conan_packed_with_arj.gfx

will generate "conan_packed_with_arj.gfx" which can then be depacked using mode 4/7 routines.

  • Like 2
Link to comment
Share on other sites

Can't edit my post above so let's make another one for PackFire and Shrinkler.

 

Test 1:

 

PackFire (large model): 24k

PackFire (tiny model): 42k

Shrinkler: 23k

 

Test 2:

 

PackFire (large model): 3183

PackFire (tiny model): 3471

Shrinkler: 3188

 

So, PackFire seems pretty much equivalent to arjbeta mode7 for the graphics files, and for the levels file both PackFire and Shrinkler beat everything by a large margin. However, like I said in the previous post, I don't think I'll be subjecting people to slow depack speeds :).

  • Like 1
Link to comment
Share on other sites

On 10/11/2021 at 8:07 PM, ggn said:

Certainly, and my mistake for not clarifying this in my post above. The file arjbeta generates is an actual ARJ achive, so it adds a lot of unnecessary header. This can be removed by using the "dumparj" command in the archive I linked. So, 


dumparj conan.arj conan.gfx conan_packed_with_arj.gfx

will generate "conan_packed_with_arj.gfx" which can then be depacked using mode 4/7 routines.

unfortunately, I am not able to properly unpack the data with this 68k procedure, I suspect I have a wrong stream of packed data.
could you please attach the packed conan.gfx data?

 

Link to comment
Share on other sites

Okay, I had a go at depacking the files myself and found a small "gotcha" of dumparj: it actually adds a 4 byte header which has the original file size (in big endian). So, in 68k code, the correct way to call the depacking routines is:

 

                lea     m4_pack,A1
                lea     m4_unpack,A0
                move.l  (A1)+,D0
                jsr     decode_m4

                lea     m7_unpack,A0
                lea     m7_pack,A1
                move.l  (A1)+,D0
                bsr     unarj

So, from the packed buffer we read the unpacked size, advance by 4 bytes, then call the depack  routines. Just for posterity I uploaded the packed files as well as the source code for both depackers, including a small addition by me to call the depackers and compare the bytes to the reference unpacked file.

 

Apologies for the confusion! Goes to show I shouldn't be writing hasty posts without testing...

conan.m4 conan.m7 UNPACK.S

  • Like 1
Link to comment
Share on other sites

On 10/14/2021 at 12:31 PM, ggn said:

I uploaded the packed files as well as the source code for both depackers

 

thanks to this procedure (and the mini emulator 68k on 6502 - the decompressor code on this emulator took over 1KB and after optimization to 6502 only about 200 bytes) we read the compressor data format, 6502 decompressor by Konop, Rastan, xxl

 

in general ... the procedure is rather slow, and the compressor itself does not sin with efficiency, but I am placing it as a chronicler ;-)

 

unarj4-gfx.obx

  • Like 1
Link to comment
Share on other sites

As you can easily see ARJ m4 wastes bits encoding the amount of data to copy from the dictionary, for example to encode length 255 it needs 14 bits. It also has a limitation for the offset to 4 KB - this is already an overkill even for 8bit machines.

 

Source for the decompressor available here: https://xxl.atari.pl/arj4-decompressor/

  • Like 2
Link to comment
Share on other sites

  • 2 weeks later...
  • 1 month later...
  • 1 year later...
On 11/4/2021 at 6:07 PM, xxl said:

Unfortunately this thread will no longer be updated by me, if there is something new it will be directly on the site: https://xxl.atari.pl/

 

in the menu on the right are procedures to use in your own programs.

 

Hi @xxl,

 

   Thanks for posting the code on the linked web-site. It says copyright at the bottom of the linked web-page, am I free to use the code as I wish, or are there any restrictions, license rules, etc?

  

Link to comment
Share on other sites

On 1/26/2023 at 2:03 AM, E474 said:

Thanks for posting the code on the linked web-site. It says copyright at the bottom of the linked web-page, am I free to use the code as I wish, or are there any restrictions, license rules, etc?

for every line of code you have to drink one beer. or not if you don't want to.

  • Like 1
  • Haha 4
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...