Jump to content

Photo

Inflate/Deflate Has anyone optimized these routines


52 replies to this topic

#1 peteym5 OFFLINE  

peteym5

    Stargunner

  • 1,509 posts
  • Location:Buffalo NY USA

Posted Wed Apr 5, 2017 11:33 AM

Has anyone optimized "Inflate.asm" so it can inflate into RAM faster or get the routine to use less space? I did a few things with it over the years to see if it can inflate any faster. But this program is outside my normal knowledge dealing with compression. Many times I experimented with something to optimized it, can get garbage loaded into RAM. I did see it was originally inefficient with CPU usage, had many extra JSR calls that could be removed. Anybody else do anything with this program?



#2 CharlieChaplin OFFLINE  

CharlieChaplin

    River Patroller

  • 2,411 posts

Posted Wed Apr 5, 2017 12:42 PM

Hmmmm, whenever the inflate routine was used in an A8 program, it was stated later: "inflate routine by FoX", so you most likely have to ask FoX about it... ;-)  His real world name is Piotr Fusik and you can find some code and info at Github:

 

https://github.com/pfusik/zlib6502

https://github.com/p...834d1fe3acebbfa

 

And no, I am not a programmer, so I never do anything with ASM sources (or any other programming sources)...



#3 phaeron OFFLINE  

phaeron

    River Patroller

  • 2,135 posts
  • Location:USA

Posted Wed Apr 5, 2017 9:59 PM

The main reason that decoding Deflate takes so much space is that it uses no less than three Huffman trees per dynamic block: one tree for literal/distance code, a second tree for the length codes, and a third tree to encode the first two (along with run-length encoding!). You can reduce this by making concessions such as hardcoding some or all of the trees and limiting encodings. For instance, restricting distances to 256 bytes would simplify the copy routine. These are difficult to take advantage of without customizing the compressor, however.

 

Before doing that, though, you might want to just check if LZ4 will work; its compression ratio won't be as good, but that may be made up for by a smaller decompressor. Deflate also tends to get poorer compression ratios on small streams due to the relatively large header on each dynamic block, so LZ4 might even get better compression at Atari-scale.


Edited by phaeron, Wed Apr 5, 2017 9:59 PM.


#4 vitoco OFFLINE  

vitoco

    Chopper Commander

  • 169 posts

Posted Thu Apr 6, 2017 5:46 AM

I've designed a LZ4-like decoder/loader, which requires a specially formatted XEX file. It would be used in old and small cartridges. It is an unfinished project as the menu for a multicart is still under development. The encoder was written in perl language using a LZ4 module, and it builds a LZ4X file as required using a standard XEX. Another perl script builds the .CAR image using those LZ4X pre-encoded files (games).

#5 xxl OFFLINE  

xxl

    Dragonstomper

  • 963 posts
  • Location:Rabka-Zdrój /Poland

Posted Thu Apr 6, 2017 6:37 AM

LZ4 for atari: http://xxl.atari.pl/lz4-decompressor/

 



#6 pirx OFFLINE  

pirx

    Moonsweeper

  • 323 posts
  • Location:Poland

Posted Thu Apr 6, 2017 6:47 AM

You may also look at https://bitbucket.or...mizer/wiki/Home-- I found exomizer faster and shorter than deflate/inflate. And compressing a bit better, too. Simply better in every way : ).

 

I guess you know https://github.com/pfusik/zlib6502-- this code is excellent and I would sincerely doubt there is an easy optimisation at all without changing the format as described by phaeron above.

(edit, already mentioned by Charlie : )


Edited by pirx, Thu Apr 6, 2017 6:55 AM.


#7 peteym5 OFFLINE  

peteym5

    Stargunner

  • Topic Starter
  • 1,509 posts
  • Location:Buffalo NY USA

Posted Thu Apr 6, 2017 7:25 AM

Thankyou, I will be doing some bench marking on this modified Inflate routine today. I had ported it to MADS ASSEMBLER and tested it with a game I am working on. I did discover with Deflate/Inflate, the larger the file or block is the better the compression ratio gets, so it is not efficient at blocks under 500 bytes. Things like fonts, large blocks of texts, or sections of the program that need to load into RAM over 2K long is what I had been using it for.

 

As many do know, I had been using deflate on cartridge games for the last 3 years to prevent them from using more expensive components. Get around the need to figure out bank switching and figuring out how to fit stuff into each bank. Makes it easier to debug. Cartridges end up being $40 instead of $50 or $60.

 

I looked into Exomizer when I was figuring out how to fit Venture onto a 16K cartridge, but the routine + compressed block ended up over 16K. Plus I had problems figure out how to get it to decompress to desired memory locations. But I got inflate work, I wanted to phase out the 32K Venture Cartridge as soon as possible, so figuring out Exomizer got shelved.



#8 xxl OFFLINE  

xxl

    Dragonstomper

  • 963 posts
  • Location:Rabka-Zdrój /Poland

Posted Thu Apr 6, 2017 7:32 AM

I found exomizer faster and shorter than deflate/inflate. And compressing a bit better, too. Simply better in every way : ).

 

 

really?

 

source file: gpl3.txt - 35147 bytes

exomizer - 12382 bytes + ~256 bytes, decompression time 2.6 sec

deflate - 11559 bytes + ~512 bytes, decompression time 3.6 sec

LZ4 - 15622 bytes + ~150 bytes, decompression time 1.1 sec



#9 Irgendwer OFFLINE  

Irgendwer

    Stargunner

  • 1,109 posts
  • Location:Germany

Posted Thu Apr 6, 2017 7:58 AM

deflate - 11559 bytes + ~512 bytes, decompression time 3.6 sec

 

deflate (with zopfli) - 11425 bytes (just for completeness)



#10 peteym5 OFFLINE  

peteym5

    Stargunner

  • Topic Starter
  • 1,509 posts
  • Location:Buffalo NY USA

Posted Thu Apr 6, 2017 9:41 AM

I did a benchmark with Venture which uses a 26231 byte binary down into 15778 compressed deflate file. Tested the original routine and decompressed the file into RAM in about 3.5 seconds. I applied this newer version by FoX and it takes about 4 seconds. I am not sure everything I did to modify the original Venture version, but I copyied the getbit/getbyte sections from FoX into my modified version and get it down to about 3.3 seconds. It will take me some time to figure out everything I did and what Fox did over the years. This was something I did 3 years ago and just left it alone because it work. Just recently looked into speeding it up more.



#11 peteym5 OFFLINE  

peteym5

    Stargunner

  • Topic Starter
  • 1,509 posts
  • Location:Buffalo NY USA

Posted Thu Apr 20, 2017 12:04 PM

I am sticking with Inflate/Deflate. Other compression routines do not compress as well, even though they may be faster. The recent version FoX did cut down the size and got  it to work a little faster. Most of the games I use it for needs to decompress segments under 2k and done in a split second, so it is unnoticeable. When you need to decompress stuff over 3k, I start noticing delays. I usually turn off the DMA and NMI so it does it faster. This does give your programs another level of protection from hackers, just compress routines, have it inflate, do the checks it needs before executing the main program. It adds an extra step for them to hack your stuff. I have a few segments that are over 8k that take several seconds to inflate. Been looking for ways to improve on that.


Edited by peteym5, Thu Apr 20, 2017 12:04 PM.


#12 Irgendwer OFFLINE  

Irgendwer

    Stargunner

  • 1,109 posts
  • Location:Germany

Posted Thu Apr 20, 2017 1:51 PM

This does give your programs another level of protection from hackers, just compress routines, have it inflate, do the checks it needs before executing the main program. It adds an extra step for them to hack your stuff.

 

Come on, you can do better: First "LZ4" then "deflate" - it just adds an extra step for the hackers.

Maybe you find some obfuscating routines too - for the hackers, but be sure you have spend at least the same amount of time to fix bugs and incompatibilities for the CUSTOMERS!



#13 CharlieChaplin OFFLINE  

CharlieChaplin

    River Patroller

  • 2,411 posts

Posted Thu Apr 20, 2017 5:12 PM

Come on, it is 2017, no need to protect any A8 program against hackers. Whatever you do - a hacker will be able to undo. Even if you create the best copy protection ever made on the A8 (better than EA, Synapse, etc.), someone will be able to hack/crack it. As irgendwer already said, better spend your time on gamecode and bugfixing. Here is something to relax for you...

 

 

 

 

 



#14 _The Doctor__ OFFLINE  

_The Doctor__

    River Patroller

  • 2,030 posts
  • Location:10-0-11-00:02

Posted Thu Apr 20, 2017 5:23 PM

This video contains content from Eagle Rock. It is not available in your country....

 

better in all ways version...

 

 

What's more is the song is not technically disco which is nice!


Edited by _The Doctor__, Thu Apr 20, 2017 5:29 PM.


#15 peteym5 OFFLINE  

peteym5

    Stargunner

  • Topic Starter
  • 1,509 posts
  • Location:Buffalo NY USA

Posted Thu Apr 20, 2017 6:05 PM

I hate when people drag these threads off subject. I am looking for ways to the improve Deflate/Inflate decompression times. We all do not like really long delays waiting for a game to load or waiting for the next level. I do not think we can get a better compression ratio than Inflate. The next Secretum Labyrinth will be using compression more intensely.  I figured out how to combine several songs into larger RMT files, and get a better compression ratio. About twice as many songs can be put onto a 64K ROM cartridge. That is just one thing I am looking at expanding on with this new game.



#16 Stephen ONLINE  

Stephen

    Quadrunner

  • 6,042 posts
  • A8 Gear Head
  • Location:Akron, Ohio

Posted Thu Apr 20, 2017 6:36 PM

 

Come on, you can do better: First "LZ4" then "deflate" - it just adds an extra step for the hackers.

Maybe you find some obfuscating routines too - for the hackers, but be sure you have spend at least the same amount of time to fix bugs and incompatibilities for the CUSTOMERS!

Nah - there are no honest customers in the 8-bit land.  Just dirty pirates who steal and spread all the new shit.  As much as I want this game, I absolutely will not support anyone with this mentality.  I will continue to PayPal donations to people who release their stuff for free, and don't consider people guilty until perceived innocent.



#17 _The Doctor__ OFFLINE  

_The Doctor__

    River Patroller

  • 2,030 posts
  • Location:10-0-11-00:02

Posted Thu Apr 20, 2017 7:41 PM

Yeah all those titles are on the Warez House BBS anywayz... :)     The dark underbelly of the interwebz on dah telnetz.  Those secret decoders are workin' full time!

 

Deflate/Inflate sales optimization rationale ...

I think some people don't realize they lose money sometimes. By an estimate of comments here and on other 8 bit forums I think it costs upwards of 27 percent in sales. If I were to take them at their word. By reputation score and consistent posting I tend to think it's a fact that they will avoid the titles.... It is a shame as more money would mean a happier coder and crew. I once refused to sell a truck on principle but that was a one off, not policy. Is that far enough back on topic for you? Old Atari commercial.. do the math!

 

in any event I await the post:  Inflate/deflate was optimized by so and so on sucha date the optimization was the code snippet. blah blah

 

The off topic video posting looks to help the topic with some much needed lightening of atmosphere and some good advice, if not just some uplifting ditties... :)


Edited by _The Doctor__, Thu Apr 20, 2017 7:57 PM.


#18 Irgendwer OFFLINE  

Irgendwer

    Stargunner

  • 1,109 posts
  • Location:Germany

Posted Fri Apr 21, 2017 1:16 AM

I hate when people drag these threads off subject.

 

I didn't introduce the other flavour. BTW: Compression is evil. If I understand

http://atariage.com/...-8#entry3743894

correctly, I'm sure Lance not only likes to support the shipping companies and custom employees but also the Eprom manufactures too (assumed this was no subterfuge).

So just go for a bigger and more expensive part (important!) and you have short fetching times and Lance is happy too!

 

I like both of you and the fun you bring in here - really!


Edited by Irgendwer, Fri Apr 21, 2017 1:37 AM.


#19 pirx OFFLINE  

pirx

    Moonsweeper

  • 323 posts
  • Location:Poland

Posted Fri Apr 21, 2017 4:42 AM

 

really?

 

source file: gpl3.txt - 35147 bytes

exomizer - 12382 bytes + ~256 bytes, decompression time 2.6 sec

deflate - 11559 bytes + ~512 bytes, decompression time 3.6 sec

LZ4 - 15622 bytes + ~150 bytes, decompression time 1.1 sec

 

 

Hi! gpl3.txt is a bad example as it has got much lower entropy than the usual binary file with code / graphics on 8-bitter. Can not prove this right now, but my results for a non-compressed Ataridos binary file were reversed for me and exomizer was faster (no doubts here) and compressed file shorter. Two files in circulation compressed with these 2 algorithms were my 16KiB intro and Amaroute+, in both exomizer won.



#20 peteym5 OFFLINE  

peteym5

    Stargunner

  • Topic Starter
  • 1,509 posts
  • Location:Buffalo NY USA

Posted Fri Apr 21, 2017 5:48 AM

I discovered with different types of files, the compression ratio is different, Text files that decompress directly to the screen have a higher ratio than ML program files. Between the 3 decompression algorithms, I got deflate to work first with my games and that is what I been sticking with. I had a problem getting Exomizer decompressing to the desired memory location and with me needing to get things to work and get these games out, deflate ended up being the algorithm of choice. In most cases, the time it needs to decompress something is not noticeable. The game that takes slightly longer time to boot up is Venture (16K) because deflate is expanding a 20k file into RAM. Delta Space Arena, only needs to expand 4k into RAM. 

 

Compression helps keep the costs down with the cartridges. Many more games can be fit into these 16K cartridges that Video61 has in stock that sale for $39.95. Uses 2 8K Eproms. Free shipping within United States. If not, games will need to go onto bigger cartridges that cost more in parts, need to work with bank switching, and more time to put together. Flashcarts, or bigger Eproms pushes the prices higher. 


Edited by peteym5, Fri Apr 21, 2017 6:12 AM.


#21 CharlieChaplin OFFLINE  

CharlieChaplin

    River Patroller

  • 2,411 posts

Posted Fri Apr 21, 2017 8:44 AM

 

 

Hi! gpl3.txt is a bad example as it has got much lower entropy than the usual binary file with code / graphics on 8-bitter. Can not prove this right now, but my results for a non-compressed Ataridos binary file were reversed for me and exomizer was faster (no doubts here) and compressed file shorter. Two files in circulation compressed with these 2 algorithms were my 16KiB intro and Amaroute+, in both exomizer won.

 

 

Think I created more than 700 TIP animations and in the end compressed them all. Exomizer most of the time had better compression ratio than Inflate/Deflate. (Superpacker by TeBe offers Exomizer, Deflater and LZ4 so I can easily test the different results.) There is also a "Snowman" XEX demo at atarionline available with two versions / two different compression formats, the deflate version is approx. 5% shorter, but therefore takes much longer to de-compress than the other version.

 

Back to compressed games on cart - no-one wants to insert a cart and stare at a black/blank screen for 10 seconds or longer. One may think the A8 has crashed then. Alas, I do have some carts in my collection where this is the case, e.g. the Yoomp cart., it does not show the nice loading animation, like the XEX version did, it only shows a black/blank screen while "loading" and de-compressing. So please, keep the de-compressing short on cart games (5 seconds or less) or show some title/animation while decompressing and not only a black/blank screen...



#22 peteym5 OFFLINE  

peteym5

    Stargunner

  • Topic Starter
  • 1,509 posts
  • Location:Buffalo NY USA

Posted Fri Apr 21, 2017 11:40 AM

 

 

Think I created more than 700 TIP animations and in the end compressed them all. Exomizer most of the time had better compression ratio than Inflate/Deflate. (Superpacker by TeBe offers Exomizer, Deflater and LZ4 so I can easily test the different results.) There is also a "Snowman" XEX demo at atarionline available with two versions / two different compression formats, the deflate version is approx. 5% shorter, but therefore takes much longer to de-compress than the other version.

 

Back to compressed games on cart - no-one wants to insert a cart and stare at a black/blank screen for 10 seconds or longer. One may think the A8 has crashed then. Alas, I do have some carts in my collection where this is the case, e.g. the Yoomp cart., it does not show the nice loading animation, like the XEX version did, it only shows a black/blank screen while "loading" and de-compressing. So please, keep the de-compressing short on cart games (5 seconds or less) or show some title/animation while decompressing and not only a black/blank screen...

 

This is what I did with the 16K cartridges games after Venture 16K. Delta Space Arena, Laser Blast, etc  only has parts of the program/data is compressed and expended to RAM as needed and many cases you only see a blank screen under 0.5 seconds. I got Venture down to 3.5 seconds, looked at ways to get Venture under 3 seconds, either optimize it to the point to where only a small portion needs to be inflated and use the latest version of Inflate. No one has yet panicked about Venture taking too long to boot up. 



#23 Stephen ONLINE  

Stephen

    Quadrunner

  • 6,042 posts
  • A8 Gear Head
  • Location:Akron, Ohio

Posted Fri Apr 21, 2017 2:22 PM

 

This is what I did with the 16K cartridges games after Venture 16K. Delta Space Arena, Laser Blast, etc  only has parts of the program/data is compressed and expended to RAM as needed and many cases you only see a blank screen under 0.5 seconds. I got Venture down to 3.5 seconds, looked at ways to get Venture under 3 seconds, either optimize it to the point to where only a small portion needs to be inflated and use the latest version of Inflate. No one has yet panicked about Venture taking too long to boot up. 

Even the Defender 2000 cart (Jaguar) displays a Lloading screen on powerup.  Nobody seemed to mind.



#24 _The Doctor__ OFFLINE  

_The Doctor__

    River Patroller

  • 2,030 posts
  • Location:10-0-11-00:02

Posted Fri Apr 21, 2017 3:01 PM

The ideas and suggestions are to improve or help the products.. unfortunately it is taken as some sort of attack or some such nonsense... A simple "loading" or "saving" has been a courtesy since the dawn of computing.... I do not understand why it is always a conflict when any suggestion is made of any kind. As it is clear no purpose is served I am out of the thread.... and as others have done... I regretfully.... no longer visit or purchase the products or the 'NO SOUP FOR YOU' website....

 

*ss-hattery by the proprietor must affect those he comes in contact with or some sort of mind meld/mind bending occurs.

 

.... :(



#25 gozar OFFLINE  

gozar

    Dragonstomper

  • 722 posts

Posted Fri Apr 21, 2017 6:45 PM

Compression helps keep the costs down with the cartridges. Many more games can be fit into these 16K cartridges that Video61 has in stock that sale for $39.95. Uses 2 8K Eproms. Free shipping within United States. If not, games will need to go onto bigger cartridges that cost more in parts, need to work with bank switching, and more time to put together. Flashcarts, or bigger Eproms pushes the prices higher. 

 

 

If I was doing a cartridge release I'd be looking at piggybacking off the work of Electrotrains and use the STM32F4 as the cartridge:

http://atariage.com/...ller/?p=3639094

 

128K and a cost of ~$13.






0 user(s) are browsing this forum

0 members, 0 guests, 0 anonymous users