PeteE Posted August 17, 2017 Share Posted August 17, 2017 Which LZW algorithm do you use? I'm looking into using LZW and Huffman coding to compress pattern tables. My goal is something very simple to decompress since I want the decompression code to take up as little space as possible. I've made a sprite compression scheme that takes advantage of horizontal/vertical symmetry and reflection/rotation, but it won't work as well on pattern tables. Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted August 17, 2017 Author Share Posted August 17, 2017 I use a custom one. I essentially created a library of common english letter combinations that are used by the bottom 31 characters. Characters 32-127 are stored as normal. 128-255 are used to denote a leading space character for the other characters. This gets me around 18-20% compression. I tested using some LZW creator applications, but they all get way too complex too fast, attempting to encode the ENTIRE text block and use an entire word (16 to 32 bit) for encoding. Plus almost all of them strip out punctuation and are case-insensitive, which I NEED both of. My only thought is if I could use the 128-255 range for more keyword replacement, I may get slightly better results, but I would need a means to analyze a large text block and cleanly identify the most optimal matches. I've decided for now to forge forward with my current system since it's better than nothing; I can always investigate better compression later. Here's an assembly snip of my encoding/decoding algorithm: * Size: 64 bytes DECDAT DATA 0,4,8,12,16,20,24,28 DATA 32,36,40,44,48,51,54,57 DATA 60,63,66,69,72,75,78,81 DATA 84,87,90,93,96,99,102,105 * Size: 108 bytes DECTXT BYTE 3 TEXT 'the' BYTE 3 TEXT 'and' BYTE 3 TEXT 'hat' BYTE 3 TEXT 'ent' BYTE 3 TEXT 'ion' BYTE 3 TEXT 'for' BYTE 3 TEXT 'tio' BYTE 3 TEXT 'has' BYTE 3 TEXT 'tis' BYTE 3 TEXT 'are' BYTE 3 TEXT 'you' BYTE 3 TEXT 'can' BYTE 2 TEXT 'th' BYTE 2 TEXT 'he' BYTE 2 TEXT 'an' BYTE 2 TEXT 'in' BYTE 2 TEXT 'er' BYTE 2 TEXT 're' BYTE 2 TEXT 'es' BYTE 2 TEXT 'on' BYTE 2 TEXT 'ti' BYTE 2 TEXT 'at' BYTE 2 TEXT 'of' BYTE 2 TEXT 'to' BYTE 2 TEXT 'ea' BYTE 2 TEXT 'is' BYTE 2 TEXT 'it' BYTE 2 TEXT 'or' BYTE 2 TEXT 'as' BYTE 2 TEXT 'so' BYTE 2 TEXT 'we' BYTE 2 TEXT 'by' * STRING is buffer for decoded text * DECBUF is buffer for encoded text DECODE MOV R11,*R10+ LI R1,DECBUF LI R2,STRING+1 CLR R3 DC1 CLR R0 MOVB *R1+,R0 CB R0,@B32 JEQ DCEND COCB R0,@B128 JNE DC2 MOVB @B32,*R2+ INC R3 ANDI R0,>7F00 DC2 CB R0,@B32 JLT DC3 MOVB R0,*R2+ INC R3 JMP DC1 DC3 MOV R0,R4 SRL R4,7 MOV @DECDAT(R4),R5 MOVB @DECTXT(R5),R4 SRL R4,8 LI R6,DECTXT(R5)+1 DC4 MOVB *R6+,*R2+ INC R3 DEC R4 JNE DC4 JMP DC1 DCEND SLA R3,8 MOVB R3,@STRING B @SUBRET * STRING is buffer for decoded text * WORK is buffer for encoded text SPACE DATA >2020 ENCODE MOV R11,*R10+ MOVB @STRING,R3 SLA R3,8 LI R0,STRING+1 LI R1,WORK ENC1 MOVB *R0+,*R1+ DEC R3 JNE ENC1 CLR *R1 CLR R9 ENC2 LI R2,WORK MOVB @DECDAT(R9),R0 SRL R0,8 LI R1,DECTXT A R0,R1 BL @POS MOV R4,R4 JNE ENC3 INC R9 C @W32,R9 JLT ENC2 JMP ENC4 ENC3 DEC R4 BL @REPLC JMP ENC2 * R4 = Position in string of insert point * R9 = Character to insert REPLC MOV R11,*R10+ LI R0,WORK LI R1,128 RP1 CLR *R0+ DEC R1 JNE RP1 LI R0,STRING+1 LI R1,WORK MOV R4,R2 RP2 MOVB *R0+,*R1+ DEC R2 JNE RP2 SWPB R9 MOVB R9,*R1+ SWPB R9 B @SUBRET * R1 = Search string (length-byte at 0 position) * R2 = Target string (No length byte, has character 0 at end) * R4 = Return value (0 = not found, 1+ = position in string) POS MOV R11,*R10+ MOV R1,R3 MOV R2,R4 INC R3 INC R4 MOVB *R3,R5 SRL R5,8 POS1 CB *R3,*R4+ JEQ POS2 MOVB *R1,R5 SRL R5,8 MOVB *R4,*R4 JNE POS1 JMP POSNO POS2 INC R3 DEC R5 JNE POS1 MOVB *R1,R0 SLA R0,8 S R0,R4 S R2,R4 JMP POSEND POSNO CLR R4 POSEND B @SUBRET Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted August 25, 2017 Author Share Posted August 25, 2017 New blog entry up! EDIT: Or not... Yahoo is being terrible with Wordpress lately, and it appears Wordpress is refusing to publish new posts (even though it SAYS it's published). The direct link is: http://adamantyr.com/blog/2017/08/25/putting-it-together/ 1 Quote Link to comment Share on other sites More sharing options...
+Vorticon Posted August 26, 2017 Share Posted August 26, 2017 Looking really good It's great to see that you seem to have little difficulty coming back to complex code and picking up where you left off even if it's been a while. I on the other hand really suck at this, and I have to literally relearn the entire code before picking it up again... This is particularly true for ALC. 1 Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted August 26, 2017 Author Share Posted August 26, 2017 Looking really good It's great to see that you seem to have little difficulty coming back to complex code and picking up where you left off even if it's been a while. I on the other hand really suck at this, and I have to literally relearn the entire code before picking it up again... This is particularly true for ALC. It helps that I comment my code pretty heavily, and the original structure has become pretty ingrained in my head... That said, turning into a module-based structure helps because I do have to go over ALL the code. I haven't touched the travel portion of the code in years so I'm looking forward to going in and optimizing it. Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted August 31, 2017 Author Share Posted August 31, 2017 Did some analysis of my text encoder... it's actually pretty good, all things considered, with room for improvement! I copied all my text for the first disk to a large text file and did some case-sensitive searches for my suffix replacement, and the results are interesting... The "value" listed is the length of the suffix multiplied by the number of occurrences. Also note that the longer suffixes are done first, so "the" supercedes "he", which then loses count. This has been accounted for in the numbers. Suffix Count Value the 536 1608 in 635 1270 er 601 1202 you 324 972 and 319 957 es 450 900 re 439 878 ea 395 790 to 379 758 on 374 748 is 367 734 it 356 712 of 304 608 or 282 564 an 274 548 at 256 512 as 246 492 he 233 466 for 136 408 hat 134 402 ti 192 384 are 127 381 so 185 370 we 183 366 th 172 344 ent 104 312 can 76 228 ion 52 156 tio 39 117 has 39 117 by 36 72 tis 2 6 The last one, "tis", is definitely going to be replaced. I think what I'll do is replace the bottom five, requiring at least a value of 200. Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted September 3, 2017 Author Share Posted September 3, 2017 Okay, so I've written a new text encoder. It now uses 160 characters (0-31, 128-255) for dictionary key value replacement. I wrote up an encoder that takes in a dictionary list of values and produces a score sheet (how many of each was replaced) so I can actually see which ones work and which ones don't. In order to determine the appropriate n-grams, I wrote up a utility to calculate ALL the n-grams in the given block of text that comprises the first disk. I think the word usage is general enough it should be re-usable for all the disks. I also tested out using a bit-wise approach, with the most "common" letters being stored in 4-bit increments with a length nybble, while non-common characters are stored as a full 8-bit value. In practice, though, it didn't produce any better results. Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted November 14, 2017 Author Share Posted November 14, 2017 Just a heads up, you can see all my latest updates on this project on my blog! http://www.adamantyr.com/blog Solved my earlier problem with Yahoo by moving to AWS as well. 3 Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted December 26, 2017 Author Share Posted December 26, 2017 Merry Christmas! Here is a video of the title screen music... I'm still making some adjustments here and there: 7 Quote Link to comment Share on other sites More sharing options...
Opry99er Posted January 3, 2018 Share Posted January 3, 2018 Hey, sweet!!!! Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted January 3, 2018 Author Share Posted January 3, 2018 Thanks! I actually just fixed it today so it has proper note decay... It sounds SO MUCH BETTER now. I still hear the odd-ball off note here and there that I want to investigate. I know two of them I can do nothing about because the note is below the TI's range to create, but there's a few other anomalies I'd like to fix... Quote Link to comment Share on other sites More sharing options...
Opry99er Posted January 3, 2018 Share Posted January 3, 2018 Sounds epic. Cant wait to dungeon crawl listening to that! Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted January 3, 2018 Author Share Posted January 3, 2018 Music is played only during the title screen. Music and CRPGs can be done; Dungeons of Asgard does it quite well. I just decided I wanted to focus on sound for effects. Also, I have no musical writing ability, and I readily acknowledge there is serious skill and talent required to make music that doesn't distract or get annoying. Quote Link to comment Share on other sites More sharing options...
+mizapf Posted January 3, 2018 Share Posted January 3, 2018 Also, I have no musical writing ability, and I readily acknowledge there is serious skill and talent required to make music that doesn't distract or get annoying. Burger Time. Nuff said. 1 Quote Link to comment Share on other sites More sharing options...
Opry99er Posted January 3, 2018 Share Posted January 3, 2018 Hen Pecked Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted January 3, 2018 Author Share Posted January 3, 2018 Oh yeah Burgertime... All versions are maddening with the silly NEVER STOPS jingle music, but the TI is worse because Data East's software engineers apparently didn't understand they didn't HAVE to stop the music for every sound effect and start it over. 1 Quote Link to comment Share on other sites More sharing options...
+TheBF Posted January 3, 2018 Share Posted January 3, 2018 Music is played only during the title screen. Music and CRPGs can be done; Dungeons of Asgard does it quite well. I just decided I wanted to focus on sound for effects. Also, I have no musical writing ability, and I readily acknowledge there is serious skill and talent required to make music that doesn't distract or get annoying. If you composed this piece then you have musical skills. This is called polyphonic music best exemplified by J. S. Bach. Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted January 3, 2018 Author Share Posted January 3, 2018 If you composed this piece then you have musical skills. This is called polyphonic music best exemplified by J. S. Bach. I did not compose this piece. It's called "Caverns of the Heart", by Fred Nachbaur. He wrote it along with a number of other pieces for an unreleased CRPG in the early 2000's. Sadly, he passed away around 2004, but I obtained permission from his daughter to use the song, so long as I give appropriate credit in both game and any manuals, which naturally I will do. (Although that was a LONG time ago... I may have to send her the completed piece and make sure she's all right with it's use again.) Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted February 1, 2018 Author Share Posted February 1, 2018 And here is the title screen with music playing... 11 Quote Link to comment Share on other sites More sharing options...
Opry99er Posted February 1, 2018 Share Posted February 1, 2018 RoA How do I long for thee.... 1 Quote Link to comment Share on other sites More sharing options...
InfernalKeith Posted February 3, 2018 Share Posted February 3, 2018 Just trying to imagine showing this to anyone in 1983 and telling them this was their TI 99/4A at work... heads would explode! Quote Link to comment Share on other sites More sharing options...
Iwantgames:) Posted February 5, 2018 Share Posted February 5, 2018 Man this looks and sounds so good Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted April 18, 2018 Author Share Posted April 18, 2018 Okay, time for an update on this thread! I'll be showing off the game and my development tools for it at Fest West this year. The engine is about 65% complete, combat is still in "rectification" from the old code base, and I won't have it anywhere near to done in time for Fest West. So I'm spending time on tuning and polishing the rest to make sure it looks good and plays well. On a side note... If you try and visit my blog and it times out, it's due to a plug-in I installed called Jetpack to try and track metrics. The stupid thing seems to cause instability though, probably because it's trying to do something an AWS t2.micro can't really handle. I'm getting close to disabling it, for now, I've been just resetting the server when that happens because it's really useful to know how many visitors I'm getting. 2 Quote Link to comment Share on other sites More sharing options...
Opry99er Posted April 19, 2018 Share Posted April 19, 2018 Excited to see your progress, sir!!! Quote Link to comment Share on other sites More sharing options...
+adamantyr Posted July 31, 2018 Author Share Posted July 31, 2018 Some mock-screenshots of the monster statistic screen. Doing stat screens is so memory consumptive, I think I'll have just enough space to get it done! 7 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.