RMT2LZSS: convert RMT tunes to LZSS for fast playback!

[rensoup]

On 7/4/2022 at 9:29 PM, miker said:

Never mind, I hacked-in some code.

you hacked code in ?

 

Any more info on what you're trying to achieve ?

[pps]

@miker I just tested the routines I presented some months ago to be able to play back rmt2lzss files from old RMT with latest RMT 1.34 exported lzss file. Test says: It still works with latest RMT

 

https://github.com/ppsberlin/tutorial_lzssplay_g2f_rasterconverter/tree/main/rmt2lzss_new

[rensoup]

13 hours ago, pps said:

@miker I just tested the routines I presented some months ago to be able to play back rmt2lzss files from old RMT with latest RMT 1.34 exported lzss file. Test says: It still works with latest RMT

 

https://github.com/ppsberlin/tutorial_lzssplay_g2f_rasterconverter/tree/main/rmt2lzss_new

of course, LZSS just encodes/decodes raw pokey data 

[miker]

On 7/9/2022 at 10:01 PM, rensoup said:

you hacked code in ?

 

Any more info on what you're trying to achieve ?

Well, not really code. I just wanted to replace 16-bit bass table with that one made by @Synthpopalooza (and nothing else).

It worked... almost like I've planned but at last something has f##ked up and I've not used it anymore.

 

Edit: I've used LZSS export from RMT 1.34 instead, and it works like a charm. Thanks @pps.  

Edited July 11, 2022 by miker
update

[pps]

Only thing with that lzss export is, if you use music in game and want to play sfx, you can't simply play the note as in RMT 1.28. There is no ready to use player source. The new RMT has no working optimized asm sources atm too. So at least I still have to use old RMT 1.28 in games.

Sorry, but I don't understand any of these music sources enough to have this again. Esp. in POKEY stuff I am a beginner with not that many knowledge.

Edited July 11, 2022 by pps

[pps]

20 hours ago, rensoup said:

of course, LZSS just encodes/decodes raw pokey data 

Yes, I don't think lzss was the problem. Some of the beta RMT versions produced sound in different way, than before. So your tool (rmt2lzss) stopped working correct around RMT 1.3204. Now RMT has direct export 2 lzss and it works again (maybe rmt2lzss works again fine, have not tested it).

Edited July 11, 2022 by pps

[rensoup]

7 hours ago, pps said:

Only thing with that lzss export is, if you use music in game and want to play sfx, you can't simply play the note as in RMT 1.28. There is no ready to use player source. The new RMT has no working optimized asm sources atm too. So at least I still have to use old RMT 1.28 in games.

Sorry, but I don't understand any of these music sources enough to have this again. Esp. in POKEY stuff I am a beginner with not that many knowledge.

To play SFX, you could simply override the LZSS unpacked data for selected Pokey channels... @xxl has written a fast SFX engine:

 

 

7 hours ago, pps said:

Yes, I don't think lzss was the problem. Some of the beta RMT versions produced sound in different way, than before. So your tool (rmt2lzss) stopped working correct around RMT 1.3204. Now RMT has direct export 2 lzss and it works again (maybe rmt2lzss works again fine, have not tested it).

RMT2LZSS is based on the 1.28 RMT engine so yeah it doesn't work with newer versions...

[VinsCool]

There are several reasons why .rmt from new RMT versions aren't working as good as they could be with RMT2LZSS:

 

- As pointed out earlier, the RMT2LZSS engine is based on the version 1.28 RMT driver, which is no longer the same as the version I have modified to achieved a lot of additional things (with the CPU penalty resulting for the increased workload.
- Tuning lookup tables are generated directly from RMT since version 1.33, there is no need to create "tables" manually any more, so to speak, even if a module would make use of the minimal number of features still compatible with 1.28, the output would sound wrong as soon as the tuning is generated on base pitch, a different key or different temperament.

- There is also just the chance of compatibility breakdown just for the fact a bunch of the code handling effect commands were altered/repurposed for new effects, so it wouldn't work right as soon as an old RMT version is used, or vice versa, a module would need to be edited to be compatible with the 1.34 driver.

 

The good news is as soon as you work with registers dump/LZSS directly, this problem is solved since those essentially use the exact same driver for playing back.

There is an exception with the LZSS driver used with RMT 1.34 exports, which I did add a few notable features to allow more flexibility, but regardless of that, the same registers dump would work in both situations with no issue.

 

[Saint (Swiety)]

Would it be possible to set the mode to force compression of all channels at once as an option?
Without setting channel stream skip bits ?
In the case of a player, e.g. for a demo, it would allow you to disable checking whether a given channel should play or not,
especially since hardly any song in the demo uses all channels at the same time

[R0ger]

2 hours ago, Saint (Swiety) said:

Would it be possible to set the mode to force compression of all channels at once as an option?
Without setting channel stream skip bits ?
In the case of a player, e.g. for a demo, it would allow you to disable checking whether a given channel should play or not,
especially since hardly any song in the demo uses all channels at the same time

 

I just don't call decoding of the individual channels. I comment the extra calls out. Basically doing exactly what the skip code was doing (which I ripped out).

Edited March 23, 2023 by R0ger

[solo/ng]

6 hours ago, R0ger said:

 

I just don't call decoding of the individual channels. I comment the extra calls out. Basically doing exactly what the skip code was doing (which I ripped out).

It is not what what Swiety was asking for. 

 

He wants to have always all 4 channels to be streamed stream -> no need to check if a channel is streamed -> less cycles per frame (more data usage).

[rensoup]

9 hours ago, Saint (Swiety) said:

Would it be possible to set the mode to force compression of all channels at once as an option?
Without setting channel stream skip bits ?
In the case of a player, e.g. for a demo, it would allow you to disable checking whether a given channel should play or not,
especially since hardly any song in the demo uses all channels at the same time

 

Gotta say I was a little puzzled too... but my guess is: You have a tune with an intro and a non zero looping point. During the intro, not every channel is playing causing the bitstream skip bit to be set. The main part uses all the channels which clears those skip bits.

 

So you're trying to get rid of the skip bit test per channel for CPU saving ? I'm sure you're aware it's not a massive saving but perhaps you're doing some fancy stereo 400hz updates 

 

I would say it's preferable as well as quick and easy to fix it in the tune: Just insert a single note at low volume in those empty streams and they will be compressed normally.

 

So if you have a C#1 Instr Vol in Channel1 and nothing in the others, just do:

 

C#1 instr Vol-3 | C#1 instr 1 | C#1 instr 1 | C#1 instr 1

 

 

[R0ger]

15 hours ago, solo/ng said:

It is not what what Swiety was asking for. 

 

He wants to have always all 4 channels to be streamed stream -> no need to check if a channel is streamed -> less cycles per frame (more data usage).

 

Yeah, what I'm suggesting is less cycles per frame and the same amount of data usage. Why would you want all 4 streams if they are empty ?

[VinsCool]

I'm pretty sure that it's relatively simple to modify the compression/decompression procedure to only target specific channels and channel number, that would also leave more free space since fewer decompression buffers would also be needed in the process.

At least from the LZSS-SAP point of view, it seems pretty easy to do.

From the decompression side, however, that would require some specific changes to the decompression routine, and it could also be a little tricky if the same routine is wanted for more than 1 stream blocks, and they happen to target different channels, which would conflict with the setup that was made in the first place.

Note that I speak mostly from educated guess after I have been spending quite a lot of time experimenting with the compression optimisations, as well as finding new schemes in order to save a lot more memory for very large streams, so I could be wrong about what I just said above.