Jump to content
rensoup

RMT2LZSS: convert RMT tunes to LZSS for fast playback!

Recommended Posts

From the readme:

Quote

 

1.What is it ?

 

It is a WinPC tool written in C# for the Atari 8Bits to convert RMT tunes to the compressed LZSS format.


RMT was written by Raster and is available at http://raster.infos.cz/atari/rmt/rmt.htm

LZSS was written by Dmsc and is available at https://github.com/dmsc/lzss-sap

RMT2LZSS was written by Rensoupp and is available at Atariage.com

 

 

2. Why?

 

While RMT is a flexible format, it is also a CPU cycle gobbler even at 50hz/60hz (at 100hz and especially above, it's difficult to use it for anything but a static screen).

 

LZSS is potentially the future for music playback on the A8 (and perhaps other platforms ?). It is a compressed raw stream of Pokey data with a very decent compression ratio and very fast decompression speed. The memory requirements are also reasonable (at 50hz at least).

 

Combining RMT & LZSS makes it possible to play RMT tunes at a very good speed. Unfortunately LZSS only takes raw Pokey data as input so RMT tunes have to be converted to raw data (SAPR) by playing them in an emulator such as Altirra or Atari800 which requires finding the tune's end point and loop point which is a tedious task. It can get even more tedious because LZSS doesn't support looping a tune anywhere but the starting point because the data always has to be decompressed from the start, so a tune with non 0 loop has to be split in 2.

 

RMT2LZSS solves the issue by having its own version of the RMT player rewritten in C# (based on the original A8 asm source code). The player will play the tune as fast as possible (this takes a fraction of a second), automatically stop at the end of the tune, identify the loop point and split the tune if needed. The tune (or tune parts) will then be compressed to LZSS. The tune can also be exported to an obx to play on the A8.

 

RMT2LZSS supports tunes from 50hz to 200hz automatically (beware because memory requirements go up with the frequency)

 

 

2021/05/29 release

RMT2LZSS_178b.zip

 

2021/05/27 release

RMT2LZSS_178.zip

 

2021/05/24 release

RMT2LZSS_177b.zip

 

2021/05/20 release

RMT2LZSS_177.zip

 

2021/05/17 release

RMT2LZSS_176.zip

 

2021/05/15 release

LZZS2RMT_175b.zip

 

2021/05/14 release

RMT2LZSS_175.zip

 

2021/05/13 release

RMT2LZSS_174b.zip

 

2021/05/11 release

RMT2LZSS_174.zip

 

2021/05/03 release

RMT2LZSS_173.zip

 

2021/05/03 release

RMT2LZSS_172.zip

 

2021/05/01 release

RMT2LZSS_171.zip

 

2021/04/28 release

RMT2LZSS_17.zip

 

2021/04/13 release

RMT2LZSS_1642.zip

 

2021/04/07 release

RMT2LZSS_163.zip

 

2021/03/20 release

RMT2LZSS_162.zip

 

2021/03/16 release

RMT2LZSS_161.zip

 

2021/03/14 release

RMT2LZSS_16.zip

 

2021/03/12 release

RMT2LZSS_151.zip

 

2021/03/11 release

RMT2LZSS_15.zip

 

2021/01/16 release

RMT2LZSS_14.zip

 

2021/01/15 release

RMT2LZSS_13.zip

 

2021/01/14 release

RMT2LZSS_12.zip

 

2021/01/13 release

RMT2LZSS_11.zip

 

initial release

RMTLZSS.zip

Edited by rensoup
  • Like 19
  • Thanks 7

Share this post


Link to post
Share on other sites

 

Btw for playing tunes inside your own executables, you need to grab the latest version from Dmsc's github page (https://github.com/dmsc/lzss-sap). It's got a bunch of fixes/minor improvements!

 


I tested the tool on the Asma archive: I searched for the RMT4 signature and found 1070 RMT4 tunes, I converted all of them which took about 2 hours... at least I know it doesn't crash!

 

Some stats:

    total RMT size:            1070 File(s)      3,265,532 bytes    
    total LZSS16 size:            1554 File(s)      4,356,981 bytes

 

(the difference in file count is because the tools sometimes splits a tune in 2 if the loop point is non 0)

 

    90% of the compressed tunes are below 7KB.

    Longest LZSS16 tune: Sloppyjones_Fuck_the_Cold 29310 bytes (RMT 4935 bytes). The tune lasts for 3mins and plays @200hz !


LZSS16 is pretty efficient at compressing but a bunch of tunes just play the same lines over and over again for a long time so RMT would be more efficient in those cases. A 1 min LZSS16 tune at 50hz would take less space usually than RMT but as duration increases, the balance shifts in favor of RMT.


Do all of them play properly ? I don't know but perhaps you can tell me... if you're a musician you could convert a bunch of your tunes and let me know if they play fine and loop points are ok ? This would help iron out any bug in the RMT player... Who knows, from a fully working player it may be possible to move on to an elusive new tracker...


In the meantime here are a bunch of quality examples:

 

screen.png.21a176a2965ec02662177e1db8ea0d74.png

 

Clubbed to death (Makary) @200hz
Electric city (Vinscool) @50hz
Far away - planet part (Aceman) @50hz
Hybris in-game (Emkay edit) @50hz
Noisy Pillars (Ivop) @50hz
RebbStars (Premium) @150hz
Sinvitro (Aceman) @100hz
Thermostat 7 (samurai) @200hz
Thruster (Warlord) @100hz
Wings of death L2 (Miker) @100hz

 

RMT2LZSS examples.zip

  • Like 10

Share this post


Link to post
Share on other sites
53 minutes ago, ilmenit said:

This is really interesting and I see many use cases.

Absolutely, there's no reason to use the RMT player anymore, unless you really need something that takes a tiny amount of space.

 

It's also possible to think about new tracker possibilities, something that can do everything the regular and patched RMTs do -and more- in a single tracker... then 400hz playback and possibly 800hz... or mix 50/100/200hz channels...

  • Like 4

Share this post


Link to post
Share on other sites

The best part is just using 6 scanlines for a 50Hz tune.... it feels like no CPU usage at all ;) 

  • Like 3

Share this post


Link to post
Share on other sites
17 minutes ago, rensoup said:

Absolutely, there's no reason to use the RMT player anymore, unless you really need something that takes a tiny amount of space.

Common use-case for RMT is also to keep one channel for SFXes and play them as RMT instruments. RMT has very useful example for this purpose. I will need to check how the decompressor could work with such SFXes.

Share this post


Link to post
Share on other sites
21 minutes ago, emkay said:

The best part is just using 6 scanlines for a 50Hz tune.... it feels like no CPU usage at all ;) 

And only a couple of extra scanlines for 100Hz, 150Hz and 200Hz.

 

@rensoup what about 240hz? ;)

 

I could have tried it myself, but 'winetricks dotnet45' takes ages :)

 

  • Like 1

Share this post


Link to post
Share on other sites
34 minutes ago, emkay said:

The best part is just using 6 scanlines for a 50Hz tune.... it feels like no CPU usage at all ;) 

It's a little more complicated though... should have mentioned it in the readme:

 

To make the CPU time visible in all configs (50->200hz) I put the decompressor in a DLI close to the top of the screen instead of the VBi (which starts at the bottom and would make the CPU time invisible)... and yeah that part of the screen is DMA free so it's about 20-30% faster... still very reasonable though

 

This is my unrolled version which is about 20% faster (or more?) than Dmsc's original, so it takes a little more memory but uses only 3 bytes of ZP.

 

The original player also sends the data right after decompressing it, while I do it in a DLI, so this requires an extra read (hence a little slower) but it is a bit more accurate in a sense because the data is sent at more regular intervals:

 

original:

-decompress byte for 1 channel

-send byte to Pokey

-loop

 

modified (and unrolled):

DLI: send all 9 bytes to Pokey

somewhere during the frame: decompress all 9 channels

 

Either solution is ok at 50hz but at 100hz and above, you probably don't want to be decompressing in the middle of the screen if you want to use your DLIs for something else than sound.

 

32 minutes ago, ivop said:

what about 240hz? ;)

hmm... not sure what you mean ? NTSC ? It's difficult to go above 200hz unless you poll VCOUNT, right ?

 

 

  • Like 2

Share this post


Link to post
Share on other sites
46 minutes ago, ilmenit said:

Common use-case for RMT is also to keep one channel for SFXes and play them as RMT instruments. RMT has very useful example for this purpose. I will need to check how the decompressor could work with such SFXes.

The compressor skip compressing channels if they're unused.

 

You can have SFX in a different format and deal with that Pokey channel manually.

 

Or you could keep 1 voice free (so 2 pokey channels) and have a 2nd tweaked instance of the player which only decompresses SFX into that voice.

 

SFX compress well too (used them in PoP)

 

Or you could have a 4 channel tune, have an extra virtual voice and decompress SFX into that virtual voice and switch between the tune's 4th voice and that SFX voice.

Share this post


Link to post
Share on other sites
2 hours ago, rensoup said:

1.What is it ?

 

It is a WinPC tool written in C# for the Atari 8Bits to convert RMT tunes to the compressed LZSS format.

 

Thanks. Is it possible to update instructions (including an example) in the following post you did in order to know how to use this tool with asm files?:

 

G2F and RMT at the same time:

https://atariage.com/forums/topic/238154-graph2font-not-running-on-widows-81/?do=findComment&comment=4452490

 

Share this post


Link to post
Share on other sites
Just now, rensoup said:

but the tracker is still available at the original page?

Yes, it is. As well as on the new page. It is recommended to refer to the new page, as the old one is not under control of the atari club, but raster's employer who can switch the pages off whenever they want. The atari club copied the pages for preservation.

  • Thanks 2

Share this post


Link to post
Share on other sites
31 minutes ago, tane said:

Thanks. Is it possible to update instructions (including an example) in the following post you did in order to know how to use this tool with asm files?:

In your case, it doesn't make sense to use LZSS because you can already play RMTs and don't need the CPU time, you'd be playing the same tune with a different player...

 

If you're thinking about 100+hz tunes, it's not possible because it requires DLIs which would interrupt G2F's processing and mess up the picture.

  • Thanks 1

Share this post


Link to post
Share on other sites
31 minutes ago, baktra said:

Yes, it is. As well as on the new page. It is recommended to refer to the new page, as the old one is not under control of the atari club, but raster's employer who can switch the pages off whenever they want. The atari club copied the pages for preservation.

ok, I'll keep it in mind for a next release... right now I can't update that first post of course...

Share this post


Link to post
Share on other sites
2 hours ago, rensoup said:

hmm... not sure what you mean ? NTSC ? It's difficult to go above 200hz unless you poll VCOUNT, right ?

Yes, NTSC. I don't know if they exist, but a 4x NTSC song can also be made with RMT.

 

RMT: 1x/2x/3x/4x

PAL: 50/100/150/200

NTSC: 60/120/180/240

 

240Hz. That's almost like replaying samples (low drum and bass) :)

  • Like 1
  • Haha 1

Share this post


Link to post
Share on other sites
3 hours ago, ivop said:

Yes, NTSC. I don't know if they exist, but a 4x NTSC song can also be made with RMT.

 

RMT: 1x/2x/3x/4x

PAL: 50/100/150/200

NTSC: 60/120/180/240

 

240Hz. That's almost like replaying samples (low drum and bass) :)

But why stop at 240 when you can go to 400 and possibly 800hz 😀 ? with static pictures of course but there's enough juice for it... (space may become a problem though)

 

Side question, does anybody know what the NTSC switch does in RMT ? If the Tracker plays the tune through emulation, I assumed it just changed the update loop to run 60 times per second (hence moving the track lines faster) and that was it...

The player source code shows it's using VCOUNT to sync and uses 312 lines (so PAL) for computing when to update... but it seems outputting a NTSC xex with RMT will cause raster bars to roll in PAL (meaning it's using 260 lines as a basis)

 

Share this post


Link to post
Share on other sites

Oh dear, I don't know if this is intentional feature, or if I broke the program, but I attempted to convert one of my experiments and that did not go exactly as planned
Looks like when the song "loops", it literally loops the state the chip was as well? I expected it to loop but continue to "sustain" something that was happening. That probably explains some looping issues I noticed when I listened to your tests before :P 

In short I was just letting the sound go until it rolled back. That is definitely not a conversion bug, rather it simply will literally loop regardless a note was sustained or not, so if an instrument was going to, say, increase the frequency very slowly, looping an empty pattern will pretty much just loop whatever state the pattern started as? I don't know if I make any sense, I did not really expect this to work but I imagine this can be interesting.

.obx is the conversion, .xex is how it it is intended to sound like (running in PAL or NTSC won't make any difference, but it was targeting NTSC

In the meantime I'll convert some tunes :D 

16 bit distortion A tests 5.obx 16 bit distortion A tests 5.xex

Edited by VinsCool

Share this post


Link to post
Share on other sites
1 hour ago, VinsCool said:

Oh dear, I don't know if this is intentional feature, or if I broke the program, but I attempted to convert one of my experiments and that did not go exactly as planned
Looks like when the song "loops", it literally loops the state the chip was as well? I expected it to loop but continue to "sustain" something that was happening. That probably explains some looping issues I noticed when I listened to your tests before :P 

In short I was just letting the sound go until it rolled back. That is definitely not a conversion bug, rather it simply will literally loop regardless a note was sustained or not, so if an instrument was going to, say, increase the frequency very slowly, looping an empty pattern will pretty much just loop whatever state the pattern started as? I don't know if I make any sense, I did not really expect this to work but I imagine this can be interesting.

.obx is the conversion, .xex is how it it is intended to sound like (running in PAL or NTSC won't make any difference, but it was targeting NTSC

In the meantime I'll convert some tunes :D 

16 bit distortion A tests 5.obx 4.2 kB · 1 download 16 bit distortion A tests 5.xex 2.8 kB · 1 download

What your  hear there is the slight offset between wave creation by POKEY and the adressing of frequency setting. Normally it is more coarse.

Interesting though, as it shows a nice feature that is usable in a tracker that is working like rensoup's adressing method.  

Share this post


Link to post
Share on other sites

If a voice is left alone by the RMT music, will RMT2LZSS also leave it alone?  In theory you might then be able to use a modified RMT player to do the sfx portions - which in most cases should be pretty light on the CPU use.

Share this post


Link to post
Share on other sites
2 hours ago, Rybags said:

If a voice is left alone by the RMT music, will RMT2LZSS also leave it alone?  In theory you might then be able to use a modified RMT player to do the sfx portions - which in most cases should be pretty light on the CPU use.

I think it should be possible to modify RMT2LZSS to skip, say, the first channel and have a modified RMT player that only plays AUDF1/AUDC1.

Share this post


Link to post
Share on other sites

Trying to skip wine (installing dotnet45 failed), I tried running it with mono natively. When I open the file dialog and select a single rmt file, it crashes:

 

$ mono RMT2LZSS.exe 
System.NullReferenceException: Object reference not set to an instance of an object
  at System.Windows.Forms.TextBoxBase.set_Lines (System.String[] value) [0x00018] in <0e1823914d7643eeaf1207febb083a4a>:0 
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.TextBoxBase.set_Lines(string[])
  at lzss2RMT.DebugStatus.Clear () [0x00007] in <4d5f7d5e6dbe4616a61a7a9c228303f7>:0 
  at lzss2RMT.Form1.m_buttonLoad_Click (System.Object sender, System.EventArgs e) [0x000dd] in <4d5f7d5e6dbe4616a61a7a9c228303f7>:0 
  at System.Windows.Forms.Control.OnClick (System.EventArgs e) [0x0001c] in <0e1823914d7643eeaf1207febb083a4a>:0 
  at System.Windows.Forms.Button.OnClick (System.EventArgs e) [0x00024] in <0e1823914d7643eeaf1207febb083a4a>:0 
  at System.Windows.Forms.ButtonBase.OnMouseUp (System.Windows.Forms.MouseEventArgs mevent) [0x00081] in <0e1823914d7643eeaf1207febb083a4a>:0 
  at System.Windows.Forms.Button.OnMouseUp (System.Windows.Forms.MouseEventArgs mevent) [0x00000] in <0e1823914d7643eeaf1207febb083a4a>:0 
  at System.Windows.Forms.Control.WmLButtonUp (System.Windows.Forms.Message& m) [0x0007e] in <0e1823914d7643eeaf1207febb083a4a>:0 
  at System.Windows.Forms.Control.WndProc (System.Windows.Forms.Message& m) [0x0016f] in <0e1823914d7643eeaf1207febb083a4a>:0 
  at System.Windows.Forms.ButtonBase.WndProc (System.Windows.Forms.Message& m) [0x0004e] in <0e1823914d7643eeaf1207febb083a4a>:0 
  at System.Windows.Forms.Button.WndProc (System.Windows.Forms.Message& m) [0x00000] in <0e1823914d7643eeaf1207febb083a4a>:0 
  at System.Windows.Forms.Control+ControlWindowTarget.OnMessage (System.Windows.Forms.Message& m) [0x00000] in <0e1823914d7643eeaf1207febb083a4a>:0 
  at System.Windows.Forms.Control+ControlNativeWindow.WndProc (System.Windows.Forms.Message& m) [0x0000b] in <0e1823914d7643eeaf1207febb083a4a>:0 
  at System.Windows.Forms.NativeWindow.WndProc (System.IntPtr hWnd, System.Windows.Forms.Msg msg, System.IntPtr wParam, System.IntPtr lParam) [0x0008e] in <0e1823914d7643eeaf1207febb083a4a>:0 

 

$ mono --version
Mono JIT compiler version 5.18.0.240 (Debian 5.18.0.240+dfsg-3 Wed Apr 17 16:37:36 UTC 2019)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	Interpreter:   yes
	LLVM:          supported, not enabled.
	Suspend:       preemptive
	GC:            sgen (concurrent by default)

@rensoup Any idea what could be wrong?

 

Share this post


Link to post
Share on other sites
12 hours ago, VinsCool said:

Looks like when the song "loops", it literally loops the state the chip was as well? I expected it to loop but continue to "sustain" something that was happening. That probably explains some looping issues I noticed when I listened to your tests before :P 

well yeah its just a recording of the pokey stream so when it loops it goes back to the state it was in at that point... why would want to play a sound then loop on an empty pattern unless you were trying to break it 😃 ?

 

Now why the LZSS version fails to loop properly from time to time is perhaps because of what Emkay explained ?

 

or did I understand your question wrong ?

  • Like 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...