RMT2LZSS: convert RMT tunes to LZSS for fast playback!

[VinsCool]

Great news! Both for envelope speed fix and portamento 16-bit!

 

I guess that brings me to test out my spreadsheet hack that puts together tables automatically, let's get that 443,9 16-bit table and put together a quick test tune, or something of the sort, haha

[VinsCool]

Hmmm I'm already running into problems...
This was me trying to convert the same kick drum test from the other day:

See the end of this message for details on invoking \njust-in-time (JIT) debugging instead of this dialog box.\n\n************** Exception Text **************\nSystem.NullReferenceException: Object reference not set to an instance of an object
  at RMT.Player.LoadFile (System.String path) [0x0069c] in <c1b0297cda89452b81316997403c4396>:0 
  at lzss2RMT.RMTConvert.DoConvertRMT (System.String obxName, System.String filename, System.String[] textLines, RMT.Player+PlaybackSpeed& playSpeed, lzss2RMT.RMTConvert+ConvertParams convParams) [0x000bf] in <c1b0297cda89452b81316997403c4396>:0 
  at lzss2RMT.Form1.BatchConvert () [0x00331] in <c1b0297cda89452b81316997403c4396>:0 
  at lzss2RMT.Form1.m_buttonLoad_Click (System.Object sender, System.EventArgs e) [0x00029] in <c1b0297cda89452b81316997403c4396>:0 
  at System.Windows.Forms.Control.OnClick (System.EventArgs e) [0x0001f] in <6a1e633bae3948029e60057de3abb3fd>:0 
  at System.Windows.Forms.Button.OnClick (System.EventArgs e) [0x00037] in <6a1e633bae3948029e60057de3abb3fd>:0 
  at System.Windows.Forms.Button.OnMouseUp (System.Windows.Forms.MouseEventArgs mevent) [0x0009a] in <6a1e633bae3948029e60057de3abb3fd>:0 
  at System.Windows.Forms.Control.WmMouseUp (System.Windows.Forms.Message& m, System.Windows.Forms.MouseButtons button, System.Int32 clicks) [0x001c3] in <6a1e633bae3948029e60057de3abb3fd>:0 
  at System.Windows.Forms.Control.WndProc (System.Windows.Forms.Message& m) [0x005a0] in <6a1e633bae3948029e60057de3abb3fd>:0 
  at System.Windows.Forms.ButtonBase.WndProc (System.Windows.Forms.Message& m) [0x0011f] in <6a1e633bae3948029e60057de3abb3fd>:0 
  at System.Windows.Forms.Button.WndProc (System.Windows.Forms.Message& m) [0x00056] in <6a1e633bae3948029e60057de3abb3fd>:0 
  at System.Windows.Forms.Control+ControlNativeWindow.OnMessage (System.Windows.Forms.Message& m) [0x00001] in <6a1e633bae3948029e60057de3abb3fd>:0 
  at System.Windows.Forms.Control+ControlNativeWindow.WndProc (System.Windows.Forms.Message& m) [0x000b3] in <6a1e633bae3948029e60057de3abb3fd>:0 
  at System.Windows.Forms.NativeWindow.Callback (System.Windows.Forms.Message& m) [0x00025] in <6a1e633bae3948029e60057de3abb3fd>:0 

\n************** Loaded Assemblies **************\nmscorlib\n    Assembly Version: 4.0.0.0\n    Win32 Version: 4.6.57.0\n    CodeBase: file:///C:/windows/mono/mono-2.0/lib/mono/4.5/mscorlib.dll\n----------------------------------------\nRMT2LZSS\n    Assembly Version: 1.0.0.0\n    Win32 Version: 1.0.0.0\n    CodeBase: file:///Z:/home/vinscool/Desktop/RMT2LZSS/RMT2LZSS_172/Release/RMT2LZSS.exe\n----------------------------------------\nSystem.Windows.Forms\n    Assembly Version: 4.0.0.0\n    Win32 Version: 4.6.57.0\n    CodeBase: file:///C:/windows/mono/mono-2.0/lib/mono/gac/System.Windows.Forms/4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll\n----------------------------------------\nSystem\n    Assembly Version: 4.0.0.0\n    Win32 Version: 4.6.57.0\n    CodeBase: file:///C:/windows/mono/mono-2.0/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll\n----------------------------------------\nSystem.Drawing\n    Assembly Version: 4.0.0.0\n    Win32 Version: 4.6.57.0\n    CodeBase: file:///C:/windows/mono/mono-2.0/lib/mono/gac/System.Drawing/4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll\n----------------------------------------\nAccessibility\n    Assembly Version: 4.0.0.0\n    Win32 Version: \n    CodeBase: file:///C:/windows/mono/mono-2.0/lib/mono/gac/Accessibility/4.0.0.0__b03f5f7f11d50a3a/Accessibility.dll\n----------------------------------------\nSystem.Configuration\n    Assembly Version: 4.0.0.0\n    Win32 Version: 4.6.57.0\n    CodeBase: file:///C:/windows/mono/mono-2.0/lib/mono/gac/System.Configuration/4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll\n----------------------------------------\nSystem.Core\n    Assembly Version: 4.0.0.0\n    Win32 Version: 4.6.57.0\n    CodeBase: file:///C:/windows/mono/mono-2.0/lib/mono/gac/System.Core/4.0.0.0__b77a5c561934e089/System.Core.dll\n----------------------------------------\n\n************** JIT Debugging **************\n

But what is weird is that I've really put the .erti to a strict minimum, and it still fails to convert.

The I tried to convert more stuff and it looks like it will always crash like this if I attempt to convert with an .erti present, everything else seem fine otherwise.

If this can be helped here's the files I was trying to convert:
 

kick test 800hz.rmt kick test 800hz.erti

[VinsCool]

Alright so due to the problem mentioned in the post above I am unable to do more tests, however, I added a new 16-bit table to match my other tables  
I'll also include the files I wanted to test, but since I cannot test them out... well, that's the best I can do, hopefully it works as expected later for testing the portamento.

I hope my spreadsheet copy and paste did not fail on me, hahaha

 

Edit: actually I can test this with an older version and should work, I'm dumb lol
It should be all good...
Edit 2: yup all good indeed! made a mistake in the .rmt but the table works perfectly.

16-Bit Distortion A Table Test.erti CustomNoteTables v23.txt 16-Bit Distortion A Table Test.rmt 16-Bit Distortion A Table Test.obx

Edited May 12, 2021 by VinsCool
Addendum

[rensoup]

20 hours ago, VinsCool said:

Hmmm I'm already running into problems...
This was me trying to convert the same kick drum test from the other day:

Ah.. forgot to initialize the param16 array, it only gets initialized if there is something in the .erti file (which there was since I was testing it duh).

 

should be fixed now (1.74b)!

 

Any idea why RMT has 8bit and 16bit tables btw ? Why not just go with 16bit tables and drop the low byte in 8bit mode ?

 

Edited May 12, 2021 by rensoup

[VinsCool]

1 hour ago, rensoup said:

Ah.. forgot to initialize the param16 array, it only gets initialized if there is something in the .erti file (which there was since I was testing it duh).

 

should be fixed now (1.74b)!

Yep! it works again!
The speed reduction test also seems to work well, at least with my drum instrument.
 

However, I have absolutely no idea how the 16-bit parameter should work, all I have been able to produce so far are high pitched pitch bends that seem to do infinitely up or down, and that's about it, lol
I'll have to test that one a bit more because, I really don't understand what should happen there...

1 hour ago, rensoup said:

Any idea why RMT has 8bit and 16bit tables btw ? Why not just go with 16bit tables and drop the low byte in 8bit mode ?

That won't work, since there are no matches, they won't be compatible.

Example from the custom tables:

// table dist A, My own table based on my Alternate Tuning approach.
$EF,$E3,$D7,$CB,$BF,$B3,$AB,$A1,$97,$8F,$87,$7F,$77,$71,$6B,$65,
$5F,$59,$55,$50,$4B,$47,$43,$3F,$3B,$38,$35,$32,$2F,$2C,$2A,$27,
$25,$23,$21,$1F,$1D,$1B,$1A,$18,$17,$15,$14,$13,$12,$11,$10,$0F,
$0E,$0D,$0C,$0C,$0B,$0A,$09,$09,$08,$08,$07,$07,$06,$06,$05,$00,

This is so far so good, that's what is expected in Distortion A, 8-bit, for the main part, then there are the 16-bit tables associated to it:

// 16bit test table A, calculated using my spreadsheet to A=443.9hz, which is close to alternate tuning
// Might be redone another time for alternative tuning ratios
$69,$63,$5E,$59,$54,$4F,$4A,$46,$42,$3E,$3B,$38,$34,$31,$2F,$2C,
$2A,$27,$25,$23,$21,$1F,$1D,$1C,$1A,$18,$17,$16,$14,$13,$12,$11,
$10,$0F,$0E,$0E,$0D,$0C,$0B,$0B,$0A,$09,$09,$08,$08,$07,$07,$06,
$06,$06,$05,$05,$05,$04,$04,$04,$04,$03,$03,$03,$03,$03,$02,$02,

$ED,$FA,$5D,$11,$11,$59,$E4,$B0,$B8,$F9,$70,$19,$F3,$FA,$2B,$85,
$05,$A9,$6F,$54,$58,$79,$B4,$09,$76,$F9,$92,$3F,$FF,$D1,$B4,$A7,
$A9,$B9,$D7,$01,$37,$79,$C6,$1C,$7C,$E5,$56,$D0,$51,$D9,$68,$FD,
$98,$39,$DF,$8B,$3B,$EF,$A8,$64,$25,$E9,$B0,$7B,$49,$19,$EC,$C2,

This is where it gets obvious this would not work: removing either the high or low part of the 16-bit values will surely output something in 8-bit, but the frequencies will be incorrect for note values  
At best there may be a few lucky frequencies being in-tune, but that would most likely be displayed on the wrong notes.


Anyway, I'll keep experimenting and hopefully figure something out for the 16-bit portamento effect...

Edited May 13, 2021 by VinsCool
typo

[VinsCool]

Well so I couldn't figure out what I did wrong but nothing works at all with the 16-bit stuff...
Anything I attempted only produced noise, silence, or high pitched pitch bends up or down...

But!
I tried out the idea I mentioned a few days ago, that is, using the 16-bit table, split into 2 8-bit tables instead, both fed in their respective channels with their appropriate AUDCTL settings.

And well, that certainly did work! Except, it's pretty bad due to the way numbers are handled, they are not 16-bit that are split into 2 channels, but 2 independent 8-bit values, so situations where $1FFF incremented by $0001 would simply underflow the resulting output to $1F00, which causes some of the glitches in the executable.
There seems to be something else that creates some weird stutters, and I can only notice them when the sound is output as 16-bit, for some reason.
I am not exactly sure about what causes it but there seems to be some cracks even when a note is sustained with no effect at all, but for now I have no idea ?

Regardless of technical workarounds I used and the obvious problems with my method, this proves, to a certain degree, that vibrato, and portamento definitely should work in 16-bit mode.

Now, the bad news, I noticed some things that were not expected to happen (I also mentioned the random stutters above, but it could also be my potato computer right now...):

- Portamento no longer works like before in 8-bit mode, any X value higher than 1 make the output not respond, so I was stuck using XY values between $11 and $1F only.

That seems to be an intentional change, I think, so I can live with it. Songs in multi speed would have some issues, however.
- 16-bit tables are forced to be picked up even when an instrument uses only 1 of the bits for it, either 1.79mhz or Join 2+1/3+4, if they play at the same time separately?

That can be worked around if no AUDCTL setting is set for the instrument in the .erti at least, otherwise I was unable to use my 8-bit tables manually (and by the same mean, couldn't get portamento to work at all).

I am not sure if either Join 2+1/3+4 alone triggered it, or just because it detected 1.79mhz and Join were being used at the same time... ?

- Envelope step does not seem to actually make the effects such as portamento go faster or slower, they get choppier but seem to run the same otherwise. That probably is not a bug but I thought it was worth pointing out.

- Using the 16-bit table seems to revert back to 8-bit notes when volume 0 is reached, even if the other channel running at 1.79mhz keeps the same value in memory.

That is hardly a problem but that could cause several unnecessary blocks of memory filled with unwanted values, and could also affect anything else if the frequencies were used to carry some modulation.

Things such as high pass filter would get problematic as a result, since that would be used at volume 0.

In any case, my test involving double 8-bit 16-bit ( ? ) would sound wrong due to that behaviour if the 16-bit table was used instead.

Overall I guess I once again have been able to break everything by doing things that were not exactly expected lol.
Other than that, everything else seems to be all good to me! I have not yet tried out samples further at the moment, but that seems to still work fine as well with my current files.

One of these days I'll need more tuning work done as well... mainly introduce more 16-bit tables, fix what I did wrong, and document the high pass filter in more detail sometime too...
I need to get a new .txt file with all the tables synthpopalooza created for the SKCTL experimentations too, since he asked me to test them the other day... using my spreadsheet hack to format the values directly into the format used in the customnotetables.txt should make it pretty easy to copy and paste... sorry I'm rambling again now am I? ?

Anyway, here's the files I used tonight for that dumb 16-bit hacked together using 2 8-bit tables;
The 8-bit tables 1A and 1C are just the same tables used in 16-bit distortion A, split in 2, so essentially this is useless for anything else, unless someone else is also into that sort of hack... emkay?  
 

16-Bit Distortion A Table Test.rmt 16-Bit Distortion A Table Test.xex 16-Bit Distortion A Table Test.erti CustomNoteTables.txt

[emkay]

2 hours ago, VinsCool said:

Anyway, here's the files I used tonight for that dumb 16-bit hacked together using 2 8-bit tables;
The 8-bit tables 1A and 1C are just the same tables used in 16-bit distortion A, split in 2, so essentially this is useless for anything else, unless someone else is also into that sort of hack... emkay?  
 

 

 

As I wrote before : There are too much unknown variables, to have a useful solution around. 

 

That's why I'm out of the 16 bit "messing around", and do things that can actually be reached

 

 

So here some merged stuff "pokey radio"?

 

 

[emkay]

Again, thanks @rensoup for this great tool. 

[rensoup]

2 hours ago, emkay said:

Again, thanks @rensoup for this great tool. 

Well you're a rollercoaster aren't you...

[emkay]

Just now, rensoup said:

Well you're a rollercoaster aren't you...

why do you think so ?

[rensoup]

1 hour ago, emkay said:

why do you think so ?

 

[VinsCool]

1 hour ago, rensoup said:

 

This made me laugh a little too much, my face is hurting ?

 

[rensoup]

1.75 is here:

 

1.75 May 2021
    -fix for 16bit portamento depth precision (8bit & 16bit mode) 
    -renamed extended commandArg to EnvelopeArgXY16
    -added Vinscool's latest custom notetables
 

Hopefully portamento 16bit works now???

[VinsCool]

Woah you're fast!
Let's get that checked real quick, I just got my Atari next to me to test things on the real thing so that couldn't be any better timing!

[rensoup]

21 hours ago, VinsCool said:

Woah you're fast!

fast but not quite on target apparently ?

 

1.75b May 2021
    -fix for 16bit portamento not picking frequencies from the 16bit tables in 16bit mode
    -changed 16bit mode switch (now requires join channel bit + 1.79Mhz bit set + custom AUDCTL on voice 2 or 4)
 

[VinsCool]

4 hours ago, rensoup said:

    -fix for 16bit portamento not picking frequencies from the 16bit tables in 16bit mode

Oh yes, it most definitely works great now!  

Just for fun: Test 2.obx

It shows that 16-bit portamento works great indeed, and that I broke a bunch of things (like I wrote to you a moment ago on Discord, that stuff is not "urgent", heehehe ?)

So now, who is going to make the first 2 channels 16-bit tune, involving the now working portamento...? ? I need my big drums fix ?

[VinsCool]

Finally, that 16-bit stuff is paying off  
Bassline, drums and portamento stuff is all 16-bit, 4th channel is 15khz, filling some holes.
Runs at 60hz btw
 

Sketch 50 V1.erti Sketch 50 V1.rmt Sketch 50 V1.xex

[emkay]

Antivirus tells that the rmt2lzss.exe contains the "agenttesla" trojan. 

Edited May 15, 2021 by emkay

[rensoup]

3 hours ago, emkay said:

Antivirus tells that the rmt2lzss.exe contains the "agenttesla" trojan. 

? which version ? Can anyone else confirm this ?

 

I build from a virtual machine which has no network access so that seems unlikely... 

 

submitted the exe to https://opentip.kaspersky.com/ :

 

Quote

No threats detected

 

That's some seriously useless antivirus you're running...

Edited May 15, 2021 by rensoup

[rensoup]

10 hours ago, VinsCool said:

It shows that 16-bit portamento works great indeed, and that I broke a bunch of things

I see you broke the equalizer already ?

[emkay]

56 minutes ago, rensoup said:

? which version ? Can anyone else confirm this ?

 

I build from a virtual machine which has no network access so that seems unlikely... 

 

submitted the exe to https://opentip.kaspersky.com/ :

 

 

That's some seriously useless antivirus you're running...

1.75b

 

I'll wait for the next version

[VinsCool]

8 hours ago, emkay said:

Antivirus tells that the rmt2lzss.exe contains the "agenttesla" trojan. 

The "virus" is definitely a false positive. There is no "agenttesla"
I uploaded the 1.75b .exe to virustotal and nothing was found, except 1 random tool called "SecureAge APEX" that apparently found something.
After a bit of googling, it seems like "SecureAge APEX" is pretty much is notorious for detecting false positives... So nothing unusual there I suppose.

5 hours ago, rensoup said:

I see you broke the equalizer already ?

I do a lot of things that seem to break it, actually, lol
I noticed the equaliser will not show anything when I use the filter too, for some reason.
I've stopped worrying about the visualisations for a while now, because I know it will get all over the place during my experiments, hahahaha  

[miker]

12 hours ago, emkay said:

Antivirus tells that the rmt2lzss.exe contains the "agenttesla" trojan. 

Too late. All your files are belong to us. 

[+Stephen]

59 minutes ago, miker said:

Too late. All your files are belong to us. 

Damn you - commander ate all my zig.

[rensoup]

1.76
    -added support for 16bit frequency tables (only supported for portamento right now). See .erti example
    -16bit mode switch now also occurs when channel volume = 0
    -added custom directories for file dialog.
 

Slowly but surely, portamento supports full 16bit... hopefully it'll be useful too ?

 

Support for custom directories means the directories you accessed are remembered in a text file and added to the file dialog... this has been nagging me for a while!