RMT Hacking Ideas and Progress

[VinsCool]

Simple RMT Player v0.2
A very simple recreation of the the RMT .xex export, with some tiny code improvements based on the currently WIP RMT Visualizer.

Edit a handful of parameter, drop a 'music.rmt' in the same folder, then assemble with XASM.
rmtplayer.a65 is the Patch 16 Beta 7 version, but you could replace it with any other version or even the original 1.28 if you prefer.

A later version will include a simple .sap creation too.

That may be useful while exports are not working due to a technical reason being memory in the rmt.exe, while the RMT Visualizer isn't finished (I broke many things lol, but I promise I am learning!) ?

Note this version isn't very different from the one I posted above, but there were a few things cleaned up in the Simple Player.

I did my best to comment the code to make most sense of it, though I admit some things I am not 100% certain yet.

 

Some timing for out of region playback works nicely with this I think, but I cannot guarantee everything perfectly stable, so let me know how it goes! 

From what I could tell, VLINE = 130 from PAL to NTSC and 187 from NTSC to PAL gave me the best results for decent stability.

There may be some jitters here and there, especially when the SHIFT key is held, for a reason I haven't been able to comprehend yet.

Note that none of this is a problem when things are in their correct region, even multi speed songs are also working fine.

 

I haven't tested the Stereo playback yet, but it shouldn't be a problem, unless of course I did something stupid and broke the rmtplayr badly lol

Speaking of stereo, maybe this simplier project will be a good reason to finally try getting the ideas I wanted to experiment out in order to unify more code together and detect stereo setups automatically...

 

Anyway, enjoy! I hope this could be useful and easy to use as a compromise for missing exports from the .exe  

 

Simple RMT Player v02.zip

Edited September 28, 2021 by VinsCool
oops forgot to upload the files

[VinsCool]

Took me a bit, but I finally figured out how to fix the pesky timing of PAL running in NTSC that would somehow always desync when the SHIFT key is held for a reason I couldn't comprehend.
Magic scanlines: 6,58,110,162,214

So stretching 5 frames into 6 without any skips or jitters? I think I was able to get it working.

I think I also accidentally fixed the timing when a tune starts playing, so the stutter that would usually be present for a frame or 2 appears to be gone as well.

Stranded on the Surface of Io (PAL in NTSC Timing Test).obx

Here's a comparison with the original .xex export format, a little skippy when it's running in NTSC compared the slightly different approach I took.

Stranded on the Surface of Io Final.xex

Edited September 29, 2021 by VinsCool
typo

[VinsCool]

Did few more tests with the same version of the code, sounds pretty stable for most Out of Region situations I think  

 

NTSC in PAL, timing is "rolling" visually but it's so smooth there's literally no problem in the sound.

I even made sure to pick one of my wackiest tunes for the tempo itself, being frame perfectly designed, and I couldn't notice a single frame being dropped!

Table Manuscrite (NTSC in PAL Timing Test).obx

 

 

STEREO PAL 100hz in NTSC, absolutely no problem either, just like the previous PAL in NTSC, timing is perfectly stable!

I'm Sure (STEREO PAL in NTSC Timing Test).obx

 

Also not in video, tested a PAL 200hz tune in NTSC, and just as it was shown above, PAL in NTSC was a perfectly smooth experience  

Unsolved (Pal in NTSC Timing Test).obx Prehistoric Tale (PAL 200hz in NTSC Timing Test).obx

 

So here's a new version of that "Simple" RMT Player with these tiny improvements  

- Added a simple VBI routine and adjusted the timing for stability

- Added a very dumb "colour shuffle" effect in rmtplayr.a65, can be removed if you want, I only needed a visual cue for timing purposes

Simple RMT Player v03.zip

As far as I'm concerned, this re-creation of the original .xex export format is about as good as I wanted it to be, so that's probably the last final version the .xex part of it.

 

Next thing to try getting done is to assemble the rmtplayr in the .sap format.

My goal now is to achieve this using only the dasmplayer.asm re-creation with .sap specific instructions to assemble, with my patched rmtplayr.a65, a music.rmt, and XASM.

 

[VinsCool]

Got a functional SAP exporter-- with a little twist.

 

Simple RMT Player v0.4:

- Added a very crappy self made SAP export format

- Added a few hacks that allow the bypass of FASTPLAY and NTSC arguments entirely (Altirra only?)

Simple RMT Player v04.zip

 

Yep, that's right, I've sorta figured out some things with the SAP format in Altirra.

I've re-used a bunch of code from the Simple RMT Player part (.xex/.obx), to initialise the engine speed, and song speed, and bypassed the original stuff entirely, which only took into account the NTSC and FASTPLAY arguments.

In my hacked export, I run the same code I used for region check, and the same code for the song speed (1-4), turns out it was much easier than expected once I actually figured out what I was doing.

 

How to use: in dasmplayer.asm, make sure EXPORTSAP is set to 1, then, like before, assemble with XASM, make sure to use the -o argument to export .sap files, else simply rename the .obx output

 

Just a side note, everything appears to work with exports, but some parameters are "Altirra Only"-- the stand alone SAP players are a little broken by this, region cannot be detected properly (no access to certain registers I think) and for some reason the song speed code seems to be ignored... or I may have missed something else.

Either way, if the stand alone SAP players are absolutely needed, just comment out the code I added to check the region and song speed, and use the arguments in the SAP header instead  

 

Here's a handful of conversions, they might only work properly in Altirra (for now):

VinsCool - Stranded on the Surface of Io.sapVinsCool & Spring - Table Manuscrite.sapRaster - I'm Sure (STEREO, 100hz).sapMakary - Prehistoric Tale (200hz).sapVinsCool - Corridors of Time.sap

 

[VinsCool]

Alright, so I stopped slacking and did something tonight after a bit of a frustrating javascript experience earlier.

since I did have a github account I just never used until yesterday, I figured I could actually make use of it for real.

 

https://github.com/VinsCool/RMT-Patch16/releases/tag/Patch16-Beta8

 

RMT Patch16 Beta8

 

- Now on github, with all the side projects and other experimental stuff.Added Two-Tone Filter (proper) toggle, use with Command 7, enable with XY=$FE, disable with $FD, or use $FF for Volume Only Mode
- Added the RMT Visualizer in its current state, it needs more work but works well enough for now
- Added XEX and SAP export re-creation based on the disassembly of the original files, with few experimental improvements
- Included several side projects, like the POKEY Tuning Spreadsheets, or some random experiments as code
- Included the stripped rmt.exe binary, hacked from the unofficial 1.30, for use with the .obx loader by ivop, also included (sa_c6502.dll)
- Included most .dlls that may be missing, including the newer Altirra plugins
- Some cleanup before the files were dropped here, might be easier to manage now
- Reminder, this is all experimental stuff. don't be surprised of any malfunction or unexpected behaviour, I am still learning how to code so things may become a little better in the future.

 

Enjoy! Atari goes on!

 

 

[ivop]

16 hours ago, VinsCool said:

- Included the stripped rmt.exe binary, hacked from the unofficial 1.30, for use with the .obx loader by ivop, also included (sa_c6502.dll)

I have thinking about reversing the logic of the loader, so you don't need a patched binary anymore. The first time sa_c6502.dll is called, don't look for "TRACKER " at the beginning of the player, but somewhere else, like $0700. If it's not there (and it'll never be at the first call, all zeroes), run the loader and write "TRACKER " to $0700. Now suddenly, it works with Rmt 1.28 and Rmt 1.30 without patching the binaries.

 

Edit:

 

Done. See https://github.com/ivop/rmt-sa-libs/tree/main/real/sa_c6502_inject

 

It's now inject always, unless a code injection already took place. Works with unpatched Rmt.exe binaries.

Edited October 6, 2021 by ivop

[ivop]

Can't edit anymore, but it still works with the patched binaries, too.

 

Also looked into replicating the 6502 to Pokey RMT bypass that the Altirra libraries do, it's C++:

 

class IATRMTBypassLink {
public:
    virtual void LinkInit(uint32 t0) = 0;
    virtual uint8 LinkDebugReadByte(uint32 t, uint32 address) = 0;
    virtual uint8 LinkReadByte(uint32 t, uint32 address) = 0;
    virtual void LinkWriteByte(uint32 t, uint32 address, uint8 v) = 0;
};
 

I could introduce a g++ dependency to replicate this, I could compile this from within a VM and hack the Altirra sources, but I think the easiest path to have a cycle exact combination of CPU and POKEY emulation is that @phaeron adds a tracker.obx loader to his RMT CPU library. Considering there's already an XEX loader in Altirra, I think it's literally less than ten lines of code

[rensoup]

On 10/6/2021 at 8:15 PM, ivop said:

Can't edit anymore, but it still works with the patched binaries, too.

?

 

So how's that tracker GUI going  ? 

[ivop]

18 hours ago, rensoup said:

?

What does that face mean?

 

18 hours ago, rensoup said:

So how's that tracker GUI going  ? 

Life got in the way, and I decided to wait until either you release your C# RMT player as a basis for a C++ port, or mrk finishing his Javascript player, to be ported to C++, too. I need a basis to work with, otherwise I'm afraid a lot of the GUI work needs to be redone afterwards if I just create a spreadsheet-like editor.

[VinsCool]

2 hours ago, ivop said:

mrk finishing his Javascript player, to be ported to C++, too.

It's already very functional in my opinion.

Apart from few issues here and there, almost everything expected from RMT does work, and I was even able to port some of my own patches to it since the last few days ?

 

[rensoup]

2 hours ago, ivop said:

What does that face mean?

it means you re still walking the path to madness ?

 

2 hours ago, ivop said:

Life got in the way, and I decided to wait until either you release your C# RMT player as a basis for a C++ port, or mrk finishing his Javascript player, to be ported to C++, too. I need a basis to work with, otherwise I'm afraid a lot of the GUI work needs to be redone afterwards if I just create a spreadsheet-like editor.

Yeah I'm working on the dxrmt format, which I believe is important...

 

See, one optimization that Raster did was to merge empty lines and line with speed changes when possible, which doesn't work with gotoline so he had to split packed empty lines to fix goto... A necessary waste of time because of the 6502 player.

I'm guessing that was unmerged when the file was loaded in the tracker so that empty lines matched the pattern displayed on screen. If you don't unmerge, it would be more messy for editing and displaying patterns.

 

dxrmt would leave the lines unmerged.

 

Right now the player uses lists for module data (the part to be displayed/modified) but that can be fixed in a minute to use arrays, then it could perhaps be possible to communicate between C++ and C# easily.

 

 

Edited October 8, 2021 by rensoup

[ivop]

52 minutes ago, rensoup said:

it means you re still walking the path to madness ?

No. In between other stuff, I just inverted the tracker loader logic to make things easier for Vinscool and others.

 

52 minutes ago, rensoup said:

Yeah I'm working on the dxrmt format, which I believe is important...

 

See, one optimization that Raster did was to merge empty lines and line with speed changes when possible, which doesn't work with gotoline so he had to split packed empty lines to fix goto... A necessary waste of time because of the 6502 player.

I'm guessing that was unmerged when the file was loaded in the tracker so that empty lines matched the pattern displayed on screen. If you don't unmerge, it would be more messy for editing and displaying patterns.

 

dxrmt would leave the lines unmerged.

 

Right now the player uses lists for module data (the part to be displayed/modified) but that can be fixed in a minute to use arrays, then it could perhaps be possible to communicate between C++ and C# easily.

As there's no 6502 player at the end of the pipeline anymore, I'd say just write the data structures to a file, or in ASCII like dxrmt? No compression needed indeed. Each note in a pattern could consist of way more bytes, arps, vibrato, glissando, etc...

[rensoup]

14 minutes ago, ivop said:

As there's no 6502 player at the end of the pipeline anymore, I'd say just write the data structures to a file, or in ASCII like dxrmt? No compression needed indeed. Each note in a pattern could consist of way more bytes, arps, vibrato, glissando, etc...

??

 

I think we may be on the same page...

 

(But probably not for long ?)

[ivop]

1 hour ago, rensoup said:

??

I think we may be on the same page...

(But probably not for long ?)

I think we are, but it's not that easy to glue a GUI on top of existing code.

 

The hierarchy is like RMTFILE(SONG(TRACKS(INSTRUMENTS)))

 

But they are not necessarily derived objects in the OO sense.

 

You need three objects derived of a table view. Song, Track, and Instrument. Well, Instrument might need a better editor later, but just do a tableview for now.

 

The main program exists of a maximum length Song table, the maximum number of Track objects, and the same for Instrument objects. Edit at will

 

The player thread retrieves its info from these objects, and plays its merry tune. Loading and saving of projects either serialize the objects, or write it to an ASCII format, or an SQLite database.

 

Export to SAP-R and/or LZSS.

 

Rubber ducking here ?

 

But that's my idea in a nutshell.

 

Maybe I can start this somewhere next week.

[rensoup]

We may still be on the same page! 

 

btw any idea if it's easy to mix C# and C ? (at least as DLLs)

Edited October 8, 2021 by rensoup

[ivop]

16 hours ago, rensoup said:

We may still be on the same page! 

 

btw any idea if it's easy to mix C# and C ? (at least as DLLs)

I don't know. I have never used C# before, but perhaps that's not even needed.

 

Edit: it's not easy https://stackoverflow.com/questions/778590/calling-c-sharp-code-from-c/778677

 

The other way around, i.e. calling C++ from C#, is easier, but I'm not going to code in C#/.NET/mono.

Edited October 9, 2021 by ivop

[rensoup]

3 hours ago, ivop said:

The other way around, i.e. calling C++ from C#, is easier, but I'm not going to code in C#/.NET/mono.

Nobody's asking you to start C# obviously... but a C# player DLL would probably be the quickest way to get something running (if you can figure out an easy way to call C# functions)

 

You can always port it later...

[ivop]

45 minutes ago, rensoup said:

Nobody's asking you to start C# obviously... but a C# player DLL would probably be the quickest way to get something running (if you can figure out an easy way to call C# functions)

Seems the [DllExport] thing is Windows only?

Quote

You can always port it later...

Sure, but the earlier the better, before new stuff gets added?   Expanding the format and player later is just as easy in C/C++ as it is in C#.

 

I have a QtCreator setup that can compile GUI apps for Linux, Windows and Android. It cross-compiles if necessary. I prefer developing for Linux at first, use native libraries. It's just faster. Use a .so/DLL loader for Pokey emulation. Cross compile for Windows and Android later. Phaeron's Altirra RMT Pokey emulation libraries are Windows only.

 

Creating a C# .so library for Linux, not sure if that is possible, and then link with a Qt application. Getting that to work might be more work than port circa 500 lines of player code. Project loading/saving, and importing RMT files, has to be rewritten too, but that's just a chore

Edited October 9, 2021 by ivop

[rensoup]

oh well... as long as there's no 6502 player...

[VinsCool]

More to come later ?

This is becoming quite the extensive hack now... I have learned some things by attempting to reverse engineer the program, and I may be able to edit quite a lot of things now.

Here I managed to fit the binary inside the .exe by hacking some memory pointers in the .exe directly, hijacking the location of the Mono and Stereo binaries.

I will fix those two in a different location once I have a better understanding of how the PE header stuff works...

 

I also fixed the Tracker binary corruption for good now, including ivop's binary loader, it should no longer require any extra workaround this time.

Turns out I was only halfway there with the first solution I had.

Turns out, the dummy RMT8 header at the end of the file was ABSOLUTELY necessary, and thanks to the reverse engineering I finally know what actually happens with the binaries loaded in memory and where they are loaded/pointing.

Also this time the Altirra plugins combo were used! Finally, I can enjoy the best quality, and cycle accurate timing, while being able to fuck some other registers, here, the SKCTL, works in Stereo too.

 

So yeah, there may be quite an extensive rmt.exe hack coming soon, original source code or not, I have the determination to find out how this program works, all by myself.

On 10/9/2021 at 3:45 PM, rensoup said:

oh well... as long as there's no 6502 player...

? Oh LOOK, a wall! *hides*

Edited October 16, 2021 by VinsCool
hehehe

[rensoup]

[VinsCool]

Hmm I got a bit stuck, but at least I do have some progress.

 

- Managed to expand the .exe by adding an .extra section

- Moved Tracker and XEX/SAP export binaries to that location, meaning the memory limitation will no longer be a problem

- Partially reverse engineered the binary loader process for loading and exporting, so I have most of the useful memory addresses and pointers I can manipulate

 

Now about this, like I said, I have only moderate success at the moment, but it's not quite fully functional.

 

Good news: Tracker binary loading fully functional, size limitation is no longer a problem! Everything seems to work perfectly with it, played with Altirra plugins combo works great, Two-Tone Filter is alsosupported with CMD7 and XY= $FE (enable) and $FD (disable).

The tracker binary corruption experienced before (by either producing garbage or nothing at all) seems to be solved for good too, and this might also remove the necessity for ivop's loader to have its workaround in place as far as I could tell.

Bad news: Despite having all the data loaded at the correct locations, exports are currently broken, there is garbage data being written, making XEX and SAP exports excessively large, and I don't know why this happens currently, since I made sure to had the correct memory addresses and size pointing the right stuff.

Things *do* get loaded, and written in exports, but they're cut by some weird sections of of 00's or FF's, but in any case, they can technically be recovered into functional exports with hex edit, it's just cursed as hell when assembling the player from scratch is much less inconvenient lol.

So for now I suspect a memory corruption (very plausible due to the hacking around the .exe I did), or there was something important I overlooked in the partially reversed export binary loader code, which is still incomplete so it's very likely to be related too.

 

Despite the currently unusable exports, the program does work, and now there's a massively larger memory section in which I can write my own data, and edit pointers accordingly very easily.

It's just not fully functional still, hehe.

 

I will eventually figure it out.

Btw this is based on the RMT 1.30 version I did have hacked before, but now I've replaced the entire memory section where binaries were originally located with NOP instructions (easier to retrieve), added the .extra section like I mentioned above, and did a very rudimentary UI hack in the About window because I can lol

 

TL;DR, it works, but exports are broken, and I probably broke more shit because I'm good at breaking shit anyway lol

Enjoy.

RMT MEMORY HACK TEST 7-9.exe

[ivop]

4 hours ago, VinsCool said:

and this might also remove the necessity for ivop's loader to have its workaround in place as far as I could tell.

Oh well, if it's not needed anymore, it's not needed It was fun to create anyway.

 

[VinsCool]

30 minutes ago, ivop said:

Oh well, if it's not needed anymore, it's not needed It was fun to create anyway.

 

I mean the part that specifically worked around the output corruption

your loader should now work without the necessity of any built-in workaround, just load tracker.obx with the necessary parts at the end of the file and there shouldn't be anything breaking again

[VinsCool]

Well damn, this is absolutely hilarious.

 

I found what caused the exports to get bloated with garbage data, and this is incredibly dumb.

The Player binaries header (FF FF xx xx xx xx) was causing it the whole time!

 

The way exports were computed would overflow since my own code was ORGing at $B000.

Turns out, EVERYTHING I did worked perfectly, exports broke literally for that reason alone, the whole time.

All the data I moved and pointed elsewhere actually did get loaded perfectly.

 

It was all in the computation code things just broke, LMFAO.

 

I suppose now I can easily fix it, because I actually know what happened this time, hahahaha  

Ah man, 2 days fucking around in Ghidra only to come to this realisation, and just edit the header and boom, exports were proper (well not really,but it was no longer bloated with garbage data!)

 

So I just need to patch the functions for this part to avoid the overflow, and edit few of bytes I need to assemble in my player binaries, and that should all work later  

Ideally, I'd want to replace the exports entirely by my "dasmplayer" recreation, but that would be too much work for much of the same result so I'll just make use of the same data but correctly mapped in the memory.

 

[Edit] Well, maybe that could all be worked around to use my own code, actually.

The way I worked around the binaries exports from ASM directly was to first compute the Player interface, add the text section, which also contained the Rasterbar colour and song speed parameter, but I could bypass the song speed step entirely since my own Simple Player computes it on the fly, then add the rmtplayer on top to it, and finally drop the .rmt at the end of the file, and that is all.

SAP is even easier to work around, it's using exactly the same code I used for the Simple Player, but there it's only the Init data that gets loaded, slapped at the end of the file and voilà.

 

Actually, I think I'll just do that, replace the export code altogether with a much simplier concatenate function directly in the x86 disassembly, since that's more or less all my own simple XEX/SAP player does, so no ORG to edit at all, it's literally just SAP or XEX stuff + Mono or Stereo rmtplayr + RMT Module and that's all.

The SAP and XEX parts that can be edited still use the exact same memory addresses, just much less fucking around.

To prove my point, you could take any or my recent Mono or Stereo exports, regardless of the region and songspeed, and just replace the .rmt at the end of the file using an hex editor, and it will always work, just like that.  

Edited October 19, 2021 by VinsCool
quack quack I'm a rubberduck