RMT Hacking Ideas and Progress

[VinsCool]

Hey guys, so I'm moving this topic here since I already went very offtopic in my PoKEY music thread.
I finally have some good news about the progress done on this.
 

 

Took me some time but I finally found what mostly failed injecting back into RMT.exe-- my hacked code was literally not compatible with it, even though it actually works for sure.

so I have to split into 3 parts:
- TRACKER binary, that is essentially the STEREO player code without complex hacked up stuff (so it takes a lot more space to even work at all)
- Mono binary, for exports, already confirmed working with the hacked up code
- STEREO binary, for exports, already confirmed working with the hacked up code
The good news however was that I was able to get the 16-bit table to only load when matching AUDCTL value was used, and Distortion A was also used, else it won't use it.
The bad news is the TRACKER binary needed 4x more data for the code I could fit into it, so that means my other hijacked things like Distortion C 15khz table won't fit into it, unless I find a way to insert larger Atari 8-bit binaries into the .exe without breaking it.
I spent  bit of the evening to go by elimination and came to the conclusion that it really did not like when I used x as a pointer for the memory addresses, so rip my fuckery that saved 4x the amount of space it needs right now.
However, that still works fine into .xex exports, so that's only half a loss, thankfully  

[VinsCool]

Okay good news, I finally have fixed my code to be compatible with the RMT exe now.
I had to sacrifice one of my hijacked thing, but having 16-bit using AUDCTL and Distortion as a trigger operational is definitely be good enough for the time being  
I just left my earlier hackjob commented out, I do know for sure it worked in stand alone XEX exports however.

Now I really need to take a different approach in order to have tables loaded through a condition...

Hijacking after all the routines for notes and pitch were executed is not ideal at all, there has to be a way to do it differently...

Changelog for this hacked RMT version since my earlier Tuning patch 15:
- This version was based on Analmux's RMT Patch 8, so several of his changes remain in place, mainly the FILTER not muting the other channel, for example
- Most of the work was done directly in the rmtplayr.a65 code since the last patch! This will allow way more things to be done as I slowly learn how to code in 6502 ASM...
- New and improved volume lookup table, saved 128 bytes of data, at the cost of some extra cycles to be executed. Special thanks to TakuikaNinja for the idea and help implementing it! This doesn't  seem to sound any different from the old code, yet the space saved is considerable!
- Improved several of my tuning tables, based on my spreadsheet maths and formulae figured out over time.
- Re-ordered several tables based on the discovery earlier that a full 256 bytes could be addressed, so the most optimal setup for the moment has been squeezed. This is not definitive yet, so changes are likely to happen as I understand better.
- 16-bit trigger now depends on having both the correct AUDCTL value, either $50 for 1+2 or $28 for 3+4, and must also use Distortion A to work at all.
- Added a dumb check for if Command 1 was used as well, will skip the hijack if detected, in order to keep certain things like percussions functional.
- 16-bit table is loaded from the space cleared from the optimised volume code, nothing lost and nothing new!
- Distortion 6 hijack is removed entirely, so instead it's temporarily replaced with an optimised Distortion A table for 15hz bass, with matching tuning to the other ones.

More can be done, I am sure of it!
Unless I fucked things up everything should play the same as before but now with some extras available, except for the 16-bit and Distortion 6 changes done like I mentioned above.  

Also, not seen in the tracker itself, there is a Distortion C 15khz table hijack, will work by using AUDCTL 01 and Distortion C, but only in the XEX exports for now, since my earlier hijack code broke the tracker lol

Enjoy, I hope I will be able to do more soon!

RMT Patch 16 Beta2.zip

[tane]

I'm happy to feel several improvements testing some RMT. But in some cases it seems that a new set of instruments is required comparing the original and new output, in order to have a similar sound. Example at 60Hz: sketch.rmt

 

Also, is it possible to have the "SONG" map in mono as big as it appears in stereo?

 

 

Edited July 9, 2021 by tane

[VinsCool]

4 minutes ago, tane said:

I'm happy to feel several improvements testing some RMT. But in some cases it seems that a new set of instruments is required comparing the original and new output, in order to have a similar sound. Example at 60Hz:

Hey there, thanks!

I will check that in a moment, because they should be mostly compatible, but it's possible some tweaks are needed between some module files, since I did change a couple things.

 

 

 

 

6 minutes ago, tane said:

Also, it it possible to have the "SONG" map in mono as big as it appears in stereo?

Unfortunately I don't know how to change that sort of detail yet, sorry about that.

[rensoup]

19 hours ago, VinsCool said:

Took me some time but I finally found what mostly failed injecting back into RMT.exe-- my hacked code was literally not compatible with it, even though it actually works for sure.

So what was not compatible ? Is it not just a case of using more space than the original ?

[VinsCool]

35 minutes ago, tane said:

But in some cases it seems that a new set of instruments is required comparing the original and new output, in order to have a similar sound. Example at 60Hz: sketch.rmt

 

Ahh I see what happened: you tried to use the RMT Distortion 4, which was the Sawtooth in RMT Patch 8, however, it is left out from my version currently. Sorry about that!
All it's doing was re-introducing the actual Distortion 4 for some effects that may be wanted, but there is no proper tuning table assigned to it yet.

29 minutes ago, rensoup said:

So what was not compatible ? Is it not just a case of using more space than the original ?

I went by elimination but basically, what I originally did was store x in the tmp memory location, then load it again after the whole 'hijack' code, since I used x as a pointer to identify what channel was being used, so I was the able to use the same code in several places, saving a lot of space to fit what I was doing, and could be used for all 4 channels, even in stereo.
For some reason, that just kept breaking RMT, the tracker, outputting garbage in the stereo channels, and made my entire code not work at all... so I went back to the original approach of "1 at the time", which used a lot more memory.
I was still able to fit my new 16-bit hijack, but everything else that could have been used, like Distortion C at 15khz loading its table in a different location, simply didn't fit the size I am currently limited to.
That whole mess of code however was working just fine when I slapped the assembled binary onto tunes exported from RMT, so it was only specific for the tracker part.

Hopefully a much better method can be used later  

[VinsCool]

Just to make a bit more sense with how to use this RMT version, here's a quick breakdown of the features and how to use them.
RMT Distortion (I'll call them 'slots') are mapped the way that mostly makes sense to the actual POKEY Distortions are numbered, so here's what I did currently:

Slot == POKEY Distortion

0  ==  Distortion 0, Pure Table (by default)
2  ==  Distortion 2, Poly5 Table (Similar to Patch 8)
4  ==  Distortion 4, Pure Table (by default)
6  ==  Distortion A, 15khz Table (Better 15khz tuning for the bass range)
8  ==  Distortion 8, Pure Table (by default)
A  ==  Distortion A, Pure Table (Alternate Tuning, optimised to 64khz and reference for all tables), 16-bit must use AUDCTL= $50 or $28 too!
C  ==  Distortion C, Buzzy Bass Table (with matching tuning, Notes from C-1 to G-1 are mapped to the Gritty Bass tones as a filler)
E  ==  Distortion E, Gritty Bass Table (with matching tuning)

I have extra tables in the code available, but ran out of space to actually use them currently:
Distortion C 15khz, with matching tuning
Distortion E 1.79mhz, with matching tuning

In the .xex exports, the Distortion C at 15khz will work as long as AUDCTL= $01 and Distortion C are used in any channel.

This is not a definitive guide, so things are likely to change or move, over time.

Edited July 10, 2021 by VinsCool
forgot some details

[rensoup]

3 hours ago, VinsCool said:

I went by elimination but basically, what I originally did was store x in the tmp memory location, then load it again after the whole 'hijack' code, since I used x as a pointer to identify what channel was being used, so I was the able to use the same code in several places, saving a lot of space to fit what I was doing, and could be used for all 4 channels, even in stereo.

I had a quick look at your code but didn't notice anything obvious related to your usage of tmp.

 

Not related to your problem but shouldn't that ldx tmp be inside the EIF ?

    EIF
hijackfinished_    
    ldx tmp                 ; restore x from temporary location now that we are past the blocks using it as an offset

 

Strange that tmp seems only used once in the original code ? I wonder why he needed that variable at all...

 

You could try pushing x on the stack ?

txa

pha

 

then popping it

pla

tax

 

if there's nothing needed in register A of course. 

 

Does the problem occur only in stereo ?

[VinsCool]

3 hours ago, rensoup said:

Not related to your problem but shouldn't that ldx tmp be inside the EIF ?

Yeah I think I made a mistake there lol
Not that this changed anything there since I wanted that part of the code anyway.

3 hours ago, rensoup said:

Strange that tmp seems only used once in the original code ? I wonder why he needed that variable at all...

 

You could try pushing x on the stack ?

txa

pha

 

then popping it

pla

tax

I have no idea either, honestly. That was the reason I used tmp to back x up, I thought it would be "safe", and seemed to be, based on how I had things go in the Altirra debugger.
Could be worth trying to use the stack next time! I'll make sure to remember this idea  

3 hours ago, rensoup said:

if there's nothing needed in register A of course. 

 

Does the problem occur only in stereo ?

It was weirder than that, the Right POKEY would be stuck with FF as the AUDCTL infinitely, and even trying to use the tracker in mono would carry some garbage from the Right side, and so playback had glitches.
Then the whole hijack code would simply not work at all, despite XEXes working perfectly with the same code.
I stopped questioning that as soon as I identified what broke the Tracker side, and just went with what I was able to make use of lol

[Heaven/TQA]

4 minutes ago, VinsCool said:

Yeah I think I made a mistake there lol
Not that this changed anything there since I wanted that part of the code anyway.

I have no idea either, honestly. That was the reason I used tmp to back x up, I thought it would be "safe", and seemed to be, based on how I had things go in the Altirra debugger.
Could be worth trying to use the stack next time! I'll make sure to remember this idea  

It was weirder than that, the Right POKEY would be stuck with FF as the AUDCTL infinitely, and even trying to use the tracker in mono would carry some garbage from the Right side, and so playback had glitches.
Then the whole hijack code would simply not work at all, despite XEXes working perfectly with the same code.
I stopped questioning that as soon as I identified what broke the Tracker side, and just went with what I was able to make use of lol

Saves CPU cycle in critical code… I never use stack even todays in NMI and DLI or VBL code but store registers in ZP.

[VinsCool]

3 minutes ago, Heaven/TQA said:

Saves CPU cycle in critical code… I never use stack even todays in NMI and DLI or VBL code but store registers in ZP.

Makes sense, I'm only a beginner so I need to adapt to why certain things were done a specific way.  

[Heaven/TQA]

19 minutes ago, VinsCool said:

Makes sense, I'm only a beginner so I need to adapt to why certain things were done a specific way.  

Yeah but me runs into issues that I totally forget PHA etc as it’s not in my mind set since ‘86 ? 

[rensoup]

15 hours ago, VinsCool said:

It was weirder than that, the Right POKEY would be stuck with FF as the AUDCTL infinitely, and even trying to use the tracker in mono would carry some garbage from the Right side, and so playback had glitches.
Then the whole hijack code would simply not work at all, despite XEXes working perfectly with the same code.

That makes no sense... the stereo code isn't compiled in the mono exe, so it shouldn't be able to ouptut any sound from the 2nd pokey ?

 

[VinsCool]

2 hours ago, rensoup said:

That makes no sense... the stereo code isn't compiled in the mono exe, so it shouldn't be able to ouptut any sound from the 2nd pokey ?

 

The way I understand how RMT.exe works, it is using 3 PLAYER binaries, it was unclear why but I figured this out recently.

- 1 STEREO PLAYER binary for the Tracker itself, which is the one that fucks itself when I tried to hijack my code into it
- 1 MONO PLAYER binary for exports
- 1 STEREO PLAYER binary for exports, sitting only a few bytes after the MONO PLAYER

I can also confirm the last 2 work just fine with my code, both XEX and SAP, actually!

The Tracker side however is weird, the way it appears to work is that it's actually always in STEREO mode, it seems!
The only thing that changes is, when it is in MONO mode, the Left side gets panned to the center, then the Right side becomes... unused, but technically still active.
Now I still have no idea why specifically the tracker side would break with my code, but I suppose more things are involved, and most likely would overwrite the area I was trying to use... That's only a theory, but that might be the reason why it behaves differently.

Edited July 11, 2021 by VinsCool
typo

[rensoup]

1 hour ago, VinsCool said:

- 1 STEREO PLAYER binary for the Tracker itself, which is the one that fucks itself when I tried to hijack my code into it

Oh... I figured there were xex for each mode...stereomode 1/2/3

 

that's different then... it means he must be patching the code to change stereomode on the fly when you change it in the editor.

 

there are labels like this:

xstastx01

 

which aren't referenced anywhere... this may be where patching occurs... those labels may be included in the editor when it gets compiled or he may be doing a search for a certain code pattern within some range.

 

Do you always make sure these labels are assembled at the same location ?

 

1.If they are always at the mem location then your code that doesn't work may be moving those.

2.If they aren't it may be doing a search for a code pattern within some range but you extra code is moving it outside of that range?

 

in either case, it would mess up the patching and cause weird errors like you're having.

 

Well that's just a wild guess...

 

[rensoup]

or maybe he's just swapping the channels before sending it them to the pokey emulator (and not patching the xex at all... )

[VinsCool]

I have tried moving code the other day, it did no difference, so I am betting on certain things may happen on the fly, and they aren't on the Atari itself when executed.
It's hard to guess going blindfolded like that, lol

I tried looking at the very very old RMT sources but that didn't give me much clue, I first suspected hard coded memory addresses, but that is probably not what was happening since at the end I had my code do mostly the same things I wanted, but differently, and that eventually worked anyway, lol

[VinsCool]

Hijacked Distortion C 15khz table, loaded outside of frqtab: SUCCESS
 

It involved a different approach, and it's just as cursed as the other things I had attempted so far  
Some of my disgusting code involved below...

First of all, this was written in the location where the old Low Byte table for 16-bit bass was, since this is one of the only spots I have left unused:

org PLAYER-$100-$140

tabbeganddistor_15
	dta frqtabbuzzy15khz-hijacktab+$D0,$00
	dta frqtabbuzzy15khz-hijacktab+$D0,$20
	dta frqtabbuzzy15khz-hijacktab+$D0,$40
	dta frqtabbuzzy15khz-hijacktab+$D0,$a0
	dta frqtabbuzzy15khz-hijacktab+$D0,$80
	dta frqtabbuzzy15khz-hijacktab+$D0,$a0
	dta frqtabbuzzy15khz-hijacktab+$D0,$c0
	dta frqtabbuzzy15khz-hijacktab+$D0,$c0

hijacktab

; Buzzy Bass Table, Slot C, Distortion C, 48 bytes, 15khz, Optimised for bass, matches Alternate Tuning

frqtabbuzzy15khz
        dta $BC,$B2,$A8,$9E,$96,$8D,$85,$7E,$76,$70,$6A,$64
        dta $5D,$58,$53,$4E,$4B,$46,$42,$3E,$3A,$37,$34,$32
        dta $2E,$2B,$29,$26,$25,$23,$21,$1F,$1C,$1B,$1A,$19
        dta $17,$16,$15,$14,$12,$11,$10,$0F,$0D,$0D,$0C,$0C

Then, the hijack code in question:

pp2scont_check
	lda trackn_audctl,x     ; Load AUDCTL value into the accumulator
	cmp #$01                ; Compare to the value required for 15khz clock, anything else should be ignored to avoid conflicts
    bne pp2scont_normal     ; No hijack, carry on like normal
	lda reg2                ; Load the value of Instrument Slot, or RMT Distortion, currently from reg2 at this point(?)
    and #$0E
	cmp #$0C
    bne pp2scont_normal     ; No hijack, carry on like normal

pp2scont_hijack15khz
	lda tabbeganddistor_15,y
	sta nr
	lda tmp
	ora tabbeganddistor_15+1,y
	sta trackn_audc,x
	lda #>hijacktab         ; Move the tables pointer to the hijacked location, in order to load the right values
	sta nr+1
	jmp InstrumentsEffects   ; Continue like normal right there

There was just another important detail with this method, however; the pointer would then be moved for everything too but only when the Distortion C hijacked happened!
Quick and dirty fix for it:

cmd0c
	sta trackn_audf,x

	lda #>frqtab    ; Hopefully that fixes the fuckery related to hijacking different table pointers, lol. If that fails, the other instruments will still output garbage...
	sta nr+1

And there we go, the hijacked Table pointer successfully did its thing, then as soon as I got the frequency in AUDF, I revert back to the frqtab pointer in order to not break 99% of everything else LOL
This approach is obviously gross, but now I have a good idea of what I could do for it, and better than that, I might be able to use that method to do all sort of things like the 16-bit hijack and the earlier code that broke RMT for basically the same purpose  

Hopefully, I will be able to show some more results later, unless I break everything again, hahaha!
By the way, I am sorry for any botchered code there, I only have been coding since... 1 week or so? Seriously, I'm a total noob.

[VinsCool]

Raster Music Tracker Patch 16 Beta3
Changelog:
- Redone the entire hijack code using a different entry point, this made everything a lot easier to manage, and future-proof in case more tables and more hacks get added in the future
- Hijack code considerably cleaner, and should take a lot less memory and cycles to run, with hopefully no penalty on everything else.
- Got rid of the 'BASS16' code entirely, this was the base from the earlier attempt, but with the newest way to handle 16-bit, it was no longer required! saved a lot of space by the same way.
- Distortion A and Distortion C now have a fully operational switch to appropriate 15khz tuning tables, simply use the 15khz bit in the instrument to use it!
- Copied Distortion C back into Slot 6 again as a fallback for compatibility reasons, the Slot might be used for other stuff in the future, so nothing is definitive!
- Added a set of 'tabbeganddistor_extra' for the hijacked code, so mapping tables to specific slots will then be very easy to add, move, and change to different ones in the future
- 16-bit hijack code has been done in such a way that now each tables, Hi and Lo, can be used independently in their appropriate channels, automatically, using the required AUDCTL value, being $50 and $28, and must also use Distortion A
- 16-bit can also effects! just edit instruments like normal, so vibrato, detune, portamento etc can be used, however things are limited a bit due to the channels not operating as a 16-bit number
- 16-bit has been future-proofed as well, so now it would only be a simple change to, for example, allowing volume output in the Lo channels, to produce Reverse-16 sounds with the appropriate tuning tables
- There is a catch however, you have to make sure both Hi and Lo channels have the right instruments playing, else some bad (or maybe intended) sounds will be also output, an easy way to handle is to simply use the same pattern in each channels, the appropriate table will be used in each.

RMT Patch 16 Beta3.zip

[emkay]

13 hours ago, VinsCool said:

 

RMT Patch 16 Beta3.zip 3.25 MB · 4 downloads

I might give it a try. But...

 

Name me picky. 

I'm still missing the clean switch to gen 2 at 1.79MHz using the correct note table, if needed.

Gen 2 is the only "additional" waveform, that has a good range of clean notes , and it can sound like a modulation of gen. A.  

[VinsCool]

31 minutes ago, emkay said:

I'm still missing the clean switch to gen 2 at 1.79MHz using the correct note table, if needed.

 

The Distortion 2 table is intended for 1.79mhz yes, with the correct tuning too.
There is no table yet for other modes, because I ran out of space for now.

[VinsCool]

Hey guys, it's been a while since I played around the rmtplayr code, and made some pretty important progress with it since the last 2 days.
So far, I don't have much to show, except that the entire code I had made very poorly before, thanks to being a beginner programmer, has been rewritten in a much better way.

Compared to the last Beta 3 I posted here, these are the current improvements that have been done:
- Rewritten the entire hijack routine, in a much more compact and intelligent way, avoid wasting a lot of resources (still in progress, being optimised and tested as I am typing this).
- The entire logic behind the hijacked functionalities are 100% based on the global AUDCTL value, and in a priority order. Everything can now be controlled from any instrument parameter, from any channel!
- A failsafe jump has been added in case nothing is working, if a combination fails, or isn't possible, the player will jump to the "No Hijack" branch, effectively skipping any code that would write garbage in memory.

- Future proofed the routine to be able to add even more special checks easily, in a more elegant and clean way.
- Cleaned up many redundant blocks of code, mainly my own, and few bits here and there from the original code when I find a possible optimisations.
- Considerably reduced the hog of memory and cpu cycles compared to my earlier attempt, since I am still learning how to code in 6502 ASM.

It gets better, slowly, but surely.  
Things are still in progress, and I am making sure to thoroughly test everything just in case I broke something, I thought a status update was a good idea for now  

Other than that, all the changes from the last version are still present, just executed better.
For now, I'm optimising my newer and better code, and as soon as I can making everything fit within the .exe binary size limits, and confirm everything works, I'll post a new version.

I can also confirm that everything made in Patch 16 Beta 3 will work virtually exactly the same, since that one had the same idea in place,.
It used to be much crappier, and execute the hijacked stuff 'per instrument' setting, and now I can say, that simply was very inefficient and often, worse to work with.
 

Here's a very boring video to show some of the improvements I was able to make.

This is the new rmtplayr code into RMT.exe, so far things look very promising!

This is just me fucking around, 1 instrument sets the AUDCTL
Everything is being updated on the fly
That probably doesn't look like much, but it's going to be a massive improvement for ease to use.
Near the end of the video, I enabled the filter, and did a very smooth PWM with the 16-bit mode combined to it :3
Again, 1 instrument setting for all the modes, and all the tables are loaded automatically  
There was only the PWM part I edited into the EMPTY instrument to adjust the pulse width, hehehe.

I'll keep you all updated when I have something new to share!

[VinsCool]

RMT Patch 16 Beta5 is now available!

Changelog:

- Rewritten the entire hijack routine, in a much more compact and intelligent way.
- The entire logic behind the hijacked functionalities are 100% based on the global AUDCTL value, and in a priority order. Everything can now be controlled from any instrument parameter, from any channel!
- A fail safe jump has been added in case nothing is working, so if a combination fails, or isn't possible, the player will jump to the "No Hijack" branch, effectively skipping any code that would write garbage in memory.

- Future proofed the routine to be able to add even more special checks easily, in a more elegant and clean way.
- Cleaned up many redundant blocks of my own code, and few bits here and there from the original code by Raster.
- Considerably reduced the hog of memory and cpu cycles compared to Beta3, which was very unclean and poorly optimised.

Functionally, this version is identical to Beta3, with the exception of tuning tables being handled in a much more efficient way, tracking music in RMT a much more comfortable experience now
There are some caveats, however.
The 16-bit code I introduced is based on the new hijack code, with most of the same downsides from the earlier version, but it allows much more control over it, such as effect commands, no pain no gain right?  
Other than that, I have given this version a very thorough test run for most of the evening, and pretty much everything is working as expected, including Stereo POKEY, and XEX/SAP exports.

Here's a quick reminder of the way the Patch 16 RMT slots and Distortions are distributed, as well as their hijacked equivalents:

RMT Slot 0 - Distortion 0 - Pure Table
RMT Slot 2 - Distortion 2 - Poly5 Table 1.79mhz
RMT Slot 4 - Distortion 4 - Pure Table
RMT Slot 6 - Distortion C - Buzzy Bass table, for compatibility reasons (eg modules made with older RMT versions using distortion 6 for 16-bit)
RMT Slot 8 - Distortion 8 - Pure Table
RMT Slot A - Distortion A - Pure Table, 16-bit with AUDCTL==$50 and AUDCTL==$28 (Join 1+2/3+4 and/or 1.79mhz ch1/ch3) / 15khz table with AUDCTL==$01
RMT Slot C - Distortion C - Buzzy Bass table, 15khz table with AUDCTL==$01
RMT Slot E - Distortion E - Gritty Bass table

From here, I don't know what else I could to to improve what I introduced or changed.
I am at a byte exact limit for the MONO player hacked into the .exe, STEREO and TRACKER player binaries can still afford some more bytes of code, but I doubt I could fit anymore things without breaking even more things.

The rmtplayr I am currently using for the visual player project has been mostly a playground for experimenting with my own code as I am learning.
Having skipped Beta4 is not an accident-- it was just too broken to even fit into the tracker!
I've dragged over Beta5 the idea I first introduced to the visual player, and optimised the code as much as I could to make it fit, and do exactly the same thing, except in a much cleaner way to save memory  
Now that I know this works I'll backport these improvements to the visual player later, but for the now, this is the new version I wanted to share with you all today.

Thanks to everyone who were able to help me during this journey, I hope this will make more people addicted to Atari music! Hehehe  

In the .zip below, I included the patched .exe as well as the necessary .dlls, and the rmtplayr code I edited, as well as prebuilt player binaries that were inserted into the main program.

RMT Patch 16 Beta5.zip

[emkay]

Hm...

 

Is it possible to set gen.2 for one position on the Envelope, and AUDCTL gets turned from basic clocking to 1.79MHz clocking for that time?

[ivop]

Are the MONO, STEREO and TRACKER binaries in consecuetive memory in the Rmt.exe binary? If so, perhaps you can merge them?