Sound list ripper

[Davvel]

Great work everyone involved in exploiting the Programmable Sound Generator (PSG) you really know how to squeeze the TI's capabilities to the last drop.

[Asmusr]

Great work everyone involved in exploiting the Programmable Sound Generator (PSG) you really know how to squeeze the TI's capabilities to the last drop.

 

Thanks. The Sound List Ripper is a fun little app, but Tursi's VGM compressor and player is the tool that has really raised the bar for what we can do with music on the TI.

[Davvel]

 

Thanks. The Sound List Ripper is a fun little app, but Tursi's VGM compressor and player is the tool that has really raised the bar for what we can do with music on the TI.

I agree, although I still have a long way to study to appreciate your (Tursi, Rasmus, Jim, Tony K. etc...) masterpieces.

[Asmusr]

I have added support for Extended Basic CALL SOUND statements, see post #1.

 

I haven't had time to test it a lot, so let me know if you find any sound lists that cannot be converted to XB.

[+OLD CS1]

It seems to only account for changes. Persistent tones in a channel (no change, but attenuation less than >F) are lost in subsequent CALL SOUND statements.

 

Mostly noticeable on longer tunes (songs, like Ms. Pac-Man interludes, etc.) Otherwise, sound effects, short tunes, and what-not work as advertised.

[Asmusr]

It seems to only account for changes. Persistent tones in a channel (no change, but attenuation less than >F) are lost in subsequent CALL SOUND statements.

 

Mostly noticeable on longer tunes (songs, like Ms. Pac-Man interludes, etc.) Otherwise, sound effects, short tunes, and what-not work as advertised.

 

Yes you're right. I didn't account for the fact that CALL SOUND statements cut the volume on any notes that you don't repeat. I will work on a fix.

[Asmusr]

I think it's fixed now, and also the wav file export.

[+OLD CS1]

I think it's fixed now, and also the wav file export.

 

Both tested and confirmed. Most excellent.

[Asmusr]

Is there any explanation for why there are 31 volume levels in XB when the PSG only has 16?

[Tursi]

Marketing.

 

No, I have no idea!

[+OLD CS1]

Is there any explanation for why there are 31 volume levels in XB when the PSG only has 16?

 

Always wondered that. Seems like it just adds extra work to conversion between BASIC and the sound list processor.

[+OLD CS1]

Test Drive subtune 2, car selection music. Work in progress (need to flesh out the end and loop, and the sections of low bass.) Exported to wav using the latest SLR release, converted to mp3 with Lame (variable bit rate using --vbr-new.)

test drive(wip-rc1).mp3

[+OLD CS1]

For the next release I would like to request a loop or repeat function, if only for playback in the GUI.

[RXB]

I figure it is time to upload this one, as it may help decode how TI was thinking when they programmed sound lists. Here's the original documented source code for Parsec, as editable text. It may not be perfect in the GROMs, as I haven't tried to run it through a GPL Assembler (it was written for the TI version, and would require some changes to make it run in the various third-party GPL Assemblers, IIRC). The Assembly portion should assemble correctly now, according to Paul.

This is why some games on the TI were so good considering the real lack of memory the TI was fighting that other computers boasted more of for use.

I did notice in the Source code this is one hell of a strange version of GPL Assembler.

As an example is it does not use the BYTE command for DATA because you can see >0003 and >03 are both DATA but the listing shows one is a BYTE and other is a WORD.

[+OLD CS1]

This is not necessarily practical for use as presented as most of the lists are far too large. But for my purposes, it was an interesting foray into understanding conversions between other PSGs with data in VGM file format and the 9919. Then playing the results.

 

I threw together a quick-n-dirty (well, over the course of a few days quick) Rexx script to convert single-channel PSG VGMs into native ISR sound lists. It mostly does the job and allows you to see how these things work, although in some cases it generates binaries which SLR will not recognize part of, or in the case of tune 8 here, not at all. I have not dug into why this is the case and it is possible I will not ever bother.

 

Original VGM files

Sonic_the_Hedgehog.zip

 

Resulting SLR-loadable binary

Sonic_the_Hedgehog(SMS).bin

 

What they sound like in SLR (as MP3s)

Sonic the Hedgehog (9919).zip

 

And the script which generated this mess

(including some commented debugging material)

 

 

/*
Decode PSG commands from VGM
into TI-99/4 ISR-compatible binary
or binaries if dual-PSG.

(W) 2016 Alan W. Rateliff, II
  
$VER: vgm2isr.rexx 0.2 (12.9.2016) 
*/

/* First, some self-discovery */

parse source . . myself

/* Determine A/Rexx environment and set OPTIONS accordingly */

if arexx() then OPTIONS RESULTS
   else OPTIONS AREXX_BIFS AREXX_SEMANTICS RESULTS TRACE

if arg()==0 | arg(1)=='help' then do
   say 'Usage:' myself 'filename.vgm'
   say
   say 'Will parse PSG commands into "filename.bin"'
/*   say 'or if dual-PSG "filename-left.bin" and "filename-right.bin"' */
   say
   exit 5
end

parse arg filename
  
if ~open(filename, filename, 'r') then do
   say 'Cannot open' filename
   return 30
   end

/* Start checking file */

vgmfile = charin(filename,, 4)
if vgmfile~="Vgm " then do
   say 'Not a VGM file.'
   return 10
   end
  
dummy = charin(filename,, 
psg = charin(filename,, 4) /* offset $0c, 32 bits */

/* TODO: $0c is the frequency setting for the PSG.  Anything varying from the 9919's
   clock frequency or equivalent will sound off when converted, thus we will have to
   make adjustments, eventually. */

if c2d(psg,4)==0 then do
   say 'Not a VGM file for PSG.'
   return 11
   end

/* Will only check for the PSG flag at $0c/32 to be non-zero or specifically $80,
   which indicates a dual PSG file.  There are other PSG-related fields in
   the VGM spec, but we are going to ignore them.
*/

/* sound_list. = x2c('049FBFDFFF00')  // define base sound list row */
sound_list. = ''
sound_list.0 = 1

/* Open necessary output file or files */



/* FOR TESTING: only output PSG #1 data */

/* Begin building each ISR sound list row */

bytes = 0
duration = 0
row = 1
  
do until eof(filename)
   vgmbyte = charin(filename,, 1)

   select
      when vgmbyte=x2c('50') then do
         if duration > 0 then do
     /* say 'data byte after duration'
     say 'closing row' row': bytes 'd2x(bytes)' duration 'd2x(duration %735)', 'duration */
            sound_list.row = close_row(bytes,sound_list.row,duration)
    
/*     call writech 'STDOUT', 'BYTE >' || c2x(substr(sound_list.row, 1, 1))
            do byte = 2 to length(sound_list.row)
            call writech 'STDOUT', ',>' || c2x(substr(sound_list.row, byte, 1))
            end
            call writeln 'STDOUT', ''
*/    
            bytes = 0
            duration = 0
     row = row + 1
     sound_list.0 = row
            end
        
  data = charin(filename,, 1)
/*  say 'data byte' c2x(data) */
  sound_list.row = sound_list.row || data
  bytes = bytes + 1
  end
      when vgmbyte=x2c('61') then duration = duration + c2d(reverse(charin(filename,, 2)))
      when vgmbyte=x2c('62') then duration = duration + 735
      when vgmbyte=x2c('63') then duration = duration + 882
      when vgmbyte=x2c('66') then
         if row > 1 then do
     sound_list.row = close_row(bytes,sound_list.row,duration)
     sound_list.0 = row + 1
     leave
     end
      otherwise nop
      end
   end  

call close filename
  
if vgmbyte~=x2c('66') then say 'End of file reached without end of sound data marker 0x66.  File may be incomplete or corrupt.'

row = row + 1
sound_list.row = x2c('049FBFDFFF00')
sound_list.0 = row

/* Opening output files is killing me in Regina REXX.  I'll figure it out in time. */

   if ~open('out', 'out.bin', 'w') then do
      say "Can't open output file"
      exit 50
      end
     
   do row = 1 to sound_list.0
        if arexx() then call writech out, sound_list.row else call charout out, sound_list.row
      select
  when arch=='ARexx' then call writech out, sound_list.row
  otherwise call charout out, sound_list.row
      end
   end
  
   call close out

exit

   do row = 1 to sound_list.0     /* decode sound list in compound variable */
      call writech 'STDOUT', 'BYTE >' || c2x(substr(sound_list.row, 1, 1))
      do byte = 2 to length(sound_list.row)
         call writech 'STDOUT', ',>' || c2x(substr(sound_list.row, byte, 1))
      end
      call writeln 'STDOUT', ''     /* newline */
   end
     
exit
  
close_row:
   arg bytes,sound_list_row,duration
   duration = duration % 735
   if duration = 0 then duration = 1
   return d2c(bytes) || sound_list_row || d2c(duration)

  
/* Detect ARexx */

arexx:
   parse upper version arch
   return choose(left(arch,5)=='AREXX', 1, 0)
  
choose: return arg(arg()-arg(1))

 

 

 

And holy crap look at the time... been playing with this far too long and need to get to bed!!

 

EDIT: Oh, dear... I realize how much I reused some old, sloppy code in this script. Never code tired, kids! *tsk tsk tsk*

[Asmusr]

Cool, it's interesting to see how tunes like this appear in sound list format. But it also shows why this format is inadequate compared to Tursi's compressed VGM format. The same tunes that take up 107 KB in sound list format only take about 20 KB in compressed VGM. That's actually better than I would have expected considering the lack of loops or other means to reuse data. (AFAIK sound lists do support loops but only as GOTOs to absolute addresses, making them difficult to use for any practical purpose.)

[+OLD CS1]

Cool, it's interesting to see how tunes like this appear in sound list format. But it also shows why this format is inadequate compared to Tursi's compressed VGM format. The same tunes that take up 107 KB in sound list format only take about 20 KB in compressed VGM. That's actually better than I would have expected considering the lack of loops or other means to reuse data. (AFAIK sound lists do support loops but only as GOTOs to absolute addresses, making them difficult to use for any practical purpose.)

 

Yeah, you have to do the lists in code in order to do loops so you can put a label where you want the loop to be (I do that in 4Anoid.) I have not confirmed it, but I will be looking through Tursi's VGM compression code because I am pretty certain he uses a note look-up table. While I am working on my player, I am curious to see how others handle shrinking the data down. I am multiplexing the volume and note data in a single word. I am working with the concept of patterns and pattern lists versus loops and jumps as well as instruments which can be complex or simple, cross-channel transposition, manipulating channels based upon values of other channels with or without delays, and a bunch more.

 

I see now where the Parker Bros. guys were talking about being redundant. I have also been working on some songs from Last Ninja 2 and found that there is a ridiculous amount of duplication and waste matching the SID renditions. I would like to learn more about how the ColecoVision's song player works as I would like to be able to extract sounds and music from CV carts.

[Asmusr]

I should really leave this for Tursi to explain, but AFAIK he is splitting the music into 12 streams: 3 for each generator, and further, for each generator, into a note stream, a volume stream and a duration stream. To unpack each stream you would usually use a dictionary algorithm to lookup data in the buffer you have already unpacked, but on the TI we don't have room for buffers so Tursi is referring to the packed data instead, which is pure genius, especially since it works .

[Tursi]

The compression ratios are a little misleading -- VGMs have a lot of extra data in them. the first thing the converter does is play out the VGM into memory so that it has a frame-by-frame stream of the entire song. Then it runs through that and RLE encodes each channel (4 voice, 4 volume) - this generates the time streams for each channel. Every byte written to the sound chip takes two bytes in a VGM, so right off the bat they are larger than they need to be. But the simple format makes them great for tools.

 

It then strips all the channel specific information from each stream so that notes going to channel 0 look the same as channel 2, and so maybe can be reused. Same deal with volume (although I toyed a lot with packing volume into nibbles, I didn't and it uses a full byte.) So that's 8 streams - the other 4 are timing streams that simply indicate how many frames before the channel needs to change state and when it does, whether to load voice, volume, or both.

 

The packer simply looks backwards through compressed data to see if the string it's currently looking at already exists - it takes the longest string it can find up to 63 bytes. (A potential but complex extension would be to look forward as well to try and figure out the best maximum length). Playback is mostly just counting and updating pointers - it's the number of streams that cost the CPU time.

 

There are a couple of simple tweaks to try and save space. All notes (up to 256 of them) are stored in a note lookup table so that voice updates can be a single byte. The time stream tends to have very frequent updates, so a couple of dedicated codes exist to represent fixed streams of bytes in there. It doesn't do any data output smaller than a byte in order to save the time needed for bit manipulation. But that's really it.

 

There are some nice tricks by using lookups in the uncompressed data that I can't take advantage of, but this packer managed to get my target song (680Rock) to fit in the TI's RAM, if only barely. That was my goal. The hope was that a packer like this could automatically locate re-used sequences like loops and choruses, say, and so get similar benefits to having them. From watching the Classic99 heatmap during song playback, it does look like that works (if a little more scattered than I expected)... it's the time streams that tend to grow the fastest.

 

I can send you the code, OLD CS1, just ask that you consider it for private use and realize that the code is such a mess of experimentation that it may not be readable

[+OLD CS1]

I would like that, thanks. I thought I would find the source in your converter, but I admit I have not yet looked.

 

I took a few seconds to look through the binaries created by my vgm2isr and think I figured out why SLR does not like some of the results. It looks like some of these VGMs take advantage of the SN76489's latching functionality. This means extra data bytes will appear in the stream with no leading "command" or "latch" byte. Ultimately not a big deal as the conversion to ISR is more academic than practical.

 

If not academic then at least informative. I had never seen the white noise used as effectively and sound so good before examining these conversion. It has changed how I look at my percussion designs.

[+OLD CS1]

I also found in Afterburner's Final Take Off that all of the commands and values are correct but apparently not in an order SLR wants to load.

For instance, the first "row" is

BYTE >0B,>80,>00,>A0,>20,>C0,>00,>E4,>92,>B6,>D5,>F0,>01

This is all valid, but it sets values for each tone, first, then sets volumes. Entering in the data manually into SLR results in this row data

BYTE >0B,>80,>00,>92,>A0,>20,>B6,>C0,>00,>D5,>E4,>F0,>01

I am thinking I can run a quick clean-up on the rows before emitting them from the script, but this also makes me wonder how many sound lists get missed or only partially loaded into SLR because the data is seen as out-of-order. Such flexibility would probably make list detection more difficult.

[Asmusr]

I also found in Afterburner's Final Take Off that all of the commands and values are correct but apparently not in an order SLR wants to load.

 

For instance, the first "row" is

BYTE >0B,>80,>00,>A0,>20,>C0,>00,>E4,>92,>B6,>D5,>F0,>01

This is all valid, but it sets values for each tone, first, then sets volumes. Entering in the data manually into SLR results in this row data

BYTE >0B,>80,>00,>92,>A0,>20,>B6,>C0,>00,>D5,>E4,>F0,>01

 

I am thinking I can run a quick clean-up on the rows before emitting them from the script, but this also makes me wonder how many sound lists get missed or only partially loaded into SLR because the data is seen as out-of-order. Such flexibility would probably make list detection more difficult.

 

I don't think the problem is the order of the commands. Could you post the actual file you're trying to import?

[+OLD CS1]

 

I don't think the problem is the order of the commands. Could you post the actual file you're trying to import?

 

Hang on... I went through and entered the missing lines manually in to SLR, and I actually think the problem is the last "missing" row has multiple commands for the same tones (sends to tone 1 value and attenuation twice.) I missed that before. I am going to cut that line and see what happens.

[Asmusr]

One thing to note is that if you try to import very small examples SLR will reject them because it requires at least 3 rows (to filter out a lot of false positives).

[+OLD CS1]

One thing to note is that if you try to import very small examples SLR will reject them because it requires at least 3 rows (to filter out a lot of false positives).

 

Go to know, but no worries here... after conversion from VGM logs these are ridiculously large Have stuff going on at home right now so I am not focused on results in the immediate future. I will be back.