VGM Compression Tool

[Tursi]

For a long time I've been playing with this, and I think it's finally ready to release. I leaked the first version of the playback libraries over a year ago (but not the compressor!), and two demos (Chuck Rock and the Piano). Since then it has been improved and solidified to the point where I am ready to support it.

 

This is a tool that packs VGM files into a new compressed playback format, and four playback libraries, each with slightly different features:

 

- playback music only at 60hz

- playback music only at 30hz

- playback music and sound effects (with priority) at 60hz

- playback music and sound effects (with priority) at 30hz

 

The 30hz versions are intended to reduce the size of the packed audio data (slightly) and improve CPU load by processing less often. The sound effect playback is a single sound effect at a time - lower priority sound effects are discarded. All versions also provide memory locations where you can read back the current state of the four music channels (whether or not sound effects are playing, they always report the music).

 

The amount of memory needed depends on the player you choose to use, from 124 bytes of data and 608 bytes of code for the 60hz music only version, to 254 bytes of RAM and 1004 bytes of code for the 30hz music plus sound effects version. All players also require a workspace, which can be used for scratch data inbetween calls. (I store it in the scratchpad).

 

The playback libraries are intended for use with GCC and are included in my "libti99" at http://www.harmlesslion.com/software/vgmcomp, and to make life easy, here's the demo program (in both TIFILES and DSK format!) Tested in Classic99 and MESS.

 

 

vgmdemoti.zip

Edited May 16, 2014 by Tursi

[Tursi]

The playback libraries are intended for use with GCC and are included in my "libti99"

I forgot to say "BUT, raw assembly versions for Asm994A are also included, along with instructions for use."

[Asmusr]

Thank you Tursi, this is a great step forward. I'm looking forward to using it in my next game project.

[Willsy]

So, what's VGM, and how does one create the music?

[Asmusr]

So, what's VGM, and how does one create the music?

 

http://www.smspower.org/Music/Software

[Tursi]

That's funny.. I distinctly remember writing a paragraph on what VGM is and where you get it... oh well.

 

Besides the link RasmusM posted (which is the definitive archive and host for documentation of the format), VGM is also output by my MOD converter (http://harmlesslion.com/software/modconvert), and other less complete converters that I need to finish and release. You can also get standalone players and Winamp plugins for listening to the files under your OS of choice.

 

VGM is also one of the export options for the PSGMod Tracker that I was endorsing a few years ago - so anything you can load or compose in that can be exported as VGM. The compressor doesn't care if the output file is optimized or bloated, since it does its own unpacking and re-packing.

 

I didn't mention, but the compressor should also transparently handle gzipped VGM files (often, but not always, saved as ".VGZ") - although the resulting file will end up larger in that case, since it's not quite as good compression as GZIP.

[TheMole]

This is awesome. Makes me anxious to be able to pick up working on my platform game again, if only work wasn't such a time sink right now...

Just out of curiosity... How many CPU cycles does every tick take, on average?

[Tursi]

It varies a fair bit, because it unpacks anywhere from 0-12 streams every tick for each track (so it depends on how busy the tune is, and music counts as one track, an active sound effect counts as a second). I did some measurements close to the end of development, so rough numbers were giving me a range of roughly 1,000 to 10,000 CPU cycles per tick, with an average of 2,000. That translates into 333uS - 3.3ms, with an average of 665us (this was on a converted MOD, which tends to be heavier-weight than, say, a Master System tune). I have the figures in the playback source header, but I worked that out to roughly 2%-20% of the CPU, with an average of 4%. (The high end was mostly when the tune started, thus the low average. The low end would be all four channels in countdown mode, no processing.)

 

There are defines in the source that you can use with a map or symbol table to find the entry and exit points to the playback code (or you can just capture it at the user interrupt hook in the ROM) - with that you can measure the cycle counts using the Classic99 timer for the particular songs you are using.

 

One of the big drives for the 30hz mode was that 30hz music with sound effects can alternate music and sound effects, rather than processing both on the same tick. That way if both are expensive you can spread out the workload. 30hz mode can even preserve arpeggio (though it of course sounds slower) - I have a large sample tune that I used to make that work right.

[Tursi]

When I started this project, one of my goals was to get "680Rock", a MIDI medley I converted with PSGMod, to fit in RAM and play. It was the only MIDI I tried that gave results I liked out of that tool, but, as the song was just over 5 minutes long, and loaded with arpeggio, the resulting VGM was 142k. My goal then was to get it down to 24k so I could load it into high memory.

 

I did not succeed in that goal, but I /did/ get it under 32k - 31,608 bytes, so I considered that good enough. But on Friday night I decided I could complete that goal of getting it to play by tweaking the player to call a function for each byte (this will enable banking on carts eventually, which is how I justified it to myself ).

 

Then I decided I could squeeze a visualizer in there, too. And make it all load under E/A without pre-loading VDP or anything. That was /slightly/ trickier than I expected, but not by very much happily!

 

And so, probably the largest single-load app I've done (or will do) for the TI, here it is. There are 6 bytes free in low memory expansion, 26 bytes free in scratchpad (luxury!), and 26 bytes free in high memory expansion. The six bytes in low RAM and 20 bytes in high RAM are due to the 8k limit of the E/A loader and its 6 byte header (the total file is still expected to be 8k or smaller - regardless as to whether more works, I honored that limit). I could have filled either or even both spots with extra files, of course, but, that felt silly.

 

Because it uses nearly all of CPU memory, EA#5 loaders that run from RAM will probably fail. I tested with Editor/Assembler (works fine) and with the ScratchPad Loader (also works!)

 

Some people may not like the arpeggio, but, I had fun with it. (Still need to make a video for YouTube..)

 

680ROCK.zip

[TheMole]

Very cool. I do think the arpeggios are a bit much in the song itself but being able to put that much information in a single song is very impressive. Can you tell a bit more about the visualisation? As in how does it work, what do the colors and locations of the bubbles mean?

 

Since the compressor is a command line application, any chance this can easily be ported to Linux or OSX?

[Tursi]

The visualization is a simple 4-way mirror - same as many simple kaleidoscope programs. One sphere is placed for each of the three frequency channels (there is no noise in that tune), and the location is determined by using the post-swapped sound chip data (including the command bits), and doing a modulo division with the screen size. The colors are just a 'fade out', they go through light green, medium green, dark green, light red, medium red, dark red, light blue, dark blue, off. New spheres are placed at the end of every interrupt, at 60hz, and the screen is continuously updated during the time between interrupts. (I should add actual interrupts are disabled, QUIT doesn't work as I didn't have room to code it - the program polls CRU for the VDP interrupt request).

 

The screen update is performed continuously when no VDP interrupt is pending. It reads one character row into scratchpad. As it reads, it counts down any characters that are not 0, in order to do the color change. Then it writes the row back to VDP.

 

The compressor should be portable. I have to decide if I want to release the source - it's not pretty. But I probably should at some point.

[Asmusr]

If I run this demo in js99er I'm getting chaotic results, whereas the demo from post #1 works fine. Any clues about what I could be missing? I suspect it could be either the X instructions or the polling of the CRU for the interrupt.

 

Edit: Never mind, I found it. I didn't clear the cru bit when VDP status was read.

[TheMole]

Tursi, I tried using it today but the results aren't quite what I hoped for. It sounds like it's stuck playing the first note. Now, I might have screwed up the conversion of the .spf file to a C constant (I assume I can simply take the binary file and have it encoded as an array of bytes, right?). Other than that, I just converted a vgm with one song using your tool, converted it to a constant and embedded the necessary parts from your example code in my project's main function. Anything obvious I might be missing (I did enable interrupts in the main loop, of course)?

[Tursi]

Maybe post what you've got, otherwise I'm only guessing.

[Tursi]

I think that I will add that "680Rock" is /not/ an example of how to use this.. it's a very specific usage of it and uses a modified player to handle the memory access. It's just something I needed to get done for myself.

[TheMole]

Maybe post what you've got, otherwise I'm only guessing.

 

Nevermind, I figured it out. Turns out I was overflowing my low memory expansion by defining the music as const unsigned char. Dropping the const puts it in high memory and that works perfectly.

Cool stuff!

[TheMole]

Maybe this is not the right topic, but since this is such a cool tool and Kevan is pushing this whole thread necromancing thing I figured why not...

 

So I've been happily importing the music in Alex Kidd as a simple byte-array defined in a header file, but since this eats into program space I need to do something better. Loading songs from disk seems like the best solution, but since I haven't worked with TI's disk system in any amount of detail ever I'm struggling to find the best way of doing it. How do I convert an .spf as outputted by Tursi's tool into something I can plop on a disk image and read into RAM from disk (efficiently) in my program? Do I turn it into an INT/FIX file with record size 1? Or is there a way to know the (in memory) size of a PROGRAM image before loading it?

[Asmusr]

Maybe this is not the right topic, but since this is such a cool tool and Kevan is pushing this whole thread necromancing thing I figured why not...

 

So I've been happily importing the music in Alex Kidd as a simple byte-array defined in a header file, but since this eats into program space I need to do something better. Loading songs from disk seems like the best solution, but since I haven't worked with TI's disk system in any amount of detail ever I'm struggling to find the best way of doing it. How do I convert an .spf as outputted by Tursi's tool into something I can plop on a disk image and read into RAM from disk (efficiently) in my program? Do I turn it into an INT/FIX file with record size 1? Or is there a way to know the (in memory) size of a PROGRAM image before loading it?

 

You can use TI99Dir to copy the PC file onto a DSK image as INT/FIX 128. I think any last incomplete record will just be filled with zeros.

[Tursi]

DIS/FIX128 was always the default format for TIFILES compatible applications importing non-native files -- but the difference between INTERNAL and DISPLAY is nil (just a flag for the application), so it amounts to the same thing. The idea is that FIXED128 is the best format for packing records since exactly 2 records fit per sector without any overhead. Extra bytes at the end won't hurt the player anyway.

 

You should be able to do an info call on a program image file, though for my own purposes I'd probably just define a fixed block of space and ensure that the file was never larger than it (as a developer). But to play it safe, the record-based approach would probably be simpler.

[TheMole]

 

You can use TI99Dir to copy the PC file onto a DSK image as INT/FIX 128. I think any last incomplete record will just be filled with zeros.

 

Thanks, that'll work for now, but I'll keep looking for a command-line program that allows me to do this (so I can integrate it in my Makefile)

[TheMole]

DIS/FIX128 was always the default format for TIFILES compatible applications importing non-native files -- but the difference between INTERNAL and DISPLAY is nil (just a flag for the application), so it amounts to the same thing. The idea is that FIXED128 is the best format for packing records since exactly 2 records fit per sector without any overhead. Extra bytes at the end won't hurt the player anyway.

 

You should be able to do an info call on a program image file, though for my own purposes I'd probably just define a fixed block of space and ensure that the file was never larger than it (as a developer). But to play it safe, the record-based approach would probably be simpler.

 

 

Good to know, I had absolutely no idea how to pick an ideal record size for this so I figured a program image might be best, but if worst case scenario you lose 128 bytes, I might just as well take the INT/FIX128 approach.

[TheMole]

I know there's a difference between the TI chip and the modified version used in the Master System, but is it normal that one loses quite a lot of detail/fidelity in the noise channel when playing these files on the TI? I'm converting (captured) vgm's for Alex Kidd, and they sound much less impressive once converted.

[Tursi]

There will be a difference, but it's rare to lose a "lot", but I don't know exactly what you mean there.

 

This converter handles the difference in playback rate for custom pitch noises (as noted in the Alex thread), but it doesn't do anything for the fixed rate. You can hear what the difference sounds like in my conversion of Rushjet's version of Butterfly for the SMS, which makes extensive use of custom noises.

 

Original version: https://www.youtube.com/watch?v=KxLXWgnU3wc

Converted: https://www.youtube.com/watch?feature=player_detailpage&v=UnpIz6CZxXY#t=167

 

So the first thing really is to know what you are working with - are the sound effects short enough that you can have a look at the data and see what they consist of?

[+InsaneMultitasker]

There will be a difference, but it's rare to lose a "lot", but I don't know exactly what you mean there.

 

This converter handles the difference in playback rate for custom pitch noises (as noted in the Alex thread), but it doesn't do anything for the fixed rate. You can hear what the difference sounds like in my conversion of Rushjet's version of Butterfly for the SMS, which makes extensive use of custom noises.

 

Original version: https://www.youtube.com/watch?v=KxLXWgnU3wc

Converted: https://www.youtube.com/watch?feature=player_detailpage&v=UnpIz6CZxXY#t=167

 

So the first thing really is to know what you are working with - are the sound effects short enough that you can have a look at the data and see what they consist of?

 

Very nice. The converted version sounds great. Who needs lyrics? I wonder if Digitally Imported would add your version to a playlist. hehehhe

[Tursi]

I know a couple of people are using this... tonight I identified a problem that was causing the first note of songs to often be corrupted. It was a side effect of the frequency packing that I added late in the compressor's life, and a bad assumption later that it was still using actual frequency codes. My fix is a little hacky, but it should work in all cases, except maybe songs with only a single note. If you have one of those... just set the frequency manually before you play.

 

Hope this helps - v101

http://harmlesslion.com/software/vgmcomp