Jump to content
Tursi

VGM Compression Tool

Recommended Posts

Unrelated to the above, though good timing, I updated the archive for the compressor with a port of the old 'quick player' tool - you can now run a Windows program and use it to load a compressed VGM (SPF) format , and it will spit out an EA#5 program for emulation or hardware.

 

  • Like 4

Share this post


Link to post
Share on other sites

I've been working on the update to this compressor when I can, as per the top of my todo list. 

 

At the moment, I've finished the suite of VGM importers that I'll be supporting: PSG, AY-3-8910, Gameboy, Atari Pokey, and NES. Now the easy stuff is done so I need to actually write the compression part and players. ;)

 

But, since I also have an ID tool and a test player, I thought I'd do a quick stream with the progress so far. I demonstrate converting from all of those:

 

AfterBurner (Master System - PSG)

Tetris (Arcade - Pokey)

Contra (Gameboy)

Street Fighter (Gameboy)

Mega Man 2 (NES)

Mr Robot (Atari 8-bit - Pokey)

Punch-Out (Arcade - NES)

Journey to Silius (NES)

Super Mario Bros 3 (NES)

Super Mario World (Gameboy)

Tetris 2 (Arcade - Pokey)

TMNT (Gameboy)

Time Pilot (Arcade - AY-3-8910 x2)

Vampire Hunter (Arcade - PSG /and/ AY-3-8910)

Wonder Boy (Arcade - PSG x2)

 

For the most part, they convert okay. SMB3 needs a little external tuning on the DMC channel (or just drop it). Good enough to start with anyway.

 

 

(Might need a high res display to read it, apologies).

 

Eagle eyes will observe that not all of the tunes can be played on the TI PSG - my intent here is to break the process into smaller pieces to improve the flexibility a bit, and reduce the complexity of maintenance. Each converter's job is just to extract the data and write out tracks that are more or less compatible with the TI PSG. This means simulating the envelope and sweep hardware (where it exists), splitting out noise and tone (where they are shared), and so forth. This, along with the possibility of multiple chips in the VGM, can lead to more channels than we can play.

 

The intent is to have a set of tools that are used between the initial conversion and the compression to tune the data - whether it is shifting frequencies, adjusting volumes, or combining channels. In addition, the intermediate format is simple text files, easily edited. Removing an unwanted channel is just deleting the file.

 

The testplayer is thus a TI sound chip without limits - any number of channels of any type, and no bass tone restrictions, so you can test your progress at each step.

 

Further to that, this allows the flexibility of outputting for multiple sound chips - like say the FourTI. ;)

 

The compression tool is next and that's where I'll see if the notes I took are any good. I'm aiming for less CPU usage with at least comparable compression by changing up the timestream concept a bit.

 

Anyway, just wanted to share now that I've finally had a little time for code. ;)

 

  • Like 8

Share this post


Link to post
Share on other sites

I'm looking forward to trying this. How are you converting the NES DPCM samples?  What does the file format look like?  It might be fun to try a few samples with my own compression tools.

Share this post


Link to post
Share on other sites

This is really exciting news!  I for one am totally psyched about the possibility of playing chiptunes from all those different systems on the TI!  I was actually in touch with Tursi just a month or so ago about this very thing.  He was kind enough to get me up and running with his .VGM quick player (which he actually created at that time as I was still using his older/now obsolete EPSGMOD quick player which was much more limiting on the size of the music files it could play).  The new quick player is great and seems to handle anything I throw at it size wise.  I built up quite a collection of Sega tunes and they sound great coming out of real iron!  I couldn't stop there though--I knew Tursi was working on a NES playback routine but I was impatient and started converting .NSF files to .VGM by copying the data out of FamiTracker and pasting into a SN76489 compatible tracker.  The results are not perfect and vary from track to track but some music actually came out pretty decently using this method.  Especially music from the Mega Man series--all that stuff sounds great, almost like the original and does not suffer too badly on the SN76489.  I even converted some Gameboy stuff using this same method.  Some of that stuff sounds amazing too, other times certain tracks have bizarre tuning issues I haven't worked out yet.

 

But I only did all this cause I didn't actually think Tursi would get around to updating his routines anytime soon.  Let alone include support for Gameboy, Atari and MSX! :)   I can't wait for the converter as it totally antiquates my previous method, firstly by simulating envelopes etc. which my method has no provision for, but also automating the entire process (no more endless copy and pasting)! :) 

 

Also I like the idea of that testplayer outputting for different sound chips other than just the TI's.  I have a good few vintage computer systems and chiptunes are an interest for me across the board.

 

Share this post


Link to post
Share on other sites
21 hours ago, PeteE said:

I'm looking forward to trying this. How are you converting the NES DPCM samples?  What does the file format look like?  It might be fun to try a few samples with my own compression tools.

I'm not.. they are just played as either noise channel or tone channel with an averaged volume. It's the same thing my MOD converter does. This works really well in Journey to Silius, pretty much not at all in Super Mario 3. ;) My VGM player is a 60hz format, so samples are a non-starter.

 

Share this post


Link to post
Share on other sites
8 hours ago, ruthven said:

But I only did all this cause I didn't actually think Tursi would get around to updating his routines anytime soon.  Let alone include support for Gameboy, Atari and MSX! :)   I can't wait for the converter as it totally antiquates my previous method, firstly by simulating envelopes etc. which my method has no provision for, but also automating the entire process (no more endless copy and pasting)! :) 

 

Also I like the idea of that testplayer outputting for different sound chips other than just the TI's.  I have a good few vintage computer systems and chiptunes are an interest for me across the board.

 

It's still going to be a while - I have a lot left to do. But this has been years coming in my eyes. ;)

 

The Testplayer is still a TI sound chip, I only removed the limitations. The idea is to come reasonably close to the final - the noise frequencies will possibly shift, and the number of channels will be restricted, but otherwise the data is already in TI PSG format. The expanded capability makes it easier to test the results of various conversion options one step at a time, without needing to go all the way to the end product.

 

Also, there aren't very many Pokey files in VGM format - at least not at VGMRips.net. And fewer NES than I expected - NSF is still the dominant format there. But one step at a time.

Share this post


Link to post
Share on other sites

Revisiting from February; I grabbed a few files from the "SN76489"  section here https://vgmrips.net/packs/chip/sn76489  but am still running into trouble.  I made sure only to download single chip files (e.g., no mixed or dual chip rips).   Here is what happens during the conversion attempt. Any thoughts?  I attached a sample file as well. 

 

D:\xdt99\>vgmcomp -v "01 Title.vgz"
v106 - 12/27/2016
Reading 01 Title.vgz - 7068 bytes
Signature not detected.. trying gzip (vgz)
Decompressed to 37112 bytes
Reading version 0x151
weird refresh rate 0

 

01 Title.vgz

 

 

Share this post


Link to post
Share on other sites

That's becoming more common... it's saying that instead of 50hz or 60hz, the frame rate field is zero. The specification allows this for songs that don't change speed based on the vertical blank - but back when I wrote the tool I never found any at all. The new tool allows you to ignore weirdness but apparently I don't have that in the old one yet, so I'll go ahead and update that...

 

It's updated at the usual sites: http://harmlesslion.com/software/vgmcomp

 

 

  • Thanks 1

Share this post


Link to post
Share on other sites
47 minutes ago, Tursi said:

The new tool allows you to ignore weirdness but apparently I don't have that in the old one yet, so I'll go ahead and update that...

 

It's updated at the usual sites: http://harmlesslion.com/software/vgmcomp

 

 

That did the trick!  Thank you.   FYI, the direct link didn't work but I knew where to go and how to get there :)

 

Share this post


Link to post
Share on other sites

I had a little random fun with the updated vgmcomp.  Along with the _C and _P files (from convert9918), my ramdisk program will now load a  _V binary file which is the raw output from the VGM compressor converted to a 'program image' file via TI99Dir.  The _V file is loaded into VRAM then copied to RAM, the picture files are loaded into VRAM, and an iteration of your music player does the rest. (I hit spacebar at 30 seconds to halt play).  Thank you for the tools :)

 

  • Like 3

Share this post


Link to post
Share on other sites

awwww, how fun! :) 

 

Guess I need to get Ramdisk implemented in Classic99 so I can play with this.. ;)

  • Like 4

Share this post


Link to post
Share on other sites
7 hours ago, Tursi said:

awwww, how fun! :) 

 

Guess I need to get Ramdisk implemented in Classic99 so I can play with this.. ;)

Not like a PC does not have the memory or space for it, I am running 16 Gig RAM in mine right now.

I used to have a CorComp RAM Disk as number 5 and Horizon 3000 RAM Disk with 4 more RAM Disk A, B, C, D and CorComp Disk Drive 1 to 4, added in with Horizon SCSI 7 Drives.

Share this post


Link to post
Share on other sites
9 hours ago, RXB said:

Not like a PC does not have the memory or space for it, I am running 16 Gig RAM in mine right now.

I used to have a CorComp RAM Disk as number 5 and Horizon 3000 RAM Disk with 4 more RAM Disk A, B, C, D and CorComp Disk Drive 1 to 4, added in with Horizon SCSI 7 Drives.

yeah... I loved my ramdisks. Almost failed my last semester of CompuCollege* because I bought my first one and then ran out of gas money, but still glad I did it. ;)

 

* Yeah, seriously, that's what it was called, but it was /something/. ;)

 

  • Like 1

Share this post


Link to post
Share on other sites

As of tonight I was able to get the new compression implemented. I didn't get the unpack test written, but IF I did it right, it's beating the old compression level slightly (my test file got 10184 bytes vs 10806 on the old), but with a much simpler packing that has 3 fewer streams and no special cases, so it'll hopefully unpack with notably less CPU. The main compression gains come from adding a 16-bit RLE and implementing a forward search that actually appears to function, more or less.

 

More details when I know it actually works. ;)

 

  • Like 2

Share this post


Link to post
Share on other sites

Pleased to say that I got the packing test code written tonight and, at least on the PC, the packing is confirmed accurate. Next step is to integrate it into the test player, and if that works, to get the TI playback code written. Then, although the toolset won't be complete, I can at least release an alpha as it'll be far enough along for people to try it.

 

The unpacking is MUCH simpler than the old, which had suffered a bit from evolution. It incorporates a double-RLE, which was one of the things missing in the old system, and this seems to help at least bit. The main goal, though, was to reduce CPU usage (and maybe memory usage). It should succeed. No need to read on unless you're curious about the system.

 

The original packer, as I noticed, suffered a bit from evolution. The basics were pure, at least to start. The concept was to divide the components of audio into separate streams without channel information so that they could take advantage of redundancies. So, there were 3 tone streams, 1 noise stream, 4 volume streams, and 4 "time" streams to manage the whole thing. Each stream was compressed using a simple command byte which could specify a run of up to 64 bytes of either inline data, RLE, a short back reference (offset within a stream) or a long back reference (offset anywhere in the song).

 

After that, each stream simply contained the next byte of data, while the timestream incorporated an RLE of sorts on a per-voice basis. Each timestream byte specified up to 64 frames of idle, and then two bits indicated whether to load tone, volume, both or neither. Timestreams furthermore had an additional 6 special codes to indicate common repeated patterns - a very weak RLE for just 6 combinations of count and data. As such, playback code needed to track unpack state for 12 streams, as well as additional counts for the timestream. To save a little more data, notes are encoded into a 256-entry note table so that they require only one byte instead of 2.

 

This worked, but used up to 30% of the CPU on moderately complex songs. The timestream compression suffered if there were lots of volume changes, which is normal for the types of music I like to encode. And no RLE was possible in back-references, meaning that a back-reference needed to contain raw strings to compress later data correctly.

 

The new system keeps the stream concept, but adapts it. There are still 4 tone streams (well, 3 tone and a noise) and 4 volume streams, but there is only a single timestream which manages the tone channels only. 4 bits are used for time, and 4 bits indicate which voices to load. (Frequency sweeps will still hit the timestream hard, but these are less common than volume sweeps). Tone channels still use the 8-bit note table. Volume channels maintain their own count using their spare nibble, meaning instrument effects no longer hammer the timestream and much less code is required to handle one volume channel changing (in addition only 1 byte is consumed per volume change instead of 2).

 

The packing was modified slightly as well. The "short back-reference" was discarded in favor of a 16-bit RLE, which helps for arpeggio and volumes that warble between two levels. And there are no more magic codes for the timestream.

 

This seems to work pretty well. You still need to manage pack state for 9 streams, but that's 25% down from 12, so even if all else were equal that'd be a win. Each of the 9 streams also requires a counter for their delay value, but I think it'll fit a lot better. I still need to write this code, but I think it'll be a lot quicker to execute. 

 

It's up on my github for those who like to play, but of course it's not ready for primetime yet. But you can play with the converters and packer written so far. https://github.com/tursilion/vgmcomp2

 

  • Like 9

Share this post


Link to post
Share on other sites

Posted an update to the above. I've now got the compressed file loading and playing in the test player, which let me find a couple of compression bugs and proves that the resulting file /is/ playable. I took the opportunity tonight to tune the compression a bit and it's now beating the old one for size in all but one of my test files (and that one is a real monster):

 

Quote

LetsPlay.VGM: 266.98 seconds (MOD convert)
  OLD: 10,806 bytes (40.48 bytes per second)
  NEW: 9,623 bytes (36.05 bytes per second)

 

Outrun.VGM: 174.23 seconds (MOD convert)
  OLD: 8,536 bytes (46.33 bytes per second)
  NEW: 8,061 bytes (43.76 bytes per second)

 

AfterBurner.VGM: 84.03 seconds (Master System)
  OLD: 3,116 bytes (37.08 bytes per second)
  NEW: 3,064 bytes (36.46 bytes per second)

 

680Rock.vgm: 307.2 seconds (MIDI convert)
  OLD: 31,608 bytes (102.89 bytes per second)
  NEW: 32,245 bytes (104.96 bytes per second)

 

So excluding the stress test that is 680Rock, you can expect slightly better compression than the old tool. If you budget for 50 bytes per second of audio you'll be solid. Next I need to code the TI player and prove the assertion that it will take less CPU (and hopefully less RAM) to play it back, but so far, so good!

 

Test files are at the link above for Windows, of course. Finished tools for VGM ID, VGM import of AY, Gameboy, NES, Pokey, and of course PSG (not 100% sure on Gameboy and NES yet, but they seem close enough to start), Prepare for PSG, compression tool and test player.

 

So right now the only import is from VGM, but you have some variety of options. The basic workflow is a few more steps than the old one, with the intent that it gives you the ability to manipulate each step more precisely. Eventually, for a real workflow, you'd write a makefile or a batch file. ;)

 

- First - optionally, you run vgm_id to see what sound chip(s) is used in the VGM you want to pull from.

- Second, run vgm_psg2psg (or nes2psg, or ay2psg, or pokey2psg, or gb2psg) to extract the music. This will write a plain text file for every channel. The files are simple - each row is a single 60hz frame and record the tone and volume. You can mix and match, or edit (just keep the row count the same in them all).

- Third, though the tools don't exist yet, you can process the converted audio - pitch shift, adjust noise, merge channels, etc etc

- Fourth, run prepare4PSG and pass in the four channels you want to play out - this writes a new text file with the four channels per row, and enforces the restrictions of the PSG

- Fifth, run vgmcomp2 on the .psg file you created in step 4. Like the old tool, you can combine multiple files into a single bank of tunes and sound effects. This tool is a fair bit slower than the old one, but it's also early in development.

 

You can also run "testPlayer" with any of the above output files - it understands a group of channels from psg2psg, individual channels specified literally, the combined .psg file and the compressed .scf file. Except for the SCF file, you can mix and match and exceed the restrictions of the actual sound chip - this is to help you get the audio into a state you're happy with, eventually.

 

Anyway, like I say, next step is to get the TI playing it so I can take some CPU measurements.

 

 

  • Like 2

Share this post


Link to post
Share on other sites
On 4/30/2020 at 4:44 AM, Tursi said:

\ The concept was to divide the components of audio into separate streams without channel information so that they could take advantage of redundancies. So, there were 3 tone streams, 1 noise stream, 4 volume streams, and 4 "time" streams to manage the whole thing. Each stream was compressed using a simple command byte which could specify a run of up to 64 bytes of either inline data, RLE, a short back reference (offset within a stream) or a long back reference (offset anywhere in the song).

 

 

 

 

First, I read through this with great interest on friday.

 

I have some questions about it.

 

I understand that VGM is a raw capture of the sound chips writes, right?

 

So once you have it separated into voices, you categorize the tone frequencies into 256 levels? This would seem ok for musical notes, but what happens to portamento? or vibrato? How many steps between tones?

 

Another thing I am confused about... AY vs PSG. Does PSG always mean SN76489 and not sometimes AY-8910-3? Because for a while, since chiptunes DJs had called them both PSGs, I thought the MSX had the 76489, when it turns out it's the AY-8910-3. The translated Japanese docs call the AY a PSG or an SSG when they're not defining it as "the cascade of the horizontal output."

 

Is the translation from AY to PSG mostly about changing the commands or do you lose precision?

 

 

  • Like 1

Share this post


Link to post
Share on other sites

I understand that VGM is a raw capture of the sound chips writes, right?

 

Correct. VGMs are sample accurate to 44khz, however, my player is only for 60hz. (There will be a way to target 50hz, but it'll be a hack like the old one). The converters do their best to accommodate any sample timing in the VGM, which is more and more common actually as they disconnect from the vertical blank and hard-code the timing instead.

 

So once you have it separated into voices, you categorize the tone frequencies into 256 levels? This would seem ok for musical notes, but what happens to portamento? or vibrato? How many steps between tones?

 

You may be surprised to learn that in most cases, it's still fine. The first thing to remember is that there are only 1024 possible frequencies on the TI sound chip. (Though only 10 or so are completely unusable for tones, and even they are useful for tuned noise). (The AY8910 is 4096, mind!)

 

A portamento between, for instance, B01 and C11 (using the E/A notation, and giving away my complete lack of music theory) is down near the range that has the greatest resolution, and yet has only 51 possible steps. With a 60hz update rate, using all 51 steps would take nearly a full second, so it's likely to skip some just out of temporal resolution. Most songs use a very small count of frequencies - from my test songs above:

 

Read 5042 lines from 'ab.psg'
Songbank contains 47/256 notes.

 

Read 16019 lines from 'letsplay.psg'
Songbank contains 37/256 notes.

 

Read 11054 lines from 'outrun.psg'
Songbank contains 56/256 notes.

 

Read 18432 lines from '..\debug\680rock.psg'
Songbank contains 49/256 notes.

 

.. and even if I combine all four:

Read 18432 lines from '..\debug\680rock.psg'
Read 11054 lines from 'outrun.psg'
Read 16019 lines from 'letsplay.psg'
Read 5042 lines from 'ab.psg'
Total of 50547 rows from 4 songs
Song 0 adds 49 notes to note table
Song 1 adds 51 notes to note table
Song 2 adds 19 notes to note table
Song 3 adds 23 notes to note table
Songbank contains 142/256 notes.

 

In the case that there are so many pitch slides that the system does run out of notes, the old tool would decimate the slides by dividing the available frequencies by 2, or sometimes 3. This usually still sounded fine because of the speed of the slides. In this system a specific tool will exist to do that.

 

Another thing I am confused about... AY vs PSG. Does PSG always mean SN76489 and not sometimes AY-8910-3?

 

Yeah, they are all PSGs by definition. I'm mostly just abbreviating the terminology in the VGM spec, probably incorrectly, but in my case, "PSG" always means the TI SN76489 and compatibles, and "AY" always means the "AY-3-8910" and compatibles. I just hate trying to remember the numbers. The documentation is more clear.

 

Is the translation from AY to PSG mostly about changing the commands or do you lose precision?

 

There's some loss. The toolchain is about creating soundbanks for the PSG, and the PSG is not 100% compatible with the AY. The AY-3-8910 has a wider frequency range and a dedicated counter for the noise channel (although it trades the noise volume control for that). The AY mixer also adds a bit of a wrinkle, and the AY has an envelope generator that the PSG does not. The output resembles what the AY would produce, in the same way as converting from NES or Gameboy.

 

The only reason I'm adding AY support to everything is to support the CollectorVision Phoenix, which has both the TI PSG and an AY-compatible sound chip. Part of the drive is that the toolchain will allow the creation of data for both -- however, the AY playback is really going to be aimed at playing back PSG-based data on the AY chip, and won't take advantage of its extra capabilities.

 

 

 

 

 

  • Like 1

Share this post


Link to post
Share on other sites

Hi @Tursi - Does the player support the periodic noise mode for the PSG?  I'm hearing white noise instead, with this file: title.psg

This is from my own NES APU converter which makes the out-of-range tone 3 triangle wave into periodic noise.

Share this post


Link to post
Share on other sites
6 hours ago, PeteE said:

Hi @Tursi - Does the player support the periodic noise mode for the PSG?  I'm hearing white noise instead, with this file: title.psg

This is from my own NES APU converter which makes the out-of-range tone 3 triangle wave into periodic noise.

It does. Lots of room for bugs. ;)

 

I see type 3 noises in the output file there, which should be periodic. Is it the test player or a later phase that you are losing the periodic at?

 

I don't know if I'll get to play with it tonight. 😕

 

Share this post


Link to post
Share on other sites
12 hours ago, PeteE said:

Hi @Tursi - Does the player support the periodic noise mode for the PSG?  I'm hearing white noise instead, with this file: title.psg

This is from my own NES APU converter which makes the out-of-range tone 3 triangle wave into periodic noise.

I took at look at it - you're right. At some point in my development I got the meaning of bit 2 in the noise generator backwards and started treating 'set' as periodic, and clear as white noise, which of course is inverted. I fixed my player and your tune plays correctly now. I'm going to go through and fix my thinko so I don't do that again in the future. ;)

 

  • Like 3

Share this post


Link to post
Share on other sites

I'm been kind of reluctant to work on this because the next task - writing the player - meant writing and debugging decompression code in assembly language, and I didn't really want to. I got reasonably far, and then just put it away and went off to play video games instead. ;)

 

I came back to it, and decided to do a C version first instead. This was something I could debug on the PC, and then port with confidence to Coleco and TI. It may be less efficient than hand-coded assembly, but our GCC is actually pretty good, so we'll try it. Can always hand-tune the assembly if it looks like too much.

 

Anyway... tonight I finished the sample code and the PC-based player. I also finished the basic toolchain for the AY and (after the above conversation stuck in my head) renamed references to the "PSG" to the "SN" so it was a little more clear. I found an AY emulator so that I could test my assumptions against something real, and happily the SN and AY toolchains both play compressed files with success.

 

Next step will be to port the player to TI and Coleco, and then it'll be usable. I have a crappy little demo planned for that, which will let me measure CPU and memory usage. (Cause after all this, I really still don't know ;) ). Then I'll try and get as many of the little editing utilities done as I can to round off the package.

 

The documentation has been keeping up to date as well, I just need to get a little bit closer (so I know nothing under the hood is going to change) and then I can clean that up and release it as well.

 

All the code and sample apps are up at Github, you can do all the basic building and running on the PC, there's just no console players yet. ;)

 

  • Like 3

Share this post


Link to post
Share on other sites

"and then just put it away and went off to play video games instead. ;)"

I completely understand this, I've been doing some assembly work, and if I don't go over it while trying to sleep for example, I'll always put it on the shelf.

Then like magic I get an algorithm just like that. Sometimes months of sleeping on it.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...