Apple II tape formats

[SmileyDude]

Hi Everyone --

 

I've been playing around with the Apple II tape format again (this time with a little Apple II emulator that I wrote), and I was curious if there was any sort of "standard" for distributing tape images in the Apple II world. I've been using .wav format with just the pure audio in it, but it seems to me that for emulation and archival purposes, that a more compact format could be made. Something similar to the .tzx format over on the ZX Spectrum side, for example.

 

Does anyone know of such a format for Apple II emulators? I realize that the .dsk format is the more popular format, but tapes have the advantage of being able to be used with real Apple II/II+/IIe without having a working Disk II, card and floppy disk around. Hence my interest in tapes at the moment. Ultimately, I plan on making an iPhone app that would be able to hold a bunch of these tape images (Apple II at first, but nothing saying that other formats couldn't be added later) and then get hooked to a real machine for loading.

 

I do know about the Apple Game Server, which is fantastic. But their distribution format is limited to wav files and a text dump from the monitor, neither of which is very compact.

 

Thanks!

[The Usotsuki]

WAV's really all that's supported...

[SmileyDude]

WAV's really all that's supported...

 

That's a bummer. Maybe I'll try to come up with a format that would compress down a bit more and lend itself to be loaded directly via the emulator. Using 800k to store a one line basic program when most of the wav file is the 770hz header tone is crazy. Even with today's storage options, that could add up fast. My local copy of the game server archive is a gig and a half, which would be a lot of space being used on my iPhone if I wanted them over there.

 

I hate to think what MP3 or AAC would do to a tape file. Especially the ones at the Apple II game server that have the fast loader on them. I might be overthinking it, but it seems like there would be plenty of opportunities for a compressor tuned for music to garble up a tape file.

[The Usotsuki]

I store WAV files compressed with Zip, maybe wav+gz might be an option. (Not the most efficient; but >0)

[carlsson]

I'm trying to think outside of the box now...

 

If you're planning an app that will assist loading programs, why not copy the bootstrap idea in ADTPro, so the iPhone at first will transmit a small program that loads on the Apple, and then a list of disk images (yes) present on the phone. The Apple user then gets to traverse into an image and select a file that will be transmitted over audio. I realize it takes a good while to load programs that way, as I've had the joy of transferring those 140K disk images using only audio as my A2 clone lacks a working serial port.

 

Assuming the Apple ][ operates tapes at 1200 baud (higher? lower?), it would result in just below 9 kB of data per minute? It means a 700 MB / 80 minute CD-R would hold about 700 kB, i.e. roughly a ratio of 1 MB audio file per kilobyte actual program. So on a 8 GB memory card you should be able to fit 8 MB of Apple ][ games stored as CD quality audio (44.1 kHz 16-bit stereo), which would be way above your actual needs.

 

Another idea instead of compressing with mp3 or zip, would be to downsample the WAV to perhaps 11 kHz mono, 8-bit samples which I would imagine is enough. That is assuming the original WAV files you're handling are of a much higher quality. While the sound quality loses more on doing that, I think the risk of data loss is less from downsampling than lossy compression?

[SmileyDude]

I'm trying to think outside of the box now...

 

If you're planning an app that will assist loading programs, why not copy the bootstrap idea in ADTPro, so the iPhone at first will transmit a small program that loads on the Apple, and then a list of disk images (yes) present on the phone. The Apple user then gets to traverse into an image and select a file that will be transmitted over audio. I realize it takes a good while to load programs that way, as I've had the joy of transferring those 140K disk images using only audio as my A2 clone lacks a working serial port.

 

Assuming the Apple ][ operates tapes at 1200 baud (higher? lower?), it would result in just below 9 kB of data per minute? It means a 700 MB / 80 minute CD-R would hold about 700 kB, i.e. roughly a ratio of 1 MB audio file per kilobyte actual program. So on a 8 GB memory card you should be able to fit 8 MB of Apple ][ games stored as CD quality audio (44.1 kHz 16-bit stereo), which would be way above your actual needs.

 

Another idea instead of compressing with mp3 or zip, would be to downsample the WAV to perhaps 11 kHz mono, 8-bit samples which I would imagine is enough. That is assuming the original WAV files you're handling are of a much higher quality. While the sound quality loses more on doing that, I think the risk of data loss is less from downsampling than lossy compression?

I am already using 11kHz, mono and 8-bit -- and even that is overkill in someways. The Apple II only had 1-bit audio for the cassette port, so the other 7-bits are just wasted space. For normal Apple II tapes (i.e, written with the Apple routines in the monitor code), the highest frequency tone is 2500hz. That would suggest that going down to 5000hz for the WAV file would be enough.

 

I just did a quick test -- even with 11khz/mono/8-bit, a 1-byte save file still takes up 44kb. Obviously for longer files, this gets much better -- adding another byte to my save file only adds about 50 bytes or so to the WAV file. But a 50:1 increase seems quite silly when we could just store it as a binary blob and let the serving app just convert it on the fly to the audio format. This also means that as improvements are made, like higher bitrates, the app can be upgraded without having to upgrade the storage format. You could just set the slider to the higher bitrate and use that.

 

Anyway, I'll be doing some more experiments on my end to see what can be done.

[Osgeld]

so you want to use computer power to convert a binary to audio on the fly to cut the overhead of a 44kb file?

[SmileyDude]

so you want to use computer power to convert a binary to audio on the fly to cut the overhead of a 44kb file?

I want to use computer power to convert to optimal audio on the fly so that I can take advantage of either better tape decoding software on the target machine or a faster CPU without having to re-encode for every possibility. The 44kb file was just one byte encoded using the defaults for a regular Apple II cassette. If you try to use higher baud rates, you have to increase the sample rate as well. Of course, you should get a shorter length file at the same time. But looking at the higher baud versions vs the lower baud versions of the files I have, the higher baud ones still end up being a bit larger byte wise, despite being shorter time wise.

 

If you look on the site I linked above, they have at least 5 different WAV files for each game they offer -- basic and monitor load versions in both hifi and lofi, which are different speeds, along with a CD optimized version as well. That collection of games takes up 1.5GB alone -- even if you throw away half of the files as being redundant, you are still looking at 750MB of storage being tied up for what's probably less than 1 MB of actual data. And since I'm thinking of making an iPhone app, where the device size goes down to 8GB (which is really more like 6GB of available storage), it would be a bit silly to use that much space that the user probably wants to have for music, videos, photos and other apps, especially if it's not needed.

 

And really, generating the audio is *not* that computationally hard -- remember, the ~1MHz 6502 in the Apple II was generating and decoding this audio in realtime. I already have code that will take a binary file and generate a WAV file that can be used with a real Apple II.

 

So yeah, a little code, a 50:1 or more savings in space (before applying other forms of compression like LZW even), I think it's a fair trade off. Does it make as much sense on a high end desktop/laptop with 1TB + of storage? Probably not. But there are other devices that are much more limited in storage in use these days, so why not?

 

Plus, we are talking about retro computing here -- this is almost certainly nobody's full time job. We are all here for the fun and the challenge of it and to scratch our own particular itches. If we tried to justify any of this beyond that, we would all be coming up short

 

Edit: One more thought I had -- a 44kb file is almost *all* of the RAM the Apple II had. I know we live in a bounty of computing power, but it still feels a bit wrong to represent a tape image meant to load onto an Apple II with a file that takes up way more space than a room full of Apple IIs had available in RAM.

Edited July 24, 2014 by SmileyDude

[SmileyDude]

If you're planning an app that will assist loading programs, why not copy the bootstrap idea in ADTPro, so the iPhone at first will transmit a small program that loads on the Apple, and then a list of disk images (yes) present on the phone. The Apple user then gets to traverse into an image and select a file that will be transmitted over audio. I realize it takes a good while to load programs that way, as I've had the joy of transferring those 140K disk images using only audio as my A2 clone lacks a working serial port.

 

I would love to see the details on loading a disk image over a tape -- the Apple II doesn't even have enough RAM to hold a entire disk in memory, right? But at the very least, I can see this being useful as a way to bootstrap up an Apple II system where you have no formatted floppy disks available. Use the tape interface to load in DOS 3.3 or ProDOS, insert a blank disk, format and install the system on it. That would be pretty useful in my opinion.

 

I do like the idea of having the iPhone act as a server of images. Using the tape out port to communicate back to the iPhone would be pretty cool and even allow for multi-load games. I'll have to think about that one some more.

[The Usotsuki]

Yeah, unless you got a GS, you're not holding a whole 140K disk in one pass.

[Rybags]

IMO just roll your own format.

 

Have header/descriptors that summarise the bitrate, length etc for each data block then just pack the data as the computer would finally see it per block.

 

Line of thinking here being keeping the data as compact as possible without actual compression taking place, and allowing the original media to be reproduced if required.

[SmileyDude]

One more thought of why to use a binary format rather than a WAV. A wav file can contain any audio information, not just Apple II tape images (obviously). By using a binary format, there has been some validation on the underlying data before hand and it should mean that images can have a higher success ratio than wav files. I've ran into quite a few tape images off the web that were bad -- mostly because they were captures off the original cassette and they failed for the same reasons the cassettes originally did back in the 80s. I've been able to clean up some of these dumps off line, but some are beyond repair for me at the moment. A binary file completely bypasses at least those failure points and gives some assurance that the data is there.

[SmileyDude]

IMO just roll your own format.

 

Have header/descriptors that summarise the bitrate, length etc for each data block then just pack the data as the computer would finally see it per block.

 

Line of thinking here being keeping the data as compact as possible without actual compression taking place, and allowing the original media to be reproduced if required.

Some level of compression would be nice -- even if it's just basic stuff. For example, instead of putting in 5 seconds of the leader, there should just be a indicator that there is 5 seconds of the standard leader. This would allow for complete reproduction of the original media if necessary, while also not filling up the file with useless data.

 

I'm thinking that wrapping the entire file in LZW or similar compression afterwards might be a good second step. Similar to how a lot of emulators zip their ROMs for storage purposes. Not required, but could be nice to have. I'll worry about that later

[Osgeld]

I would mention it on the apple II user group since the authors of ADT, Game Server (and JACE), and applewin are usually quite active

 

https://groups.google.com/forum/?fromgroups#!forum/comp.sys.apple2

 

cause I know there has been work done on it before, and if you do end up rolling your own it would be nice to use it in emulators instead of "yet another standard"

[Rybags]

http://a8cas.sourceforge.net/format-cas.html

The Atari CAS format - might be some useful ideas in that document.
Stuff like leader and inter-record gaps might be described by a value/length type of header.

Does Apple II tape use start/stop bits at all? If not, then the overall format would be somewhat simpler. Atari normally uses start/stop so there are special desriptors that allow for data that doesn't.

Edited July 24, 2014 by Rybags

[carlsson]

Regarding transferring disk images over audio, ADTPro does it in multiple passes. As you'd only load one program at a time, it wouldn't be half as bad but of course if you already have some kind of client software in the Apple ][ computer, you would not really be able to load a full 48 kB program on top of that. I don't know the memory map, if it'd be best to store the client as high into the memory as possible if regular software loads starting from a low address.

 

Actually you could skip the A2 client and have the iPhone app generate audio on the fly from a selected file inside a disk image, in case tape images in digital format don't exist and you're not about to roll your own format. I figure most games were not available on tape anyhow, so even if they're single-filed, most of them would reside inside disk images. Either you would have to convert disk to tape format first, then on the fly generate audio from the tape format, or combine the two steps and skip the temporary tape storage format. Please pardon me if I'm missing or misunderstanding part of your idea/task.

[datajerk]

If you look on the site I linked above, they have at least 5 different WAV files for each game they offer -- basic and monitor load versions in both hifi and lofi, which are different speeds, along with a CD optimized version as well. That collection of games takes up 1.5GB alone -- even if you throw away half of the files as being redundant, you are still looking at 750MB of storage being tied up for what's probably less than 1 MB of actual data. And since I'm thinking of making an iPhone app, where the device size goes down to 8GB (which is really more like 6GB of available storage), it would be a bit silly to use that much space that the user probably wants to have for music, videos, photos and other apps, especially if it's not needed.

 

If your end goal is an iOS app that streams tapes to the Apple II, then just store the code as compressed binaries and generate the audio as needed.

 

It was my intention to have the Apple Game and Disk Server Online dynamically generate the audio, but I ran out of time and opted for static pre-generated audio files (that and I have 200GB free on that server). Someday I'll update both sites to something more dynamic.

 

The c2t source that the Apple Game and Disk Server Online uses is available from here: http://asciiexpress.net/files/

 

Tape @ 1333 bps is painfully slow, I'd recommend using c2t @ 8000 bps with compression enabled.

[datajerk]

Use the tape interface to load in DOS 3.3 or ProDOS, insert a blank disk, format and install the system on it. That would be pretty useful in my opinion.

 

Already been done: http://asciiexpress.net/diskserver/

 

Out of date demo (it's much better now): http://asciiexpress.net/diskserver/video.mp4

 

 

Edited July 25, 2014 by datajerk

[datajerk]

Using the tape out port to communicate back to the iPhone would be pretty cool and even allow for multi-load games. I'll have to think about that one some more.

 

I started to explore this as well, but ran out of time. I had a number of challenges getting a good recording from the Apple II via iOS (existing apps).

 

Here's a possible workaround, have the Apple II post up a QR code, use the iPhone camera to read it and fetch the audio code. E.g.: http://asciiexpress.net/gameserver/gameserverclient.mp4

[datajerk]

 

I'm thinking that wrapping the entire file in LZW or similar compression afterwards might be a good second step. Similar to how a lot of emulators zip their ROMs for storage purposes. Not required, but could be nice to have. I'll worry about that later

 

Again done. Look at the c2t source. There is an option to compress. The compression is 6502 simulated by c2t to judge if it would be faster to just transfer vs. transferring the decompress code + binary + decompress time. Examples of a compressed binary and an uncompress binary:

Input:  Binary game without DOS header that should be loaded at $801 as fast
        as possible while being compatible with all Apple IIs.

Command:

c2t -8c moon.patrol,801 moon.patrol.aif

Reading moon.patrol, type binary, segment 1, start: 0x0801, length: 18460

Writing moon.patrol.aif as Apple II formatted aiff.

start: 0x7226, length: 18393, deflated: 0.36%, data time:18.95, inflate time:6.83
WARNING: compression disabled: no significant gain (18.11)

To load up and run on your Apple II, type:

        LOAD

NOTE:  Compression was disabled because it didn't help.

Input:  Binary game without DOS header that should be loaded at $800 as fast
        as possible while being compatible with all Apple IIs.

Command:

c2t -8c super_puckman,800 super_puckman.wav

Reading super_puckman, type binary, segment 1, start: 0x0800, length: 30719

Writing super_puckman.wav as Apple II formatted wave.

start: 0x886C, length: 12691, deflated: 58.69%, data time:13.25, inflate time:5.79

To load up and run on your Apple II, type:

        LOAD

Edited July 25, 2014 by datajerk

[datajerk]

Does Apple II tape use start/stop bits at all? If not, then the overall format would be somewhat simpler. Atari normally uses start/stop so there are special desriptors that allow for data that doesn't.

 

No, but the COSMAC VIP did. Here's a quick hack to support the VIP with start/stop (parity really) bits: http://asciiexpress.net/files/VIP/

 

You could adapt to other formats.

Edited July 25, 2014 by datajerk

[SmileyDude]

Again done. Look at the c2t source. There is an option to compress. The compression is 6502 simulated by c2t to judge if it would be faster to just transfer vs. transferring the decompress code + binary + decompress time. Examples of a compressed binary and an uncompress binary:

 

Oh, this is pretty neat, but I was thinking of compression of the binary tape file itself, not the data within the tape file. But I can see both being useful.

 

I'm going to have to check out c2t some more. I've written some code to generate tape audio before -- nothing too fancy. Lately I've been using the little Apple II emulator to generate tape images, but again limited to what the stock firmware will do. Definitely want to push the limits a bit more on the tape interface.

[datajerk]

 

Oh, this is pretty neat, but I was thinking of compression of the binary tape file itself, not the data within the tape file. But I can see both being useful.

 

I'm going to have to check out c2t some more. I've written some code to generate tape audio before -- nothing too fancy. Lately I've been using the little Apple II emulator to generate tape images, but again limited to what the stock firmware will do. Definitely want to push the limits a bit more on the tape interface.

 

The WAV and AIF formats (if computer generated using c2t or emulators) are very compressible. I just did a quick test and was able to reduce the size of an Apple II WAV file by 38x (1277K to 33K).

 

I've spent too much time trying to squeeze the most out of the Apple II audio jacks. The best I could do was 9600 bps (no compression). However, that speed is not stable on all Apple IIs. 8000 bps has been demonstrated to be stable with all Apple IIs (that have been reported) as well as Virtual ][ emulation. When/if I have the time I may try again to get 9600 bps stable.

 

The c2t source has all the 6502 assembly source as well. Good luck with your project.

[rolanyang]

This might be going off on a tangent from the cassette audio thread, but I was wondering if one could offload data from the Apple to a mobile phone or computer+camera faster by utilizing a stream of QR codes. This guy toyed around with the idea: http://stephendnicholas.com/archives/310

His calcs say 43KB/s theoretical max throughput. I imagine the computing expense of generating the codes on a 6502 might be a bottleneck here. What do you think?

 

~Rolan