STAC (Software Tape turbo with error recovery) sources

[ijor]

STAC is a custom tape system I designed in the mid 80's. It was very popular in South America and spawned a myriad of clones, mostly at Chile, where Atari was very strong. Note that most clones just reused the main concept, and not copied the actual implementation.

 

STAC stands for Super Turbo Auto Corrector in Spanish. Is has a software turbo. But I learned soon enough that the reliability of the turbo depends on many external factors. So STAC implements other features that I considered even more important.

 

The key feature in STAC was error recovery. The Atari tape system is slow enough that it is almost unbearable to wait until some software finishes loading. But worse than this was being very close to complete the loading and suddenly getting a load error. You had to rewind and start all over from the beginning. At this point what you really wanted is to throw the tape, the recorder, and even the computer out of the window.

 

To avoid this, STAC implements a simple error recovery. If the loader detects an error the user is prompted to rewind just a little bit. The loader then resyncs with the tape, hopefully this time the error is not repeated, and continues loading. Most errors are typically soft errors and the retry logic works very well.

 

STAC also implements a simple zero cost RLE compression. The data is decompressed on the flight while loading. I was amazed how much this saved in some cases.

 

Finally, although not strictly a STAC feature, I adapted for tape several titles that were available only on disk.

Edited July 5, 2017 by ijor

[ijor]

These are the original STAC sources. They include the original comments in mixed Spanish and English language. Sorry, but that's how I used to write the comments at the time. If I'll find the time I will try to expand and elaborate the comments. I'm not 100% sure this is the latest version. But if it is not, it should be very close. They assemble with AMAC (Atari Macro Assembler). Shouldn't be so difficult to adapt to other assemblers.

 

The source is for the STAC recorder, the software that writes using the custom tape system. The loader is embedded in the recorder.

 

STACAUT.ASM : Main recorder source

STPRES : Main loader source

PANT.ASM : Load screen

TOS.ASM : Custom tape routines.

HIGH.ASM : SIO high speed xfer routines. Only for the recording system.

 

The recorder was distributed on disks that were optionally copy protected. The protection was not too sophisticated but the disks were recorded in enhanced density. The screen content was also protected to prevent changes. But of course, nothing is fool proof and I was aware about some cracks.

 

Stac-Sources.zip

[ijor]

Some CAS images on STAC system available here: http://atariage.com/forums/topic/267365-stac-images/

 

[vitoco]

Wow! Impressive! Thank you!

 

I'm from Chile, and by the end of 1988 I was told about some games that were created using a "STAC system" that came from Argentina. The other turbo system at the time was the Injektor, which has similar features but required a modified tape unit (the Atari XL12-Injektor). As I didn't own a tape unit, I never paid enough attention to them. I even thought that both systems where the same.

 

A local game provider asked me if it was possible to create a similar system to his store, because the existing copiers were slow (compared to them) and some games could be loaded using some copiers but not with others. I had read somewhere about a simple way to change the tape recording speed, so I acceped the challenge. The result was SITRE, which I documented (English by Google Translate), updated and partially released last year.

 

[Franco Catrin]

This is awesome in many levels!!!

 

I was behind a CAS with your loader for many years and I got one just a few days/weeks ago. I reverse-engineered your loader when I was a kid and I always had trouble understanding a part that seemed like an alternative bitrate estimator to me. I haven't checked your code yet and I don't know if I should jump directly into it or I'll try to repeat the reverse-engineering, it was just too fun.

 

If that was a bitrate estimator, your loader is unique. All turbo/error recovery clones (including mine) didn't made that part, only tuned the SIO to use a faster bitrate at recording.

[ijor]

Some technical details for those interested. But please bear in mind that I didn't touch this stuff for about 30 years.

STAC uses a software turbo that by default it records at 990 bps. I experimented with higher values, and the software can be easily configured for other rates. As a matter of fact sometimes higher rates were actually used for recording. The loader extends and improves upon the OS auto baud detection. But the reliability of higher rates depend on some things that sometimes you couldn't control, including the recording system, the tape, and of course, the user's player.

The error recovery is straightforward and uses a simple mechanism. Each block is numbered. Whenever an error is detected the tape is stopped and the user is prompted to rewind a couple of blocks. Then the loader scans blocks until the expected block number is matched. At this point the loader resyncs and retries reading the block that previously failed.

The block size is a compromise between speed and reliability. Larger blocks load faster because there is a fixed overhead per block including the IRG (inter record gap). But larger blocks have more chances to get hit by an error and make the error recovery less efficient. STAC uses a block size of 200 bytes, about 50% larger than the standard Atari block size of 128 bytes. I'm afraid I don't remember how and why I decided for this exact size.

IRG (Inter Record Gap), the gap between blocks, is much smaller than the short standard one used by the OS. It can't be reduced much more because that would interfere with the resync logic that requires more than one byte time without transitions on the line to reliably detect the start of a block.

The loader uses RAM under the ROM for the purpose of leaving standard RAM to the loaded program as much as possible. This wasn't a significant limitation because computers with less than 64K RAM (400/800) were very rare at the region.

The recording system uses a simple custom disk file system. Each directory entry, besides the name of the software, size, and location on disk, included the load and run address, and a number of flags that modified some of the behavior of the loader. The actual file data must be already compressed and without any kind of headers. The recorder can't understand a file in standard Atari binary format.

Data is compressed with a simple RLE method and decompressed on the flight while loading. Compression was performed by a separate tool that so far I couldn't locate the source. As I recall there were a couple of generations of the compressor, and the latest one run on an Atari ST.

[ijor]

I'm from Chile, and by the end of 1988 I was told about some games that were created using a "STAC system" that came from Argentina. The other turbo system at the time was the Injektor, which has similar features but required a modified tape unit (the Atari XL12-Injektor). As I didn't own a tape unit, I never paid enough attention to them. I even thought that both systems where the same.

 

Hi vitoco,

 

I'm afraid I don't know or remember all the clones that implemented a similar error recovery. I'm not sure, but I believe injektor was developed by Pedro, a friend of mine from your country, and his father. Pedro actually happened to be the first to release a STAC clone because I personally showed him my system very early. He and his team used to come to visit to my country rather frequently at the time. For me it was so great because at the time he was the only one I knew that I could sustain a high technical conversation about the Atari. I even told him how STAC works. Of course that he wrote his own implementation.

 

Congratulations for your work as well

[ijor]

I reverse-engineered your loader when I was a kid and I always had trouble understanding a part that seemed like an alternative bitrate estimator to me. I haven't checked your code yet and I don't know if I should jump directly into it or I'll try to repeat the reverse-engineering, it was just too fun.

 

Hi Franco,

 

Yep, STAC has an auto baud detector similar in some ways than the one used by the OS. I seemed to remember that it uses tables again, as the OS, just extended for higher rates. But I can see in my own sources that it actually uses a formula. I think now I used tables in some earlier tests, but I'm afraid I don't remember too much about that. I haven't done anything with STAC for close than 30 years! LOL.

 

Btw, the bitrate detector counts frames and, of course, it checks if the system is PAL or NTSC to compute the bitrate from frames and scan lines. But this will fail if the computer happens to have a mixed ANTIC-GTIA pair, one PAL and the other NTSC. Don't remember either if I was aware at the time about the existence of those "hybrids".

 

Reserve engineering is sometimes too fun indeed. Glad you enjoyed reverse engineering my work. Really. I will post disks images with the STAC recorder on the other thread. Some of those disks have an earlier version than the source I posted. And I'm afraid I didn't use to keep all the historic versions as it is common practice nowadays. So reverse engineering might still find something not fully documented. LOL

[Franco Catrin]

May be you don't realize but you are some kind of God to us. Your STAC inspired a lot of people here in Chile, I even got my first job thanks to a clone system that I did after reading your code (I was 14). Not to mention the users!! They just adore the turbo + error recovery system. Vitoco made another clone that was way more popular than mine, and I knew about him because of that.

 

I think I'll do the reverse engineering again and then compare my findings with your source, just for fun.

[José Pereira]

May be you don't realize but you are some kind of God to us. Your STAC inspired a lot of people here in Chile, I even got my first job thanks to a clone system that I did after reading your code (I was 14). Not to mention the users!! They just adore the turbo + error recovery system. Vitoco made another clone that was way more popular than mine, and I knew about him because of that.

 

I think I'll do the reverse engineering again and then compare my findings with your source, just for fun.

Hi Franco. Nice to see you here again and hope that me, MrFish and you can talk soon ...

[Franco Catrin]

Hi José!! We have to go back on track!

[Heaven/TQA]

never heard of such tape system... I was using 1010 standard tapes (still remember my first cassette with Blue Max / Dimension X piano loading music) and always was wondering why the damned VIC20 or c64 had "turbotape"

[Faicuai]

Wow! Impressive! Thank you!

 

I'm from Chile, and by the end of 1988 I was told about some games that were created using a "STAC system" that came from Argentina. The other turbo system at the time was the Injektor, which has similar features but required a modified tape unit (the Atari XL12-Injektor). As I didn't own a tape unit, I never paid enough attention to them. I even thought that both systems where the same.

 

A local game provider asked me if it was possible to create a similar system to his store, because the existing copiers were slow (compared to them) and some games could be loaded using some copiers but not with others. I had read somewhere about a simple way to change the tape recording speed, so I acceped the challenge. The result was SITRE, which I documented (English by Google Translate), updated and partially released last year.

 

Had to recently check my barely used Japan-made 1010 (which I keep for a handful of titles that only run with Tape recorder), and had to replace the belts, which thank God I purchased a while ago, "just in case".

 

At this point, I decided to give SITRE a try, as a good workout for the 1010 (only tape recorder I would recommend), and... oh boy!!!!

 

What an AWESOME, KICK-ASS piece of work SITRE is!!!! I recorder Super Breakout, Yoomp, Full-title Zaxxon, Video Blitz demo, Ocean Detox and Matrix rain-demo (almost 200 Kbytez) in a single, dreadful 60mins tape... which it managed to read with ZERO (NONE) errors !!!That was better than my lifetime of tape-readings I managed to painfully extract from the worthless piece-of-shit 410 that I got with my first Atari 8bit (a 400).

 

Absolutely impressed!!!!

[rcgldr]

Probably too late now, but assuming bad reads didn't entirely stop the read process, then some type of ECC (error correction code) could have been used. The simplest scheme would be to XOR each block and put that XOR block at the end of the tape. This would work only if one of the blocks was bad, XOR'ing all the good blocks read would recreate the one black that wasn't read. The next step would be to group the data on tape into groups of 16, 15 blocks of data, one XOR block, so that any one block bad out of the 16 could be recovered. Higher level ECC could be used, but the cpu time could be excessive, and probably not needed for a device with a relatively low error rate. Wiki article:

 

https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction

Edited June 10, 2018 by rcgldr

[vitoco]

Had to recently check my barely used Japan-made 1010 (which I keep for a handful of titles that only run with Tape recorder), and had to replace the belts, which thank God I purchased a while ago, "just in case".

 

At this point, I decided to give SITRE a try, as a good workout for the 1010 (only tape recorder I would recommend), and... oh boy!!!!

 

What an AWESOME, KICK-ASS piece of work SITRE is!!!! I recorder Super Breakout, Yoomp, Full-title Zaxxon, Video Blitz demo, Ocean Detox and Matrix rain-demo (almost 200 Kbytez) in a single, dreadful 60mins tape... which it managed to read with ZERO (NONE) errors !!!That was better than my lifetime of tape-readings I managed to painfully extract from the worthless piece-of-shit 410 that I got with my first Atari 8bit (a 400).

 

Absolutely impressed!!!!

 

I'm glad that you've found SITRE useful.

 

 

Probably too late now, but assuming bad reads didn't entirely stop the read process, then some type of ECC (error correction code) could have been used. The simplest scheme would be to XOR each block and put that XOR block at the end of the tape. This would work only if one of the blocks was bad, XOR'ing all the good blocks read would recreate the one black that wasn't read. The next step would be to group the data on tape into groups of 16, 15 blocks of data, one XOR block, so that any one block bad out of the 16 could be recovered. Higher level ECC could be used, but the cpu time could be excessive, and probably not needed for a device with a relatively low error rate. Wiki article:

 

https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction

 

ECC could not be used by these tape loaders for many reasons:

 

- Data has a XEX structure, with headers, ponters and so. It is not just a huge memory data sliced in blocks, where you could go back and fix the identified bad block in memory using the checksum data. Bytes have to be processed sequentialy to know where to load the next chunk, where to stop the tape device to run an initialization routine in the computer and then to resume, etc....

 

- The calculation of the timming parameters to read a block of data from tape is made at the begining of each block, identifying some control bits at the beginning of the block. An error in that calculation makes the loader to read more tape to fill the block of data, reading beyond the real end of the physical block in the tape... then, the next block couldn't be available to be read.

 

- When there is a problem with the tape (bad quality, overuse, whatever), some bits might be weak, resulting in a bad block. As these loaders have some checksum data in every block, the most simple way to recover a block is to read it again immediatelly, thus the "rewind the tape 3 numbers of the counter" and then retry. If it fails again, and again, it's better to discard the tape.

 

So, the way that tapes are used in the Atari makes impossible to apply ECC even inside every single block.

[ijor]

ECC could not be used by these tape loaders for many reasons:

Beg to differ. I think it can be used. If it is efficient or not is another issue, but certainly it can be used.

 

- Data has a XEX structure, with headers, ponters and so. It is not just a huge memory data sliced in blocks, where you could go back and fix the identified bad block in memory using the checksum data ...

Not always it's a XEX structure with blocks. My loader doesn't use binary file blocks. So may be you can't use this in every case, but in many cases you can.

 

- The calculation of the timming parameters to read a block of data from tape is made at the begining of each block, identifying some control bits at the beginning of the block. An error in that calculation makes the loader to read more tape to fill the block of data, reading beyond the real end of the physical block in the tape... then, the next block couldn't be available to be read.

I don't see this as a problem at all. If you compute a wrong bit rate then you would likely hit a framing error early on. And in the worst case, you have the IRG to always be able to detect the physical end of the block.

 

- When there is a problem with the tape (bad quality, overuse, whatever), some bits might be weak, resulting in a bad block. As these loaders have some checksum data in every block, the most simple way to recover a block is to read it again immediatelly, thus the "rewind the tape 3 numbers of the counter" and then retry. If it fails again, and again, it's better to discard the tape.

Sometimes retrying helps, sometimes it doesn't and it always fails. And the user typically has a single copy and many times it can't discard it and get another recorded tape. Error correction might be useful in that situation.

 

Again, not so sure ECC is a very efficient method here. It is certainly not efficient using a single block at the end of the tape because errors on multiple blocks are quite common. But it is IMHO a valid idea.

 

[vitoco]

Beg to differ. I think it can be used. If it is efficient or not is another issue, but certainly it can be used.

 

Well, you are right. It can be used and it could solve some problems without the need to retry anothyer read to a block, but under certain conditions.

 

 

Not always it's a XEX structure with blocks. My loader doesn't use binary file blocks. So may be you can't use this in every case, but in many cases you can.

 

When there is a XEX structure in the data, the recovery might be valid only if the extra data is inside the same block. (*)

 

I don't see this as a problem at all. If you compute a wrong bit rate then you would likely hit a framing error early on. And in the worst case, you have the IRG to always be able to detect the physical end of the block.

 

I haven't read yor code, but I guess that your loader has its own loadind routines, so the timming could be calculated at the very first block and then used for the remainin ones. My code relies in OS routines, so the timming is calculated for every block. And if the weak bits are the first ones of a block, the estimated lenght of the block could be after the IRG (or many of them)... the following well read block won't be the expected and a rewing would be required anyway. If this appens to a block, then EEC like in (*) won't be useful.

 

Sometimes retrying helps, sometimes it doesn't and it always fails. And the user typically has a single copy and many times it can't discard it and get another recorded tape. Error correction might be useful in that situation.

 

These loaders were used by stores that sold pirated games, so it was not that difficult or expensive to get a new copy of our favourite game.

 

Again, not so sure ECC is a very efficient method here. It is certainly not efficient using a single block at the end of the tape because errors on multiple blocks are quite common. But it is IMHO a valid idea.

 

I forgot to mention that issue: multiple blocks failures... an ECC method that validates the whole load against an extra block won't be useful.

 

[ijor]

I haven't read yor code, but I guess that your loader has its own loadind routines, so the timming could be calculated at the very first block and then used for the remainin ones. My code relies in OS routines, so the timming is calculated for every block. And if the weak bits are the first ones of a block, the estimated lenght of the block could be after the IRG (or many of them)... the following well read block won't be the expected and a rewing would be required anyway. If this appens to a block, then EEC like in (*) won't be useful.

Correct, I am not using OS routines. But no, I am measuring BPS on every single block, and that shouldn't be a problem anyway.

 

As I said, if the measured BPS is so wrong, then most likely that Pokey will report a framing error early during the block. You can discard the block immediately when getting a framing error well before physical block ends. And in the very unlikely event that a framing error won't be produced, you can still detect the IRG because in the middle of the block the signal can't be idle that long. And when you measure the BPS at the start of the block, you can also check for a reasonable value. If the measured BPS is that low (as it would need to be to really attempt to keep reading after the physical end of the block), then you already know something is wrong with that block. So you should never get in the situation you mention in the first place.

 

Yes, this will require a custom routine accessing the hardware directly instead of calling the OS. I don't think this is a big problem.

 

These loaders were used by stores that sold pirated games, so it was not that difficult or expensive to get a new copy of our favourite game.

Sorry, but I think this is an unreasonable answer. rcgldr suggested an idea of using some kind of error correction. We are not talking about a specific distribution as it was at that period in our region of the world. And even in that case when getting a new copy of the tape wouldn't be expensive, I think it is more than obvious that is much more convenient to be able to use the very same tape right away.

 

 

[Faicuai]

Well, at this point I like Vitoco's approach better, because it is more holistic, integral and universal, in nature.

 

I have always been wary of the "direct-hardware-access mantra", as the incompatibility that its future-broken approach engendered proved to be a pain-in-the-ass, both in PAST and PRESENT time, especially now with the emergent wave of next-gen HW upgrades, and their corresponding low-level code design and support, which always run aground when having to deal with direct-hardware relics. This approach is what pretty much fragmented the Atari scene into smaller, unconnected islands of one-time hacks and "vertical" knowledge that, for the most part, could not survive the test of time, unfortunately. There were exceptions, of course.

 

Now, with respect to ECC support on tape-loading code, I intuitively sense that there has to be a "coded-density" sweetspot for the actual combination of [encoding+media] on hand... one at which error correction and overall throughput efficiency could be sustained.

 

In other words. what is the MAXIMUM bits-per-hertz that could be encoded-and-recorded (and then read/corrected) with the Atari computer (as host) and the Atari 1010 (with OEM HW)? Has anyone explored or measured this?

 

I ask this because, in my humble opinion, this should be the actual beginning or foundation of this discussion, and it would help us evaluate the worthiness of ECC, or any any other error-correction approach.

 

At this point, my intuition tells me retry-by-rewind is the overall best way to handle tape-loading crap, assuming the encoded density does not push the magnetic-media capabilities to its bare limits...

 

As for SITRE, I have gone way beyond what I reported here... and in multiple attempts to load ~200 KBytes of data, have gotten anywhere from 3 to ZERO (none) total errors, and the retry-by-rewind has ALWAYS worked, every single f-time. Kudos!!!

[ijor]

I have always been wary of the "direct-hardware-access mantra", as the incompatibility that its future-broken approach engendered proved to be a pain-in-the-ass, both in PAST and PRESENT time, especially now with the emergent wave of next-gen HW upgrades, and their corresponding low-level code design and support, which always run aground when having to deal with direct-hardware relics.

But for tapes ??? Do you really expect a lot of new development in Atari tape hardware ???

 

And in this case you do want, and even need, to go to the hardware at least partially anyway. The OS is not designed for what we are doing. You can reuse portion of the OS, which might be even riskier than going directly to the hardware because you depend on specific OS versions. And the OS is not reliable for handling higher bit rates, let alone that is buggy even for the standard bit rate.

 

Now, with respect to ECC support on tape-loading code, I intuitively sense that there has to be a "coded-density" sweetspot for the actual combination of [encoding+media] on hand... one at which error correction and overall throughput efficiency could be sustained.

In other words. what is the MAXIMUM bits-per-hertz that could be encoded-and-recorded (and then read/corrected) with the Atari computer (as host) and the Atari 1010 (with OEM HW)? Has anyone explored or measured this?

There is no simple, absolute or unique answer to this. It depends a lot on how it is recorded, on the tape, and of course, on the specific user device. Some are better than the other.

 

I ask this because, in my humble opinion, this should be the actual beginning or foundation of this discussion, and it would help us evaluate the worthiness of ECC, or any any other error-correction approach.

At this point, my intuition tells me retry-by-rewind is the overall best way to handle tape-loading crap, assuming the encoded density does not push the magnetic-media capabilities to its bare limits...

Well, I never said the retry-by-rewind is a bad approach. I was the one who invented this method after all. Jeje. I just said that the error correction idea is valid and it can certainly be used.

 

In my case, I am using compression, which in many cases is more useful than anything else. I am decompressing on the fly with zero cost. If using error correction that wouldn't be possible anymore. You can't decompress on the fly if the data is not valid yet. That's a similar problem vitoco mentioned when used a XEX structure. It would need to be decompressed at the end, or at least after the error was corrected. But then it won't be zero cost anymore. It might take some time to decompress several blocks, and you need space to store the temporary compressed data.

 

I would say that depends on the specific case. If you don't need a XEX structure, and if compression won't reduce too much the number of blocks, then error correction might be a reasonable alternative. But I think we agree that a single redundant block is not enough. You need one every so many blocks. This would increase overall loading time, of course.

 

To be honest, if I were back in time I might explore alternative synchronous modulation. We talked about this long ago. It might be interesting to see if it is possible to do RLL on tapes. That would mean a huge saving on the effective bit rate. But even using "simple" MFM encoding will be an improvement.

Edited June 20, 2018 by ijor