a8rawconv, a new raw disk conversion utility

[phaeron]

Spent some time rewriting a8rawconv. Version 0.6 is attached.

 

The first main change is that the tool can now read .ATR and .ATX images and encode them to disk streams. This makes it possible to write ATX images back to physical disks with copy protection intact. The encoder supports missing sectors, bad sectors (CRC errors), deleted sectors, weak sectors, and long sectors. Because of fiddly issues with ATX sector timing, some tracks may require manually adjusting the write timing to fit a long track (-p 95), and types of tracks with overlapping sectors may be hopeless. The sector positions will differ a bit from the original, as well, so this isn't a pristine copy. However, it's good enough that I've been able to boot several VAPI images on my 1050 drive. MFM ED/DD formats are also supported, for a faster way to get 130/180K images onto a real disk than an SIO2PC or 10502PC disk copy.

 

The second main change is that the tool can directly read or write to a floppy through a SuperCard Pro device. This is done by using a special path for input or output, like scp0:96tpi or scp1:48tpi. For read, this is mainly just handy for spot checking tracks, as generally it's better to grab a full image with the SCP software for archival or repeated extraction tests. However, this is more useful for writing as a8rawconv can directly control the write splice points. I've added support for writing .scp images, but have run into problems with errors on sectors crossing the index mark when writing those images through the SCP tool; this avoids the extra step and bypasses that problem. I want to specially thank Jim Drew for providing SCP hardware to test with, which turned out to be rather simple to drive directly.

 

In theory, this version is able to do a conversion from KryoFlux stream format (track*.*.raw) to SuperCard Pro format (*.scp), but it's a bit rough at the moment - YMMV.

 

Other changes:

• Fixed writing of double-density ATR images.
• Fixed incorrect encoding of long sectors in ATX images.
• MFM decoder can now output bit decode debug spew (-vvv).
• Reduced angular position threshold for merging sectors (WTF would you write 12 versions of sector 1 on a track??).

SCP direct access is also supported under Linux, although I have only tested it in a VM -- compile only compileall.cpp. No snickering about my crappy serial port code. I didn't have a really easy way to detect the FTDI interface there, so it simply defaults to /dev/ttyUSB0; override if necessary (scp0:96tpi:/dev/ttyUSB1). You may also need to run as root to have permissions to write to the serial port.

 

a8rawconv-0.6.zip

[Farb]

Nice work, Phaeron!

 

Is there a way for this tool to make an image from side 2 of a Kryoflux imaged "flippy disk"?

[phaeron]

I was under the impression that this wasn't possible because the drive would need to seek to track -8...?

[sup8pdct]

How does it work with fuzzy sectors? Where each time it is read, different data is returned.

 

James

[Farb]

I was under the impression that this wasn't possible because the drive would need to seek to track -8...?

I physically modified a drive to do it with Kryoflux. But when I point a8rawconv at the files for the second side, it doesn't find any data. I can provided you an image of a two sided disk image if it would help.

[phaeron]

How does it work with fuzzy sectors? Where each time it is read, different data is returned.

 

James

 

The program decodes all sectors it can find, then clusters them by angular position. If it finds multiple reads that give the same data, it encodes the sector as stable. Otherwise, it finds the farthest stable point and encodes that as the weak offset within the sector, using the data from the first copy.

 

On encode, it writes out flux transitions on the timing boundary between a valid 0 and 1, so noise causes bits to flip back and forth when read by the drive.

 

I physically modified a drive to do it with Kryoflux. But when I point a8rawconv at the files for the second side, it doesn't find any data. I can provided you an image of a two sided disk image if it would help.

 

Yeah, if you post a sample I can probably get this implemented -- might be as simple as just reading the flux stream backwards.

[+wood_jl]

Amazing stuff! Bravo.

[phaeron]

Version 0.7 attached with support for reading tracks backwards -- use -r for decoding the flipped side.

 

a8rawconv-0.7.zip

[Farb]

Version 0.7 attached with support for reading tracks backwards -- use -r for decoding the flipped side.

 

Wow, that was fast! Great job!

[malers]

Hello,

 

did anyone compile a8rawcon on MacOS (Yosemite)?

 

Thanks

Michael

 

 

[Farb]

I just tried out a8rawconv with SCP direct reads and writes -- seems to work great. As always, awesome job!!

 

I did run into an interesting problem. I tried imaging a disk 3 different ways (same drive/same disk for each attempt):

 

1. Image using Kryoflux to stream file, use a8rawconv to convert stream to ATX -- success

2. Image using SCP to flux file (.scp), use a8rawconv to convert .scp file to ATX -- fails with BOOT ERROR

3. Image using SCP/a8rawconv to go straight from disk to ATX -- success

 

I've run into this problem several times where an SCP flux file won't create a valid ATX but a Kryoflux stream file created using the same drive/disk will. In at least one case, I tried writing the SCP flux file back to a real disk and that wouldn't boot either so I suspect it might be a problem with SCP. The SCP "failures" are sporadic -- some disks appear to image fine and others don't.

 

On a side note, I've noticed SCP flux file creation is quite fast compared to a Kryoflux flux image or even a a8rawconv/SCP direct read. I'm speculating here, but perhaps the SCP flux file doesn't have enough redundant reads of the same sectors for a8rawconv to extract valid data?

 

Anyway, here are the dumps of the 3 above approaches if there's interest in analyzing what's going on:

 

https://drive.google.com/drive/#folders/0B7VbqNUi-LBvSVVJZ29lWW5rNmc

[phaeron]

You've only got one revolution per track in the SCP image. I need to put in a warning for this. Atari tracks are not index mark aligned and imaging only one rev per track is almost guaranteed to drop a sector.

[Farb]

You've only got one revolution per track in the SCP image. I need to put in a warning for this. Atari tracks are not index mark aligned and imaging only one rev per track is almost guaranteed to drop a sector.

 

So does this mean the "Copy Mode" should always be set to "Splice" for A8 disks? That seems to be the only way get the "# of Revolutions" parameter to be enabled. I also assume the "# of Revolutions" should be changed from 2 to 5 per your prior suggestion?

[phaeron]

Yeah, you need to use 5 revs on splice mode. 2 can work, but it's marginal, and if you've got an old disk in the drive you only want to do one pass on it if possible.

[JimDrew]

You've only got one revolution per track in the SCP image. I need to put in a warning for this. Atari tracks are not index mark aligned and imaging only one rev per track is almost guaranteed to drop a sector.

 

Is this always the case for data disks formatted with the Atari drive? I was under the impression that the index pulse was always used for starting/stopping the formatting of disks. I don't have any Atari "data disks", and ALL of the commercial Atari disks I have were produced using the index mark to start/stop the tracks, so they duplicate just fine in INDEX mode.

 

If Atari disks formatted on Atari drives do not actually use the index pulse to start/stop the track during the format, then you would need to use SPLICE mode, with 2 revs. Anymore than 2 revs is a waste as the data will always be the same, but you need two revolutions to encompass a full track that does not have write a splice somewhere in it. If you image with more than 2 revs, the SCP software will write a disk in SPLICE mode using the last 2 revs.

Edited February 15, 2015 by JimDrew

[phaeron]

Is this always the case for data disks formatted with the Atari drive? I was under the impression that the index pulse was always used for starting/stopping the formatting of disks. I don't have any Atari "data disks", and ALL of the commercial Atari disks I have were produced using the index mark to start/stop the tracks, so they duplicate just fine in INDEX mode.

 

Yes, for both the 810 and 1050. The index sensor isn't even hooked up to the FDC -- it is simulated using the RIOT.

 

Some commercial disks were indeed created with index-aligned tracks, but not all of them. This is part of the sector layout map for Dimension X:

 0 (18) |  6   8  10  12 14  16 18      1   3  5   7  9   11 13  15 17  2  4
 1 (18) |    8  10  12 14  16 18       1  3   5  7   9  11  13 15  17 2   4  6
 2 (18) |  8   10 12  14 16  18      1  3   5  7   9  11  13 15  17 2   4  6
 3 (18) | 8  10  12 14  16 18      1   3  5   7  9   11 13  15  17 2   4  6
 4 (18) |   10 12  14 16  18      1  3   5  7   9  11  13 15  17 2   4   6  8
 5 (18) | 10 12  14 16  18      1  3   5  7   9  11  13 15  17 2   4  6   8
 6 (18) |   12 14  16 18      1   3  5   7  9   11 13  15 17  2  4   6   8  10
 7 (18) | 12  14 16  18     1   3  5   7  9   11  13 15  17 2   4  6   8   10

 

If Atari disks formatted on Atari drives do not actually use the index pulse to start/stop the track during the format, then you would need to use SPLICE mode, with 2 revs. Anymore than 2 revs is a waste as the data will always be the same, but you need two revolutions to encompass a full track that does not have write a splice somewhere in it. If you image with more than 2 revs, the SCP software will write a disk in SPLICE mode using the last 2 revs.

 

No, 5 revs should always be used for an unknown or old disk. Using only two revs means that there will only be one complete image of the sector that crosses the read/write point and it will not be recoverable if that read happens to be bad, and it will hamper a8rawconv's ability to detect weak sectors. If the disk is old and deteriorating, you don't want to have to do another read pass on it with different settings if it turns out that the two-rev image wasn't good enough.

[JimDrew]

In working with C64, Amiga, Atari ST, PC, TRS-80, and a few other formats I have never had to use more than 2 revolutions. The only way data can change from revolution to revolution is if the disk or head is dirty. The flux data itself (even what is considered by many as "weak bits") won't actually change on its own.

 

Granted, if the disk or head is dirty you can use multiple passes to extract good data when found. Everyone should be cleaning their heads every time they start imaging disks. I clean the heads frequently, sometimes between every disk image - depending on the quality of media I am imaging.

 

Thanks for the info on the Atari disks. What I read on the internet appears to be in error, stating that the index sensor is connected and used during formats. I guess I will have to connect my 800XL and 1050 drive and figure out how to format a blank disk and then look at it.

 

I looked at the Floyd of the Jungle image, and you can easily see it is not index aligned by looking at the flux display in SCP's analyzer.

[JimDrew]

OK, so where do I find out more info about the Atari drives and how they interact with an Atari computer? Can the Atari ever see the CPU in the disk drive or is it limited to some type of serial communications? The reason I ask is that I have a product that I am going to release for the C64 community called uDrive (micro-drive). It is a cycle exact 6502/dual 6522/data separator emulation of a 1541 drive. I will be adding 1571 and 1581 disk drive support as well. I can emulate the 6507 (which is the cousin to the 6502), and all of the support hardware that I see in the 1050's schematic. But if copy protection couldn't poke around at the CPU or memory, then some simple code is all that would be necessary to use ATR/ATX/SCP image files with it plugged into an Atari computer.

Edited February 15, 2015 by JimDrew

[phaeron]

In working with C64, Amiga, Atari ST, PC, TRS-80, and a few other formats I have never had to use more than 2 revolutions. The only way data can change from revolution to revolution is if the disk or head is dirty. The flux data itself (even what is considered by many as "weak bits") won't actually change on its own.

 

With all due respect, this is demonstrably false. The whole point of "weak bits" is to produce data coming out of the drive that is interpreted differently by the FDC on each revolution, and I have SCP images of such disks that show markedly different flux transition patterns between different revolutions on the same track. If what you said was true, the protection wouldn't work, because the FDC would see the same pattern each time and produce a stable read. I don't doubt that the flux pattern read in one spliced revolution would work when written back to the disk verbatim, but that's because any of the captured patterns is inherently unstable on the disk medium. From a decoding and analysis standpoint, it is more reliable to capture multiple revolutions to determine if this effect was intentional or due to a dirty drive or deteriorating disk, and a8rawconv can and does compare the same sector from all revolutions to tell you if it is seeing a stable sector, a marginal sector that read OK only some of the time, or a sector that was bad and different every single time.

 

Besides -- what is the point of trying to economize on captured revolutions? This is for imaging and archival, not high-volume mastering. Saving a couple of minutes and a few hundred kilobytes of space per disk at the cost of having a less comprehensive image doesn't seem like a good tradeoff.

[phaeron]

OK, so where do I find out more info about the Atari drives and how they interact with an Atari computer? Can the Atari ever see the CPU in the disk drive or is it limited to some type of serial communications? The reason I ask is that I have a product that I am going to release for the C64 community called uDrive (micro-drive). It is a cycle exact 6502/dual 6522/data separator emulation of a 1541 drive. I will be adding 1571 and 1581 disk drive support as well. I can emulate the 6507 (which is the cousin to the 6502), and all of the support hardware that I see in the 1050's schematic. But if copy protection couldn't poke around at the CPU or memory, then some simple code is all that would be necessary to use ATR/ATX/SCP image files with it plugged into an Atari computer.

 

Attached is a work-in-progress version of my Hardware Manual. Information about the Atari disk drive serial protocol is in Chapter 9.6, 810 Disk Drive, and the physical disk in Appendix B, Physical Disk Format.

 

Unlike the C64 disk drives, the standard serial protocol for communication with Atari disk drives does not allow direct access to the CPU or memory on the disk drive. None of Atari's drives had code upload capability, and the XF551 couldn't even execute code from RAM, being 8048-based. Third-party disk drives did have this capability, but the standard drives did not and copy protection mechanisms could not depend on it. The vast majority of protected disks targeted the 90K single density format supported by the 810 disk drive. The serial protocol only supports basic commands to read or write sectors by virtual sector number (1-720), to retrieve controller and FDC status bytes, and to do a full disk format. As you suspected, the protocol is too high level to require full simulation of the 6507-based controller -- it is only necessary to simulate the behavior at the protocol level. However, the protocol does provide enough information to:

• Detect the timing of the end of a sector (very commonly used)
• Retrieve phantom sectors with duplicated sector addresses based on the timing of the read sector request
• Determine if a sector is missing, has a deleted data mark, is a long sector (>128 bytes), or has a CRC error
• Retrieve the data from a sector that has a detected CRC error, or the first 128 bytes of a long sector

atari4.pdf

[JimDrew]

I can read and write "weakbits" (that will change every revolution when read by a native disk drive) by imaging just a single revolution using the SuperCard Pro hardware, and some software magic. So, multiple revolutions are not required to duplicate the flux exactly. Captured patterns are not "unstable", they are perfectly stable and predictable - even though they appear to change.

 

The biggest complaint I get about SuperCard Pro is the image size and how long it takes to image a disk. This is one reason why I have the ability to set a "preservation mode" or not. The data is virtually the same, but the non-preservation data is altered slightly to produce image files that will compress substantially better than an unaltered preservation image. The Amiga emulators can actually load and use a .scp image file that is compressed with .zip, .7z, etc. 7-zip (.7z) is hands down the best compressor you can use for flux image files. The non-preservation files can be compressed to just less that 10% of their original size.

 

SuperCard Pro is a flux level copier, with some analyzing and conversion for Amiga and C64 disks. I don't know anything really about Atari disks other than they are FM. My flux->FM decoding is not correct in the analyzer I have spent my life working on GCR and MFM formats only, so FM is something I have never had to work with before. I need to fix the flux->FM decoder, at which point I could make a routine that did error checking on the data for FM based disks.

 

I did look at the Atari 810 and 1050 schematics and I can see how the 6532 does make a fake index pulse. Based on some info, I guess you just send commands to the drive and it responds.

 

Thanks for that info. I may peek at your source code for your program to see how you convert my flux data to FM (unless you convert flux to raw sector data, which won't do me any good). I am not a C programmer, but I will try to figure it out.

Edited February 15, 2015 by JimDrew

[JimDrew]

Interesting read. So, it appears emulating a 810 or 1050 would be a very easy thing to do compared to a 1541 or other CBM drive where you can access the CPU directly.

 

I never designed SuperCard Pro to be anything more than a means to dump flux data, but it has turned into quite a bit more than that. Several people have made utilities that access the SCP hardware directly and/or use the .scp disk image format.

 

The HxC floppy drive emulator software (free download) will convert to/from .scp image file format to/from numerous (dozens) of different disk types, but Atari 400/800 format is not supported. :\

 

Well... one thing that just dawned on me is that Atari disks spin at 288 RPMs. I need to adjust the flux conversion times accordingly for Atari 400/800 disks. Maybe my FM decoder works, but the bitcell zones are wrong.

Edited February 15, 2015 by JimDrew

[phaeron]

288 RPM vs. 300 RPM shouldn't be an issue. It'll reduce the tolerances for decoding, but the majority of disks should still decode fine. A common practice for imaging Atari disks used to be to read them on a PC using a DOS-based program called Anadisk; this was using 300 RPM equivalent timings but it was close enough for the FDC to be able to pull the data successfully. Reportedly it was also possible to go the other way with writing to Atari disks from the PC, although sometimes there would be problems with the PC overrunning sectors on tracks formatted by an Atari drive.

[BillC]

Interesting read. So, it appears emulating a 810 or 1050 would be a very easy thing to do compared to a 1541 or other CBM drive where you can access the CPU directly.

 

I never designed SuperCard Pro to be anything more than a means to dump flux data, but it has turned into quite a bit more than that. Several people have made utilities that access the SCP hardware directly and/or use the .scp disk image format.

 

The HxC floppy drive emulator software (free download) will convert to/from .scp image file format to/from numerous (dozens) of different disk types, but Atari 400/800 format is not supported. :\

 

Well... one thing that just dawned on me is that Atari disks spin at 288 RPMs. I need to adjust the flux conversion times accordingly for Atari 400/800 disks. Maybe my FM decoder works, but the bitcell zones are wrong.

It's not that the discs have to spin at 288RPM, it's that most Atari drives do so. The XF551 drive uses standard mechanisms which spin at 300RPM, its read/write speed is increased to compensate for the faster rotation.

[JimDrew]

288 RPM vs. 300 RPM shouldn't be an issue.

 

It's actually a huge issue. If you don't take into account the drive speed when decoding the data, it definitely will not decode correctly. A 4us pulse at 300RPMs is nowhere near 4us at 288RPMs. My decode for GCR and MFM looks at the drive speed and adjusts the decoder accordingly. I do not do that for Atari 400/800 disks (or images) that are being viewed in the analyzer, which is likely why the decoded data looks funny. Most people are using 360RPM drives for reading/writing disks with SuperCard Pro. So, going from 288RPMs to 360RPMs is a big jump!

Edited February 19, 2015 by JimDrew