a8rawconv, a new raw disk conversion utility

[F#READY]

Some ED protections only contain (a lot of) double sectors. I once experimented with it and got 30 full sectors on each track.

 

IMHO it would still be nice to have an open documented image format. It could be based on the ATP proposal or something completely different. As long as it's documented and open source it would make me happy

 

http://www.ataripreservation.org/websites/freddy.offenga/ATP16.htm

[phaeron]

v0.91 attached. Found a pretty nasty bug in the KryoFlux stream read routines -- the index pulse tracking code was using the file position instead of the stream position, and so the index pulses were sliding by a small amount for each revolution. The reason is a subtle quirk of the KryoFlux stream format where the index pulses are timed by a stream position that does not include OOB data like the index pulses themselves. It was barely noticeable with most images but became more noticeable with voy's 20 revolution dumps. This was causing more variance in sector position than expected and thus difficulties when trying to write the KryoFlux images out via SuperCard Pro.

 

For ATX images with 5 revs, the error in sector positions is likely very small but I would still recommend reconverting the dumps. (You did keep the original raw dumps, right?!) This only pertains to KryoFlux streams -- SCP images are different and weren't affected.

 

I've done some experiments with extending the ATX format and the results look promising -- got an SD disk image that had extra chunks in it to be accepted by Altirra, Atari++, Atari800, and the VAPI version of A8WP, and a protected ED image to boot on a modified emulator. Here's the current plan:

• Add two new track chunks, one with the same format as the existing sector list and another with new fields. The two will have the same number of sector entries in the same order and will be required together as the two combined create the extended sector list, which contains a superset of and supercedes the existing sector list when supported. Existing readers see the compatible SD sectors in the existing sector list, new readers ignore it and use the two-chunk extended sector list. This allows the set of sectors in the original list to be restricted to what existing ATX readers can support while reducing the changes needed to support the extended sector list, since the first chunk is a direct replacement for the old sector list.
• The new fields in the second chunk will contain (so far): sector ending angular position, weak data offset, MFM encoding flag, User Data Address Mark flag ($F9/$FA DAM), long sector storage flag, physical sector size, side address field. The last one is to support emulating the FDC's READ ADDRESS command. Additional fields in the future would be in another parallel chunk.
• SD disks (track 0 FM) would be encoded with standard ATX headers, while ED/DD disks would use a different signature to force incompatibility. The version field and any other fields in the file header are probably too inconsistently treated by parsers to use.
• Double-sided disks are still TBD. I don't have a drive that can do DSDD, and don't think I have a test image, and there's also the issue that apparently different drives used different virtual->physical sector ordering.
• There will be a real spec this time.
• The aim is not to create another preservation format. ATX is pre-decoded to allow easy conversion to the SIO disk interface, and it's too messy to support that and all of the possible quirks at the physical layer. The goal is to encode enough data that software reading the disk through the SIO disk interface cannot tell the difference from the original disk on a real drive.
• It's also not a goal to create a format that's easily consumable on the Atari itself. Wouldn't be going crazy with something like XML, but the ATX format is already too unfriendly to the Atari with misaligned data chunks that have to be read out-of-order. An entirely new format could be a lot better laid out for this, but it's taken way too long even to get ATX adopted widely enough much less yet another disk format.
• If all goes well, a8rawconv would then be able to default to "ATXv2" with the result being backwards compatible for SD disks.

 

 

a8rawconv-0.91.zip

[Farb]

Thanks for the update and for making everyone aware of the issue.

 

From my side I've identified 269 ATX files that will need to be re-created. I do archive the raw dumps but it will obviously take some time to re-convert them all.

 

Does this only effect ATX or ATR as well?

 

Oh, and I would agree that a backwards-compatible "v2" ATX is probably preferable to a brand new disk format given there is at least some support for ATX in the wild. I'm sure much of the reason for lackluster ATX support (Altirra being the notable exception) has been lack of documentation. If the v2 format fully documents both the "v1" and "v2" formats it should help out a lot.

[serj]

Avery, you can make a simple gui for this program ?

for example - "file open", "save as", etc.

[phaeron]

Does this only effect ATX or ATR as well?

 

ATX only. ATR just has the decoded sector data without timing info, so it is unaffected.

 

Avery, you can make a simple gui for this program ?

for example - "file open", "save as", etc.

 

Sorry... this is too technical of a tool.

[F#READY]

A backward compatible ATX sounds fine to me as long as the whole thing is documented.

 

I wouldn't call ATX, PRO, ATP or any other derivative format a "preservation format". It's best practice to save the raw source and have several "publication formats" e.g. for emulators.

[HiassofT]

Wow, these are some weird disk images. Mixed SD+ED, and looks like one even has remnants of DOS formatting on it. Is it actually possible to read mixed FM and MFM on a stock drive? I don't see provision for it in the 1050 firmware and the first couple of games I have tried worked fine after simply removing the MFM sectors.

A stock 1050 will happily read mixed SD+ED disk with a small trick. You need to write-protect the floppy disk and then can simply switch between densities by sending the appropriate format command ($21 for SD, $22 for ED).

 

It's also quite easy to create such disks on a stock 1050 without any additional hardware:

- format the whole disk in ED

- write data to the drive (use higher sector numbers for this, eg 500-1040)

- format the disk in SD and turn off the drive after it's formatted the first 20 (or so) tracks

- write (bootable) data to the disk (lower sector numbers)

 

Voila, now you have a disk with SD at the beginning and ED at the end. Of course you can also do it the other way round

 

Mixing SD and ED on the same track is a little bit more trickier and you need a modified (Happy etc) drive for that.

 

so long,

 

Hias

[floppydumper]

I was experiencing a crash (segfault) when working with some macgcr disks (SCP dumps of 3.5" Mac LD disks). The issue appears to be that in the scp_read() function, it's hardcoded to only read in 40 tracks (0-39), but 3.5" disks have 80 tracks. As a result, it segfaults later in the code when reading the uninitialized tracks 40-79.

 

I've resolved it in my local copy and verified that it seems to work (I think...) with this change. Just FYI.

[Clay Cowgill]

On the a8rawconv command line, what is the correct formatting in order to override serial port detection and point to a particular port?

 

I have a (working with the SCP software) Supercard Pro that results in an "unable to autodetect SCP COM port" when I try:

 

a8rawconv -of auto -d auto scp0:96tpi output.scp

 

I've tried things like "scp0:96tpi:8" (for COM8) as well as "...:COM8", but nothing seems to work. Wondering if it needs to be some kind of Windows-esque hardware path or maybe not even a "COM"?

 

Thanks,

-Clay

[phaeron]

scp0:96tpi:com8 is the syntax to use for connecting to COM8.

 

If this doesn't work, check Device Manager to see that the COM port is actually registered and enabled, and that its name is indeed COM8. a8rawconv normally autodetects the SuperCard Pro COM port by scanning for the SCP-specific COM driver, so if the autodetect fails it implies that the SCP COM driver is not installed. This driver should have been installed, but it's possible to get into a weird situation where it isn't since IIRC the SCP software uses a slightly different driver (VCP). You need the COM driver for a8rawconv to work.

[Clay Cowgill]

scp0:96tpi:com8 is the syntax to use for connecting to COM8.

 

If this doesn't work, check Device Manager to see that the COM port is actually registered and enabled, and that its name is indeed COM8. a8rawconv normally autodetects the SuperCard Pro COM port by scanning for the SCP-specific COM driver, so if the autodetect fails it implies that the SCP COM driver is not installed. This driver should have been installed, but it's possible to get into a weird situation where it isn't since IIRC the SCP software uses a slightly different driver (VCP). You need the COM driver for a8rawconv to work.

 

Ah, that was the clue I needed. I blew away the "SCP-JIM" driver and let windows just use the default FTDI one-- up and running now. Thanks for the help!

[JimDrew]

phaeron,

 

I just grabbed your latest version and read the manual. I guess you didn't know that the SuperCard PRO hardware has always supported reading the back side of disk just by flipping it over - even if there is just a single index hole. You just need to turn off the index flag during a read or write. The drive will read/write data just fine. A few drives need a jumper change, but just about all of them work without doing anything. This is why I have never worried about doing any type of "flippy" support. If data is stored on the disk as a true double-sided disk, it will be backwards when you read it. That's easy enough to fix by reversing the flux stream before analyzing it. Prior to writing, you would have to reverse the flux stream as well.

 

An easy way to look for the SuperCard PRO hardware on a COM port is to just send the command to get the hardware info to every COM port and look for a response.

Edited December 22, 2015 by JimDrew

[JimDrew]

In Window's Device Manager, go to "Universal Serial Bus Controllers" and locate the entry called "USB Serial Converter". Right-click that entry and select Properties. Under the advanced tab select "Load VCP". That instructs Windows to load a virtual com port (a COMx port). If that is not selected, you will not have a COM port setup for the SuperCard PRO board! Click OK and exit the Device Manager. You will need to remove and re-insert the USB connection to the SuperCard PRO board in order for Windows to force the loading of the VCP.

 

I use the .dll to access the board with the SCP software, not the COM port. It's faster and there are specific buffer commands and such that I use to improve the transfer speed that are not available as a COM port only interface.

Edited December 22, 2015 by JimDrew

[_The Doctor__]

I think this thread needs to be marked of high importance and post at http://atariage.com/forums/topic/231835-a8rawconv-a-new-raw-disk-conversion-utility/?p=3323983

and we be re-directed to that post... it is a big deal indeed.

[phaeron]

0.92 update:

• Fix for long sector encoding in ATX files. The CRC error bit now matches the CRC state of the sector and the data request (DRQ) bit is no longer set. This allows the 1050's behavior to be emulated. Accurate 810 emulation requires the flags to be modified, but that's better than encoding 810 behavior, which loses the CRC status. This fixes Fight Night decoding.
• Bits 2-7 of the sector size byte in the address field are now ignored -- tested on real 1050.
• Fixed crash in macgcr decoder on empty tracks that floppydumper reported. Mac GCR decoding isn't actually usable yet; at some point I need to go back and actually implement proper support for 80 tracks and two sides across the board.

 

a8rawconv-0.92.zip

[Farb]

Thanks for the update, phaeron! I'll check it out shortly.

[bani]

writing seems not to work. tried on two completely different drives, 360k (48tpi) and 1.2m (96tpi).

 

a8rawconv disk.atr scp0:96tpi:com3

 

a8rawconv disk.atr scp1:48tpi:com3

 

both drives result in unreadable disks on real atari 800 with atari 1050.

 

i am able to use both drives to read and convert to atr/atx just fine.

[phaeron]

Try reading back the disks with a8rawconv first to see if it's an encoding issue or a drive compatibility issue.

[bani]

it seems it cannot read back its own output. a8rawconv writes the disk but reads its own output back as a massive bunch of errors. verified on both drives.

Edited May 20, 2016 by bani

[phaeron]

Are you losing sectors outright or getting CRC errors randomly on sectors? In particular, can you check the sector layout on the readback with -l to see if there is a pattern, like losing sectors at the track start/end?

[bani]

now i'm not so sure the scp is even writing correctly at all. i can't get the disk archiver to copy a disk properly. looks like writes are writing complete randomness. don't think this is a8rawconv problem.

Edited May 22, 2016 by bani

[Zarxx]

writing seems not to work. tried on two completely different drives, 360k (48tpi) and 1.2m (96tpi).

 

a8rawconv disk.atr scp0:96tpi:com3

 

a8rawconv disk.atr scp1:48tpi:com3

 

both drives result in unreadable disks on real atari 800 with atari 1050.

 

i am able to use both drives to read and convert to atr/atx just fine.

 

Hi bani,

 

I'm not sure if this helps but I use a Teac FD-55GFR - 84 track, 360 rpm drive and I can read and write perfect floppies from SCP and ATX images via a8rawconv (even the 34 sector EA protection (from SCP image only)). I'm going through the ATX originals currently and most write back fine, a few with timing protections won't even after tweaking. I've been updating Farb. I use Super Archiver on a 1050 to check the data, the only time I see random weak/CRCs are when the disk is going bad. This is using SCP on an old Win XP laptop and Win 10 desktop. Atari 400/800 profile, Splice, 5 revolutions, other settings default.

 

FYI, I have a KryoFlux too but it's gathering dust. I can create raw/atx images fine but no matter how I write back the raw image, the floppy is full of CRCs (at least 1 per track). If anyone is interested in archiving/writing images back to disk, using a SuperCard Pro and phaerons wonderful utility is the way to go.

[ijor]

writing seems not to work. tried on two completely different drives, 360k (48tpi) and 1.2m (96tpi).

 

a8rawconv disk.atr scp0:96tpi:com3

a8rawconv disk.atr scp1:48tpi:com3

 

both drives result in unreadable disks on real atari 800 with atari 1050.

 

I realize it is not your problem, or at least not your only problem ...

 

But please be aware everybody, that writing back on a 96 TPI drive usually doesn't work. You won't be able to read it with an Atari drive, unless you were using virgin unformatted disks as they come from factory.

[phaeron]

 

But please be aware everybody, that writing back on a 96 TPI drive usually doesn't work. You won't be able to read it with an Atari drive, unless you were using virgin unformatted disks as they come from factory.

 

I would upgrade this to a "might not work." I've been using a 96 TPI drive for my experiments, and have had surprising success with reading the disks on a 1050, even without bulk-erasing the disk first. Also, if this is the problem, the 96 TPI drive should at least be able to read back its own output.

[ijor]

 

Also, if this is the problem, the 96 TPI drive should at least be able to read back its own output.

 

As I said, I know it's not his problem. But, and just for completeness ...

 

Some 5.25" HD drives, depending on the configuration, can't write reliable on DD disks. Note that 96 TPI and HD is not quite the same thing. There are 96 TPI drives that are DD only.