Jump to content

Photo

a8rawconv, a new raw disk conversion utility


144 replies to this topic

#1 phaeron OFFLINE  

phaeron

    River Patroller

  • 2,781 posts
  • Location:Bay Area, CA, USA

Posted Sat Nov 15, 2014 2:13 AM

I've been helping a friend archive some Atari 8-bit disks lately, and ran into a problem with the limited imaging paths available, which tend to only work with unprotected or minimally protected disks and aren't suited well to producing A8-specific image formats, or require elaborate and slow imaging steps with real A8 hardware. Attached is a8rawconv, an attempt to rectify that. It's a command-line utility to convert raw flux transition streams produced by imaging hardware to usable disk images.

 

Currently, the input can either be a set of KryoFlux raw track streams (track**.0.raw) or a SuperCard Pro image (*.scp). Unlike the base conversion utilities available for those hardware, a8rawconv attempts to analyze and decode unusual sector patterns seen in A8 protected disks, including phantom (duplicated) sectors, deleted sectors, sectors with CRC errors, and weak sectors. The output format is an ATX (VAPI) disk image, in order to preserve those unusual properties and also encode sector timing and track skew information. For unprotected disks, the tool can also produce a plain ATR disk image.

 

The main disk geometry supported by a8rawconv is a standard 40 track, 18 sector/track single density (FM) image compatible with the 810 drive. It can also decode enhanced and double density (MFM) tracks, but ED can currently only be encoded into ATR and DD is mostly untested as I don't have an image of a DD disk or a DD-capable drive to produce one.

 

The tool is invoked simply:

a8rawconv kryoflux-image/track00.0.raw disk.atx
a8rawconv supercardpro-image.scp disk.atr

This will automatically scan for both FM and MFM encoded tracks, merge duplicate sectors, and write the disk image. Recording 5 disk revolutions during imaging is recommended to ensure a good read and that weak sectors can be distinguished from stable sectors. Additional command-line options are available to dump additional data like sector layout and tweak the decoding process.

 

Source and Win32 binary are included. Although not really tested, it's written in standard C++ and should compile on other platforms.

 

Attached Files



#2 F#READY OFFLINE  

F#READY

    Chopper Commander

  • 100 posts

Posted Sat Nov 15, 2014 6:04 AM

This is great Phaeron! Thanks a lot :-)

I got both Kryoflux and SCP and won't mind testing/improving the program.

The raw/scp files are perfect for long time preservation, so i would always keep the original flux files together with the extracted/converted disk images.

Again, great work!!!

F#READY

#3 Farb OFFLINE  

Farb

    Dragonstomper

  • 726 posts
  • Location:Frankfurt, Germany

Posted Sat Nov 15, 2014 3:50 PM

This is fantastic news! Thanks for your work on this.

 

Do you know the Kryoflux command line settings you used to produce the stream file? I had gone through the trouble of imaging my entire library at one point to discover I had a missing command line parameter that made the dumps useless for preservation purposes (-i I think?)

 

Not to derail things too much, but how do you find the SCP format vs. the Kryoflux stream format?



#4 phaeron OFFLINE  

phaeron

    River Patroller

  • Topic Starter
  • 2,781 posts
  • Location:Bay Area, CA, USA

Posted Sat Nov 15, 2014 4:58 PM

I don't actually have a KryoFlux or SCP myself and didn't do the imaging, so I can't say what parameters were used. The KryoFlux stream dumps were done at 96tpi dual-sided and probably done on default preservation settings. This is overkill for Atari single-density, but they were done under the mantra of "image first, ask questions later." The tool processes even tracks on side 0.

 

The KryoFlux and SCP stream formats are conceptually similar. SCP is a much cleaner and easier to parse format, having a thorough specification, structured headers, and an easy to parse sample encoding. The KryoFlux stream format, on the other hand, requires a separate file per track with no headers and has a more complex variable-length encoding. The index mark encoding is particularly a PITA as it is out of band and requires going back to match the index mark against the sample positions and timings.

 

At the flux level, SCP has a slight advantage as it records sample timing at 25ns precision vs. KryoFlux's 42ns. Both are overkill for decoding as one bit cell in FM is 4us, which gives almost 100 samples per bit cell at KryoFlux's resolution. In tests with the same disk imaged on the same drive on both KryoFlux and SCP devices, I didn't see any difference in decoding -- in both cases the sample precision was far more than required and the bit cell timings were very stable. For kicks I pushed a 360K DOS disk through the converter and didn't see any issues with 2us MFM decoding either, although obviously it produced a nonsensical output ATR file.

 

The Atari 8-bit's high level disk interface means that the disks can't get as crazy as on platforms like the Amiga or Apple II where the bit stream can be manipulated directly, so IMO as long as the stream decodes stably there's not as much to be worried about with regard to preservation. However, you do want to preserve the stream files regardless, as there is still a lot of information not preserved in even ATX, such as bit cell rate. Just with a couple images I've already seen some crazy stuff, such as overlapping sectors where the IDAM of one sector is inside of the data region of the last sector (!).



#5 JimDrew OFFLINE  

JimDrew

    Chopper Commander

  • 112 posts

Posted Sat Nov 15, 2014 10:33 PM

Nice job! I will need to find an Atari emulator to test the .atr files with. The last Atari 400/800 emulator I played with was Wacke, written by employees at my company. That was quite a long time ago. What PC version Atari 400/800 emulation do you recommend?

Edited by JimDrew, Sat Nov 15, 2014 10:33 PM.


#6 Farb OFFLINE  

Farb

    Dragonstomper

  • 726 posts
  • Location:Frankfurt, Germany

Posted Thu Nov 20, 2014 1:27 AM

I've created a .scp of Tumble Bugs as a test. When the image is written back to a real floppy disk it boots and plays (albeit with some title screen corruption that I'm hoping Jim can help identify). When I run it through the converter and create an ATX it crashes during the loading of the splash screen.

 

The image is available here:

 

https://drive.google...SVVJZ29lWW5rNmc

 

Oh, and @JimDrew: if you haven't found out already, Altirra is the way to go!


Edited by Farb, Thu Nov 20, 2014 1:28 AM.


#7 Mclaneinc OFFLINE  

Mclaneinc

    Retro Madman

  • 6,717 posts
  • Location:Northolt, UK

Posted Thu Nov 20, 2014 9:57 AM

Same crash here, tried every trick to make sure Altirra is as like a real machine possible..



#8 phaeron OFFLINE  

phaeron

    River Patroller

  • Topic Starter
  • 2,781 posts
  • Location:Bay Area, CA, USA

Posted Thu Nov 20, 2014 10:56 PM

Issue was with the stride for fishing tracks out of the .SCP image, since this image was generated with different settings. I think the images I tested with were 96 tpi whereas this is 48 tpi. Put a hack into v0.4 to allow both images to decode, but need to work out correct format interpretation.

 

This disk is recorded with a poor interleave, btw:

 0 (18) | 1   8   15 4   11  18 7   14  3  10  17  6  13  2   9  16  5   12

If I'm right, this disk should read at about two thirds of normal speed on a non track buffered drive.

 

Attached Files



#9 Farb OFFLINE  

Farb

    Dragonstomper

  • 726 posts
  • Location:Frankfurt, Germany

Posted Thu Nov 20, 2014 11:46 PM

That image was made using SuperCard Pro's C64/C128 setting because there seems to be an issue with the Atari 400/800 one. I believe Jim is looking into it. I can confirm that the 0.4 build creates a working ATX.

 

On another note, I don't know if this is helpful or not, but I uploaded a Kryoflux stream dump of "Canyon Climber" that Ijor had used a home-made converter to create an ATX out of. I've uploaded the ATX he made as well as the ATX that a8rawconv made. They are different so I don't know if it provides any useful information. All are available at the same link as above.

 

I have other examples of Kryoflux stream dumps and ATX images he generated from them if it helps at all.


Edited by Farb, Thu Nov 20, 2014 11:56 PM.


#10 Farb OFFLINE  

Farb

    Dragonstomper

  • 726 posts
  • Location:Frankfurt, Germany

Posted Thu Nov 20, 2014 11:54 PM

Oh, and I also just uploaded a Kryoflux stream dump of the same Tumble Bugs disk. a8rawconv made a working ATX of it as well.



#11 phaeron OFFLINE  

phaeron

    River Patroller

  • Topic Starter
  • 2,781 posts
  • Location:Bay Area, CA, USA

Posted Fri Nov 21, 2014 3:21 AM

Thanks, this is helpful. There are a number of ways that ATX files can vary while holding the same information, so a raw byte compare difference is not surprising. Was Ijor's image made from this same dump or separately? I had heard that the existing images were made with a modified 1050 drive, but if this is from the same stream then it proves that the ATX sector angular position unit is smaller than 1/26042th of a track.



#12 Larry OFFLINE  

Larry

    River Patroller

  • 4,112 posts
  • Location:U.S. -- Midwest

Posted Fri Nov 21, 2014 6:11 AM

I do have a Kryoflux and as with the original intent of the ATX files, it would be great to actually be able to write the files to a real disk.

 

Any realistic chance that this new conversion utility can help get us further toward that goal?  Maybe with the Mega Speedy?

 

I also remember that Steve Tucker was working on being able to write PRO images to a Happy Drive, but I don't remember how that all turned out.  Anyone have any info on that?

 

-Larry



#13 R.Cade OFFLINE  

R.Cade

    Stargunner

  • 1,327 posts
  • Location:Augusta, Georgia, USA

Posted Fri Nov 21, 2014 7:50 AM

Kryoflux can now write any streams directly back to disk as of the latest release...



#14 Farb OFFLINE  

Farb

    Dragonstomper

  • 726 posts
  • Location:Frankfurt, Germany

Posted Fri Nov 21, 2014 9:55 AM

Was Ijor's image made from this same dump or separately?

 

Ijor's image was made from the same dump as far as I know. I provided him access to all of my Kryoflux stream dumps and he provided me with ATX files that he created from them and he also submitted them to Atarimania.

 

He had a program that supposedly made ATX files using a Happy 1050 but I could never get that to work successfully.


Edited by Farb, Fri Nov 21, 2014 9:56 AM.


#15 Farb OFFLINE  

Farb

    Dragonstomper

  • 726 posts
  • Location:Frankfurt, Germany

Posted Fri Nov 21, 2014 9:58 AM

I also remember that Steve Tucker was working on being able to write PRO images to a Happy Drive, but I don't remember how that all turned out.  Anyone have any info on that?

 

PRO is a lossy format so attempting to write those images back would be interesting to say the least :-)



#16 Farb OFFLINE  

Farb

    Dragonstomper

  • 726 posts
  • Location:Frankfurt, Germany

Posted Fri Nov 21, 2014 10:55 PM

I created both SuperCard and Kryoflux dumps of F-15 Strike Eagle. Running an ATX created from either with 0.4 of the converter in Altirra doesn't work. The former causes a boot error and the latter causes Altirra to crash.

 

Raw dumps are available at my Googld Drive link above.



#17 phaeron OFFLINE  

phaeron

    River Patroller

  • Topic Starter
  • 2,781 posts
  • Location:Bay Area, CA, USA

Posted Sat Nov 22, 2014 12:00 AM

Hmm, thanks. The SCP image converted fine for me, but strangely, it only has 38 tracks in the file whereas the KryoFlux image has 40 valid tracks. The KryoFlux image isn't decoding because one sector is not being found, unless I stretch the decoder clock with -p 110. I've seen this happen with other disks, so I need to work on the decoder PLL.



#18 playermissile OFFLINE  

playermissile

    Chopper Commander

  • 249 posts

Posted Sat Nov 22, 2014 12:03 AM

Compiles under linux using gcc 4.7 (fails with 4.6 or earlier):

gcc -std=c++0x -o a8rawconv a8rawconv.cpp

The following results are all under linux running the atari800 emulator version 3.1

 

Converting TumbleBugs.scp to an atr briefly flashes the splash screen before locking up.

 

F15 Strike Eagle converted from the scp does seem to work -- it asked for the authentication code 7 and I pressed a random key. I don't remember what I pressed but I was flying and could control the airplane and fire the guns. I don't remember the controls well enough but it seemed to work.

 

Mouskattack from the kryoflux crashes to the monitor.

 

Canyon Climber from the kyroflux freezes at the title screen. The CanyonClimber-a8rawconv.atx in the google drive folder does work so I don't know what's different there.

 

Agent USA from the kryoflux freezes immediately.



#19 Farb OFFLINE  

Farb

    Dragonstomper

  • 726 posts
  • Location:Frankfurt, Germany

Posted Sat Nov 22, 2014 1:38 AM

The SCP image converted fine for me, but strangely, it only has 38 tracks in the file whereas the KryoFlux image has 40 valid tracks.

 

I went back and looked at the SCP program. I didn't notice that it defaults to 38 tracks in C64/C128 mode. I've redumped the F15 disk with 40 tracks and re-uploaded it. I was able to make a working ATX of it -- might have been user error on my part.



#20 phaeron OFFLINE  

phaeron

    River Patroller

  • Topic Starter
  • 2,781 posts
  • Location:Bay Area, CA, USA

Posted Sat Nov 22, 2014 4:28 AM

Huh, those results are quite a bit different than what I'm seeing -- Canyon Climber and Mouskattack work fine in Atari800 3.1.0 for Win32. What are you getting for the conversion results for Mouskattack? I'm just seeing an unformatted track 3 with no missing or phantom sectors or CRC errors. The -l switch will show the sector layout of each track. I did a test compile of v0.3 in an Ubuntu VM and didn't see any different behavior with a gcc-compiled version of the tool.



#21 Larry OFFLINE  

Larry

    River Patroller

  • 4,112 posts
  • Location:U.S. -- Midwest

Posted Sat Nov 22, 2014 7:55 AM

Kryoflux can now write any streams directly back to disk as of the latest release...

 

Thanks!  I just downloaded the new files and docs.  I'll check them out.  I must admit, I've haven't checked in probably a year or so.

 

-Larry



#22 playermissile OFFLINE  

playermissile

    Chopper Commander

  • 249 posts

Posted Sat Nov 22, 2014 1:28 PM

Huh, those results are quite a bit different than what I'm seeing -- Canyon Climber and Mouskattack work fine in Atari800 3.1.0 for Win32. What are you getting for the conversion results for Mouskattack? I'm just seeing an unformatted track 3 with no missing or phantom sectors or CRC errors. The -l switch will show the sector layout of each track. I did a test compile of v0.3 in an Ubuntu VM and didn't see any different behavior with a gcc-compiled version of the tool.

 

Ahhh, my bad. I was saving the files as .atr rather than .atx and just ignoring the warnings. Running mousekattack again using atx gives me:

$ ./a8rawconv mouskattack/mouskattack-stream00.0.raw mouskattack.atx
A8 raw disk conversion utility v0.4
Copyright (C) 2014 Avery Lee, All Rights Reserved.
Licensed under GNU General Public License, version 2 or later.

Writing ATX file: mouskattack.atx
WARNING: No sectors found for track 3 -- possibly unformatted.
18 missing sectors, 0 phantom sectors, 0 sectors with errors

and both Mousekattack and Canyon Climber work.

 

I forgot to mention: nice work!  :)



#23 JimDrew OFFLINE  

JimDrew

    Chopper Commander

  • 112 posts

Posted Tue Dec 2, 2014 10:30 AM

I will look at the fixing the formats that use the standard IBM 3740 format (Atari 400/800, TRS-80, CP/M, IBM PC, etc.) this week.  I am changing the copier/imager so that either side (or both) of the disk can be copied/imaged.  In the case of Atari 400/800 disks, you only need to image one side.  That will greatly reduce the file size.  Right now, the C64/128 disk mode works fine for creating images that can be converted with Phaeron's utility, but the disk type ID is wrong.  The first 16 bytes of the header are not part of the checksum'd data, so you could edit the header to be Atari 400/800 when you are done.

 

You don't need a true PLL emulation.  We have the flux data, so there is no guessing about what is to come.  The converted FM data can really only be one of two time values, +/- a certain percentage for drive speed wobble and any write precomp transition (which does not occur with Atari 400/800 disks).  I use a +/-48% bitcell time window for converting flux to MFM/GCR.   In the case of an Atari ST disk (MFM) the bitcells will be either 4us, 6us, or 8us long.  There is 2us between them.  So, in my code a value of 4us could be as low as 3.06us or as high as 4.94us.  Anything between 4.94us and 5.06us really shouldn't be occurring and would have to be considered invalid.  It's a simple window comparator, and it works exceptionally well.  I have been able to recover Amiga disks that have no chance of reading on a real Amiga, no matter what you do.



#24 phaeron OFFLINE  

phaeron

    River Patroller

  • Topic Starter
  • 2,781 posts
  • Location:Bay Area, CA, USA

Posted Fri Dec 5, 2014 2:45 PM

That's mostly what I do, but I have a case that's not decoding properly without timing tweaks. This is the raw dump from the bit decoder of the start of an ID field for a sector on track 39:

 C7 FE | delta = +122 | 0  <-- ID address mark
 FE 8F | delta =  -5 | 1
 FE 8F | 253 | 1788985
 8F FC | delta = +126 | 0
 FC 1F | delta =  -1 | 1
 FC 1F | 245 | 1789238
 1F F8 | delta = +118 | 0
 F8 3F | delta =  -9 | 1
 F8 3F | 135 | 1789483
 3F F1 | delta =  +8 | 1
 3F F1 | 135 | 1789618
 F1 7F | delta =  +8 | 1
 F1 7F | 245 | 1789753
 7F E2 | delta = +118 | 0
 E2 FF | delta =  -9 | 1
 E2 FF | 248 | 1789998
 FF C4 | delta = +121 | 0
 C4 FF | delta =  -6 | 1
 C4 FF | 134 | 1790246
 FF 89 | delta =  +7 | 1
E3 7F C7 FE 8F FC 1F F8 3F F1 7F E2 FF C4 FF 89 | 15.95
 FF 89 | 125 | 1790380
 89 FF | delta =  -2 | 1
 89 FF | 127 | 1790505
 FF 13 | delta =  +0 | 1
 FF 13 | 128 | 1790632
 13 FF | delta =  +1 | 1
 13 FF | 125 | 1790760
 FF 27 | delta =  -2 | 1   <-- track number (39)
 FF 27 | 136 | 1790885
 27 FF | delta =  +9 | 1
 27 FF | 246 | 1791021
 FF 4E | delta = +119 | 0
 4E FF | delta =  -8 | 1
 4E FF | 258 | 1791267
 FF 9C | delta = +131 | 0
 9C FF | delta =  +4 | 1
 9C FF | 249 | 1791525
 FF 38 | delta = +122 | 0
 38 FF | delta =  -5 | 1
 38 FF | 257 | 1791774
 FF 70 | delta = +130 | 0
 70 FF | delta =  +3 | 1
 70 FF | 246 | 1792031
 FF E0 | delta = +119 | 0
 E0 FF | delta =  -8 | 1
 E0 FF | 191 | 1792277
 FF C0 | delta = +64 | 0
FF 13 FF 27 FF 4E FF 9C FF 38 FF 70 FF E0 FF C0 | 15.88
 C0 FF | delta = -63 | 1
 C0 FF | 322 | 1792468
 FF 80 | delta = +195 | 0
 80 FE | delta = +68 | 0   <-- extraneous zero bit
 FE 01 | delta = -59 | 1   <-- side number (bad clock bits)
 FE 01 | 254 | 1792790
 01 FC | delta = +127 | 0
 FC 03 | delta =  +0 | 1
 FC 03 | 252 | 1793044
 03 F8 | delta = +125 | 0
 F8 07 | delta =  -2 | 1

Left two columns are the FM clock and data shift registers. Lines with two numbers are flux transitions and have the sample delta and absolute sample numbers, respectively. Lines with "delta" indicate new bits being shifted in, along with the current timing error. This is a SuperCard Pro dump on a 360 RPM drive, so nominal bit cell time is 128 samples (4ms/bit * 288 RPM / 360 RPM / 25ns/sample = 128 samples/bit).

 

The trace starts at the modified byte that starts the IDAM, C7 clock bits with FE data. We then shift in 16 bits and get a clean track number (FF / 27). The next byte goes wrong, though, with an extra zero bit being decoded due to a flux transition that is nearly half a bit cell off. This then shifts the rest of the bit decoding and breaks the sector ID. Now, you might think this is just a disk going bad, which is possible. What bothers me is that by stretching the bit cell timing to 110% of average, this sector decodes cleanly. That leads me to believe that there's a bit more going on here that could lead to decoder improvements even if this is a marginal disk.

 



#25 JimDrew OFFLINE  

JimDrew

    Chopper Commander

  • 112 posts

Posted Sun Dec 7, 2014 9:39 PM

It's entirely possible that a bit cell is "faded" and ends up becoming two samples instead of one. How are you converting flux to FM?  I have a really easy method for MFM (and GCR), but since I have never worked on FM until the SuperCard Pro project I have very little experience with it.  For MFM you just contruct a bit stream based on the conversion time for each bit ie.: 4us = 10, 6us=100, 8us=1000.  So 4us/4us/4us/4us = 10101010 = 0x55.  I doubt my FM conversion is correct with the analyzer because I am not use to looking at that type of data and I just guessed at what should be appropriate for decoding.  I would like to come up with a routine that built the bit stream like I do for MFM and GCR.  That would give you a way to look at the image files as FM and decoded data (like we can for MFM/GCR) using SCP's analyzer.  You might try it to see if it's junk or similar to what you are seeing.  Please let me know, this is one area I would like to improve for sure!

 

One thing to keep in mind, IBM 3740 protocol allows for a non-standard number of bits in the sync comparator.  This is the same as the WD1772 and certain combinations of extra bits will cause the decoder to do different (and predictable) things.  Since the Atari 400/800 didn't have access to the FDC itself, I doubt this is the case here but you never know... it could be part of a copy protection routine where maybe there are two sectors in a row with the same number and they wait for the first bad sector and immediately fetch the next one which is good.  I am not sure what tricks the Atari 400/800 had for protection, but I would assume they are similar to all other FDC constrained systems.


Edited by JimDrew, Sun Dec 7, 2014 9:39 PM.





0 user(s) are browsing this forum

0 members, 0 guests, 0 anonymous users