a8rawconv, a new raw disk conversion utility

[Savetz]

17 minutes ago, telengard said:

I'm trying to image some working Atari disks using a8rawconv and an SCP.  Some image and convert (with lots of warnings about missing sectors, etc), to .atx and work in Altirra.  Some have similar errors and don't even boot, even though they do on my Atari w/ a 1050.  I'm not sure exactly how to interpreter the output as to what is bad or not.   I've attached the output of one of the non-working imaged disks.  Any insight is greatly appreciated.

 

Oh, and my drive is a Panasonic 5 1/4 JU-475-4EAF with flippy mod.

 

thanks

errors.txt 4.62 kB · 1 download

 

I am not the expert at this, but I have SCPd hundreds of disks and have a system that does pretty well.

 

This little script does the SCPing, converts it to ATX, then runs a8diskutil's sector-summary function to give an easier-to-understand picture of how the disk reading went.

 

$ cat ~/scripts/flux 
#!/bin/sh

a8rawconv -p 104 scp0:96tpi:/dev/cu.usbserial-SCP_JIM $1.scp
mkdir -p working
a8rawconv $1.scp working/$1.atx
a8diskutil sector-summary working/$1.atx

 

Usage: flux Synapse_Synfile

 

You can get a8diskutil here:

http://www.a8preservation.com/downloads/a8diskutil-darwin-amd64-0.8.6.zip
http://www.a8preservation.com/downloads/a8diskutil-win-amd64-0.8.6.zip
http://www.a8preservation.com/downloads/a8diskutil-win-i386-0.8.6.zip
http://www.a8preservation.com/downloads/a8diskutil-linux-amd64-0.8.6.zip

 

[phaeron]

1 hour ago, telengard said:

I'm trying to image some working Atari disks using a8rawconv and an SCP.  Some image and convert (with lots of warnings about missing sectors, etc), to .atx and work in Altirra.  Some have similar errors and don't even boot, even though they do on my Atari w/ a 1050.  I'm not sure exactly how to interpreter the output as to what is bad or not.   I've attached the output of one of the non-working imaged disks.  Any insight is greatly appreciated.

 

Oh, and my drive is a Panasonic 5 1/4 JU-475-4EAF with flippy mod.

 

thanks

errors.txt 4.62 kB · 2 downloads

This is a really poor read -- unless you're copying a copy protected disk you should not be getting all those warnings about sectors that didn't read cleanly, especially if one of the passes did get a good sector. This means that the disk drive is barely reading the disk. A test you should try is to find a spare disk, format it in Atari DOS, and then read it back. You should get an absolutely clean read off a freshly written disk like that. If not, then something is wonky with the setup and needs to be adjusted.

 

[_The Doctor__]

make sure the heads and disk are clean also..

[telengard]

On 10/6/2021 at 11:11 PM, phaeron said:

This is a really poor read -- unless you're copying a copy protected disk you should not be getting all those warnings about sectors that didn't read cleanly, especially if one of the passes did get a good sector. This means that the disk drive is barely reading the disk. A test you should try is to find a spare disk, format it in Atari DOS, and then read it back. You should get an absolutely clean read off a freshly written disk like that. If not, then something is wonky with the setup and needs to be adjusted.

 

I did this and it was absolutely clean as you mentioned:

0 missing sectors, 0 phantom sectors, 0 sectors with errors

Glad to know my rig is OK.  I'm surprised all of these disks I'm using are seemingly OK and are so hard to image.  Guess there are some errors lurking in there.  

Is there a way to force the program to continue trying to ready dodgy/weak sectors, kinda brute force?

 

thanks everyone for all of the great information/suggestions!

[Savetz]

3 minutes ago, telengard said:

I did this and it was absolutely clean as you mentioned:

0 missing sectors, 0 phantom sectors, 0 sectors with errors

Glad to know my rig is OK.  I'm surprised all of these disks I'm using are seemingly OK and are so hard to image.  Guess there are some errors lurking in there.  

Is there a way to force the program to continue trying to ready dodgy/weak sectors, kinda brute force?

 

thanks everyone for all of the great information/suggestions!

 

The software can't do that. Wish it could. (AppleSauce FDC can do that, and it's a great feature.)

 

But your next step is to clean the disk. Carefully, carefully open the disk sleeve (either by lifting the flaps or carefully, carefully, cutting the back edge of the sleeve), remove the media -- making sure you know which side is up -- wash it under warm water, dry with a lint-free cloth, insert into a clean donor sleeve, and then try reading it again with the software.

 

-K

[mrtwo]

I am apparently very late to the flux party but I would like to say thanks for everyone who has posted great info regarding this.  I currently have a greaseweazle v4 which I purchased to attempt to create floppy disks for different vintage systems.  I have created SCP files of a8 disks but I have yet to write one that boots.  It is good to see the author has included support for Supercard Pro in a8rawconv and I was wondering if it might be possible to do the same with greaseweazle?

 

I have been getting tips on how to create working Atari floppy disks using my Tandon TM65-2L drive which is a 360K pc drive.  I am going to try working with a8conv as well as increasing my revs per track which is currently set to 3.

 

Any advice on this would be greatly appreciated!

 

Anybody know if there is a central repository for settings people have used successfully to create diskettes from SCP files for different systems like Apple II, Atari 8 bit, C64, etc?  Thanks!

Edited October 25, 2021 by mrtwo

[+Peter Rabitt]

Good Day!!!

I have both the SCP and Greaseweazel flux copiers and as I have not use them (the scr for about a year or more and never used the gw.) was wondering which would be the best..

The problem I have is that I found my flashback back ups of the old BBS I ran back in the 80's and was going to restore them...Well best plains of mice and men, the first one gave a DIR and tried to read the disk but failed about 1/2 way... the sec,third etc do not ever have DIR...

I was hoping the the flux image could read the low level, that the disk drive could not, and maybe recover the files...

Any ideas!!

 

Peter

 

[BillC]

5 hours ago, Peter Rabitt said:

Good Day!!!

I have both the SCP and Greaseweazel flux copiers and as I have not use them (the scr for about a year or more and never used the gw.) was wondering which would be the best..

The problem I have is that I found my flashback back ups of the old BBS I ran back in the 80's and was going to restore them...Well best plains of mice and men, the first one gave a DIR and tried to read the disk but failed about 1/2 way... the sec,third etc do not ever have DIR...

I was hoping the the flux image could read the low level, that the disk drive could not, and maybe recover the files...

Any ideas!!

 

Peter

A flux backup of the disks MAY be successful, this depends on the condition of the disks in question, as you say you have used the SCP you should have a compatible PC type 5-1/4" drive.

 

The disks may be good with the Atari drive having issues, as the disks are old they may have shed some of the oxide coating which resulted in read failures due to dirty heads.

 

The first step would be to visually examine the surface of the disks looking for obvious defects or contamination, rotating the disk in the jacket to check the entire surface, any contamination should be removed, single sided disks only have DATA on the bottom side. Thorough cleaning of the heads with high purity(>95%) isopropyl alcohol would be the next step.

 

 

 

 

 

[+Peter Rabitt]

Thanks for the info. Just in the last day a very important problem came up and all my time is on that but will get back soooooon...

Peter

[keirf]

I have a user report that a8rawconv won't work with KF streams produced by Greaseweazle:

Unable to match index mark against stream position

 

This is probably because Greaseweazle index-cues its output, and places an index mark at the very start of the stream (literally stream position = sample counter = index counter = 0).

 

But a8rawconv uses start of stream as a sentinel to detect when it is unable to place an index mark. Could a8rawconv be fixed to check for and allow an index mark at start of stream?

 

For now the workaround is to convert to scp and pass that to a8rawconv.

 

I can provide an example KF stream if need be, but should be easy to generate using the gw command line tool (gw convert foo.scp foo_track00.0.raw for example).

Edited July 29, 2022 by keirf

[BillC]

9 hours ago, keirf said:

I have a user report that a8rawconv won't work with KF streams produced by Greaseweazle:

Unable to match index mark against stream position

 

This is probably because Greaseweazle index-cues its output, and places an index mark at the very start of the stream (literally stream position = sample counter = index counter = 0).

 

But a8rawconv uses start of stream as a sentinel to detect when it is unable to place an index mark. Could a8rawconv be fixed to check for and allow an index mark at start of stream?

 

For now the workaround is to convert to scp and pass that to a8rawconv.

 

I can provide an example KF stream if need be, but should be easy to generate using the gw command line tool (gw convert foo.scp foo_track00.0.raw for example).

I reported the issue to you via PM in the Greaseweazle group on Facebook, the lack of timing information affects other software than a8rawconv. As a Greaeweazle user I do dump to SCP, but the current version of a8rawconv needs the track header data table edited to extract dumps of head1/sideB. No modification of KF created flux dumps is needed for decoding either side.

 

Since a8rawconv is able to decode ATR/ATX images from KF created raw dumps, but not those created by a GW, this leads me to believe the lack of timing information in the GW dumps is the issue.

 

This doesn't affect only a8rawconv, there are utilities for C64 images(likely for other formats as well) that also require this information.

One of the utilities used by the C64 user I was working with, who discovered this timing information issue, to create a G64 uses can only access KF .raw dumps, SCP dumps need to be converted to KF .raw. Can be overcome but adds an extra step to creating a G64 image.

 

There are workarounds in both cases, which lowers the priority for the issue, but still thought you should be made aware of it.

 

Have you confirmed the GW conversion from SCP to KF includes the timing information missing when dumped as KF from disk?

If it can produce a complete KF dump when converting from SCP why not when reading from disk?

 

Edited July 29, 2022 by BillC

[keirf]

On 7/29/2022 at 5:40 PM, BillC said:

I reported the issue to you via PM in the Greaseweazle group on Facebook, the lack of timing information affects other software than a8rawconv. As a Greaeweazle user I do dump to SCP, but the current version of a8rawconv needs the track header data table edited to extract dumps of head1/sideB. No modification of KF created flux dumps is needed for decoding either side.

 

Since a8rawconv is able to decode ATR/ATX images from KF created raw dumps, but not those created by a GW, this leads me to believe the lack of timing information in the GW dumps is the issue.

 

This doesn't affect only a8rawconv, there are utilities for C64 images(likely for other formats as well) that also require this information.

One of the utilities used by the C64 user I was working with, who discovered this timing information issue, to create a G64 uses can only access KF .raw dumps, SCP dumps need to be converted to KF .raw. Can be overcome but adds an extra step to creating a G64 image.

 

There are workarounds in both cases, which lowers the priority for the issue, but still thought you should be made aware of it.

 

Have you confirmed the GW conversion from SCP to KF includes the timing information missing when dumped as KF from disk?

If it can produce a complete KF dump when converting from SCP why not when reading from disk?

 

I have just released a new version of Greaseweazle tools which includes timing information in Kryoflux stream files.

 

However I doubt this will fix the a8rawconv issue, which I have already diagnosed as a different root cause, that being an edge condition which a8rawconv should handle.

[J D Potter]

I'm trying to use a8rawconv (very nice software, IMHO) to help me resurrect some source code of old projects of mine (APACLOAD, Maze of Agdagon, Crab Nebula) that I retained on 5.25" floppies. Specifically I'd had an XF-551, and created these in DSDD. But now all I have is a 1050 (that's been converted to SSDD), and an old stock Mitsumi D503 drive (without controller). I found this software to scan and save floppies as SCP using mostly parts I had lying around: Drawbridge, also the GitHub page (I've had to make some source code changes to accept Atari 8-bit floppies). I've then used a8rawconv to convert them to ATR files for my use. This has worked fine for 40-track / 90 kB SSSD or 180 kB SSDD disks, either one-sided, or "flippy", but my D503 head appears to only work on the bottom surface, not the top surface. So, after studying a8rawconv, I found I could "flip" my DSDD disk and read the top surface with my D503's working head, then use a8rawconv's "-r" option to create an SCP file that's been reversed. So I wrote more code (within the Drawbridge solution) that interleaves the tracks correctly (as I understood it). Not quite correctly...

 

The remaining problem is that a8rawconv warns me that the first sectors (on track 0.1) report it is seeing track 0, when the program's expecting to find track 4. And all tracks on the "top" surface after 0 report a track number mismatch, always by four. I seem to remember reading about this peculiarity somewhere here on the forums; can anyone clue me in on what Atari DSDD disks look like, and where I've gone wrong? Does the encoding method change between FM and MFM on the top surface?

 

I've hand edited the headers in the SCP "track header offset table" moving up the real track 4 to be where track 0 would have been (and all tracks through 39, for the top surface) and re-decode this SCP with a8rawconv. That allows much more of the tracks to be found, but 72 sectors (or four tracks) from the top surface never show up anywhere (and tracks 36–39, since they've been "moved up" end up intentionally blanked in the resulting ATR file). These missing tracks 0–3 are the continuation files (sectors $2D1–$318) that are from the top surface, when I view it with a program that reveals all the sector data.

 

Thanks in advance...

 

Jeff Potter

Your 8-bit friend from at least 1980

Also a big fan of Altirra and Virtualdub

[BillC]

1 hour ago, J D Potter said:

The remaining problem is that a8rawconv warns me that the first sectors (on track 0.1) report it is seeing track 0, when the program's expecting to find track 4. And all tracks on the "top" surface after 0 report a track number mismatch, always by four. I seem to remember reading about this peculiarity somewhere here on the forums; can anyone clue me in on what Atari DSDD disks look like, and where I've gone wrong? Does the encoding method change between FM and MFM on the top surface?

The heads on a DS drive are not lined up vertically, head 0 on the bottom is 1/12"(4 tracks for 48TPI, 8 tracks for 96TPI) offset from head 1 on the top. This means that when you flip the disk all tracks are 1/12" from where the drive expects them to be.

 

I have noticed on one of my Mitsumi D503 that if the metal piece on the upper head is missing that there is a gap between the heads when the latch is closed, this may be your issue.

 

 

[J D Potter]

BillC:

 

Indeed, my D503 appears to have a "naked" head on top (it might have always had that since I received it; it's so long ago I forget). What does the metal piece look like, and where does it fit?

 

When I stick a plastic ruler on the other side of the two heads, no floppy inserted, but the latch is closed, I can see at least 2mm of space between the heads.

 

So how do I continue?

[BillC]

7 hours ago, J D Potter said:

Indeed, my D503 appears to have a "naked" head on top (it might have always had that since I received it; it's so long ago I forget). What does the metal piece look like, and where does it fit?

It's a U shaped piece of metal that fits on top of the head, the cut-out piece making it a U is to prevent interference with the bar over the head when the latch is open.

The top R/W head is on mounted on a very thin flexible foil and there is a pad on the underside of the U, this pad pushes the head into contact with the disk when the latch is closed

Edited October 25, 2022 by BillC

[ijor]

On 10/24/2022 at 3:12 PM, J D Potter said:

I'm trying to use a8rawconv (very nice software, IMHO) to help me resurrect some source code of old projects of mine (APACLOAD, Maze of Agdagon, Crab Nebula) that I retained on 5.25" floppies. Specifically I'd had an XF-551, and created these in DSDD. But now all I have is a 1050 (that's been converted to SSDD), and an old stock Mitsumi D503 drive (without controller). I found this software to scan and save floppies as SCP using mostly parts I had lying around: Drawbridge, also the GitHub page (I've had to make some source code changes to accept Atari 8-bit floppies). I've then used a8rawconv to convert them to ATR files for my use. This has worked fine for 40-track / 90 kB SSSD or 180 kB SSDD disks, either one-sided, or "flippy", but my D503 head appears to only work on the bottom surface, not the top surface. So, after studying a8rawconv, I found I could "flip" my DSDD disk and read the top surface with my D503's working head, then use a8rawconv's "-r" option to create an SCP file that's been reversed. So I wrote more code (within the Drawbridge solution) that interleaves the tracks correctly (as I understood it). Not quite correctly...

 

In first place let's agree on the terminology. While it is true that the first head is the bottom one, and the first recorded side is the back one; it is more common to follow the disk labels and refer to the front side as the one that matches the label on top of the disk, which is actually recorded on the back surface. To avoid ambiguity I will try to use first and second sides, instead of bottom, top, front or back.

 

If I understand correctly, you are trying to read the second side of a double sided disk with the first head, by flipping the disk. This the opposite of the more common operation, which is to read the flippy side, without flipping, with the second head.

 

As @Billc described, the two heads are not exactly aligned one with the other. And as you probably read already, when reading a flippy side with the second head you must read 4 tracks (at 48 TPI) outwards from the normal track 0 position. But here, again, you are doing the opposite operation. I never tried that, but I think you need to perform the opposite than when reading a flippy side. You need to read 4 inner tracks beyond track 39 (the last track position counting from 0), track 40-43. So you should use the proper parameters with whatever tool you are using to perform the reading. But I'm not sure that every drive can read 4 extra inner tracks. Encoding is always MFM on both sides of any Atari DSDD disk.

 

Then, if you succeed reading the second side correctly, you would need somehow to merge it with the front side. Not sure there is any tool that would be able to decode both sides together in this case. You need to merge both sides correctly to produce a valid ATR logical image.

 

If you are in doubt, you might post the the original, unmodified, SCP image and we might take a look.

[BillC]

4 hours ago, ijor said:

Encoding is always MFM on both sides of any Atari DSDD disk.

This is true since enhanced and double density are MFM, but users should also be aware that single density is FM.

It doesn't seem to matter with flux controllers though, the same settings work for all 3.

Edited October 26, 2022 by BillC

[J D Potter]

ijor: When I inserted the floppy into the sleeve intentionally flipped, using (Rob Smith's) ArduinoFloppyDiskReader software, it was under the impression that I was reading the bottom surface with the first head (Head 0 in the software) and didn't index four tracks into it. Then using a8rawconv with the -r command option, it reversed the direction of all 40 tracks.

 

I wrote the changes to ArduinoFloppyDiskReader  myself, where track-by-track it interleaves bottom/top surface tracks in the SCP file. I'll release those changes to be open source, if I ever get it working

 

So, a8rawconv goes to where it thought the first track is located, it finds a track identifying itself at "0" (I could trace a8rawconv in debug mode, and sure enough it finds the "00" in the track field) it complains the "04" should be there (and every track after that has the same issue, being too low by four.

 

So am I hearing that the "real" lowest number track should be "04" and the highest one (instead of being 39 decimal) should read 43 (decimal)?

 

I'm also using the "HxCFloppyEmulator v2.5.6.6" to examine the SCP contents. My outputs for tracks 04 to 39 (decimal) look "good" to it, and tracks 0 to 3 look unintelligible. I compare this behavior to SSSD SCP files I've created with my D503 have all shown to be "good" for all tracks based on that software.

 

I'll be trying to move my upper head on the D503 closer to the floppy disk media, and see if that gets me closer to correct operation (rather than flipping disks and using my interleave scheme).

 

Thanks for all of the advice.

[ijor]

28 minutes ago, J D Potter said:

ijor: When I inserted the floppy into the sleeve intentionally flipped, using (Rob Smith's) ArduinoFloppyDiskReader software, it was under the impression that I was reading the bottom surface with the first head (Head 0 in the software) and didn't index four tracks into it. Then using a8rawconv with the -r command option, it reversed the direction of all 40 tracks.

...

I'm also using the "HxCFloppyEmulator v2.5.6.6" to examine the SCP contents. My outputs for tracks 04 to 39 (decimal) look "good" to it, and tracks 0 to 3 look unintelligible.

 

As I was saying, you need to read 4 tracks inwards. When you flip a DS disk, and because of the displacement between both heads, the track 0 of the disk ends at the head position for track 4.

 

If you position the head at any track < 4, it will attempt to read outwards beyond track 0 of your disk. That's why you get garbage on your first 4 tracks. You need to tell the floppy reader software to read tracks 4-43, not tracks 0-39. But again, not sure your drive would be able to read tracks that far inwards.

 

Edited October 26, 2022 by ijor

[J D Potter]

All right, I've got something that works 98% of the time (2 to 12 bad sectors on the top side 720 sectors). For one, my Mitsumi D503, on startup, bangs the read/write head onto the track 0 stop. Encoding immediately after that ends up with variations in disk RPM that always fail to create a good image (be it natural bottom side, or when I flip the disk inside the sleeve, so the index hole can be detected for the top side). I can't start at track 4 and make it to track 43 without removing a piece in the drive that stops the rotary-based stepper after track 39. And despite my best efforts, it seems that the top head is unresponsive, when tested by IMD, an old DOS program that seems to work well for 3.5 inch drives I've tested.

 

I create the bottom surface data from the unmodified disks, and create the top surface data with disks flipped within the original sleeve. The best results occur when I start by manually parking the head at track 0 (furthest from center) before applying power. For the top surface, I ask it to proceed (inwards) to track 4, and start copying tracks until it reaches track 43. That usually ends with a loud "clack" as the head carriage hits the far (closest to center) mounting screw for the rails that the head rides on. I can check the disk RPM with the program HxCFloppy Emulator; all seemed to be normal. Then I use a8rawconv's ability to do a read of my (top) SCP with the "-r" flag to reverse the track numbers and data within each track, creating a new SCP file with only top surface data.

 

I modified the ArduinoFloppyDiskReader software to perform the interleaving of the top and bottom tracks. There is a manual step I do with a Hex editor (Hexplorer, for Windows) where I take the forty track pointers (04 to 43) and move them up to the slots reserved for 00 to 39 within the SCP header file; it takes one minute. Then my "SCP Merge" routine (I learned by studying the routines in Rob Smith's excellent program) stitches the bottom and top head information. Then I run a8rawconv to finally convert the merged SCP file into an ATR file.

 

This tests out pretty good with my two samples. The bad sectors always occur on the last track (43) on the top surface, which I'm almost sure is due to the ramming of the head into the mount near the center of the drive. Since the Atari 8-bit XF-551 always ran sectors 1-$2D0 on the bottom (outside to inside of disk) then $2D1-$5A0 on the top (inside to outside), the bad sectors in the top surface are always from $2D1-$2E2; these are the 18 sectors within track 43 (i.e., just past the center). It's a hacked-together system, but it's got some of my old source mode intact. Thanks for the help!

[ijor]

21 hours ago, J D Potter said:

All right, I've got something that works 98% of the time (2 to 12 bad sectors on the top side 720 sectors).

I'm glad you could recover most of the content. To be honest, it worked out better than I expected.

 

If the missing data is very important, you probably can get help from somebody with a fully working double sided drive . In such case you may want to mention where you are located.

[telengard]

I'm trying to write Apple ][ .dsk files to physical floppy.  I had success with this and my .atx files, and on the SCP forums it seems this is the way to go for A2 as well.

 

Not having much luck, so I want to ensure I'm invoking the program correctly:

 

a8rawconv.exe -if do mecc_computer_inspector.dsk scp0:96tpi

 

I'm using tested, good media (and have tried other disks as well) and am using this with an SCP and a 96 tpi 5.25 drive.  The process writes out 35 [0-34] tracks, and erases 5 [35-39].  However, none of the disks I've tried boot.

I've also tried using HxC to convert the .dsk to an .scp and writing that, same results.

 

Hopefully, I'm just missing a setting, any help appreciated.

 

[telengard]

Answering my own question as I've been tinkering a bit.  I found a post on the HxC forums that said the HxC tool recognizes Apple DOS disks correctly if they are named w/ a .do extension.  So, I renamed my .dsk, exported it as an scp using the HxC tool and was able to write that using a8rawconv!

 

Glad to have it working, be curious to know if there is a way to go straight from .dsk to a floppy.

[phaeron]

7 hours ago, telengard said:

Answering my own question as I've been tinkering a bit.  I found a post on the HxC forums that said the HxC tool recognizes Apple DOS disks correctly if they are named w/ a .do extension.  So, I renamed my .dsk, exported it as an scp using the HxC tool and was able to write that using a8rawconv!

 

Glad to have it working, be curious to know if there is a way to go straight from .dsk to a floppy.

Hey, sorry for the delay in answering, glad that you got it working in the meantime. There isn't supposed to be a difference between using -if do and .dsk file, and renaming that .dsk file to .do. In both cases, they should interpret the file with Apple II DOS 3.3 ordering.

 

If you still have a case where this is an issue, I'd like to see diagnostics on what's going on, such as whether the program reports it is reading a different format than "Apple II disk image (DOS 3.3 ordering)" or if a8rawconv can't read back its own result in both cases. I checked the current version with output to an .scp file and couldn't see a difference in the output, aside from the timestamp at the end of the file.