a8rawconv, a new raw disk conversion utility

[Savetz]

9 minutes ago, Farb said:

And, just to be certain, you ran this with the SuperCard Pro board plugged in via USB?

 

At least in prior versions of MacOS, a file would appear in the /dev directory when the USB cable was plugged in and disappear when it was unplugged. We need to figure out what device shows up. I'll check with my Mac notebook shortly and see if I see anything.

 

Yes, plugged into USB.  I even rebooted the Mac for good measure.

 

-K

 

 

[+DrVenkman]

54 minutes ago, Savetz said:

 

Yes, plugged into USB.  I even rebooted the Mac for good measure.

 

-K

Are you using a USB hub? If so, plug the board straight into the Mac. If you already are doing that, try a second USB port.

 

Another thing to try: with the Terminal window open, check the contents of the /Dev/ directory without the device plugged in. Then plug it in, refresh the window and compare the contents. If your USB controller in the Mac detects the device at all, it should create an entry for it under some name. 

[Farb]

I confirmed that the right device appears on my Mojave installation when I plug the card in.

 

It looks like the SCP card uses an FTDI chip. It is possible, as previously mentioned, that you need to install a driver but I had thought that MacOS came with an FTDI driver already installed. Take a look here:

 

https://www.ftdichip.com/Drivers/VCP.htm

 

Note: Make sure you unplug the SCP card before installing the driver.

[Savetz]

46 minutes ago, DrVenkman said:

Are you using a USB hub? If so, plug the board straight into the Mac. If you already are doing that, try a second USB port.

 

Another thing to try: with the Terminal window open, check the contents of the /Dev/ directory without the device plugged in. Then plug it in, refresh the window and compare the contents. If your USB controller in the Mac detects the device at all, it should create an entry for it under some name. 

 

No hub.

 

Oh! When it's plugged in, the following appear in /dev:

 

cu.usbserial-SCP_JIM
tty.usbserial-SCP_JIM

 

Note that based on the error message, a8rawconv seems to be looking for cu.usbserial-SCP-JIM with dash instead of underscore.

 

—K

 

 

[+DrVenkman]

49 minutes ago, Savetz said:

Note that based on the error message, a8rawconv seems to be looking for cu.usbserial-SCP-JIM with dash instead of underscore.

 

Okay, you're getting somewhere! If you have the know-how, you can modify the source code to replace the device name with the one actually created by your Mac when you plug it in and then recompile it. Alternately, you can use a Terminal command to create a symlink (essentially a Unix "shorcut" ) with a dummy name using a dash that points to the real device filename created when you plug in the device.

 

So in the /Dev/ directory, at the command prompt type:  

 ln -s cu.usbserial-SCP_JIM cu.usbserial-SCP-JIM

You may need to do this as a superuser (using the sudo prefix - been a few years since I did this on a Mac).

 

This should create a symbolic link (an alias or shortcut) that points to the correct device name. That way when the software is trying to access the device by the name containing the dash, it will actually be redirected to the device name with the underscore. 

[phaeron]

Not sure if this was changed as the original versions I made don't specifically have Mac support, but you should be able to override the device path directly on the command line:

a8rawconv scp0:48tpi:/dev/cu.usbserial-SCP_JIM test.atx

 

[Savetz]

1 hour ago, phaeron said:

Not sure if this was changed as the original versions I made don't specifically have Mac support, but you should be able to override the device path directly on the command line:

a8rawconv scp0:48tpi:/dev/cu.usbserial-SCP_JIM test.atx

 

 

That seems to work!! (I didn't try the symlink thing, tried this first and it worked.) Thank you.

 

Now I seem to have a new problem. I've tried a couple of non-copy protected disks, and this always happens:

 

Reading track 37
Reading track 38
Reading track 39
Writing single density ATR file: backed_up_disk.atr
WARNING: Missing sectors not supported by ATR format. Writing out null data for track 1, sector 1.
WARNING: Missing sectors not supported by ATR format. Writing out null data for track 1, sector 2.
WARNING: Missing sectors not supported by ATR format. Writing out null data for track 1, sector 3.
WARNING: Missing sectors not supported by ATR format. Writing out null data for track 1, sector 4.
 

...all the way through...

 

WARNING: Missing sectors not supported by ATR format. Writing out null data for track 39, sector 17.
WARNING: Missing sectors not supported by ATR format. Writing out null data for track 39, sector 18.
 

The resulting ATR files are 92,176 bytes but cause an unrecoverable error in my emulator, Atari800MacX.

 

-Kevin

[Farb]

Is your drive a 48tpi or 96tpi drive?
 

The example phaeron gave is for 48tpi drives.

[tschak909]

Yes, more specifically, what model drive are you using?

-Thom

[Savetz]

30 minutes ago, Farb said:

Is your drive a 48tpi or 96tpi drive?
 

The example phaeron gave is for 48tpi drives.

Ah! It's a flippy-modified PC drive.

 

Changing the command to

./a8rawconv -p 104 scp0:96tpi:/dev/cu.usbserial-SCP_JIM backed_up_disk2.atr

 

worked perfectly, and the resulting file boots in the emulator! Woo!

 

Thank you!

 

Now. The end goal here is to archive things for the software preservation project. I suppose I need to use a different command for flux-level images that will satisfy the needs of that project?

 

—K

[Savetz]

Also: can I use this setup to read Apple II disks? TRS-80? If so, how?

 

thanks. ya'll are great.

—K

[tschak909]

You can try to specify a destination file of type .scp to dump a supercard pro format disk. I can't remember if a8rawconv can write kryoflux format (which is _VERY_ unweildy!, there is a seperate file for each track, SCP has it all in one file) by specifying a directory/track00.raw 

 

For Atari disks, ATX as an output format is most useful.

 

You can try to archive other disks in the SCP format (it is a flux level format). If not, there are some CLI tools that are available to compile to get SCP format disks natively.

 

-Thom 

[phaeron]

KryoFlux format support is read-only, SCP format support is read/write.

 

It's best to archive in two-step by imaging to SCP first and then converting the SCP image to ATR/ATX. SCP is a raw flux dump of the disk as produced by the SCP hardware, while converting to ATR/ATX requires some interpretation of the flux data. Using a two-step process ensures that everything is captured in one pass on the disk so that no further reads are needed if something turns out to be busted or missing with the decoded image. This is particularly important with disks that are old and brittle or limited access.

 

a8rawconv can image 5.25" Apple II disks, though it is not well tested. It will overdump tracks when imaging to SCP -- 40 instead of 35 -- but this is harmless. You can then convert the image to DOS 3.3 sector order (DO/DSK) or raw nibble (NIB)  images for emulator usage. ProDOS order images are not currently supported. My limited experience is that the Apple II disk checksum algorithm is extremely flimsy and prone to missing errors, so it is very necessary to manually check decoded images.

 

You should be able to image a TRS-80 disk to SCP if it is 40 track, single sided. There is currently no decoded image support for this type of format.

 

[Savetz]

6 hours ago, phaeron said:

converting the SCP image to ATR/ATX

Great. How? Is there another tool for that?

 

-K

[phaeron]

14 minutes ago, Savetz said:

Great. How? Is there another tool for that?

Nope, you just give a8rawconv the SCP image instead of the SCP device:

 

a8rawconv image.scp disk.atx

 

[Fuji-Man]

Just got a scp, cannot open serial port. VCP installed in USB serial converter. COM13 shows in device manager using FTDI 2.12.18.0, running a8rawconv gives me the following error.

 

A8 raw disk conversion utility v0.92
Copyright (C) 2014-2016 Avery Lee, All Rights Reserved.
Licensed under GNU General Public License, version 2 or later.

Unable to open serial port: COM13

 

Any hints or directions greatly appreciated.

[phaeron]

4 hours ago, Fuji-Man said:

Just got a scp, cannot open serial port. VCP installed in USB serial converter. COM13 shows in device manager using FTDI 2.12.18.0, running a8rawconv gives me the following error.

 

A8 raw disk conversion utility v0.92
Copyright (C) 2014-2016 Avery Lee, All Rights Reserved.
Licensed under GNU General Public License, version 2 or later.

Unable to open serial port: COM13

 

Any hints or directions greatly appreciated.

You might need to override the device name to force a colon after it due to the high number, e.g: scp0:96tpi:com13:

[Fuji-Man]

On 4/18/2020 at 12:15 AM, phaeron said:

You might need to override the device name to force a colon after it due to the high number, e.g: scp0:96tpi:com13:

a8rawconv.exe scp0:96tpi:com13: test.atx gave the same error.

Wondering if I need to use a different drive or ???.

[phaeron]

37 minutes ago, Fuji-Man said:

a8rawconv.exe scp0:96tpi:com13: test.atx gave the same error.

Wondering if I need to use a different drive or ???.

Try this source name, then:

scp0:96tpi:\\.\COM13

 

[Fuji-Man]

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

 

C:\Users\rory\Downloads\a8rawconv-0.92>a8rawconv.exe scp0:96tpi:\\.\COM13 test.atx

A8 raw disk conversion utility v0.92
Copyright (C) 2014-2016 Avery Lee, All Rights Reserved.
Licensed under GNU General Public License, version 2 or later.

Unable to open serial port: \\.\COM13
C:\Users\rory\Downloads\a8rawconv-0.92>

[+DjayBee]

I have an image of a disk with a gone-bad sector. It is decoded as "weak" by a8rawconv with $4C stable bytes.

Hoping that only a few bytes have gone-bad, I would like to see which bytes have different data read.

 

Is it possible to dump the data for all five reads of the sector from a Kryoflux file?

[ijor]

1 hour ago, DjayBee said:

I have an image of a disk with a gone-bad sector. It is decoded as "weak" by a8rawconv with $4C stable bytes.

Hoping that only a few bytes have gone-bad, I would like to see which bytes have different data read.

 

Is it possible to dump the data for all five reads of the sector from a Kryoflux file?

 

Hi DjayBee,

 

It might be possible, but it probably won't be enough as typically even a few bad transitions is enough to corrupt all the rest of the sector.

 

Send me the raw dump (or a link to it). I might be able to recover.

[phaeron]

I finally got around to cleaning up some of the WIP code that I had for a8rawconv and implementing some missing support for different disk geometries than 40 track single-sided. Attached is a8rawconv 0.94. There's a lot of changes under the hood for the multiple disk geometry support, so there may be bugs lurking, but the intent is to try to do the right thing by default -- default Atari decoding uses 40 track single-sided, decoding to ADF or VFD selects 80 track double-sided, etc.

 

Changelog:

• Updated help file, now has a TOC.
• Windows 64-bit build included.
• Double-sided and 80-track formats are now supported (-g). Dumping up to 42 tracks at 48 TPI or 84 tracks at 96/135 TPI is supported (-g 84,2).
• New decoded format support: Atari XFD (read/write), Apple II 5.25" ProDOS order (read/write), Apple II / Mac 3.5" 400/800K DSK (write only), PC 160K-1.44M VFD/FLP (write only), Amiga 880K ADF (write only).
• Double-density ATR images and XF551-compatible double-sided encode/decode are now supported (-g 40,2).
• Sector interleave can be changed for encoding (-i). -i xf551-hs, for instance, will encode double-density disks with 9:1 interleave instead of 15:1, and -i none will use 1:1 interleave.
• Added support for high-density 2us FM / 1us MFM timing and high-density drive select (-H).
• 179X/279X MFM decoder now has relaxed sector size and side address field validation to better match FDC behavior.
• Address CRC errors are now encoded properly from ATX to raw flux.
• Added flux timing analyzer to display a histogram of flux transition spacing relative to expected (-analyze).
• ATX image handler supports enhanced density tracks and extended long sector size encodings, sectors outside of the normal range, and better compatibility for long sector error encodings. The creator ID is now set.
• SCP hardware handler now auto-handles COM ports above COM9, decodes hardware errors to readable messages, shuts off the drive on an error, and allows lowering revolutions for reading below 5 (-revs).
• When reading from an SCP directly, a8rawconv automatically switches to 8-bit flux timing encoding when the SCP hardware runs out of memory with the default 16-bit encoding. This can be required for unformatted tracks or for 5 revs on a 1.44M disk (200ms / 3us avg * 5 * 2 = ~660K > 512K).
• Add heuristics to try to detect the track data list layout of an SCP image to work around ambiguity in the file format spec. If this fails, the layout can be overridden with the -if switch, e.g. -if scp-ds40.
• SCP write format updated to latest 2.20 spec -- 84 tracks now supported, timestamp and footer added. Drive type is varied based on geometry to try to provide best compatibility depending on whether tracks are supposed to be physically spaced at 48 or 96/135 TPI.
• SCP images written from synthesized flux timings by encoding sector data are marked as normalized non-preservation.

a8rawconv-0.94.zip

[Farb]

Thank you for continued work on this, @phaeron!

 

It is so nice to see proper enhanced density ATX support!

[Farb]

FYI, it seems like the -r option is not working properly anymore in 0.94. It's showing Kryo B sides as blank.