Atari 8-bit Software Preservation Initiative and ATX Format Anomalies

[jamm]

As part of the work done to add ATX support to FujiNet, I wrote a little ATX parser/analyzer program. Source for this is available here:
https://github.com/kaelef/Atari-ATX-Parser

 

I'm providing some notes below based on what I found in the ATX files included in the January 25, 2020 version of the Atari 8bit Preserved Software archive.

All my notes are based on the documentation provided at
http://www.a8preservation.com/#/guides/atx

which I assume is the most accurate source of documentation available on the ATX format.

 

Also, can someone provide a list of what programs are responsible for each of the "creator" codes found in the ATX headers?  Here's the list I have, including both what's documented and not, based on results of files in the archive:

 

$01 TX_CR_FX7
$02 ATX_CR_FX8
$03 ATX_CR_ATR
$10 ATX_CR_WH2PC
$74 UNKNOWN

 

There are several cases where the UNKNOWN_SKEW (0x0100) flag is set on tracks. What does this really indicate (i.e. how should that be handled, if at all)?  Ignoring it seems to have no negative repercussions.

 

ATX Header End Field and Host Records
Creator 0x10 (ATX_CR_WH2PC) adds a HOST record at the end of every file. The contents of this record are undefined in the documentation. The size of the record (8 record header bytes + 40 bytes of data) is not included in the ATX header "end" value. Whether or not this is a bug is debatable since the "end" value isn't clearly defined, but it seems to me like it should be included. In all other cases, the ATX header "end" value is equal to the size of the ATX file itself, so it serves as a nice verification that the whole file is present.

 

ATX Header End Field Incorrect
There are a couple of instances of images with creator ID 0x01 (ATX_CR_FX8) where the ATX 'end' field off by 8 bytes. In all other cases, the 'end' value correctly matches the size of the entire image file. Examples:

 

• Colonial Conquest v1.1 (1985)(SSI)(US).atx (8 bytes short)
• Metric and Problem Solving v1.2 (1982)(APX)(US)[f][BASIC].atx (8 bytes over)

 

Action Classics - Volume I (1987)(Activision)(US)[!].atx
This is not an ATX file - it has an ATR header.

 

Bad Angular Position Values
There are several instances of images with creator ID 0x01 (ATX_CR_FX8) where the angular position of a sector is greater than the expected maximum of 26041. Interestingly, with only a couple of exceptions (see below), this only happens 17 sectors into the table. Examples:

 

• Adventure on a Boat! (1981)(subLOGIC)(US)[f][BASIC].atx
  • Tracks 13, 27; sector index 17
• Blue Max - 2001 (1984)(Synapse Software)(US)[f].atx
  • Tracks 10, 13; sector index 17
• Drelbs (1983)(Synapse Software)(US)[f][OS-B].atx
  • Track 2, 5, 6, 10, 14, 17, 22, 23, 26, 31; sector index 17
• Laser Hawk (1986)(Red Rat Software)(GB)[f].atx
  • Tracks 1, 11, 13, 37; sector index 17
• Pac-Man (1984)(Datasoft)(US)[f][!].atx
  • Tracks 2, 7; sector index 17
  • Tracks 5, 10; sector index 18
• Print Shop Graphics Library 1 (1985)(Broderbund Software)(US)[f][!].atx
  • Tracks 0, 4, 10, 11, 13, 16, 22, 26, 28, 32; sector index 17
• S.A.G.A. #06 - Strange Odyssey v5.0-119 (1982)(Adventure International)(US)(Side B)[!].atx
  • Track 0; sector index 15
  • Track 7; sector index 17
• Spaceport (1983)(Umbrella Software)(CA)[f].atx
  • Tracks 1, 4, 12, 15, 18, 24, 27, 30; sector index 17
• Spy vs. Spy Vol. II - The Island Caper (1985)(Databyte)(PAL)(GB)[XL-OS].atx    
  • Tracks 1; sector index 17
• Zeppelin (1983)(Synapse Software)(US)[f][810 only].atx
  • Tracks 0, 1, 6, 15, 20, 21, 23, 30; sector index 17

 

Undocumented Sector Flag
There are several instances of images with creator ID 0x01 (ATX_CR_FX8) where sectors are marked with bit #2 (0x02). The function of this flag is not documented, and it doesn't get set by any other creators. This only seems to happen in cases where the CRC_ERROR (0x08) and LOSTDATA_ERROR (0x04) flags are also set. Examples:

 

• Boulder Dash II (1985)(Databyte)(PAL)(GB)[XL-OS].atx
  • Track #37: Sectors that also set the CRC_ERROR and LOSTDATA_ERROR flags
• Mr. Robot and His Robot Factory (1986)(Databyte)(PAL)(GB)[!][XL-OS].atx
  • Track #37: Sectors that also set the CRC_ERROR and LOSTDATA_ERROR flags
• Spy vs. Spy Vol. II - The Island Caper (1985)(Databyte)(PAL)(GB)[XL-OS].atx    
  • Track #37: Sectors that also set the CRC_ERROR and LOSTDATA_ERROR flags

 

Random Angular Position Values
There are several instances of images with creator ID 0x10 (ATX_CR_WH2PC) where the angular position for most of the sectors seems completely random and often greater than the expected maximum of 26041. Throwing the values out and replacing them with '0' allows the game to load. Examples:

 

• Crisis Mountain (1982)(Synergistic Software)(US).atx
• Galactic Chase (1981)(Spectrum Computers)(US).atx
• Lords of Karma (1981)(Avalon Hill)(US).atx

 

Invalid Sector Numbers

There are a couple of instances of images with creator ID 0x01 (ATX_CR_FX8) where sectors have an invalid sector number.  These are marked as single density disk images, however, several tracks include sector numbers greater than 18. Since the invalid sector numbers only go up to 26, it seems like the individual track is MEDIUM/ENHANCED density, although it doesn't have the expected MFM flag (0x02) if that was the intention. Examples:
 

• Max + Magic World (1997)(Sikor Soft)(PL)(Side B)(Magic Word).atx
  • Track 37: Sector numbers between 19 and 26
  • Track 38: Sector numbers between 19 and 26
• Moscow 1993 (1994)(Sikor Soft)(PL)[!].atx
  • Most tracks have sector numbers greater than 18. Maybe this one simply has the wrong density header value and should be marked as "ENHANCED DENSITY". I tried changing just the density header value, but it still wouldn't load in Altirra.

 

 

Edited September 7, 2020 by jamm

[ijor]

2 hours ago, jamm said:

As part of the work done to add ATX support to FujiNet, I wrote a little ATX parser/analyzer program...

 

Most of this is my fault. And most of it, for historical reasons. Take in mind that the earlier ATX images and tools were created many years ago. Among other things, this was years before the advent of the Kryoflux and the SCP. And it was done mostly for personal backup, not for actual preservation purposes at that time.

 

2 hours ago, jamm said:

Also, can someone provide a list of what programs are responsible for each of the "creator" codes found in the ATX headers? 

We need to create a new "official" database of creators. The only ones I know so far besides my own tools are Phaerons's ones. As you might have read on some other threads, we are adding some new extensions to the specs, so that would be a good opportunity to address a full list of creators as well. Btw, please keep in mind that the creator field is 16 bits.

 

2 hours ago, jamm said:

ATX Header End Field and Host Records
Creator 0x10 (ATX_CR_WH2PC) adds a HOST record at the end of every file.

 

I used this record for remote debugging purposes when people used the legacy Vapi imaging tools on a Happy or similar enhancement. Ignore this record. In general, you should ignore any record you don't know.

 

2 hours ago, jamm said:

There are a couple of instances of images with creator ID 0x01 (ATX_CR_FX8) where the ATX 'end' field off by 8 bytes.

That seems to be a bug on some versions of the tools.

 

2 hours ago, jamm said:

Bad Angular Position Values

I'll need to check that.

 

2 hours ago, jamm said:

There are several instances of images with creator ID 0x01 (ATX_CR_FX8) where sectors are marked with bit #2 (0x02).

 

This is the DRQ bit of the FDC status register. Most bits on the sector header status are the corresponding ones on the FDC status register. Some tools set this bit for "big sectors" together with the DATA LOST bit, because some Atari drives will have this bit set on the status register when reading such a sector. The actual behavior depends on the exact drive and firmware.

 

2 hours ago, jamm said:

There are a couple of instances of images with creator ID 0x01 (ATX_CR_FX8) where sectors have an invalid sector number. 

Some tools encode sectors with invalid sector number if they are present on the disk. You should normally ignore them.

 

2 hours ago, jamm said:

Max + Magic World (1997)(Sikor Soft)(PL)(Side B)(Magic Word).atx

• Track 37: Sector numbers between 19 and 26
• Track 38: Sector numbers between 19 and 26

Those tracks are FM (single density) and the sectors you see are still present on those tracks.

 

2 hours ago, jamm said:

Moscow 1993 (1994)(Sikor Soft)(PL)[!].atx

• Most tracks have sector numbers greater than 18. Maybe this one simply has the wrong density header value and should be marked as "ENHANCED DENSITY". I tried changing just the density header value, but it still wouldn't load in Altirra.

 

This is an enhanced density disk that for some reason was included in the torrent missing the MFM flag. It does work for me if I set the flag, at least it loads. Note that what it matters is the MFM flag at the track header.

 

 

Edited September 7, 2020 by ijor

[jamm]

1 hour ago, ijor said:

 

Most of this is my fault. And most of it, for historical reasons. Take in mind that the earlier ATX images and tools were created many years ago. Among other things, this was years before the advent of the Kryoflux and the SCP. And it was done mostly for personal backup, not for actual preservation purposes at that time.

 

Thanks for the replies, @ijor - I assumed many of these were due to old/early versions of programs generating different output.  Since these ATX files are in a "preservation" archive, I thought it'd be useful to document cases where we might want to re-archive those particular entries if possible.

 

Quote

This is an enhanced density disk that for some reason was included in the torrent missing the MFM flag. It does work for me if I set the flag, at least it loads. Note that what it matters is the MFM flag at the track header.

I did check, and none of the tracks have the MFM flag set.

 

Out of curiosity, I went ahead and did a quick check of the ATR files in the archive, too.  Here are a couple of issues I found:

 

Napoleon in Russia (1988)(Datasoft)(US)[!].atr

This is actually an ATX file.

 

Silicon Dreams (1986)(Firebird)(GB)[!][req 64K].atr

The number of paragraphs is wrong in the header (it's recorded as 0x1680 but it should be 0x2080).  The similarly-named "Silicon Dreams (1986)(Rainbird)(GB)[!][req 64K].atr" is actually identical to this file, except it has the correct number of paragraphs in the header.

 

 

[Mclaneinc]

Thanks people, with such an initiative there's always going to be hiccups with incorrect files and copying anomalies but this new focus should rid / clarify many of the anomalous entries which is a major step towards a true collection.

 

Well done and thank you.

[Farb]

Thanks for pointing out the problems you found. A number had already been corrected in the database and I've fixed the others that I am able to. As @ijor said, some of them are ATX files that were made a long time ago.

 

14 hours ago, ijor said:

We need to create a new "official" database of creators. The only ones I know so far besides my own tools are Phaerons's ones.

$74 is the creator for my a8diskutils.

[ijor]

21 hours ago, jamm said:

Moscow 1993 (1994)(Sikor Soft)(PL)[!].atx

Quote

This is an enhanced density disk that for some reason was included in the torrent missing the MFM flag. It does work for me if I set the flag, at least it loads. Note that what it matters is the MFM flag at the track header.

I did check, and none of the tracks have the MFM flag set.

 

Yes, may be I didn't explain myself correctly. The tracks indeed don't have the MFM flag set, and that is a mistake. If you set the flag on all the tracks, then it loads. At least it does to me.

 

On 9/6/2020 at 9:44 PM, jamm said:

There are several cases where the UNKNOWN_SKEW (0x0100) flag is set on tracks. What does this really indicate (i.e. how should that be handled, if at all)?  Ignoring it seems to have no negative repercussions.

Forgot to answer this one. It indicates that the skew alignment (alignment between the tracks) is unknown. This could be for several reasons, such as because it was processed by hardware or software that can't measure the skew alignment. There is nothing you can do about that, it just an informational flag.

 

[jamm]

43 minutes ago, ijor said:

 

Yes, may be I didn't explain myself correctly. The tracks indeed don't have the MFM flag set, and that is a mistake. If you set the flag on all the tracks, then it loads. At least it does to me.

Oh, I see - I'll try that out, thanks.

 

Quote

Forgot to answer this one. It indicates that the skew alignment (alignment between the tracks) is unknown. This could be for several reasons, such as because it was processed by hardware or software that can't measure the skew alignment. There is nothing you can do about that, it just an informational flag.

Thanks for the clarification!