Extending the ATR file format

[cyco130]

Hi,

 

Everytime I try to write an SIO2PC style program I find myself struggling to calculate a reasonable geometry information (number of sides, tracks and sectors per track) from the very little data that ATR provides. Moreover, different emulators interpret the same data differently.

 

Currently ATR header bytes 12, 13 and 14 (counting from 0) are not used neither by APE (it uses bytes 8-11 and 15) nor by SIO2PC 4.05 (it uses bytes 8-11).

 

I was wondering what would other people think about using some of this bytes to provide geometry information about the image. We could even squeeze some flags in there to support SpartaDosX's 512 bytes/sector format, for example a bit for signaling that the first 3 sectors are not 128 bytes.

 

Apparently APE and Atari800 just write back what they find in those bytes. So, it would be reasonably backwards compatible. In the long run maybe we can succeed in getting the emulator writers to support these extensions.

 

Even one byte could be enough, lets say for byte 12 (this is just a sketch, the track counts are from SpartaDos 3.2g XINIT):

 

bits 0-3: number of tracks on disk
               0: 1 (harddisk, enhanced density information must be zero)
               1: 35 single sided
               2: 40 single sided
               3: 77 single sided (8 inches, enhanced density information must be zero)
               4: 80 single sided
               5: 35 double sided
               6: 40 double sided
               7: 77 double sided (8 inches, enhanced density information must be zero)
               8: 80 double sided
               9-15: reserved
bit 4:    disk is 1050 enhanced density
bit 5:    reserved
bit 6:    first 3 sectors are not 128 bytes long
bit 7:    the geometry information is valid (if zero, we calculate the geometry info in the old way)

 

Any thoughts? (Especially from peripheral and/or computer emulator writers!)

[Rybags]

Sounds good, but the whole can thing fall over again once the next wall comes up.

 

Ideally, the format initially would have used offset pointers for various descriptor field blocks, which is a way of being able to extend a standard without breaking compatability.

 

Of course, going that way now would break stuff even more.

 

Alternatively, just use the remaining spare bytes as magic number + flags, then have extra descriptor data right at the end of file.

 

But then that falls over a bit too if you were to want to incorporate such features as extendable images.

[cyco130]

Sounds good, but the whole can thing fall over again once the next wall comes up.

 

We still have 2 spare bytes for the next 2 walls The ideal solution would be to design a whole new extensible file format with optional copy protection and compression support, ATP looked promising but I'm not sure if it's still active. Anyway, I don't see that happening in the near future. So I'm just proposing a quick hack until that time comes

 

Even including a header size field in the original format would make extensions possible. But, alas, I don't think Nick Kennedy ever thought that his format would be a standard in the years to come...

[jacobus]

UnixCoffee and I were tossing around an idea to embed a checksum for each .ATR - thereby producing a unique identifier that could be keyed to a global database (this was in the context of upgrades to my front end for Atari800Win - all emulator settings could be stored in a database and the user would never have to fiddle with the different compatibility settings)

[drac030]

I once wrote an 8-bit emulator for Falcon/TT, and yes, I thought about providing exact geometry information. The XLD file my emulator used consisted of:

 

1) 16-byte header: 4 byte-signature "EXLD" (as in "Emulated XL Disk") + 12-byte PERCOM block defining the geometry of the disk.

 

2) actual data (sector_size indicated by the header * number_of_sectors indicated by the header).

 

By the way, ATR format is just plainly brain-damaged, e.g. it expresses the disk size in 16-byte increments (instead of just bytes, or, if not, in 128-byte increments, i.e. the smallest sector size). It would be most useful to simply throw it off instead of expanding it, and write a tool to convert existing ATR files to a new format.

 

We could even squeeze some flags in there to support SpartaDosX's 512 bytes/sector format, for example a bit for signaling that the first 3 sectors are not 128 bytes

 

I think that such a flag is not necessary. I think that sector_size > 256 could mean just this (i.e. that first 3 sectors are not 128-byters). ATR files, as I said above, are crap regarding the format design, but nevertheless they are capable of holding a 512-byte-per-sector images. Currently SIO2BSD supports such images (SIO2BSD is sort of APE for BSD/Linux), and we (me & trub) also have a customized version of Atari800Win which does them too (it has been made for testing purposes, when SDX 4.39 was being developed).

[jaybird3rd]

The ideal solution would be to design a whole new extensible file format with optional copy protection and compression support, ATP looked promising but I'm not sure if it's still active. Anyway, I don't see that happening in the near future. So I'm just proposing a quick hack until that time comes

 

I think that's an excellent idea. The ATR format has its limitations, as you point out, but the alternatives haven't caught on or have issues which make them unsuitable as direct replacements. The PRO standard that APE/ProSystem uses offers good copy protection emulation, but it's a proprietary format. The ATX/VAPI/whatever format isn't suitable either because the imaging tools are still unavailable to the public, and neither the plugin for APE or the VAPI mini-SIO2PC tool work very well in my experience; the format is basically useless outside of (patched) emulators.

[jaybird3rd]

Currently SIO2BSD supports such images (SIO2BSD is sort of APE for BSD/Linux), and we (me & trub) also have a customized version of Atari800Win which does them too (it has been made for testing purposes, when SDX 4.39 was being developed).

Not to go off-topic, but I just wanted to say thanks for mentioning SIO2BSD. I wasn't aware of it before, but I've been looking for a suitable SIO2PC solution for FreeBSD, and this is just what I need. Thanks again!

[phaeron]

The ATX/VAPI/whatever format isn't suitable either because the imaging tools are still unavailable to the public, and neither the plugin for APE or the VAPI mini-SIO2PC tool work very well in my experience; the format is basically useless outside of (patched) emulators.

 

I'm afraid I don't see the difference here -- what's the difference between having a format for which the imaging tools haven't been released, and a format for which no imaging tools exist yet? The ATX format is basically known at this point, even weak sector information.

 

There is, unfortunately, one big problem with ATX/VAPI, and that is as far as I know it does not have support for non-SD sector sizes. The most important information it does have, though, is sector timing, which no other Atari disk format that I know of supports.

[pajero_pn]

I was wondering what would other people think about using some of this bytes to provide geometry information about the image. We could even squeeze some flags in there to support SpartaDosX's 512 bytes/sector format, for example a bit for signaling that the first 3 sectors are not 128 bytes

 

 

It is possible to conclude for length it exactly ATR. It see example.

 

Has not to change format of file presently meaning ATR !

[+Larry]

I'm evidently missing something. This ATR shows up in APE as 721 sectors and double density. Won't boot; can't be recognized by any Dos I've tried. Maybe that's the point? Or just to use this ATR to read the header info?

-Larry

 

I was wondering what would other people think about using some of this bytes to provide geometry information about the image. We could even squeeze some flags in there to support SpartaDosX's 512 bytes/sector format, for example a bit for signaling that the first 3 sectors are not 128 bytes

 

 

It is possible to conclude for length it exactly ATR. It see example.

 

Has not to change format of file presently meaning ATR !

[drac030]

Here:

 

http://www.ibi.uw.ed...aco/bigdisk.zip (ZIP file)

 

is a valid, 32 MB ATR file for use with SIO2BSD and the A800Win version I mentioned. It is empty (except having been formatted for SDX 4.42).I don't know, however, if APE could open it

 

EDIT: URL fixed

 

EDIT2:

 

Edited December 24, 2009 by drac030

[cyco130]

UnixCoffee and I were tossing around an idea to embed a checksum for each .ATR - thereby producing a unique identifier that could be keyed to a global database (this was in the context of upgrades to my front end for Atari800Win - all emulator settings could be stored in a database and the user would never have to fiddle with the different compatibility settings)

 

APE has an ATR format extension to add a CRC32 to the images. See here.

 

 

I think that such a flag is not necessary. I think that sector_size > 256 could mean just this (i.e. that first 3 sectors are not 128-byters). ATR files, as I said above, are crap regarding the format design, but nevertheless they are capable of holding a 512-byte-per-sector images. Currently SIO2BSD supports such images (SIO2BSD is sort of APE for BSD/Linux), and we (me & trub) also have a customized version of Atari800Win which does them too (it has been made for testing purposes, when SDX 4.39 was being developed).

 

Agreed. Thanks for the attached image in your later post, I will analyze it and try to support 512 bytes/sector in AspeQt the way you do. Would it make sense if I supported the XLD format too? Or is it deprecated?

 

I think that's an excellent idea. The ATR format has its limitations, as you point out, but the alternatives haven't caught on or have issues which make them unsuitable as direct replacements. The PRO standard that APE/ProSystem uses offers good copy protection emulation, but it's a proprietary format. The ATX/VAPI/whatever format isn't suitable either because the imaging tools are still unavailable to the public, and neither the plugin for APE or the VAPI mini-SIO2PC tool work very well in my experience; the format is basically useless outside of (patched) emulators.

 

What exactly does this VAPI format look like? There isn't much info here. Just a DLL with no header. I suppose it's not an open format. So, no better that PRO if that's the case.

 

I'm evidently missing something. This ATR shows up in APE as 721 sectors and double density. Won't boot; can't be recognized by any Dos I've tried. Maybe that's the point? Or just to use this ATR to read the header info?

-Larry

 

I suppose that's the point. The image is broken, its first three sectors are 256-byte long instead of 128 as it should be. See here.

 

 

 

 

BTW, would you BSD people mind to check if AspeQt compiles there? I guess Qt is available in BSDs and I tried to make the I/O code posix-compliant. I would happily try to solve any problems that keep it from running.

[cyco130]

The ideal solution would be to design a whole new extensible file format with optional copy protection and compression support, ATP looked promising but I'm not sure if it's still active. Anyway, I don't see that happening in the near future. So I'm just proposing a quick hack until that time comes

 

I think that's an excellent idea. The ATR format has its limitations, as you point out, but the alternatives haven't caught on or have issues which make them unsuitable as direct replacements.

 

Sorry, I forgot to add:

 

I think a new format must have the following properties to be useful:

 

* It must be an open format with a single clear definition.

* There must be crossplatform command line and GUI tools to convert existing ATR's to the new format and back (when possible).

* There must be a ready to use crossplatform library to manipulate them, licensed under something like zlib/png license so that it can be used even by commercial programs.

* It must have compression support (zlib comes to mind).

* It must have copy protection support with necessary tools to extract them using a 1050-2-PC style cable.

* It must support all known Atari disk formats.

* In it's simplest form (no compression, no copy protection), it must be as easy to use as ATR, or even easier.

* It must be extensible so that it can cope with future extensions.

* Not a requirement but it would be cool if there were native Atari programs to extract them from, or write them to real disks (like SCOPY or DISKCOMM).

 

I've got some ideas on this. I will post them when things materialize.

Edited December 25, 2009 by cyco130

[drac030]

Agreed. Thanks for the attached image in your later post, I will analyze it and try to support 512 bytes/sector in AspeQt the way you do. Would it make sense if I supported the XLD format too? Or is it deprecated?

 

XLD: I don't think so, it's use is limited to one emulator which isn't even finished (I lost the motivation when I realized, that a stock Falcon030 is just too slow, even if the program was written in 100% 68030 assembler).

 

The format you propose:

 

* It must have compression support (zlib comes to mind).

 

Why internal compression format? An image can always be compressed with ZIP, and it probably will be compressed anyways, because in this form it can be safely downloaded from the web. And, besides, internal compression will make it difficult/slow to create/use such an image on the Atari itself. Even if I think I saw zlib for cc65, it seems to me as an unnecessary complication.

 

Besides:

 

the new format should, IMHO, absolutely drop the ATR habit of keeping first 3 sectors in different length than the rest of the imaged sectors have. The only advantage is to save 384 bytes, which is nothing. The disadvantages are: 1) complications in implementation (the perpetuous problems with programs treating/creating ATR files in incompatible ways), 2) inability to hold CP/M images, in which the first sectors are 256-byters.

 

Of course, an APE-type program should read/write the first 3 sectors as 128-byters on a 256-byter disk image by default. It would be nice, though, if it allowed to optionally read/write them in full extent.

 

EDIT:

 

I don't know if you're seriously going to expand the ATR format or just introduce a new one. But in the latter situation, including an explicit PERCOM block in the image header would be probably a good idea.

 

BTW, would you BSD people mind to check if AspeQt compiles there? I guess Qt is available in BSDs and I tried to make the I/O code posix-compliant. I would happily try to solve any problems that keep it from running.

 

Will do after Christmas.

Edited December 25, 2009 by drac030

[cyco130]

XLD: I don't think so

 

Ok.

 

Why internal compression format? An image can always be compressed with ZIP, and it probably will be compressed anyways, because in this form it can be safely downloaded from the web. And, besides, internal compression will make it difficult/slow to create/use such an image on the Atari itself. Even if I think I saw zlib for cc65, it seems to me as an unnecessary complication.

 

I was counting on cc65's zlib too. I never used it though, but it would be nice to make a diskette image fit in a file in another diskette.

 

the new format should, IMHO, absolutely drop the ATR habit of keeping first 3 sectors in different length than the rest of the imaged sectors have.

 

Couldn't agree more. I hate that test code all over the place.

 

I don't know if you're seriously going to expand the ATR format or just introduce a new one. But in the latter situation, including an explicit PERCOM block in the image header would be probably a good idea.

 

I was just thinking loud. I should be focusing on the core functionality first. But in the long run, why not?

I once read somewhere that the percom block (byte 8 I think) had some info stating that the sectors are starting from 0 and the first 3 sectors are full size. But I can't find it right now. If this info is indeed reliable, it can be a way to support CP/M disks with full size sectors 1-3.

 

Will do after Christmas.

 

Neato I'll probably make another release before new year, this time I'll double check the Posix compatibility.

[drac030]

I once read somewhere that the percom block (byte 8 I think) had some info stating that the sectors are starting from 0 and the first 3 sectors are full size. But I can't find it right now. If this info is indeed reliable, it can be a way to support CP/M disks with full size sectors 1-3.

 

This is "bit 1 = 1, sectors 1-3 have full extent". But in reality it is like disk geometry in early AT-BUS drives: this bit is always 1, because these sectors are *physically* always 256-byters in DD, it is only the disk drive which cuts this to 128 bytes when sending/receiving data to/from the computer. So this bit is useless, most drives I saw return $FF in byte 8 and ignore it when written :/

[phaeron]

the new format should, IMHO, absolutely drop the ATR habit of keeping first 3 sectors in different length than the rest of the imaged sectors have.

 

Couldn't agree more. I hate that test code all over the place.

 

I don't understand why this is a problem -- a sector offset/size table takes care of this. You'll need to handle the more general case of this anyway if you plan to handle any non-standard/protected disk formats. Otherwise, you've potentially got data in the image that is fabricated dummy data since it effectively didn't exist in the original disk as the Atari could only see it.

[drac030]

the new format should, IMHO, absolutely drop the ATR habit of keeping first 3 sectors in different length than the rest of the imaged sectors have.

 

Couldn't agree more. I hate that test code all over the place.

 

I don't understand why this is a problem (...)

 

You possibly didn't read the whole argument. The problem is inconsistent implementation of this "feature" of the ATR format. As a result, you have first to find out, if a DD image's first sectors size is 128 bytes or 256 bytes. I guess this is the "test code" cyco130 is referring to. The only reasonable solution IMHO is to make a cut on this.

Edited December 25, 2009 by drac030

[cyco130]

I don't understand why this is a problem -- a sector offset/size table takes care of this. You'll need to handle the more general case of this anyway if you plan to handle any non-standard/protected disk formats. Otherwise, you've potentially got data in the image that is fabricated dummy data since it effectively didn't exist in the original disk as the Atari could only see it.

 

As drac030 pointed out the DD ATR files are just plain nightmare. A few posts earlier pajero_pn attached an example here. The first three sectors are 256 bytes long, padded with zeroes. I saw even worse files with 384 padding bytes at the end. I believe there are more buggy ATR tools than non-buggy ones. Flattening out this difference would make this a lot easier for everybody.

 

With protected image formats one has to deal with much more complicated schemes of course, but I think the unprotected version of a possible new format should be as simple as it can be. A sector offset table would complicate the things for 8-bit programs because standard Atari Dos does not support random access.

 

PS: I couldn't thank you enough for virtualdub and altirra. Both are huge contributions to the society

[phaeron]

I don't understand why this is a problem -- a sector offset/size table takes care of this. You'll need to handle the more general case of this anyway if you plan to handle any non-standard/protected disk formats. Otherwise, you've potentially got data in the image that is fabricated dummy data since it effectively didn't exist in the original disk as the Atari could only see it.

 

As drac030 pointed out the DD ATR files are just plain nightmare. A few posts earlier pajero_pn attached an example here. The first three sectors are 256 bytes long, padded with zeroes. I saw even worse files with 384 padding bytes at the end. I believe there are more buggy ATR tools than non-buggy ones. Flattening out this difference would make this a lot easier for everybody.

 

With protected image formats one has to deal with much more complicated schemes of course, but I think the unprotected version of a possible new format should be as simple as it can be. A sector offset table would complicate the things for 8-bit programs because standard Atari Dos does not support random access.

 

Oh, sure. You don't have to remind me of the ATR mess, as I just landed in another instance of it in the VBXE thread. The problem with ATR, though, is not that it has the mixed sector sizes, but that it has no metadata telling you how many sectors are present or of each size. I can't speak to the difficulties of writing an 8-bit tool, but from my perspective, knowing the exact numbers of boot sectors and total sectors is far more important than making sure the format is regular, because it's the ambiguity that's causing the problems.

 

(By the way, if you think ATR is bad... try EXE/XEX....)

[drac030]

it's the ambiguity that's causing the problems.

 

This is exactly what we are speaking about. E.g.: bytes 2,3,6 of the header contain "the size of the disk image in 16-byte increments (aka paragraphs)". Ok, so this is the file's size - including or excluding the header?

 

And, besides, why the size is expressed in "paragraphs"? Or, why on earth the header contains the size of the file, and it does not contain the number of actual sectors?

 

Atari disk images do not need to be complicated or ambiguous at all. A disk is a series of sectors numbered from 1 to n (max. 65535, thus far), and there is no question how many boot sector it contains, because the problem of boot sectors occurs only in double density, and in that density the problematic sectors are never more (and never less) than 3. So the simplest solution is to store all sectors continuously assuming their constant size (i.e. 256 bytes in DD), and provide a pointer (seek offset) to the first byte of the sector 1 in the header (plus, obviously, the necessary geometry information).

 

(By the way, if you think ATR is bad... try EXE/XEX....)

 

The COM file (aka EXE, or, for PC users, "XEX") is at least well defined and not ambiguous at all.

Edited December 26, 2009 by drac030

[HiassofT]

Hi!

 

The problem is inconsistent implementation of this "feature" of the ATR format. As a result, you have first to find out, if a DD image's first sectors size is 128 bytes or 256 bytes. I guess this is the "test code" cyco130 is referring to. The only reasonable solution IMHO is to make a cut on this.

The best solution IMO is to just ignore ATRs which have the first 3 sectors stored with 256 bytes. These images are just plain broken, and once developers stop supporting these broken ATRs they will hopefully vanish. There are programs available to fix these broken ATRs, so it's really not a big problem - people just need to run the fixer over their ATRs (in batch mode) and everything's fine. I did this several years ago.

 

Actually, the ATR format spec isn't that ambiguous at all (I had to deal with a lot worse specs...). Here's the chapter from Nick Kennedy's SIO2PC doc (the "readme.txt"):

 

STRUCTURE OF AN SIO2PC ATARI DISK IMAGE:

 

It's extremely simple. There is first a 16 byte header with the following

information:

 

WORD = special code* indicating this is an Atari disk file

WORD = size of this disk image, in paragraphs (size/16)

WORD = sector size. (128 or 256) bytes/sector

WORD = high part of size, in paragraphs (added by REV 3.00)

BYTE = disk flags such as copy protection and write protect; see copy

protection chapter.

WORD=1st (or typical) bad sector; see copy protection chapter.

SPARES 5 unused (spare) header bytes (contain zeroes)

 

After the header comes the disk image. This is just a continuous string of

bytes, with the first 128 bytes being the contents of disk sector 1, the

second being sector 2, etc.

So, it's quite clear that the first sector(s) have to be 128 bytes and the 16-bytes header isn't included in the "image size" parameter. Storing the size in "paragraphs" is an relict of x86 real-mode (assembler) programming (x86 segment registers address the memory in 16-byte increments).

 

The SIO2PC software had an annoying bug: if you created a 180k DD image it calculated the size wrong and added 384 bytes at the end of an image. One had to be very careful with this because SIO2PC let you access sector 721 of these images and this could possibly corrupt other loaded images. A workaround was to create a "custom DD image" with 720 sector, in this case SIO2PC did it's math correctly - still quite annoying.

 

so long,

 

Hias

[drac030]

the ATR format spec isn't that ambiguous at all (...). Here's the chapter from Nick Kennedy's SIO2PC doc (the "readme.txt"):

 

STRUCTURE OF AN SIO2PC ATARI DISK IMAGE:

 

It's extremely simple (...)

 

The SIO2PC software had an annoying bug: (...)

 

And, what do you think, where this bug came from? It was caused by that "extremely simple" and crystal clear specification. It is IMHO not very difficult to foresee, that any new tools which may exist in the future will make the same bug come back again and again, because programmers will fall into this same trap again and again. And all this hassle just to save precious 384 bytes of ATR file size - well, really, I still think that the only reasonable solution is to abandon ATR format and get rid of that crap once for good.

[HiassofT]

Hi!

 

And, what do you think, where this bug came from? It was caused by that "extremely simple" and crystal clear specification. It is IMHO not very difficult to foresee, that any new tools which may exist in the future will make the same bug come back again and again, because programmers will fall into this same trap again and again. And all this hassle just to save precious 384 bytes of ATR file size - well, really, I still think that the only reasonable solution is to abandon ATR format and get rid of that crap once for good.

Yes, it's all a mess, introduced by the original ugly hack of drives truncating the first 3 sectors to 128 bytes because the Atari OS couldn't deal with it (i.e. boot these disks). I'm not a big fan of the ATR format, either, but I think it's still the best currently available format for (standard) Atari disks (XFD, DCM, ATX, PRO are a lot worse). And you can't really blame the format spec for programmers making mistakes :-)

 

Concering current Atari disk images, there's no big benefit of padding the first 3 sectors to 256 bytes (like the drives do). Usually you have one single routine in your code that calculates the offset inside the file and the size of a given sector. Padding the first 3 sectors to 256 bytes will simplify offset calculation, but you still need this routine (or an "if" statement) to determine how many bytes to transfer. I'm sure programmers will also find a way to get this wrong :-)

 

But, concerning future image formats (like support for 512 bytes sectors), I think there's no point of extending the ATR format. Creating a new image format (ideally and extendable one) would be the way to go.

 

so long,

 

Hias

[+poobah]

But, concerning future image formats (like support for 512 bytes sectors), I think there's no point of extending the ATR format. Creating a new image format (ideally and extendable one) would be the way to go.

 

Absolutely.