Jump to content

Photo

New disk format required ?


21 replies to this topic

#1 Alfred OFFLINE  

Alfred

    Moonsweeper

  • 291 posts
  • Location:Elmwood, Ontario

Posted Sun Mar 11, 2018 2:20 PM

I was reading the SDX 4.48 manual one day and I believe I read that it supports 512 byte sectors. I'm not positive, but doesn't the MyIDE interface also support 512 byte sectors ? Are there any other interfaces that support >256 byte sectors and/or more than 65536 sectors ?

 

Either way, it occurs to me that we could use a new disk format that supports larger sectors and (maybe) larger drive sizes.  The first question might then be, what are the limits of the current interfaces, and what might be a logical maximum to include future development ?

 

I know from when I was talking to Bob during the development of the Black Box he said that it supported three more bits for the sector number. With 256 byte sectors that would yield a maximum partition size of 128MB using 524K sectors. What are the maximum partition sizes for these other interfaces like MyIDE or the SiDE stuff ?

 

The second question is, are they prevalent enough to warrant a new ATR-style format for the emulators. I think they are, if only to allow for the development of software via an emulator for either current hardware or that which may yet appear.

 

If there is some other file type like ATR that does all this, then by all means, enlighten me as to what it is, and then we can forget this post.

 

 



#2 Nezgar OFFLINE  

Nezgar

    Dragonstomper

  • 753 posts
  • Location:Regina SK Canada

Posted Sun Mar 11, 2018 2:30 PM

Most DOS's for the Atari 8-bits have a software limit of 16 bits for sector ID, which means maximum 65536 sectors. At 128 bytes per sector, this is 8MB, and 256 bytes its 16MB (normal max for ATR's and conventional Atari hard drive partitions), and yes 512 bytes per sector with newer versions of SpartaDOS X = 32MB. This works for sure on SIDE2, maybe for MyIDE2, with an added speed boost too (because it doesn't have to split the physical 512 byte sectors)

 

There's a framework in the 2.1 SpartaDOS file system for the concept of 'clusters' to allow larger partitions while still maintaining a 16-bit allocation map, and same sector size, but this is not yet implemented in the drivers.

 

The ATR file image format itself is conventionally limited to 16MB, but the header descriptor seems like it could be extended for larger sectors and size...

 

Edit: 65535 sectors max I think is an actual hard limit of the RAM SIO handler, as well as SIO protocol itself, hard to break out of that limit unless using a totally new handler type which abstracts from the sectors at a higher level, simiar to RespeQt PC-LINK, and the read-only FAT driver.


Edited by Nezgar, Sun Mar 11, 2018 2:36 PM.


#3 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 13,492 posts
  • Location:United Kingdom

Posted Sun Mar 11, 2018 2:43 PM

Ultimate 1MB's PBI HDD BIOS (and thus SIDE/SIDE2), Incognito's CF card adapter, and IDE Plus 2.0 all support 512 byte sectors. They also all support 28-bit LBA, so the perceived 65,535 sector limit (as mentioned above) is really a limitation of DOS. The firmwares of the aforementioned devices provide an extension to the 16-bit DCB sector number to take full advantage of large physical and logical disk capacities.

SDX already provides FATFS.SYS, a FAT driver which can access partitions larger than the 32MB allowed by the existing SDFS implementation. Sadly the FAT driver is still read-only, but there is already ample firmware support in all popular hard disk host adpaters (at least those which support APT and XDCB protocols) for large disks.

#4 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 13,492 posts
  • Location:United Kingdom

Posted Sun Mar 11, 2018 2:49 PM

65535 sectors max I think is an actual hard limit of the RAM SIO handler, as well as SIO protocol itself, hard to break out of that limit unless using a totally new handler type which abstracts from the sectors at a higher level, simiar to RespeQt PC-LINK, and the read-only FAT driver.


The XDCB is an extension to the DCB designed by Drac030 and has been adopted by all APT firmwares. As mentioned, it provides up to 32 bits for sector numbers and the implementation in his DracOS 65C816 OS even provides a 24-bit buffer address (also supported by extensions to the aforementioned hard disk host adapters).

#5 Alfred OFFLINE  

Alfred

    Moonsweeper

  • Topic Starter
  • 291 posts
  • Location:Elmwood, Ontario

Posted Sun Mar 11, 2018 3:04 PM

I take it then, since no one has said so, that there is not a current file format, like ATR which supports these larger size disk images.


Edited by Alfred, Sun Mar 11, 2018 3:05 PM.


#6 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 13,492 posts
  • Location:United Kingdom

Posted Sun Mar 11, 2018 3:12 PM

The usual ATR header supports a 24-bit count of the number of 'paragraphs' (16 byte chunks) in the container, so I reckon that's a limit of 256MB. However, it's unlikely you'd want to deal which such a large volume on a serial-connected peripheral, and since the currently addressable maximum hard disk size is 137.4GB, it usually just makes sense to make an ISO of the HDD.

Edited by flashjazzcat, Sun Mar 11, 2018 3:16 PM.


#7 Alfred OFFLINE  

Alfred

    Moonsweeper

  • Topic Starter
  • 291 posts
  • Location:Elmwood, Ontario

Posted Sun Mar 11, 2018 3:26 PM

Where do you get 137GB from ? 16 million sectors of 512 bytes ?

 

I guess the other question would be for Phaeron is, does Altirra support an ATR with 16 million sectors ? Offhand, I don't think it does.


Edited by Alfred, Sun Mar 11, 2018 3:29 PM.


#8 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 13,492 posts
  • Location:United Kingdom

Posted Sun Mar 11, 2018 3:28 PM

I guess. The Internet says that's the 28-bit LBA limit.

#9 Alfred OFFLINE  

Alfred

    Moonsweeper

  • Topic Starter
  • 291 posts
  • Location:Elmwood, Ontario

Posted Sun Mar 11, 2018 3:42 PM

I dunno, 16MB of 512 gives an anser of 8192MB to me. And the definition of ATR that is here http://atariage.com/...rmat-reference/says there are only 65536 paragraphs. I also see an old thread http://atariage.com/...tr-file-format/ about extending the format, but nothing came of it that I can tell.

 

So, it seems that we might need a new format that can handle larger sector counts, and variable sector sizes.



#10 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 13,492 posts
  • Location:United Kingdom

Posted Sun Mar 11, 2018 4:02 PM

Um... I said the maximum hard disk size is 137.4GB, and the maximum ATR size is 256MB (assuming 24-bit paragraph count). The ATR size is governed by the metadata in the header of the container. Logical and physical hard disks have no such limitation. Subsequently I suggested that the 256MB ATR limit wasn't worth worrying about, since anything bigger than that is likely to be a hard disk partition anyway. All the modern devices deal directly with hard disk partitions (no ATR intermediary), although most of them are also capable of mounting ATRs (including ATRs with 512 byte sectors) contained inside of said partitions.

I guess the other question would be for Phaeron is, does Altirra support an ATR with 16 million sectors ? Offhand, I don't think it does.


Obviously not. It supports ATRs with 65,535 sectors, so a maximum of 32MB. But that's an arbitrary limit of the 16-bit DCB sector number, not the geometry data in the ATR header.

Edited by flashjazzcat, Sun Mar 11, 2018 4:15 PM.


#11 Alfred OFFLINE  

Alfred

    Moonsweeper

  • Topic Starter
  • 291 posts
  • Location:Elmwood, Ontario

Posted Sun Mar 11, 2018 4:15 PM

I wasn't aware that ATR supported 512 byte sectors. So what is the byte size sector size field set to in the case of 512 byte sectors ?



#12 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 13,492 posts
  • Location:United Kingdom

Posted Sun Mar 11, 2018 4:18 PM

The MSB of the sector size is set to $02 for 512 byte sectors.

#13 Alfred OFFLINE  

Alfred

    Moonsweeper

  • Topic Starter
  • 291 posts
  • Location:Elmwood, Ontario

Posted Sun Mar 11, 2018 4:25 PM

Right, yes, I was blinded by the $80 or $100 comment field. nm.



#14 phaeron OFFLINE  

phaeron

    River Patroller

  • 2,482 posts
  • Location:USA

Posted Sun Mar 11, 2018 4:26 PM

Altirra does not support ATR images bigger than 64K sectors. There is no standard SIO command interface to access such disks, so you would not be able to use them anyway with any interface implemented in the emulator. For large hard disk images both .bin and .vhd are superior anyway as they don't misalign the sectors and VHD also supports sparse allocation, geometry metadata, and differencing disks. You can also remount VHDs in all modern OSes to directly access FAT32 partitions.



#15 Alfred OFFLINE  

Alfred

    Moonsweeper

  • Topic Starter
  • 291 posts
  • Location:Elmwood, Ontario

Posted Sun Mar 11, 2018 6:02 PM

When you say Altirra supports VHD files, I have a few questions:

 

1. Do you mean VHD as used by Microsoft such as the disk images used for Hyper-V (not including the VHDX ones) or do you mean some custom Atari format that uses the same extension letters ?

 

2. How can such VHD files be attached ? Do they require that they be attached via an interface like MyIDE, or can they be directly attached as D1: etc ? I'm ignoring the issue regarding DOS support, just asking in terms of raw device i/o.

 

3. Assuming they can be attached directly as D1:, how (or if) is the issue of sectors 1-3 handled ? Does Altirra just do what the DCB implies (ie. length =128 so only reads/writes 128) or does Altirra ignore the length and return the full sector size as specified in the VHD and its up to the software to partition the returned data ?



#16 phaeron OFFLINE  

phaeron

    River Patroller

  • 2,482 posts
  • Location:USA

Posted Sun Mar 11, 2018 6:18 PM

When you say Altirra supports VHD files, I have a few questions:
 
1. Do you mean VHD as used by Microsoft such as the disk images used for Hyper-V (not including the VHDX ones) or do you mean some custom Atari format that uses the same extension letters ?


It is the same format, mountable in Windows 7+ Disk Management / Diskpart.exe and in all major VM software. Altirra only supports plain VHD, not VHDX.
 

2. How can such VHD files be attached ? Do they require that they be attached via an interface like MyIDE, or can they be directly attached as D1: etc ? I'm ignoring the issue regarding DOS support, just asking in terms of raw device i/o.


Hard disk interfaces only. The VHD format is inefficient for small disk images.
 

3. Assuming they can be attached directly as D1:, how (or if) is the issue of sectors 1-3 handled ? Does Altirra just do what the DCB implies (ie. length =128 so only reads/writes 128) or does Altirra ignore the length and return the full sector size as specified in the VHD and its up to the software to partition the returned data ?

 

Emulators can't use the size from the DCB, as this is not visible for transfers over the SIO bus. The intended transfer length is only visible if an OS-level patch hook is being used.

 

There are no short boot sectors for the hard disk interfaces. For the floppy disk drive protocol, Altirra uses 3 128-byte boot sectors for sector sizes below 512 bytes and no boot sectors for 512 bytes and above. The latter is required by SpartaDOS 4.40+, which assumes a 512 byte boot sector. There is a weird case of ATR files with 8192 byte sectors, which are used by some cartridge flashing programs when flashing from an SIO2PC style device -- not sure if anyone tries to boot off disks with such huge sectors.



#17 guus.assmann OFFLINE  

guus.assmann

    Moonsweeper

  • 346 posts
  • Location:Netherlands City EDE

Posted Mon Mar 12, 2018 2:45 PM

Hello Alfred,

 

Maybe I've missed something?

Why would we need another disk format?

The 8-bit Atari computer is working just fine right now.

What would be the benefit?

Or maybe in other words: Do we need it or is it just because we can?

 

BR/

Guus



#18 ProWizard OFFLINE  

ProWizard

    River Patroller

  • 4,626 posts
  • Into Stampede
  • Location:In the cattle!

Posted Tue Mar 13, 2018 1:08 AM

Respect also has the 8192 bytes / sector 'large' atr feature. I use this with flashing The!cart. 128MegaBytes. Not the sizes mentioned here, but still.

@Guus : although I used to think like that... I never understood why I would want a 32Mb partition on a8, now I use it .... and it is full now too, while I absolutely would love to have more space on that partition. So... although I agree that a8 system is already excellent, It is never finished of course.

#19 HiassofT OFFLINE  

HiassofT

    Stargunner

  • 1,057 posts
  • Location:Salzburg, Austria

Posted Tue Mar 13, 2018 7:42 AM

Um... I said the maximum hard disk size is 137.4GB, and the maximum ATR size is 256MB (assuming 24-bit paragraph count)

Actually ATR allows up to 64GB maximum (32 bit paragaph count, 2^36-1 bytes max size).

See the SIO2PC docs http://pages.suddenl...5bdu/readme.txt

STRUCTURE OF AN SIO2PC ATARI DISK IMAGE:

It's extremely simple. There is first a 16 byte header with the following
information:

WORD = special code* indicating this is an Atari disk file
WORD = size of this disk image, in paragraphs (size/16)
WORD = sector size. (128 or 256) bytes/sector
WORD = high part of size, in paragraphs (added by REV 3.00)
BYTE = disk flags such as copy protection and write protect; see copy
protection chapter.
WORD=1st (or typical) bad sector; see copy protection chapter.
SPARES 5 unused (spare) header bytes (contain zeroes)


It looks like APE uses a slight variation with the high part of size stored as a single byte and 2 bytes CRC where the MSB of the size and the disk flags are stored. Yes, the nice thing about standards is that there are so many of them...

I wouldn't use ATR for storing harddrive images though, as phaeron wrote there are far better, standard formats available for that purpose.

so long,

Hias

#20 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 13,492 posts
  • Location:United Kingdom

Posted Tue Mar 13, 2018 7:54 AM

It looks like APE uses a slight variation with the high part of size stored as a single byte and 2 bytes CRC where the MSB of the size and the disk flags are stored. Yes, the nice thing about standards is that there are so many of them...


Exactly. It's long been debated, and you just have to pick one and assume it's wrong. ;) Better yet, try to ignore what's not part of the basic spec.
 

I wouldn't use ATR for storing harddrive images though, as phaeron wrote there are far better, standard formats available for that purpose.


I agree. And as I wrote earlier, in most situations where one would like to use an ATR for backups, the amounts of data involved are not well suited to the serial devices commonly used for hosting ATRs. Even if a "PIO2PC" existed, if would probably be better to cut ties with ATRs and use VHDs, etc. Sure there are some cases where ATRs are practical for backing up partitions (put a 32MB ATR in a FAT partition on a CF card, mount it using U1MB/SIDE2, and perform a raw sector backup at reasonable speed), but more often than not it's easier to just put the media in the PC and back it up that way.



#21 HiassofT OFFLINE  

HiassofT

    Stargunner

  • 1,057 posts
  • Location:Salzburg, Austria

Posted Tue Mar 13, 2018 8:29 AM

Exactly. It's long been debated, and you just have to pick one and assume it's wrong. ;) Better yet, try to ignore what's not part of the basic spec.

Basic spec before formats branched away seems to be 2 bytes for paragraphs, 2^20-1 ~=1MB :)

The APE CRC bytes are usually 0, and if they are set one can rather easily guess if there are 3 or 4 size bytes by looking at the real file size and/or the sector size.
 
It's a bit academic since there are currently no users of ATRs larger than 128MB, but with 8k sectors a 32768 sectors image would already be above what the APE format can represent.

As we don't have plans to create a 256MB or larger The!Cart this is going to stay academic for a quite while and the simple approach of checking just 3 size bytes should also be fine for now :)

so long,

Hias

Edited by HiassofT, Tue Mar 13, 2018 8:29 AM.


#22 flashjazzcat ONLINE  

flashjazzcat

    Quadrunner

  • 13,492 posts
  • Location:United Kingdom

Posted Tue Mar 13, 2018 9:33 AM

AFAIK, SIO2SD doesn't even support 512 byte sector ATRs yet, so container portability already becomes an issue even with 32MB images.






0 user(s) are browsing this forum

0 members, 0 guests, 0 anonymous users