SpartaDOS X MyIDE Driver

[+Stephen]

Well, looks like I won't get to test until I get my internal SDX mod installed.

[Shawn Jefferson]

I believe a utility will be provided with the next SDX release which will allow users to add files to the 8K "user" area of CAR: (including a customised CONFIG.SYS). This is what I'm using at the moment, but it's in beta and is not for public release.

 

That is awesome!

[rchennau]

Did I read this is written in C and compiled with CC65?

[flashjazzcat]

Did I read this is written in C and compiled with CC65?

No, it's written in MADS assembler. You may be thinking of my remark that the UI library would be well suited to a CC65 implementation.

 

I've finally got the difficult part of FDISK written, so the driver will follow shortly. FDISK allows for visual allocation of partition space in unallocated areas, much like the Windows disk manager. It was surprisingly tricky to write, mainly because of all the hefty 32 bit arithmetic (well, 28 bits get used...).

Edited September 25, 2010 by flashjazzcat

[+Stephen]

--SNIP-- I believe a utility will be provided with the next SDX release which will allow users to add files to the 8K "user" area of CAR: (including a customised CONFIG.SYS). This is what I'm using at the moment, but it's in beta and is not for public release.

That's the best news about SDX I've heard in a while. Your MyIDE driver and Sparta Commander also made the top 3

[flashjazzcat]

That's the best news about SDX I've heard in a while. Your MyIDE driver and Sparta Commander also made the top 3

Good to hear. It's Kyle Dain's driver, really... I just amended it and added FDISK (now 2,730 lines of code and counting ).

[ascrnet]

hi flashjazzcat,

 

I wonder if this building will serve this myid cart with ide flash module?

 

 

Regards

[flashjazzcat]

I wonder if this building will serve this myid cart with ide flash module?

I wonder too... I have never used the MyIDE+Flash cart. I'm not sure if there's an SDX build for it (?)

 

Considerable progress here:

 

 

 

I estimate I'll release a beta in the next day or two (the driver code still needs aligning to the partition editor). We need this software testing with a range of HDDs/CF cards/MyIDE carts/internal adaptors: I only use CF cards on a machine with a modified Phi2 so we'll doubtless need many amendments to get this running on a wide variety of set-ups. It ought to correctly identify CF cards and subsequently allow the use of 512 byte sectors. FDISK will also perform a series of one-off verification tests on the media and decide whether to use "Fast" I/O, or I/O which compensates for timing instability.

 

I hope FDISK was worth the effort, considering it's an app for "occasional" use!

 

NOTE: Can't get this to run on Altirra, hence screenshots from real hardware.

Edited September 27, 2010 by flashjazzcat

[+bf2k+]

I wonder if this building will serve this myid cart with ide flash module?

I wonder too... I have never used the MyIDE+Flash cart. I'm not sure if there's an SDX build for it (?)

 

 

I could certainly use an SDX build for it. I'd be willing to try it out. Drac030?

[flashjazzcat]

I could certainly use an SDX build for it. I'd be willing to try it out. Drac030?

Drac030's away. Maybe ask Trub about it.

 

We're getting close here:

 

 

 

FDISK now reads and writes the partition table, and Hias' beta LBA MyIDETool successfully IDs it on the CF card. FDISK needs a bunch of "safety net" features added before it's a finished app (i.e. a two step confirmation process when writing out the partition table), but it works reliably and all the numbers line up. The program correctly IDs the size of the disk, whether the adapter is internal/external, works with master/slave devices, and supports 512 byte sectors where the capability is detected. Just some work to do on the driver and then we'll put this out for testing!

 

Since I'm not able to release the beta SDX Customiser, I'll try to prepare some cart/IntSDX images for folks to try out (these will have the driver built-in, and a custom CONFIG.SYS to load it).

Edited September 29, 2010 by flashjazzcat

[a8isa1]

It's kind of late in the process to chime in but have you considered leaving the MyIDE partition scheme as is (including images) and use the space that would be used for movies as the place for SDX partitions?

 

Also, I was wondering if it would be a good idea to revive the OS on disc concept. MyIDE 3.1 docs mention a space reserved on disc for a copy of the OS.

 

I realize my idea might be a logistical nightmare but there is so much space on CF that it seems a waste not to keep bootable partitions and images even if they are used only occasionally.

 

Just doing a little thinking out loud (or in type)...

 

You boot SDX. Your driver checks for a hotkey and finding it loads OS from the special region on the disk. From then on the system acts just like MyIDE+flash. Your SDX partitions are outside the normal partitions and will never be touched (unless you have a tool to transfer data between both types of partitions).

 

Thanks for listening to my opinion.

 

-Steve Sheppard

[flashjazzcat]

It's kind of late in the process to chime in but have you considered leaving the MyIDE partition scheme as is (including images) and use the space that would be used for movies as the place for SDX partitions?

I considered that for about a day before I threw my arms up in dispair at the CHS partitioning scheme, with each partition having to start on a cylinder boundary and the expensive calculations. At that point I decided on an LBA scheme, and since this is something that both Sijmen and Hias were already working on, it seemed a sensible way to proceed. Between us, we have drawn up a new LBA draft partition scheme, by no means limited to SDX. Images (which are ignored by the SDX FDISK) will be stored in specially flagged partition entries. We have a new MyIDETool to support the new format. I know little of how Sijmen is progressing with his LBA OS.

 

The new scheme is not an afterthought to the existing format, but a chance to go back to the drawing board. Also - from a personal perspective - my primary intention was to create something which would make MyIDE behave as near as possible like a "standard" hard disk interface (i.e. MIO or IDEa).

 

So yes - it was considered.

 

Also, I was wondering if it would be a good idea to revive the OS on disc concept. MyIDE 3.1 docs mention a space reserved on disc for a copy of the OS.

 

I realize my idea might be a logistical nightmare but there is so much space on CF that it seems a waste not to keep bootable partitions and images even if they are used only occasionally.

 

Just doing a little thinking out loud (or in type)...

 

You boot SDX. Your driver checks for a hotkey and finding it loads OS from the special region on the disk. From then on the system acts just like MyIDE+flash. Your SDX partitions are outside the normal partitions and will never be touched (unless you have a tool to transfer data between both types of partitions).

The main problem with the soft-OS, from my point of view, is that it works with next to none of my software, and certainly not with SDX. That's not to say the new format won't support such things: I have no idea what Sijmen has in mind. The SDX driver and FDISK are simply ways of interfacing the new LBA partition scheme with SpartaDOS X. The partitioning scheme is entirely adaptable to other uses outside of SpartaDOS X. RAM-based operating systems, images, bootable partitions, hotkeys, sector-based movies, etc are entirely outside of my remit. Hopefully the new scheme will support all of those things for those who want to use them, but none of them belong in the SDX driver. I wanted to:

 

1) Banish hotkeys and missing ROM vectors: done.

2) Triple or quadruple the MyIDE DOS read/write speeds: done.

3) Stream video via the FMS instead of from raw sectors: done.

 

To all intents and purposes, the SDX driver simulates a fast PBI-compliant HDD. Only the fact that any software attempting to read the HDD using the OS SIO vector fails reminds you it's a MyIDE interface. Outside of SDX, it should be business as usual once the new OS is written. There's also plenty of space in the new disk headers for external flags and option bytes, so SDX FDISK can be told to avoid certain disk regions, partitions, etc.

Edited September 29, 2010 by flashjazzcat

[a8isa1]

It's kind of late in the process to chime in but have you considered leaving the MyIDE partition scheme as is (including images) and use the space that would be used for movies as the place for SDX partitions?

I considered that for about a day before I threw my arms up in dispair at the CHS partitioning scheme, with each partition having to start on a cylinder boundary and the expensive calculations. At that point I decided on an LBA scheme, and since this is something that both Sijmen and Hias were already working on, it seemed a sensible way to proceed. Between us, we have drawn up a new LBA draft partition scheme, by no means limited to SDX. Images (which are ignored by the SDX FDISK) will be stored in specially flagged partition entries. We have a new MyIDETool to support the new format. I know little of how Sijmen is progressing with his LBA OS.

 

The new scheme is not an afterthought to the existing format, but a chance to go back to the drawing board. Also - from a personal perspective - my primary intention was to create something which would make MyIDE behave as near as possible like a "standard" hard disk interface (i.e. MIO or IDEa).

 

So yes - it was considered.

 

Also, I was wondering if it would be a good idea to revive the OS on disc concept. MyIDE 3.1 docs mention a space reserved on disc for a copy of the OS.

 

I realize my idea might be a logistical nightmare but there is so much space on CF that it seems a waste not to keep bootable partitions and images even if they are used only occasionally.

 

Just doing a little thinking out loud (or in type)...

 

You boot SDX. Your driver checks for a hotkey and finding it loads OS from the special region on the disk. From then on the system acts just like MyIDE+flash. Your SDX partitions are outside the normal partitions and will never be touched (unless you have a tool to transfer data between both types of partitions).

The main problem with the soft-OS, from my point of view, is that it works with next to none of my software, and certainly not with SDX. That's not to say the new format won't support such things: I have no idea what Sijmen has in mind. The SDX driver and FDISK are simply ways of interfacing the new LBA partition scheme with SpartaDOS X. The partitioning scheme is entirely adaptable to other uses outside of SpartaDOS X. RAM-based operating systems, images, bootable partitions, hotkeys, sector-based movies, etc are entirely outside of my remit. Hopefully the new scheme will support all of those things for those who want to use them, but none of them belong in the SDX driver. I wanted to:

 

1) Banish hotkeys and missing ROM vectors: done.

2) Triple or quadruple the MyIDE DOS read/write speeds: done.

3) Stream video via the FMS instead of from raw sectors: done.

 

To all intents and purposes, the SDX driver simulates a fast PBI-compliant HDD. Only the fact that any software attempting to read the HDD using the OS SIO vector fails reminds you it's a MyIDE interface. Outside of SDX, it should be business as usual once the new OS is written. There's also plenty of space in the new disk headers for external flags and option bytes, so SDX FDISK can be told to avoid certain disk regions, partitions, etc.

 

Ah. OK. Thanks.

 

I guess I'll just have to adapt.

 

Actually I don't do anything DOS intensive. I've been sticking with MyDOS in no small part because MyPicoDOS is just too convenient and because up to this point I wouldn't be able to use SDX and Action! on the same machine (well not without adding a second cart port). Candle's new combination cartridge (with SDX and OSS carts) may change my mind though and SDrive is perfectly adequate for storing games.

 

-Steve Sheppard

[flashjazzcat]

Actually I don't do anything DOS intensive. I've been sticking with MyDOS in no small part because MyPicoDOS is just too convenient and because up to this point I wouldn't be able to use SDX and Action! on the same machine (well not without adding a second cart port). Candle's new combination cartridge (with SDX and OSS carts) may change my mind though and SDrive is perfectly adequate for storing games.

This is why IntSDX is so handy. Of course, with my external MyIDE interface I still can't plug in a cart, although I've considered adding a pass-thru. The alternative would be an internal interface which hooks up to the CPU or cart pins (IntSDX sits in the OS socket). However, with MyIDE+SDX driver being a rather "imperfect" solution (re: the lack of native hardware support without the driver), I'd be reluctant to build it inside the machine. I'm lucky in as much as I don't currently require any cart based apps, although Candle's new cart certainly looks exciting.

 

When I've finished the MyIDE driver, I aim to set up my IDEa interface, which - although more bulky - is 100% standards compliant. Whether that will completely userp the MyIDE setup remains to be seen.

Edited September 30, 2010 by flashjazzcat

[ivop]

The program will no longer run on any emulator, but...

 

Why not? Are my (atari800) and Phaeron's (Altirra) IDE emulations not complete enough or is it something else?

[flashjazzcat]

The program will no longer run on any emulator, but...

 

Why not? Are my (atari800) and Phaeron's (Altirra) IDE emulations not complete enough or is it something else?

Well - I'm sorry I haven't used Atari800 in a long while, so I was perhaps wrong in saying that. When the driver's finished, please try it with that emulator. I don't have time to set it up.

 

However, I still can't get it to run on Altirra. I finally managed to get MyIDE emulation and SDX to co-exist, but the same FDISK which runs perfectly on real hardware locks the emulator up. Don't have time to delve into why it doesn't work at the moment. MyIDE is a sensitive interface, and I guess if things aren't 100% timing perfect, the emulation will go awry. Of course, it may simply be that the emulator offers no way to simulate Internal SDX and an external MyIDE interface.

 

I have conditionals in the FDISK source code so I can quickly compile a "demo" version which runs on Atari800WinPlus.

Edited September 30, 2010 by flashjazzcat

[flashjazzcat]

I was nearly at the end of my string with this driver today, until I discovered a stupid, stupid bug (relative branch based on Z flag being set; write sector wasn't leaving Z set before RTS, so relative branch never executed after a sector write, causing catastrophic results). Why does stuff like that always take 2-3 hours to track down?

 

Anyway, first RWTEST with the proper driver, reading the partition table from sector 0 of the disk and calculating the sector bounds and LBA offsets for each partition, yields this:

 

DOS writing: 32676.2496 B/sek

DOS reading: 51056.64 B/sek

DOS average: 41866.4448 B/sek

 

This is with a 65535 sector partition, 256 bytes per sector. 512BPS partitions are much faster (60KB/s reads), but I haven't implemented the CFA I/O routines for them yet. I estimate that with a tradeoff of further code optimisation against error checking, the I/O speed should remain about the same.

 

The driver code itself is surprisingly small; currently about 800 bytes (more or less the same as Kyle's original driver).

Edited September 30, 2010 by flashjazzcat

[twh/f2]

The driver code itself is surprisingly small; currently about 800 bytes (more or less the same as Kyle's original driver).

 

It's really cool to watch this thread and to get updates almost daily! thanks for that!

 

yesterday I got my Transcend IDE flash modules. Now I'm waiting for the MyIDE/Flash cart. I was following this thread right from the beginning, but did not get a clear understanding about the combination MyIDE/Flash and your SDX driver. Is this something you would like to support with your project? I understand that you have internal SDX and a single MyIDE cart.

 

As I hope Trub/Drac030 build/compile a new SpartaDosX AtariMax-image which includes your new driver. Then it would be hopefully possible to boot into SDX with the driver already in the cartridge, which then automatically recognizes the partitions on the flash module.

 

grtx,

\twh

Edited September 30, 2010 by twh/f2

[flashjazzcat]

It's really cool to watch this thread and to get updates almost daily! thanks for that!

 

yesterday I got my Transcend IDE flash modules. Now I'm waiting for the MyIDE/Flash cart. I was following this thread right from the beginning, but did not get a clear understanding about the combination MyIDE/Flash and your SDX driver. Is this something you would like to support with your project? I understand that you have internal SDX and a single MyIDE cart.

 

As I hope Trub/Drac030 build/compile a new SpartaDosX AtariMax-image which includes your new driver. Then it would be hopefully possible to boot into SDX with the driver already in the cartridge, which then automatically recognizes the partitions on the flash module.

Regularly sharing my progress helps me to maintain my sanity, I think.

 

In theory, the only two drivers required are for internal/external MyIDE (unless the hardware registers for MyIDE+Flash are different). I found it easier to create efficient code by having a separate driver for the internal and external interface. Other than that, it's really down to whether there's an SDX image for the desired hardware.

 

As I've said, I'm lucky enough to have the means to put the driver on the cart image. So testers can either boot the driver off an SIO device, or if they have EPROM burners, use special SDX images (which I will prepare at length). Note that the software I have only produces autoboot flashers for IntSDX, and these are somewhat unreliable ATM.

 

I now have 512 byte sectors working:

 

DOS writing: 55383.4738 B/sek

DOS reading: 62838.9415 B/sek

DOS average: 59111.2076 B/sek

 

These are easily the highest speeds I've yet achieved. The discrepancy between read and write speeds is so much greater with 256 byte sectors owing to DRQ wait code in the write loop for the smaller density. I'll see if I can eradicate that. 512bps is flying in both directions, however. All I need to add is retry code, which should barely impact on the speeds (and I have no way of testing my retry logic, since evidently the drive is ready 100% of the time: I'll depend on others for that). It's quite possible to stream video direct from the hard disk via the FMS at around 10FPS with bandwidth like this, rendering the MyIDE movie format unnecessary for SDX/MyIDE users. The CF card seems to switch between CF and 16-bit modes without problems: 512 byte and 256 byte sectors happily co-exist on the same disk. I haven't implemented 128 byte sectors yet, but that's hardly pressing with SDX hard drives.

 

I do get a feeling we'll get very mixed reliability reports. I've gone through two timing mods to get CF cards working properly, and the I/O code had to change every time (it's at its purest at the moment, apart from the glitch with 16-bit mode writes).

Edited September 30, 2010 by flashjazzcat

[a8isa1]

It's quite possible to stream video direct from the hard disk via the FMS at around 10FPS with bandwidth like this, rendering the MyIDE movie format unnecessary for SDX/MyIDE users.

 

It was my understanding the greyscale movie player pumps data at 61,440 bytes/sec, 7,680 bytes/frame (including audio) * 8 frames/sec.

 

If your driver does manage more bandwidth I'd like to see full frame videos rather than more frames/sec, but that's just my preference

 

-Steve

[flashjazzcat]

It was my understanding the greyscale movie player pumps data at 61,440 bytes/sec, 7,680 bytes/frame (including audio) * 8 frames/sec.

 

If your driver does manage more bandwidth I'd like to see full frame videos rather than more frames/sec, but that's just my preference

I'm sure it does, but I understood that the MyIDE movie player (if that's what you're referring to) requires movies to be placed outside the filing system, accessed by direct sector reads.

 

Using the VS movie player, I can achieve full-screen playback of the "MARIO" video at 8.7FPS, with the movie held as a 2.5MB file on an SDX partition, with no direct sector reads. Admittedly there's no audio, but the bandwidth is still 62838 bytes per second through the FMS.

[flashjazzcat]

The driver now defines the global "MYIDE" symbol, which points to a user-accessible vector/data table in RAM (draft format):

 

+0: jmp u_readsector

+3: jmp u_writesector

+6: LBA sector # (4 bytes)

+10: BPS (bytes per sector)

+11: verify flag

+12: sector buffer (pointer)

+14: partition table (pointer)

+16: drive table (offsets into the partition table for each drive #)

 

This proved essential, since although FDISK can (usefully) be used without the driver installed, when the driver is installed, toggling CFA mode without telling the driver knocks things out of sync (since the driver still thinks the media is in whichever mode it was in before FDISK ran). Now, FDISK checks for the "MYIDE" symbol, and if found, it sets the BPS variable when it turns CFA mode off (where applicable).

 

The data table also meant that the "LISTPART" command line utility which displays information about all the partitions could be coded in 300 or so bytes. It simply extracts the partition table pointer and prints the entries on the screen.

 

Naturally, u_readsector and u_writesector are only intended for apps which need to perform "low level" disk reads/writes without regard to partition boundaries (they are quite perilous routines in that sense). Most of the time, software requiring direct sector I/O should just use the DCB and the sparta SIO vector. I'm already thinking about ways to keep the partition table out of the reach of rogue/out of control software. It might be handy to have the option to keep it in the extended DOS bank. I'll speak to Draco or Trub about this in due course.

 

Another nice feature which (for a change) worked first time today was setting the boot drive. In FDISK, you can flag any single partition as "bootable", and when the computer is next booted, the driver sets the SDX "BOOT" environment variable to the root folder of the desired drive number, as well as setting CURDEV to the boot drive #. This means that if you set the boot drive to "C:", AUTOEXEC.BAT (not CONFIG.SYS, since it's never read from disk with this system) will be loaded from drive C:, which will also be the default drive.

 

I believe the following have expressed an interest in testing the driver:

 

twh/f2

bf2k+

Stephen

Larry

Fres

Shawn Jefferson

Hias

rdea6

ascrnet

Steve?

 

Not that it's a limited beta; I might release something into the wild on Sunday.

Edited October 1, 2010 by flashjazzcat

[flashjazzcat]

Are my (atari800) and Phaeron's (Altirra) IDE emulations not complete enough or is it something else?

No luck running these programs on Altirra. Where is the MyIDE emulation in Atari800 of which you speak? I've looked through the documentation and there's no mention of it.

[dwhyte]

Are my (atari800) and Phaeron's (Altirra) IDE emulations not complete enough or is it something else?

No luck running these programs on Altirra. Where is the MyIDE emulation in Atari800 of which you speak? I've looked through the documentation and there's no mention of it.

 

It's in the CVS sources for atari800.

[HiassofT]

Jon: If you like I can test your software with atari800, I'm running the CVS version of atari800 here on my Linux box.

 

Ah, wait, another idea: There are ready compiled (Win32) versions of atari800 with Tomasz Krasuski's SDL/OpenGL patch, the ide* files are the same as in current CVS

http://students.mimuw.edu.pl/~tk197881/atari800/

 

so long,

 

Hias