Spartados X and FatFS... Issues

[Kyle22]

18 hours ago, _The Doctor__ said:

Here's a request to someone with an IDE+

I'm wondering if there are any problems with transfers under the control of its BIOS. I don't have one handy, but it could help with unraveling another piece of the puzzle.

Test ALL possibilities.

 

[flashjazzcat]

7 hours ago, Kyle22 said:

Test ALL possibilities.

Use Altirra if you don't have the hardware.

 

BTW: You've been asked twice for details on how to reproduce the issue without FATFS and have chosen to ignore the question.

Edited October 5, 2019 by flashjazzcat

[Jeffrey Worley]

5 hours ago, flashjazzcat said:

Use Altirra if you don't have the hardware.

 

BTW: You've been asked twice for details on how to reproduce the issue without FATFS and have chosen to ignore the question.

I haven't had enough time on the Altirra emulator to have complete confidence in it, nor to know quite how to do a few things that would be helpful, so til' now I've been using real hardware.  Using an Emulator feels like a cheat, also,  ?  Though I know it is just a great tool.  One thing I'm unsure of is how to mount a 'real' side2 disk, not partition, onto the emulator.  Apropos of that, how would I break out the APT partition from the card?  dd it to a file like I did the whole-card I linked you to?

 

Here's a real kicker, at least to me, and maybe sheds light on why 448 and '9 fail in fatfs but 447 works.:

 

This, from the 448 "whatsnew.txt" file (part of the distro):

 

<SNIP>

 

* In case of accessing file systems larger than 32 MB (like these handled
  by FATFS) now the IDE+2.0/U1MB PBI XDCB protocol will be used instead of
  the old IDEa extended SIO addressing protocol. In either case this only
  applies to the PBI devices, serial drives are unaffected.

<SNIP>

 

So there was a major change in this rev.  They decided to use XDCB instead of IDEa for the fat32 access code backbone and, it appears, broke it?

 

Best,

 

Jeff
 

Edited October 5, 2019 by Technoid Mutant
TYPO

[flashjazzcat]

1 hour ago, Technoid Mutant said:

I haven't had enough time on the Altirra emulator to have complete confidence in it, nor to know quite how to do a few things that would be helpful, so til' now I've been using real hardware.  Using an Emulator feels like a cheat, also

If you have the disk image you previously sent me with instructions on how to replicate the issue, I'll take another look at it. I had to delete the file, unfortunately, since it was rather large and my 240GB SSD was stuffed with video footage. You may have complete confidence in Altirra, however, and it's more useful to observe the issue in emulation since one can then investigate the code using the debugger, so rather than being a cheat, observing the problem in emulation is manifestly more useful than doing so on real hardware. It's almost certainly a logic issue which will behave identically in emulation, too (otherwise I wouldn't have been able to reproduce the issue previously using the disk image and instructions you provided).

1 hour ago, Technoid Mutant said:

One thing I'm unsure of is how to mount a 'real' side2 disk, not partition, onto the emulator.

System->Configure System->Devices->'Add...'. Add your SIDE2 device, and to that (subordinate in the tree structure) add a VHD or ISO as a device on the 'CompactFlash Bus'.

1 hour ago, Technoid Mutant said:

Apropos of that, how would I break out the APT partition from the card?  dd it to a file like I did the whole-card I linked you to?

There's no need: you'll be dealing with complete, raw physical disk images the whole time. DD the whole card.

1 hour ago, Technoid Mutant said:

So there was a major change in this rev.  They decided to use XDCB instead of IDEa for the fat32 access code backbone and, it appears, broke it?

Yes: I noticed you pointing this out before and it's at least plausible that this has something to do with it. The question is whether the device firmware is failing, or the FMS side of things. The error code appears to originate from inside the FMS driver, but this may simply mean that the driver encountered nonsensical file system data which might in turn have arrived because of a firmware problem. This is why it would probably be useful to establish whether the errant behaviour can be duplicated using a different IDE host adapter.

 

It would be interesting if Kyle observed the same issue without using FAT partitions at all, since SDFS is limited to 32MB partitions, and the XDCB is only employed when using partitions larger than 32MB (i.e. 65,535 512 byte sectors). The XDCB provides DAUXA and DAUXB and is described on page 4 of this document. Previous versions of SDX used 'DUNUSE' to store bits 16-23 of the LBA sector number (this was the method employed by the 'IDEa' host adapter). DUNUSE is still supported by IDE Plus, SIDE.SYS and the U1MB PBI BIOS, but setting bit 7 of DDEVIC causes the firmware/driver to use DAUXA and DAUXB instead, to account for bits 16-31 of the LBA sector number. So: it could be that DOS improperly populates one of these locations, forgets to set bit 7 of DDEVIC at some point, or maybe the XDCB is a complete red-herring. Who knows until IDE Plus has been tested. If Kyle observed the issue when using 32MB SDFS partitions, perhaps DUNUSE was not properly cleared when the XDCB is NOT being used.

 

I would like to at least eliminate the U1MB PBI BIOS as a source of problems, since there is a firmware update pending and this seems like an ideal opportunity to address any lingering issues. As soon as I hear or observe that this is a DOS issue, I can go ahead and release the firmware update.

Edited October 5, 2019 by flashjazzcat

[Jeffrey Worley]

1 hour ago, flashjazzcat said:

If you have the disk image you previously sent me with instructions on how to replicate the issue, I'll take another look at it. I had to delete the file, unfortunately, since it was rather large and my 240GB SSD was stuffed with video footage. You may have complete confidence in Altirra, however, and it's more useful to observe the issue in emulation since one can then investigate the code using the debugger, so rather than being a cheat, observing the problem in emulation is manifestly more useful than doing so on real hardware. It's almost certainly a logic issue which will behave identically in emulation, too (otherwise I wouldn't have been able to reproduce the issue previously using the disk image and instructions you provided).

System->Configure System->Devices->'Add...'. Add your SIDE2 device, and to that (subordinate in the tree structure) add a VHD or ISO as a device on the 'CompactFlash Bus'.

There's no need: you'll be dealing with complete, raw physical disk images the whole time. DD the whole card.

Yes: I noticed you pointing this out before and it's at least plausible that this has something to do with it. The question is whether the device firmware is failing, or the FMS side of things. The error code appears to originate from inside the FMS driver, but this may simply mean that the driver encountered nonsensical file system data which might in turn have arrived because of a firmware problem. This is why it would probably be useful to establish whether the errant behaviour can be duplicated using a different IDE host adapter.

 

It would be interesting if Kyle observed the same issue without using FAT partitions at all, since SDFS is limited to 32MB partitions, and the XDCB is only employed when using partitions larger than 32MB (i.e. 65,535 512 byte sectors). The XDCB provides DAUXA and DAUXB and is described on page 4 of this document. Previous versions of SDX used 'DUNUSE' to store bits 16-23 of the LBA sector number (this was the method employed by the 'IDEa' host adapter). DUNUSE is still supported by IDE Plus, SIDE.SYS and the U1MB PBI BIOS, but setting bit 7 of DDEVIC causes the firmware/driver to use DAUXA and DAUXB instead, to account for bits 16-31 of the LBA sector number. So: it could be that DOS improperly populates one of these locations, forgets to set bit 7 of DDEVIC at some point, or maybe the XDCB is a complete red-herring. Who knows until IDE Plus has been tested. If Kyle observed the issue when using 32MB SDFS partitions, perhaps DUNUSE was not properly cleared when the XDCB is NOT being used.

 

I would like to at least eliminate the U1MB PBI BIOS as a source of problems, since there is a firmware update pending and this seems like an ideal opportunity to address any lingering issues. As soon as I hear or observe that this is a DOS issue, I can go ahead and release the firmware update.

I'm working on it now with a plain 800xl with a rambo 256 board, an actual ICD Rambo, with Ceramic gold-top Fujitsu rams no less.  I copied one of the suspect files and it worked, so we may be talking U1MB.  I'll let you know more in a little bit.  The file I linked you to is still in the same place on Google drive.  https://drive.google.com/open?id=11MJOyFQnlgWC_38PzsRoR8TBSaEruFop

 

Sorry it's so big.  We were diagnosing for partition issues at the time.  I can pretty confidently omit that as a cause of the fault now, but the image is handy and will work.

 

Best,

 

Jeff

[flashjazzcat]

1 minute ago, Technoid Mutant said:

I'm working on it now with a plain 800xl with a rambo 256 board, an actual ICD Rambo, with Ceramic gold-top Fujitsu rams no less.

So what device is hosting SDX and which hard disk and mass storage drivers are you using? We can't really conclude anything without that information.

2 minutes ago, Technoid Mutant said:

The file I linked you to is still in the same place on Google drive.

Thanks: downloading now. No worries about the size. I should have just kept the ZIP file, but I didn't expect to need it again.

 

[flashjazzcat]

OK: Managed to provoke error 144 with FATFS.SYS v.0.86L by attempting to copy E:\G\GAUNTL_1\GAUNTL_1.ATR to the root of drive D: (4). Here's the immediate source of the error:

 

 

Since DUNUSE is included in sector bounds checking when the XDCB is not used (which it is not here: DDEVIC = $31), we can see that SPARTA.SYS has issued an out of bounds sector read request (DCOMND = $52) on the destination drive (DUNIT = 4):

 

The requested 24-bit sector number is $0E2401, but since the D: partition is only 32MB in size (actually 65,534 or $FFFE sectors), the PBI driver is quite correctly detecting that the request is out of bounds.

 

It's conceivable that the driver is issuing this spurious read request because of a prior error (nonsensical information returned by a prior sector IO request on the same volume), but I consider it highly unlikely since all IO on volume D: requires only 16-bit sector numbers and the XDCB is not used. The question is why does DUNUSE contain the value $0E? The PBI BIOS did not put it there. Why SDX would put it there, meanwhile, I do not know since the E: volume is 1.8GB in size, so the XDCB should be employed if the DOS changelog is to be believed. In any case, at no point during communications with a 32MB HDD partition should DUNUSE contain anything but zero, but here it is containing a non-zero value. This would cause problems regardless of the presence of FATFS.SYS; the only question being who wrote the value to DUNUSE in the first place. I can trigger a conditional break-point to try and figure this out, but I doubt the U1MB firmware is to blame here.

 

Edited October 5, 2019 by flashjazzcat
typos

[Jeffrey Worley]

55 minutes ago, flashjazzcat said:

So what device is hosting SDX and which hard disk and mass storage drivers are you using? We can't really conclude anything without that information.

Thanks: downloading now. No worries about the size. I should have just kept the ZIP file, but I didn't expect to need it again.

 

The file I've dialed it down to, not the only one of course, but one that always fails or always works, depending on SDX version greater than 447 or if 447, respectively, is games>atlantis>atlant_1.xex  That file is the target for testing.

 

With the side on my Rambo 800xl and the Side2 cart running your latest firmware (449c), the file fails to copy to a Side2 hosted hard disk partition of any description, with Error 139 Device NAK.  On the same machine with the same configuration, not even a reboot or reset, you can copy this file to ramdisk O:, or to an SIO2pc hosted disk or to my 1050.  Nothing wrong with the file, it runs fine.

 

Swapping the Rambo XL for my u1mb XL and with the same Side2 cart, but running SDX 449c from the U1MB of course, the file fails to copy with an error 144 to the Side2 hosted hard disk, but does copy fine to any other sort of drive (not hosted on the side2).

 

Oh, so this is for sure not a U1MB issue.

 

Jeff

 

Edited October 5, 2019 by Technoid Mutant

[flashjazzcat]

Tried FATFS v.0.87L and the same operation does not generate an error at all here, so go figure. The specific issue described above must have been fixed in the next revision.

 

For the record, here is FATFS v.0.86L writing $0E to DUNUSE when setting up the DCB for a write to drive 4:

 

 

This does not happen with the next driver iteration, so if there's a problem with the more recent version of FATFS, it's something else.

[Jeffrey Worley]

1 minute ago, flashjazzcat said:

Tried FATFS v.0.87L and the same operation does not generate an error at all here, so go figure. The specific issue described above must have been fixed in the next revision.

 

For the record, here is FATFS v.0.86L writing $0E to DUNUSE when setting up the DCB for a write to drive 4:

 

 

This does not happen with the next driver iteration, so if there's a problem with the more recent version of FATFS, it's something else.

May I have a copy of fatfs .87L?  I've not found it anywhere, gave up trying at some point.

 

 

[flashjazzcat]

7 minutes ago, Technoid Mutant said:

but one that always fails or always works, depending on SDX version greater than 447 or if 447, respectively, is games>atlantis>atlant_1.xex  That file is the target for testing.

Where is this file? There is no 'GAMES' folder on drive E:.

7 minutes ago, Technoid Mutant said:

your latest firmware (449c)

That's the SDX version number: nothing to do with my driver. SIDE.SYS should report its own version number upon loading.

4 minutes ago, Technoid Mutant said:

May I have a copy of fatfs .87L?  I've not found it anywhere, gave up trying at some point.

You already have it: it's in the root directory of your 'A:' partition, which is where I loaded it from. '86L' is in the 'DOS' folder on the same drive.

Edited October 5, 2019 by flashjazzcat

[flashjazzcat]

12 minutes ago, Technoid Mutant said:

games>atlantis>atlant_1.xex

OK: found it by using the loader's search facility (in the E:\A\ATLANTIS\ folder as seen by DOS). Copies to the root of D: without issue using '87L'.

 

[Jeffrey Worley]

12 minutes ago, flashjazzcat said:

OK: found it by using the loader's search facility (in the E:\A\ATLANTIS\ folder as seen by DOS). Copies to the root of ? without issue using '87L'.

 

I'm sorry.  I don't have that disk in any form other than the big dd file.  Can you send the .87L file and I'll move it right over and test.  Else it is gonna be a mission.  Wow.  If it turns out to be that I'm gonna be embarassed.

 

Best,

 

Jeff

[flashjazzcat]

Here you go:

 

fatfs0.87L.zip

[Jeffrey Worley]

Um, it looks like the problem is fatfs086L.  I just tried 087L FJC sent me and it works like a peach on the file I was using for diagnostics.  I'm going to run this through its paces, but I have the feeling that this is the fix.  Grimace.

 

Best,

 

jeff

 

[Kyle22]

10 hours ago, flashjazzcat said:

BTW: You've been asked twice for details on how to reproduce the issue without FATFS and have chosen to ignore the question.

There's been so much crazy stuff going on here so I haven't had a chance yet. I'll REALLY try to make time tomorrow.

 

[flashjazzcat]

11 minutes ago, Technoid Mutant said:

Um, it looks like the problem is fatfs086L.  I just tried 087L FJC sent me and it works like a peach on the file I was using for diagnostics.  I'm going to run this through its paces, but I have the feeling that this is the fix.  Grimace.

No need for a grimace. We've accomplished more in one evening regarding discovery of what caused the 86L issue than anyone else but the driver's author himself in the space of two years (since it appears he already fixed the problem in 87L). The question now is whether all related 'Error 144' issues are caused by the same bug, and whether all can be overcome by simply updating to the latest FATFS version.

5 minutes ago, Kyle22 said:

I'll REALLY try to make time tomorrow.

That would be immensely helpful - thanks. Since an update to FATFS.SYS very conspicuously fixes the issue in the scenario explored in recent posts, being able to reproduce the same or a similar issue without the involvement of that driver would be quite significant.

 

At a guess, I'd say that the later version of FATFS.SYS follows DOS's example and uses the XDCB for >32MB volumes, while the older driver uses 24-bit addressing with the high-order bits in DUNUSE. SPARTA.SYS in older SDX versions also used DUNUSE in the sense that it would clear it (since no SDFS partition is larger than 32MB), but newer builds of SPARTA.SYS may not bother, since it's expected that the newer FATFS.SYS is used, and the newer FAT driver doesn't write to DUNUSE. Thus the combination of the newer SPARTA.SYS with the older FATFS.SYS causes problems. If SPARTA.SYS in SDX 4.49c cleared DUNUSE prior to writing to the 32MB SDFS volume, no problem would exist with the older FAT driver either.

[_The Doctor__]

this may be the message where the updated fatfs was posted to help out... its possible to put the rev as part of the filename keeping a mix up from happening... but that would make everyone have to re image their roms each time it changes to update their configs...

it appears the text hasn't been updated in the arc file either..

 

this thread has provided *very* useful information none the less... some things to double check,  no need for a grimace!

that was some very nice teamwork running down one of the issues.

Edited October 5, 2019 by _The Doctor__

[Jeffrey Worley]

So far, so good.  I've copied over 64 megabytes of files today, not including hits on specific targets.  No faults found with .087L and sdx 4.49c.  I'll keep plugging away for a bit, trying to find fault, but I think it really fixes things.

 

Best,

 

Jeff

[+Nezgar]

7 hours ago, _The Doctor__ said:

this may be the message where the updated fatfs was posted to help out... its possible to put the rev as part of the filename keeping a mix up from happening... but that would make everyone have to re image their roms each time it changes to update their configs...

it appears the text hasn't been updated in the arc file either..

This thread has been a great read thanks for everyones efforts here. I just had to comment wow - looks like @rdea6 made the sleeper post of the thread at the beginning by suggesting trying .087L.

 

Still trying to figure out where .087L came from? My current copy of the toolkit on my side2 has the 3893 byte version of FATFS.SYS, which means its .086L. I just downloaded it again from the SDX upgrade project site, and it is still contains the 3893 bytes (.086L) version, so that discounts rdea6's guess that it was from there...

[Roydea6]

1 hour ago, Nezgar said:

Still trying to figure out where .087L came from?

 

 

[+Nezgar]

Thanks @rdea6 it all makes sense now. 

 

I saw another interesting observation earlier in that thread - @Faicuai found that using COPY with the /S switch (to disable ANTIC) helped, in this case FATFS.SYS was not involved:

 

[flashjazzcat]

1 hour ago, Nezgar said:

I saw another interesting observation earlier in that thread - @Faicuai found that using COPY with the /S switch (to disable ANTIC) helped, in this case FATFS.SYS was not involved

Totally different issue: stalled high speed SIO transmission.

[Jeffrey Worley]

6 hours ago, Nezgar said:

Thanks @rdea6 it all makes sense now. 

 

I saw another interesting observation earlier in that thread - @Faicuai found that using COPY with the /S switch (to disable ANTIC) helped, in this case FATFS.SYS was not involved:

 

I used both modes and it didn't make a difference (DMA on or OFF) with regard to the failures.  With .087L it doesn't make a difference either - works properly either way, just works faster with DMA off.

 

Best,

 

Jeff

[flashjazzcat]

6 minutes ago, Technoid Mutant said:

I used both modes and it didn't make a difference (DMA on or OFF) with regard to the failures.  With .087L it doesn't make a difference either - works properly either way, just works faster with DMA off.

See my previous post. The DMA setting concerns high speed SIO failures. Disabling the screen allowed the SDX SIO driver to run fast enough to keep pace with low divisors. It has nothing whatsoever to do with the DUNUSE issue concerning FATFS v.0.86L.