AspeQt: Yet another SIO2PC/APE clone

[drac030]

fjc: you could combine the original approach of circular sector links with your new code. Maybe it would be enough to simply extend the "circle" so that it could never entirely fit in the SDX sector cache. The cache may hold maximum 16 sectors, but just in case anything changes in this respect, I would suggest to make a circular link of say 100 sectors. This would probably fix both problems, i.e. incompatibility with SDX and file size limit.

[flashjazzcat]

fjc: you could combine the original approach of circular sector links with your new code. Maybe it would be enough to simply extend the "circle" so that it could never entirely fit in the SDX sector cache. The cache may hold maximum 16 sectors, but just in case anything changes in this respect, I would suggest to make a circular link of say 100 sectors. This would probably fix both problems, i.e. incompatibility with SDX and file size limit.

 

That's weird - I was just thinking the exact same thing while I was feeding the cats. So the offset into the file would be (simplified): (((sector - base offset) + (pass * 100)) * 125). Neat. Maybe this will prove satisfactory to all.

[flashjazzcat]

OK:

 

http://atari8.co.uk/...0.8.4-test2.zip

 

http://atari8.co.uk/...4-test2-src.zip

 

I tested this with 2 files, both of over 100KB, and the size of the destination file was correct in both cases. I simply loop back to sector 68 when the "next" sector is computed to be 360, so the range here is 292 sectors - more than ample, I think, to ensure the SDX cache is fully flushed by the time repeating sector numbers come around.

 

Mirroring more than one folder simultaneously may be "broken" (I'll test this later), and I suspect "AtariFileNo" should be a fifteen element global array to make this work. I can make these changes later if need be, once the basic functionality is verified.

 

Thanks Konrad for your insightful suggestion: clearly only a small amount of extra effort was required to avoid further frustration.

 

SDX's copy progress bar extending to 129% was slightly disconcerting at first, but I assume this is because it's impossible to represent a file size greater than 999 sectors in AtariDOS, and this is where the COPY command derives the total expected size of the file.

Edited October 25, 2012 by flashjazzcat

[drac030]

It is possible (file's size field in the directory has two bytes, so it can count up to 65535 sectors), but I am not sure what DOS 2.0/2.5 et consortes will then display in the directory.

[flashjazzcat]

It is possible (file's size field in the directory has two bytes, so it can count up to 65535 sectors), but I am not sure what DOS 2.0/2.5 et consortes will then display in the directory.

 

You're right of course. And the answer is that DOS 2.5 truncates >999 to 999, while DOS 2.0S appears to wrap the number around, so that - for example - a 600KB file ends up with a reported size of 672 sectors.

 

So it may be worthwhile removing the code which limits the sector count to 999, since SDX can use it and AtariDOS appears not to care (unless we have occasion to read 600KB files in DOS 2.0S!). Removing the truncation certainly cures the SDX progress indicator problem, which now doesn't exceed 99% when copying a 158KB file (1,289 sectors). Not sure where the other 1 per cent went, although the destination file size is correct.

Edited October 25, 2012 by flashjazzcat

[atari8warez]

No problem.

I have difficulty accepting that the inability to access files of over 80KB in length is a bug,.....

 

Even though I was expecting that you would act the way you just did, i still don't like the tone of your voice. First calm down, and next read further:

 

What Fatih did was clever and it allowed us to handle any file correctly within the limits of Atari DOS (max 64 dir entries and only root directory -- which are ATARI DOS limitations). However, and read this carefully Atari DOS allows a maximum file size of 88,750 bytes, not 80xxx bytes.

 

Here's a little calculation for you:

 

Maximum Atari DOS file size on a SD disk = 720 - 1 (inaccessible sector) - 1 VTOC sector - 8 directory entry sectors = 710 sectors x 125 bytes/sec = 88,750 bytes

 

if an implementation does not allow at least that many bytes on a file IT IS NOT COMPATIBLE WITH ATARI DOS whether you like it or not.

 

Fatih never claimed SDX compatibility nor even functionality with it. That was why it is only a partial implementation, however the part that worked worked CORRECTLY.

 

Whether I like the word "broken" or not, the truth is his implementation is now broken, therefore I can not allow the software to be released with this "fix".

 

You could share your "fix" anywhere with anyone as you said, but It won't be in SOURCEFORGE where the official release will live.

 

AND I CAUTION USERS OF ASPEQT THAT YOUR FIX WILL NEVER BE SUPPORTED OFFICIALLY UNTIL CORRECTED.

 

You are still welcome to contribute should you choose so after correcting the fix, but it was premature for you to put this into into the credits screen and change the release number without even getting an official approval of release (see photo). I choose not to further discuss this matter as your ego seems to take priority over your desire to put out a solid solution.

[drac030]

Removing the truncation certainly cures the SDX progress indicator problem, which now doesn't exceed 99% when copying a 158KB file (1,289 sectors). Not sure where the other 1 per cent went, although the destination file size is correct.

 

AtariDOS does not provide file's length in the directory with better accuracy than 1 sector. So, ATARIDOS.SYS, while converting a directory entry to SpartaDOS format, has no choice but take the count of sectors the file occupies, multiply it by 125 or 253 depending on the density, and sell to SPARTA.SYS as the actual file's length. This is the very value, which COPY.COM assumes later as 100%. But in fact, most files saved on AtariDOS file system are shorter than sector_count*(sector_size-3), and this is where the 1 percent goes.

[flashjazzcat]

Even though I was expecting that you would act the way you just did, i still don't like the tone of your voice. First calm down, and next read further:

 

Ray, I'm calm - so calm that I've fixed it up.

 

What Fatih did was clever and it allowed us to handle any file correctly within the limits of Atari DOS (max 64 dir entries and only root directory -- which are ATARI DOS limitations). However, and read this carefully Atari DOS allows a maximum file size of 88,750 bytes, not 80xxx bytes.

Here's a little calculation for you:

Maximum Atari DOS file size on a SD disk = 720 - 1 (inaccessible sector) - 1 VTOC sector - 8 directory entry sectors = 710 sectors x 125 bytes/sec = 88,750 bytes

if an implementation does not allow at least that many bytes on a file IT IS NOT COMPATIBLE WITH ATARI DOS whether you like it or not.

 

The pictures you posted showed files larger than that (FLEXIN.ATR, 123,875 bytes long for instance), so I don't accept any of this as a valid retort. You were using files far bigger than this perceived maximum to prove your point.

 

You are still welcome to contribute should you choose so after correcting the fix, but it was premature for you to put this into into the credits screen and change the release number without even getting an official approval of release (see photo). I choose not to further discuss this matter as your ego seems to take priority over your desire to put out a solid solution.

 

Ray, you're obsessed with egos. I didn't expect you to replace all the source code on SourceForge with the contents of my ZIP file, but to selectively incorporate relevant parts of the new code as you wished, if you saw fit.

 

What's actually coming across loud and clear to me here is not that I have a fragile ego, but that you'd actually prefer - for whatever reason - to have fixed this issue all on your little lonesome. In fact I encourage you to do so if you're insistent on reinventing the wheel. But what's really been grinding my gears throughout this whole process (which is supposed to be fun, after all, and which all spawned from a fix I had to implement for myself because no-one else had the time / inclination / nous to do so), is that every time I've presented something, I've clearly described not only the details of the implmentation, but also its limitations (such as the all-important file size), but you give the impression of having read my posts through a blue funk, and instead of making a helpful suggestion on how things might be done better, you post screen shots of the fix working exactly as designed, with comments about how this and that is "broken", and then I'm forced to make reference time and again to stuff I've already explained but to which you appear to be completely oblivious. And eventually - as usual - we reach this stage.

 

Would you download a beta version of some DOS, read the accompanying text file whose to-do list has the entry: "512 byte sector support", then post a screenshot of the FMS not working with 512 byte sectors? Because you repeatedly do the precisely equivalent thing, not only here, but in other topics on this forum, and that's why people get irritated and tired of repeating themselves. In addition, when you (apparently) fail to grasp some salient point, and the point is reiterated, you take this as a personal attack, or a demonstration of egos or "cliques" in action, and then complain that - yet again - you've been cast as the agitator for some unknown reason.

 

I took you off the block list especially to contribute to this project Ray, but you have somehow managed to make me regret this after a mere four days.

 

AtariDOS does not provide file's length in the directory with better accuracy than 1 sector. So, ATARIDOS.SYS, while converting a directory entry to SpartaDOS format, has no choice but take the count of sectors the file occupies, multiply it by 125 or 253 depending on the density, and sell to SPARTA.SYS as the actual file's length. This is the very value, which COPY.COM assumes later as 100%. But in fact, most files saved on AtariDOS file system are shorter than sector_count*(sector_size-3), and this is where the 1 percent goes.

 

Again - obvious now it has been explained. Thanks for doing so.

Edited October 25, 2012 by flashjazzcat

[atari8warez]

Blah, blah blah....

I am not going to repeat myself and have no desire to contribute to this senseless argument. I will implement the fix if i see it works 100% compatible with Atari DOS and SDX as it claims it does fix both, anything less and it goes to the back burner.. You should seriously consider why you are having trouble finding a paying day job...... Nough said!...

Edited October 25, 2012 by atari8warez

[atari8warez]

Update:

 

version 0.8.4

 

+ Added Slovak translation (Victor Cech).

+ Fixed a path separator issue with Linux/Mac OS

 

Many thanks to Victor Chec for Slovak language translation.

Edited October 25, 2012 by atari8warez

[ivop]

You should seriously consider why you are having trouble finding a paying day job...... Nough said!...

 

That's low and uncalled for. Have you seen post #428? It seems you are constantly one or two posts behind what fjc said.

[atari8warez]

As long as he doesn't change his attitude, I don't care how many posts I am behind..... had enough of his crap already.... I am almost certain that it is his behavior blocking his luck with finding a job, I wouldn't hire him if I was an employer and I see nothing low about stating this fact.. and yes it was indeed called for giving his past and present attitude on these forums... I've always tried to be nice and control myself, but enough is enough.....

Edited October 26, 2012 by atari8warez

[mrnukem]

Update:

 

version 0.8.4

 

+ Added Slovak translation (Victor Cech).

+ Fixed a path separator issue with Linux/Mac OS

 

Many thanks to Victor Chec for Slovak language translation.

 

I can't thank you enough for this program. I have my 130 XE with SIO to Serial interface set up next to my Windows system and I always have a game or some demo, slide show going using ASPEQT. I just thought since I am using it daily and have been for a year or more I should take the time to say thank you for your hard work.

[atari8warez]

I can't thank you enough for this program. I have my 130 XE with SIO to Serial interface set up next to my Windows system and I always have a game or some demo, slide show going using ASPEQT. I just thought since I am using it daily and have been for a year or more I should take the time to say thank you for your hard work.

 

Thank you for the nice words, however majority of the credit goes to Fatih Aygun's work, he did an excellent job on AspeQt and I am just trying to improve it further within the limits of my abilities. In any case I will continue doing so in the future as long as life allows me to do so . I must also acknowledge everybody who contributed to this project one way or another and many thanks go to them as well.

 

I also have a setup like yours and my Atari 130XE became part of the permanent fixtures on my desktop, and I love it.

Edited October 26, 2012 by atari8warez

[Artie Stolks]

Mac Binary of 0.8.4 anyone?

[atarixle]

I stopped building/trying to build it for Mac OS X, as it's pretty un-compilable.

 

Additionally, Carsten told me, that the serial-port in Mac OS X has to be accessed in a different way than in Linux. So we both guess, that AspeQt wouldn't work at all. Althru, I never got to run my only successfully built version of AspeQt 0.8.1 (is it the lack of a correctly working serial-2-usb-driver for OS X or the faulty serial-access in OS X).

 

You could try to get it to run in Wine (it runs perfectly in Wine on Linux, and at least the GUI appears in Mac OS X, which is a good sign).

 

An btw, the sources for 0.8.4 are still missing on SourceForge.net ...

Edited November 6, 2012 by atarixle

[atari8warez]

An btw, the sources for 0.8.4 are still missing on SourceForge.net ...

 

It is there, revision 61 under trunk. aspeqt.pro

[atari8warez]

Folder Imaging Issue with AspeQt:

 

Current AspeQt Folder imaging implementation (feature known as PC-Mirror in APE parlance) presents sectors of a file to the Atari in a circular fashion.

 

For example a file is assigned 5 sector numbers say from 400-404 and when Atari requests the first sector from the file, sector no 400 is returned along with the first 125 bytes of data from the file and the next sector number is set to 401.

 

When Atari requests the second sector (i.e. sector no 401) AspeQt sends the next 125 data bytes along with next sector number of 402. This continues until sector number 404 is returned to the Atari along with next sector no of 400 (this is where it wraps back to the first sector in a circular fashion).

 

When all data bytes of the file are served the next sector number is returned as 0 to indicate end-of-sectors.

 

The reason why 5 sectors are used per file is because AspeQt uses sector numbers from 400 to 719 for all 64 possible directory entries (720 - 400 = 320 available sectors / 64 = 5 sectors per file). The way it determines which sector group belongs to a file is by integer division of the sector number by 5. (i.e. 400 - 400 / 5 = 0, 401 - 400 / 5 = 0,....., 404 - 400 / 5 = 0, where 0 is the file number).

 

This way AspeQt makes sure that it can serve 64 (Atari DOS maximum) files regardless of the file size (within the DOS limits) of each file from a given PC Folder. This works well for Atari DOS 2.0/2.5, MyDos 4.45, SpartaDos 3.2/3.3, as they don’t cache incoming sector numbers and don’t care what sector number comes next.

 

SDX however is another story, circular sector numbers don’t play nice with SDX as it seems to cache incoming sector numbers and goes haywire when a duplicate sector number is found in the sector cache. So the initial reaction by me and by others was to increase the number of sectors used in the circular scheme from 5 to at least 16 to overflow the cache. I implemented the change and this at first seemed to fix the problem with the cache but introduced a new one. The first sector of a file is now dropped and the second sector is duplicated at the very beginning of the target file when doing a file copy.

 

Like this:

 

Source file sectors: 400, 401, 402, 403, 404, 405,……., 415

Target file sectors: 401, 401, 402, 403, 404, 405,……., 415

 

After some debugging with AspeQt, I am still not sure why this happens but since the scheme works fine with other DOS I tend to think that this is another issue with the SDX file system as it only happen with SDX formatted disks, disks in Atari format work fine so ATARI.SYS driver does not seem to be the source of the problem, or at least it appears that way.

 

I am just not sure what that might be, and hoping that Konrad or someone with a better knowledge of SDX’s inner workings then I, can shed some light….

 

P.S: Since the current scheme of serving files from folders work with almost all pre SDX Dos, I will be releasing a new version enhanced by Write Support in the upcoming days/weeks depending on my schedule. SDX support may have to be written separately unless we overcome this issue in an Atari DOS compatible way.

Edited December 9, 2012 by atari8warez

[phaeron]

On a whim I took a crack at implementing virtual folders in Altirra, and I think I see now why the modified AspeQt is running into problems. In short, the SDX copy command reads the first data sector twice:

 

HOOKSIO: Intercepting disk SIO status req.: buf=02EA

HOOKSIO: Intercepting disk SIO read PERCOM req.: buf=5A11

HOOKSIO: Intercepting disk SIO read: buf=5A11, len=0080, sector=0001, status=01

HOOKSIO: Intercepting disk SIO read: buf=5A11, len=0080, sector=0168, status=01

HOOKSIO: Intercepting disk SIO read: buf=5A11, len=0080, sector=0169, status=01

HOOKSIO: Intercepting disk SIO read: buf=5C11, len=0080, sector=016A, status=01

HOOKSIO: Intercepting disk SIO read: buf=5E11, len=0080, sector=016B, status=01

HOOKSIO: Intercepting disk SIO read: buf=6011, len=0080, sector=016C, status=01

HOOKSIO: Intercepting disk SIO status req.: buf=02EA

HOOKSIO: Intercepting disk SIO read PERCOM req.: buf=5A11

HOOKSIO: Intercepting disk SIO read: buf=5A11, len=0080, sector=0001, status=01

HOOKSIO: Intercepting disk SIO read: buf=5C11, len=0080, sector=0168, status=01

HOOKSIO: Intercepting disk SIO read: buf=5C11, len=0080, sector=0169, status=01

HOOKSIO: Intercepting disk SIO read: buf=5E11, len=0080, sector=016A, status=01

HOOKSIO: Intercepting disk SIO read: buf=6011, len=0080, sector=016B, status=01

HOOKSIO: Intercepting disk SIO read: buf=5A11, len=0080, sector=016C, status=01

VDISK: Reading sector 28 (sector 0 of file 24 / tarzan.xex)

VDISK: Reassigning cached link at sector 67

HOOKSIO: Intercepting disk SIO read: buf=5C11, len=0080, sector=001C, status=01

HOOKSIO: Intercepting disk SIO status req.: buf=02EA

HOOKSIO: Intercepting disk SIO read PERCOM req.: buf=5E11

HOOKSIO: Intercepting disk SIO read: buf=5E11, len=0080, sector=0001, status=01

HOOKSIO: Intercepting disk SIO read: buf=5E11, len=0080, sector=0005, status=01

HOOKSIO: Intercepting disk SIO read: buf=6011, len=0080, sector=0006, status=01

HOOKSIO: Intercepting disk SIO read: buf=5A11, len=0080, sector=0004, status=01

HOOKSIO: Intercepting disk SIO write: buf=5A11, len=0080, sector=0004, status=01

HOOKSIO: Intercepting disk SIO read: buf=5A11, len=0080, sector=0001, status=01

HOOKSIO: Intercepting disk SIO write: buf=5A11, len=0080, sector=0001, status=01

HOOKSIO: Intercepting disk SIO write: buf=6011, len=0080, sector=0006, status=01

HOOKSIO: Intercepting disk SIO write: buf=5C11, len=0080, sector=0020, status=01

VDISK: Reading sector 28 (sector 0 of file 24 / tarzan.xex)

VDISK: Using cached link at sector 67

HOOKSIO: Intercepting disk SIO read: buf=5E11, len=0080, sector=001C, status=01

 

My implementation is diferent as it avoids reusing sectors for much longer -- it uses the rest of the disk as a pool for all files with LRU ordering for reuse -- but I limited the pool to 15 sectors and SDX was able to copy a file off of it. I'd suggest checking that the new AspeQt implementation can handle a re-read sector.

 

The SDX cache appears to be almost LRU... in this configuration I was seeing it cycle through four buffers during the copy. As you can see above, though, when reading and writing metadata it is not as well behaved.

[+Philsan]

It would be nice if AspeQt always remembers last directory used.

Now it remembers last directory used if you unmount a disk image and then change disk image.

If you change disk image without unmounting it, the program goes to AspeQt directory.

[flashjazzcat]

My implementation is diferent as it avoids reusing sectors for much longer -- it uses the rest of the disk as a pool for all files with LRU ordering for reuse -- but I limited the pool to 15 sectors and SDX was able to copy a file off of it. I'd suggest checking that the new AspeQt implementation can handle a re-read sector.

 

My amended version of Aspeqt avoids sector reuse for some 300 iterations, IIRC. I've been using this build every day without issue since October 25, and it does not appear to have any problem copying a file from a mirrored folder to an SDX formatted ATR. Of course I don't claim to have exhaustively tested the changes in anything other than my "typical use" scenarios.

 

http://atari8.co.uk/downloads/Aspeqt/aspeqt.zip

 

http://atari8.co.uk/downloads/Aspeqt/aspeqt_src.zip

 

This is built off the old 0.8.3 sources and has all the potentially offensive material removed from the About box. Maybe it can shed some light on the situation? In any case, I hope it can help in some way. If not, no problem. I don't want to intrude on or detract from what's being done here.

[drac030]

The SDX cache appears to be almost LRU... in this configuration I was seeing it cycle through four buffers during the copy. As you can see above, though, when reading and writing metadata it is not as well behaved.

 

It cycles through 4 buffers, because this is (IIRC) the default number of buffers the SDX declares. But the user can enlarge this amount to 16 buffers.

 

The first sector of a file appears to be read twice, because the ATARIDOS.SYS driver does a spurious read of what in a proper SpartaDOS FS disk would be a file map sector. Generally, the unusual behaviour of SDX is not only caused by the buffer cache, but also by the fact, that DOS 2.0 disks are handled by the SpartaDOS file system driver, which uses ATARIDOS.SYS as a plugin helper to translate DOS 2.0 structures into SpartaDOS and back.

 

I am not saying that the emulation ATARIDOS.SYS provides is perfect (you could say that spurious reads should be avoided), but still - with real disks it works without problems. Plus any performance issue is not a real problem, because it is assumed, that people using SDX will simultaneously use SpartaDOS file system most of the time.

[atari8warez]

My amended version of Aspeqt avoids sector reuse for some 300 iterations, IIRC. I've been using this build every day without issue since October 25, and it does not appear to have any problem copying a file from a mirrored folder to an SDX formatted ATR. Of course I don't claim to have exhaustively tested the changes in anything other than my "typical use" scenarios.

 

My code does away with sector reuse for 320 sectors, but the number of sectors doesn't matter as long as it is enough to overflow the cache (even 6 sectors worked on my previous tests). I will test your code anyway to see how it behaves and let you know, and don't worry about the "offensive material", the credit will be given where is due

Edited December 9, 2012 by atari8warez

[atari8warez]

Thank you all for your insights, I am still trying to digest this information and see what I can do to prevent the 1st sector problem, currently everything else works fine including mounting multiple folder images.

[phaeron]

I am not saying that the emulation ATARIDOS.SYS provides is perfect (you could say that spurious reads should be avoided), but still - with real disks it works without problems. Plus any performance issue is not a real problem, because it is assumed, that people using SDX will simultaneously use SpartaDOS file system most of the time.

 

Sorry if I implied that what ATARIDOS.SYS is doing is wrong -- it may rightfully read or write any sector on the disk in random order. What I was referring to was that the order in which SpartaDOS X uses the buffers is no longer LRU when accessing and updating non-file data, at least on a SpartaDOS X filesystem. This complicates virtual folder mapping since the way we're reusing sectors is not completely consistent in reality.

 

My amended version of Aspeqt avoids sector reuse for some 300 iterations, IIRC. I've been using this build every day without issue since October 25, and it does not appear to have any problem copying a file from a mirrored folder to an SDX formatted ATR. Of course I don't claim to have exhaustively tested the changes in anything other than my "typical use" scenarios.

 

Am I correct that this code only works for one open file at a time? It looks like the sector window is reassigned to whichever file had its initial sector read last. In theory we should be able to handle more than one simultaneous open file, although I haven't gotten around to stress testing my implementation yet.