MemCard file area: file types proposal

[+batari]

Y'all are making the standards way too hard again, and making it so nobody will continue to use the file area. I think this discussion is denigrating into the same sad state as the Stella list's. I still see nothing fundamentally wrong with Paul Slocum's original proposal.

 

All I want to do is use the file area. NO static area, not even for the so-called directory. The files themselves will contain the filename and file type as the first 9 bytes. All I ask is that nobody erase files with the first byte $BB unless you have to (e.g. if the memcard is full). Everybody else can use whatever funky scheme they want.

 

Yeah, I remember old Apple DOS 3.3, where there were only 6 types. The C64 had about that many too. And we managed just fine. In the memcard, we will see even less use than the old Apple DOS, and we are proposing at least 255 types. Even such, the onus should be on the programmers to make sure their own programs have failsafes in place to prevent bad things from happening when their files get deleted or corrupted, not on everyone else. Missing/corrupted bB files will not harm anything on the memcard by my own design, so it's not unreasonable to ask others to do the same!

 

In turn, I will not put in automatic functionality to erase any files without $BB as the first byte. If the memcard gets full, the driver will overwrite other bB files only.

[vdub_bobby]

Y'all are making the standards way too hard again, and making it so nobody will continue to use the file area.  I think this discussion is denigrating into the same sad state as the Stella list's.  I still see nothing fundamentally wrong with Paul Slocum's original proposal.

3 bytes of char data is too hard?

 

And Paul was open to discussion about his original proposal. Rereading the discussion, it seems as if folks had come around to a 8+3

 

The problem I have with a single-byte file type is this:

 

Say I write a program that saves, for example, user-created Arkanoid levels.

I build a robust file manipulation system, since the user could save unlimited number of levels in a sitting.

 

So I want to display all of the files on the memcard.

If each file has a 3-char identifier plus a filename, I display that. It might not be completely clear what games the files are attached to, but it should be clear enough.

OTOH, if each file has a single-byte identifier plus a filename, unless I build a giant lookup table into my ROM, all I can display is a hex byte plus a filename. To save two bytes out of 128 you throw a big blanket of obscurity on the file system.

 

EDIT: As it happens, the file system I have been proposing is essentially the same as Eckhard's in this email.

 

Actually, reading further in the [stella] discussion, just about everyone there settled on a 3+8 file name convention, where the "8" could vary between 1 and 8 characters in length.

 

EDIT II: Blah. I don't think I'm being clear. To try again:

I prefer a 3-character filetype to a 1-byte filetype because, absent a lookup table, a 1-byte filetype is meaningless. A 3-character filetype likely will have meaning even if you don't have a lookup table telling you that BAS (or BB_ or whatever) means bBASIC. Now in your case $BB happens to correspond nicely to batari BASIC, but let's play pretend and imagine that Paul completes the Homestar Runner RPG someday. Is it better or worse for his save states to be HSR or HRR or some random hex byte, $5D?

Edited February 3, 2006 by vdub_bobby

[+batari]

Y'all are making the standards way too hard again, and making it so nobody will continue to use the file area.  I think this discussion is denigrating into the same sad state as the Stella list's.  I still see nothing fundamentally wrong with Paul Slocum's original proposal.

3 bytes of char data is too hard?

...

EDIT II: Blah. I don't think I'm being clear. To try again:

I prefer a 3-character filetype to a 1-byte filetype because, absent a lookup table, a 1-byte filetype is meaningless. A 3-character filetype likely will have meaning even if you don't have a lookup table telling you that BAS (or BB_ or whatever) means bBASIC. Now in your case $BB happens to correspond nicely to batari BASIC, but let's play pretend and imagine that Paul completes the Homestar Runner RPG someday. Is it better or worse for his save states to be HSR or HRR or some random hex byte, $5D?

1011678[/snapback]

That cuts into the space I need for filenames and RAM storage. I already know space is going to be tight. There is a chance that I might only have 7 usable bytes free in the 128 bytes (though I hope for at least 9), which leaves me with only 4 bytes for the filename. Sure, 6 is not a lot, but much better than 4.

 

At any rate, your scheme is still compatible with mine. For your purposes, you can just assume that all 3-byte filetypes that happen to start with hex $BB are reserved for bB programs, and I will not argue anymore. It's not really "hogging" per se. Keep in mind that bB does not represent just one game.

[supercat]

OTOH, if each file has a single-byte identifier plus a filename, unless I build a giant lookup table into my ROM, all I can display is a hex byte plus a filename.  To save two bytes out of 128 you throw a big blanket of obscurity on the file system.

1011678[/snapback]

 

My initial draft proposal allowed for an easy way of getting a six-character file type for any single-byte filetype associated with a statically-allocated page. Since there's no particular requirement that applications need to be allocated entire pages, it would be possible to have an area of memory which statically allocates 8 bytes per application instead of 64.

 

Further, I've already suggested a method for having application-info records which would contain both 8x7 and 48x7 sprites (probably a more useful indication of the application name than a text string). If you think it would be too hard to retrieve them, it would be possible to define a byte of static storage as being a pointer to the application info block.

[vdub_bobby]

At any rate, your scheme is still compatible with mine.  For your purposes, you can just assume that all 3-byte filetypes that happen to start with hex $BB are reserved for bB programs, and I will not argue anymore.  It's not really "hogging" per se.  Keep in mind that bB does not represent just one game.

The problem is that they are sort of compatible, but if everyone uses a 3 char+8 scheme while you use a 1-byte code+8 scheme then programmers have to code in the special case for the bB files. Two schemes at work at the same time is just asking for collisions, no matter how compatible.

 

But ultimately, I think whoever releases the first cart that uses the MemCard file system is going to decide what the standard is.

 

And icons are cool, but the more I think about it, the more I think they are a bad idea. In theory, they allow for an almost unlimited number of filetypes.

 

In practice, the vast, vast majority of those would just be blobs or other unrecognizable, unusable, indistinguishable, seemingly random sprays of pixels. Plus it is a lot more work to require programmers (few of which are graphic artists) to come up with an icon than it is to come up with a 3-char extension. Most of the icons would probably just be letters anyway.

[brpocock]

At any rate, your scheme is still compatible with mine.  For your purposes, you can just assume that all 3-byte filetypes that happen to start with hex $BB are reserved for bB programs, and I will not argue anymore.  It's not really "hogging" per se.  Keep in mind that bB does not represent just one game.

The problem is that they are sort of compatible, but if everyone uses a 3 char+8 scheme while you use a 1-byte code+8 scheme then programmers have to code in the special case for the bB files. Two schemes at work at the same time is just asking for collisions, no matter how compatible.

1012047[/snapback]

 

Three bytes or one, it's the same problem, collision control.

 

As a variant on the "Classed Addressing" theme:

 

ID type byte $00 is reserved for "app info" blocks, à la "Desktop DB". $ff = free block.

 

Let's say ID types $01..$20 are "reserved."

 

$21..$7e (ASCII7 printable characters) are byte 1 of a multichar "type." The type must be at least 2 bytes long, consist only of chars $21..$7e, and the last byte has the high bit set. This is followed by a filename consisting only of chars $20..$7e (note, $20=SPC), again with the high bit set on the last char.

 

$7f "reserved."

 

$80..$bf are one-byte "opaque" types. Note, $bb. There is no filename or internal structure implied. The type may be multibyte or not (e.g. types like $bb, $b0) but no guarantees are made about format, &c.

 

$c0..$fe are two-byte opaque identifiers with a filename following. The second byte can be $00..$ff. The filename follows, restricted to $20..$7e, with the high bit set on the last byte.

 

Does that meet everyone's needs ... ?

[vdub_bobby]

Three bytes or one, it's the same problem, collision control.

 

As a variant on the "Classed Addressing" theme:

 

ID type byte $00 is reserved for "app info" blocks, à la "Desktop DB". $ff = free block.

 

Let's say ID types $01..$20 are "reserved."

 

$21..$7e (ASCII7 printable characters) are byte 1 of a multichar "type." The type must be at least 2 bytes long, consist only of chars $21..$7e, and the last byte has the high bit set. This is followed by a filename consisting only of chars $20..$7e (note, $20=SPC), again with the high bit set on the last char.

 

$7f "reserved."

 

$80..$bf are one-byte "opaque" types. Note, $bb. There is no filename or internal structure implied. The type may be multibyte or not (e.g. types like $bb, $b0) but no guarantees are made about format, &c.

 

$c0..$fe are two-byte opaque identifiers with a filename following. The second byte can be $00..$ff. The filename follows, restricted to $20..$7e, with the high bit set on the last byte.

 

Does that meet everyone's needs ... ?

1012101[/snapback]

Too complicated, IMO. And, again: who ever gets a cart out first is going to define the standard, unless they use a system that is too specialized.

 

EDIT: The problem with all those weird cases (what are there, 6?!), is that if someone wants to build a robust file handling system into their game they have to account for all the cases. Which is too much of a pain, so no one will ever do it.

Edited February 3, 2006 by vdub_bobby

[Eckhard Stolberg]

Are filenames or standardized file identifyers really nessessary. Each file has the same size and each memcard has the same number of files. So it would be quite easy to identify each file by it's index number. If programs just let the user select which file number to save data to or load data from, users could keep track of the contents of their memcards on a sheet of paper. Richard could even have an official printable memcard directory sheet for download on his website.

 

No matter how clever you design the file identifiers, there is still a possibility that a program accidentally loads invalid data. And if the program has to check the data anyway, why not allow it to freely identify it's own files too. That way batari could for example put $BB in the first and last byte of each bB file, and supercat could put the ASCII representaion of "GEMS" at the start of each Strat-o-Gems play save file.

 

If filenames were used, every program that wanted to be able to overwrite other program's files would have to include a text based file browser. I think that this would be unnessessary complex considering how limited the VCS is with processing power and RAM and how bad it is with displaying text. If files were only selected by their index number, even a simple 4K program could use the file area in the memcard.

 

The only thing that might be helpful would be a 128 bit list in the Atarivox/memcard status block to mark used files. That way a program could have a "This file is already in use. Do you really want to overwrite it?" type of warning when the user selects a certain file.

 

 

Ciao, Eckhard Stolberg

[vdub_bobby]

I'd agree, Eckhard, except that 128 is a lot. Assume Paul writes his music tracker and you can save songs to the MemCard. I could easily save 50+ songs to the Memcard. But lose my handwritten sheet and spend a month doing other things and when I came back I'd never find any of them again. I'd have to go through and load each of 50 songs and recreate the master list. I doubt anyone would go to the trouble and, if they had to, they'd probably be soured on ever using the Memcard again.

[Eckhard Stolberg]

I'd agree, Eckhard, except that 128 is a lot.  Assume Paul writes his music tracker and you can save songs to the MemCard.  I could easily save 50+ songs to the Memcard.  But lose my handwritten sheet and spend a month doing other things and when I came back I'd never find any of them again.  I'd have to go through and load each of 50 songs and recreate the master list.  I doubt anyone would go to the trouble and, if they had to, they'd probably be soured on ever using the Memcard again.

1012121[/snapback]

No-one would stop Paul from saving song names in his files, if he feels they are useful. And his tracker could easily check all 128 files in the background for it's file signature and only let you select proper tracker files by name in it's loading dialog. You only wouldn't be able to display these names in other programs, because they all would use their own format for naming and identifying their files.

 

Having a standard file naming scheme would only be useful when you need to be able to delete other program's files to be able to save your data to the file area on the MemCard. But if for example you wanted to create a savegame state for a RPG, but your MemCard was already filled with tracker songs that you created months ago, then filenames like "MUS.MYSONG23" probably wouldn't be too helpfull for deciding which file to delete anyway.

 

 

Ciao, Eckhard Stolberg

[supercat]

Having a standard file naming scheme would only be useful when you need to be able to delete other program's files to be able to save your data to the file area on the MemCard. But if for example you wanted to create a savegame state for a RPG, but your MemCard was already filled with tracker songs that you created months ago, then filenames like "MUS.MYSONG23" probably wouldn't be too helpfull for deciding which file to delete anyway.

1012161[/snapback]

 

Exactly. For some applications, there would be a 'logical' name to attach to a file, but in many other applications there would not.

 

Perhaps it would be good to provide a means by which someone could nuke the data from an application they no longer have, and in such case it might be nice to know what the application is, but I don't see even that as being a terribly common case.

 

The idea of using a one-byte application id derived from (static address/64) does have one difficulty I can see, which is that it precludes the possibility of giving applications static allocations smaller than 64 bytes.

 

Perhaps the application ID should be two bytes, which represent the base address of an application's static allocation (MSB first). The first byte of an application's static allocation would be a pointer to an info block (value*128+$4000) containing small and large icons as well as an application name and some space the application could use for its own purposes.

 

If things were done this way, many applications could get by with just a single byte of statically-allocated storage (one extra block of data storage would be spent when the application first stored something in the EEPROM, but that shouldn't be a problem).

 

Actually, it might also be good to move the two-byte owner ID from the start of data blocks into its own 256-byte area of flash (128 blocks of data each have a two-byte owner ID). The MSB's of each owner ID would indicate that the block was:

 

00 -- Data for use by application, that starts with a filename

01 -- Data for use by application, not starting with filename

10 -- Information about the application

11 -- Blank

 

Having two types of data blocks for each application could speed up searching if, e.g., an application supported multiple-block files. The first block of each file would have the bits set one way; other blocks would have it the other way. Each block of the file could contain a pointer to the next block.

 

It might not be a bad idea to have the data in this table duplicated in the first two bytes of each block to ease reconstruction if something goes wrong, but reading 256 bytes sequentially is apt to be much faster than reading 2 bytes from each of 128 pages.

[mos6507]

If it were possible to have a version of AtariVox that the Memcard piggybacked onto, then you could just swap memcards and allocate an entire memcard to any game that wanted to use it like a filesystem.

 

$10 per memcard is no big deal. $50 for extra AtariVox's are a different story, mostly because of the cost of the Speakjet chip. What is really needed here is removable memory.

[Richard H.]

What is really needed here is removable memory.

Not perhaps an elegant solution, but the EEPROM is socketted on rev1 AVox's. So it would be just a case swapping it out for another chip.

 

(BTW it'll take the larger 64K if that's ever required).

[mos6507]

What is really needed here is removable memory.

Not perhaps an elegant solution, but the EEPROM is socketted on rev1 AVox's. So it would be just a case swapping it out for another chip.

 

(BTW it'll take the larger 64K if that's ever required).

 

Are there ZIF sockets available in that form-factor?

[Richard H.]

Are there ZIF sockets available in that form-factor?

I've never seen any in 8 pin DIP.

[Jacob Rose]

At any rate, your scheme is still compatible with mine. For your purposes, you can just assume that all 3-byte filetypes that happen to start with hex $BB are reserved for bB programs, and I will not argue anymore. It's not really "hogging" per se. Keep in mind that bB does not represent just one game.

 

Another suggestion from Stella was that the extension and filename should be variable length 7-bit ASCII strings with the eighth bit used as a stop bit - which $BB would obviously raise, so it's compatible, so long as the name that follows also has the stop bit in bit 7 in the last character.

 

Of course, the file extension is then "technically" ASCII 3B, which is ";" - you might want to follow with ASCII 29

Edited March 28, 2006 by Jacob Rose