Atari Software Preservation Initiative - what about RAW?

[GlowingGhoul]

I thought this deserved it's own thread, given how critical this project is to the future of our hobby.

 

I appreciate the critical work the Atari Software Preservation Initiative is doing to ensure software for our beloved 8-bits is preserved.

 

However, I have some concerns about the RAW files not being made accessible to the community. I think having the RAW files in the possession of one or a small handful of individuals puts as at risk of losing the only truly accurate record of software that is becoming increasingly unavailable in it’s original disk form. It’s like archiving a “good enough” MP3 and discarding the original, lossless music data.

 

Here are my points:

 

-First and foremost, an ATX is a close approximation of the original disk. A RAW is an absolutely accurate version of the original, capable of recreating the original in as perfect a way as any available technology. An ATX can generate a close, but not exact copy of the original.

 

-A8rawconv is great but not perfect, is in active development, and has a number of self-identified limitations. There have been multiple instances when a new ATR/ATX had to be created from the RAW file because a flaw was found and corrected in a newer version of A8rawconv. What happens down the road when more flaws are found but a RAW can’t be located?

 

-Since RAW files are not required for an ATX to be added to the archive, user submitted ATX files have no associated RAW, except in the possession of the user who submitted the ATX. Once again if the RAW is needed later it may be gone forever.

 

-Problem ATXs may only become apparent years down the road, particularly if a title isn’t popular and doesn’t get used much. When someone finds a problem on level 29 because of a defective ATX, where are we supposed to turn for a “good” copy if the RAW can’t be found and floppies have turned to dust?

 

-The primary objection I hear is that RAW files are too large to disseminate. For the casual user, perhaps. But getting these RAW files in the hands of as many users as possible will offer the greatest security for software preservation. Frankly, anyone can get "hit by a bus", and the RAWs in their possession could be lost at that point.

 

-I propose that RAW files must be supplied when any new ATX is submitted for entry into the archive. I also propose setting up a web based FTP archive of RAW and ATX files, and would like to mirror to as many other Atari 8-bit FTPs as possible. If cost or usage caps are a problem, I run commercial FTP's from 24/7 servers, so I could host one if necessary.

 

To use an audio analogy: In this, perhaps the last, best effort to preserve Atari 8-bit software, do we want a lossy MP3 that's "good enough", or the bit-perfect FLAC that represents the best possible copy of the original material?

Edited December 21, 2016 by GlowingGhoul

[ijor]

In first place, let me tell you that I don't oppose to the distribution of the raw images. I don't think it is a bad idea at all. But, having said that …

- I'm not sure I agree 100% that ATX images are necessary less accurate than raw ones. They are at a different level. But lower level doesn't always necessarily means more accurate. There are some limitations and issues with ATX images, and that is mostly my own fault. We have some ideas for enhancement the format. Phaeron also has some ideas. Something will be implemented sooner or later. Most urgent is to add support for MFM disks.

- What really IMHO, matters is human testing and verification. You have seen that several images have problems or don't work. But this is mostly because Farb decided to release images even without a thorough verification. And I think, now, it was a very wise decision.

As most people know, the ATX specifications were not public for quite some time. Honestly, I did that on purpose. My idea was to keep tight control until most titles were preserved and verified. I tried to not release any image unless I was quite certain it was not only working, but it was absolutely unmodified as it came from factory. It might have been a good idea, but it didn't work out so well. On retrospective, an open approach would have been much better. And releasing not so verified images is part of an open approach.

So if there was any problem in the conversion process from RAW images, you will likely catch it after verification. Yes, verification can't never be perfect, and there is a small chance that even a verified image has a problem that you see only later. But this could happen with raw images as well. Having a raw image is no guarantee at all. Actually, a big percentage of the raw images are bad for one reason or the other.

- I have considered implementing software for compacting RAW images. Most images include lots of extra tracks. Many have more than 160 tracks, while the typical disk has only 40. Furthermore, those tracks usually have just noise, which compress very bad. So it would be possible to remove the unused tracks. It is also possible to reduce the number of revolutions. Making an image with 5 revolutions is not a bad idea just in case. But once you processed and verified, most revolutions are redundant.

- It is possible to use an intermediate format. Not fully decoded as in ATX, but neither raw flux transitions. I actually used internally something like that. I have a low level image format that typically doesn't take much more space than an ATX image. I considered proposing using that format, I even have many images on such format. But, oh well ...

All the technical issues aside, I don't think it's our main concern at this time. Our main problem now is lack of contribution. If you read the list that Farb posted not long ago, you can see that the number of titles that were confirmed from multiple redundant dumps is very small. And even unverified or unconfirmed images, lots of rather common titles are still missing. You should expect that at this point we only miss rarities, but we are very far from that.

Lastly, I think that the human factor is the most important one. I personally cooperated with multiple preservation projects in multiple platforms. I don't want to bother you will all the historical and anecdotic details of the projects. But trust me, you might have the best technical implementation, but that is only secondary. What really make a difference are the attitude, the perseverance and the motivation of the people that lead the preservation project. In this case, it is all about Farb. He took the entire burden on his shoulders. I know he even spends money from his own pocket to acquire disks and dumps them himself.

I would say then, that we should mainly support Farb. All suggestions and recommendations, even constructive critic, are valid, of course. But mostly, let's support Farb.

[_The Doctor__]

I support both Farb and Glowing Ghoul as I believe Farb said space was a big factor and time was secondary but tight at the moment in so many words....

 

Ghoul has offered free hosting with no space constraints.... now that's a wonderful thing, a proper repository to back up the torrent's atx files and the kryo flux or raw images sounds pretty cool....

 

We understand that none of them are perfect... the more of us that get involved the better off all of it will be... I personally know what getting hit by the bus is like.... when someone helps themselves to all you own and sells it for top dollar on ebay.... the whole valet seller thing is bs, they are basically fences ... sorry bout that... stay on topic stay on....

 

In any event, we need a checklist of what people need to help.... what types of hardware is required or acceptable.... is bit writer good enough? lots of questions like are vapi's good enough etc. If only the top of the page in the forum's thread could hold a quick checklist or how to link and the current titles.. Do we need a bonafide kryo system etc... I tried to get an up to date spread sheet idea but when it updates randomly or not at all what do we do? I love what Farb has done and I like to think earlier conversations lead to this wonderful thing we have... maybe we need a reliable constantly updated spread sheet with needed title high light so we can concentrate on each title in a more ordered way...so everyone looks for and sends a particular title at once....and any title that is not yet in the database be accepted at any time....all are welcome but slightly directed to help get it done.

 

ijor probably knows what I am trying to say... as he has worked on the same projects for other platforms...