IntyBASIC/JLP - guarantees on storage capacity?

[petit chou]

Hey all,

 

There isn't a whole lot of technical information on the JLP flashcarts in the IntyBASIC manual, so I wanted to ask some questions for clarification. I'm wanting to store some (overly) complicated data structures in the onboard flash memory and need to know some specific information:

 

* Are FLASH.FIRST and FLASH.LAST guaranteed to point to the same place every time the cart runs?

* Can I trust that there will always be a certain amount of continuous storage available on the cart? (i.e., will FLASH.LAST - FLASH.FIRST ever change?)

* How fast are the FLASH READ and FLASH WRITE operations? Will it be approximately the same speed on real hardware as it is in jzintv?

* Is using flash memory damaging to a physical cartridge over time or can I read and write as much as I'd like to without worrying?

 

Thanks a ton in advance!

 

(edit: fixed a typo)

Edited May 10, 2016 by petit chou

[intvnut]

I really need to go through and polish up the JLP Programmer's Guide, since it answers all of these questions more or less.

 

Some background: On a real JLP board, FIRST and LAST are fixed by two details: FIRST is determined by the first flash sector on JLP not used by the game ROM image or the JLP firmware itself. LAST is simply the last physical sector on the device. So, once the ROM image is blown into flash, FIRST and LAST never change.

 

That means:

• Whatever FIRST and LAST are the first time your program runs is also the value they will have every time it runs thereafter.
• LAST - FIRST, therefore, is also fixed.

The flash read and write commands aren't guaranteed to be fast. On a real JLP board, the following is a useful guide:

• Reads are fairly quick. Probably a few dozen CPU cycles, however I haven't benchmarked it.
• Writes can take up to ~2ms. (This is based on the datasheet numbers for writes taking 1.3ms to 1.8ms, approx., plus the overhead of making the call.)
• Erasures can take up to ~27ms. (Again, based on the datasheet numbers.)

jzIntv has very, very rough approximations for these timings, plus some purposeful randomness thrown in. The timings there are meant mainly to expose bugs, and serve as an extremely rough guide.

 

Reads don't wear the flash. Erasures, however, do. And, once you write to a row, you can't rewrite it without erasing it first. Each flash sector is good for 10,000 erase/write cycles. So, you likely want to consider doing the following:

• Write only when something's actually changed.
• Spread your writes around a ring of sectors. Embed a "version number" in your data so you can scan to find the "newest" data when reading.
• Write to flash only at points where small pauses won't be noticed in the gameplay, such as natural stopping points between levels, etc., or at explicit save points. Remember, you may need to erase something before you can write, so include both costs in your budgeting.

For example, if you spread your writes around 16 separate locations, then you're good for 160,000 E/W cycles, but your capacity drops accordingly.

 

The flash save area capacity on a JLP board is a function of the game's size. The board's total flash capacity is 256K bytes. Assume the firmware takes 16K. (Right now, it actually only takes ~5K, but let's give some headroom.)

 

The game also takes up space almost exactly equal to its .BIN size, after rounding up each 4K-word page to exactly 4K words. For example, if you have 1 word at location $5000, I still allocate $5000 - $5FFF. But if you fill $5000 - $5FFF, it takes just as much space in JLP.

 

So, for example, Space Patrol is 16Kwords, filling $5000 - $6FFF, $D000 - $DFFF and $E000 - $EFFF. It requires 32K bytes of flash ROM. So, the flash available after that is (256K - 16K (firmware) - 32K (Space Patrol)) = ~208K available for the save area.

 

 

And then there's LTO Flash!

 

LTO Flash! emulates the JLP save-game feature. However, because it's an emulation, it's slower. It's reading from external flash through a filesystem layer and a flash translation layer. Reads are typically ~8ms. Writes are typically ~9ms. Erasures are ~20ms. (Note: These timings could improve in future firmware releases. Right now, I have some very conservative code in there, preferring correctness to performance.)

 

There's an additional caveat with writes and erasures, though. Both writes and erasures actually could trigger a block allocation in the filesystem. If there's lots of free, clean space, this goes very fast. This should be the most common case by far. If the filesystem is very full, and doesn't have a clean page handy, that allocation could trigger a garbage collection cycle, and garbage collection can be quite costly.

 

This should be rare: LTO Flash! performs background cleanup operations whenever it can, so it always has pages free to allocate. However, if your game sends a lot of writes and erasures, you could create more "garbage" than there is free space. In practice, I've only seen big pauses when I've purposefully set it up to behave badly: Disable background cleanup, very full filesystem, and a purposefully-evil access pattern that leaves lots of fragmentation.

 

So I'll reiterate the design recommendation above: Only write or erase flash at points where small pauses won't be noticed. And I'll add: don't read if you can't spare half a frame-time.

 

On the plus side, LTO Flash! stores your save area as a separate data fork, and that data fork can be up to 1MB. It's not dependent on your game's size at all. You can specify the number of 1.5K flash sectors your game needs in the CFG file for the game, and that actually determines FIRST/LAST reported by LTO Flash! So, if you develop a game too big for JLP, you can still distribute it for LTO Flash! and have tons of storage.

Edited May 10, 2016 by intvnut

[petit chou]

This was absolutely amazing. Thank you so much for taking the time to give such a detailed and informative reply!

[+DZ-Jay]

One thing to be aware of is that the LTO save-state implementation is specific to the LTO Flash and PCB boards. Of course, then there is jzIntv, which supports it also; but consider that it is mostly a developer's tool. Although many non-developers use it, a lot of people still use Nostalgia or Bliss as well.

 

Now, none of this is a problem, of course, but just make sure you are aware that the feature is only available on those technologies.

 

By the way, that only applies for non-volatile RAM exclusively for saving state. Most emulators and PCB boards available (if not all) already support 30K, 42K, or more of on-board cartridge memory that can be used for game program code (ROM) and volatile game state (RAM). NVRAM has not yet standardized, that's all.

 

All that said, I've used the LTO save-state implementation in my own game and it is mostly transparent, though like Joe said, you should use it sparsely and only in places were timing is not crucial (like during level initialization or completion).

 

-dZ.

Edited May 10, 2016 by DZ-Jay

[petit chou]

One thing to be aware of is that the LTO save-state implementation is specific to the LTO Flash and PCB boards. Of course, then there is jzIntv, which supports it also; but consider that it is mostly a developer's tool. Although many non-developers use it, a lot of people still use Nostalgia or Bliss as well.

 

Now, none of this is a problem, of course, but just make sure you are aware that the feature is only available on those technologies.

 

By the way, that only applies for non-volatile RAM exclusively for saving state. Most emulators and PCB boards available (if not all) already support 30K, 42K, or more of on-board cartridge memory that can be used for game program code (ROM) and volatile game state (RAM). NVRAM has not yet standardized, that's all.

 

All that said, I've used the LTO save-state implementation in my own game and it is mostly transparent, though like Joe said, you should use it sparsely and only in places were timing is not crucial (like during level initialization or completion).

 

-dZ.

 

Thank you for the response. A question: is there a way in IntyBASIC to access the extra RAM storage one of these carts would use? Would I need to manually declare an area as write-enabled with an assembly directive and PEEK()/POKE (or equivalent) at the memory locations? I'm trying to write a roguelike and need to hold quite a bit of information in memory to make the level states at least semi-persistent.

 

I'd really love for my game to run in Nostalgia. jzintv is great, but my vision is terrible and its windowed mode only goes up to 640x480.

[freewheel]

 

Thank you for the response. A question: is there a way in IntyBASIC to access the extra RAM storage one of these carts would use? Would I need to manually declare an area as write-enabled with an assembly directive and PEEK()/POKE (or equivalent) at the memory locations? I'm trying to write a roguelike and need to hold quite a bit of information in memory to make the level states at least semi-persistent.

 

When you compile with --jlp in IntyBASIC, you'll find that you get an extra 8000 16-bit variables to play with. Which "should be plenty", but you never know. Just address it like any other variables - probably with arrays, given the size.

[petit chou]

 

When you compile with --jlp in IntyBASIC, you'll find that you get an extra 8000 16-bit variables to play with. Which "should be plenty", but you never know. Just address it like any other variables - probably with arrays, given the size.

 

Sorry, I may not have been clear. I meant to say, if I don't want to use JLP features so that I can run my game in Nostalgia or Bliss, can I still use the extra RAM somehow from a modern board?

[freewheel]

 

Sorry, I may not have been clear. I meant to say, if I don't want to use JLP features so that I can run my game in Nostalgia or Bliss, can I still use the extra RAM somehow from a modern board?

 

Ah, that makes sense. I'm not sure if the extra memory is standardized across the various boards/emulators. I've thought about this myself - if I write for JLP/jzintv/LTO Flash!, things "just work". It would be nice to be able to strip out the flash save ability and at least make a ROM of the game that "the rest of the world" can use. Without a large rewrite.

[intvnut]

 

I'd really love for my game to run in Nostalgia. jzintv is great, but my vision is terrible and its windowed mode only goes up to 640x480.

 

Download the stable dev release. The windowed mode now goes as large as you like. It's not limited to 640x480. I've personally run it at around 3200x2000 across two monitors, just to do it.

 

 

 

Sorry, I may not have been clear. I meant to say, if I don't want to use JLP features so that I can run my game in Nostalgia or Bliss, can I still use the extra RAM somehow from a modern board?

 

I don't know the granularity of control IntyBASIC has over just declaring the extra memory, vs. enabling all of the JLP functionality.

 

 

As for the other emulators, can someone call Joe Fisher, Kyle Davis, or John Dullea out of emulator-maintenance-retirement to refresh their emulators? :-) I'll happily work with them to implement or integrate new functionality into their wares. Also, I don't know who maintains the MAME Intellivision module, but I'd be willing to work with them, too. (I even just relicensed a version of my speech synth emulation to BSD for the MAME crowd.)

[petit chou]

 

Download the stable dev release. The windowed mode now goes as large as you like. It's not limited to 640x480. I've personally run it at around 3200x2000 across two monitors, just to do it.

 

You have no idea how excited that makes me. Thank you!

 

 

 

I don't know the granularity of control IntyBASIC has over just declaring the extra memory, vs. enabling all of the JLP functionality.

As for the other emulators, can someone call Joe Fisher, Kyle Davis, or John Dullea out of emulator-maintenance-retirement to refresh their emulators? :-) I'll happily work with them to implement or integrate new functionality into their wares. Also, I don't know who maintains the MAME Intellivision module, but I'd be willing to work with them, too. (I even just relicensed a version of my speech synth emulation to BSD for the MAME crowd.)

 

I know that Nostalgia at the very least supports the Intellicart standard, and it seems like it might be possible to hoodwink IntyBASIC into playing along with bankswitching by putting a label at the beginning of the initial memory for that area and just treating the bankswitched area like a gigantic array, but I don't have the technical know-how to check for sure.

 

Unfortunately I don't know any of those developers, but this definitely seems like the right place to check! Maybe we'll get lucky and someone else around here knows them and could put you in touch.

[intvnut]

 

You have no idea how excited that makes me. Thank you!

 

It also supports the Scale2x, Scale3x and Scale4x "smoothing" filters, if you like those effects.

 

 

Unfortunately I don't know any of those developers, but this definitely seems like the right place to check! Maybe we'll get lucky and someone else around here knows them and could put you in touch.

 

I know John still posts on here (JohnPCAE). I think he took IntvWin offline a long time ago. I remember the last time I looked for it, I couldn't find it.

 

Joe Fisher got into a different kind of homebrew, and that hobby grew into Man Skirt Brewing Company. I think he's parted with a lot of his classic gaming stuff, or at least the microcontroller stuff he used to build the Classic Gaming Controller. I know I ended up with some of the latter. (Thanks, Mr. Fisher!) If nothing else, I suggest stopping in for a brew if you're in his neck of the woods.

 

As for Bliss, I believe the source for the last version is online somewhere. If someone wants to take it up as a modern maintainer, that's probably the best bet to get it updated.

[+DZ-Jay]

 

Ah, that makes sense. I'm not sure if the extra memory is standardized across the various boards/emulators. I've thought about this myself - if I write for JLP/jzintv/LTO Flash!, things "just work". It would be nice to be able to strip out the flash save ability and at least make a ROM of the game that "the rest of the world" can use. Without a large rewrite.

 

The extra memory is pretty much standard (they don't make 4K or 8K ROMs any more), you just need to map it in your ROM to usable memory on the Intellivision. Even the CC3 supports it.

 

I believe IntyBASIC uses the "cart.mac" standard memory map, which supports up to 42K of RAM. You just need to switch segments manually using that "ASM ORG" trick people talk about.

 

As I said before, there is a distinction between RAM and NVRAM. The latter is a feature of the JLP only, so far.

 

-dZ.

Edited May 11, 2016 by DZ-Jay

[intvnut]

 

The extra memory is pretty much standard (they don't make 4K or 8K ROMs any more), you just need to map it in your ROM to usable memory on the Intellivision. Even the CC3 supports it.

 

I believe IntyBASIC uses the "cart.mac" standard memory map, which supports up to 42K of RAM. You just need to switch segments manually using that "ASM ORG" trick people talk about.

 

As I said before, there is a distinction between RAM and NVRAM. The latter is a feature of the JLP only, so far.

 

-dZ.

 

For the ROM memory map, that's almost entirely under the programmer's control with ASM ORG. You're not actually constrained to the cart.mac 42K map. You could go up to 50K if you don't need extra RAM, and aren't willing to page-flip. You can go higher if you want to page-flip, although that's obviously a bit more fun...

 

For the RAM memory map, I just had a look: The --cc3 flag declares extra RAM similarly to the --jlp flag, but without assuming the existence of other JLP features (MPY/DIV accelerators, flash/NVRAM support). The RAM requested by this flag should work across most (all?) emulators. I think 5-11under's board design may support that RAM also. (When I posted earlier, I didn't know for sure if there was a separate flag. I've had a chance to look at the IntyBASIC manual since then.)

 

I think that's the question petit chou was asking upthread when he asked about extra memory: Can we count on having extra RAM across a variety of emulators and boards, and get to that RAM from IntyBASIC. I think the answer is yes.

[+DZ-Jay]

 

 

I think that's the question petit chou was asking upthread when he asked about extra memory: Can we count on having extra RAM across a variety of emulators and boards, and get to that RAM from IntyBASIC. I think the answer is yes.

 

I know that's what he asked, since he was responding to my comment. The answer is yes -- as long as he understands (and I believe he does) the difference between the extra RAM available in most (if not all) emulators and PCB boards and the NVRAM exclusive to the JLP/LTO Flash.

 

I was just pointing out that the latter is not a requirement for the former.

 

-dZ.

[GroovyBee]

NVRAM exclusive to the JLP/LTO Flash.

Its also in Bee3, but I haven't done the IntyBASIC support package for it yet.

[petit chou]

 

The answer is yes -- as long as he understands (and I believe he does) the difference between the extra RAM available in most (if not all) emulators and PCB boards and the NVRAM exclusive to the JLP/LTO Flash.

 

I've got it. Thank you so much.

 

Do you happen to know what areas of memory are used by IntyBASIC? I've been looking at the memory map but I can't tell from the assembly output (my assembly skill is... questionable) what areas are being used and what are not.

[+Tarzilla]

 

I've got it. Thank you so much.

 

Do you happen to know what areas of memory are used by IntyBASIC? I've been looking at the memory map but I can't tell from the assembly output (my assembly skill is... questionable) what areas are being used and what are not.

No black and white answer to that, as shown here, courtesy of a thread you appear to have started

 

http://atariage.com/forums/topic/240347-lots-of-technical-questions-about-the-org-directive-and-memory/

 

We have discussed it many times, but unfortunately it gets lost in among the other discussions.

[petit chou]

No black and white answer to that, as shown here, courtesy of a thread you appear to have started

 

http://atariage.com/forums/topic/240347-lots-of-technical-questions-about-the-org-directive-and-memory/

 

We have discussed it many times, but unfortunately it gets lost in among the other discussions.

 

That is hilarious. I thought I'd wondered about this before! Thanks for finding that for me.

[+Tarzilla]

For newbies looking at the lst file can be overwhelming, here is a simple way of looking at it:

 

Look for the first ORG statement, in this case ORG $5000

 

By default IntyBASIC starts at the $5000 location with an allowed range of $5000 thru $6FFF

See the numbers down the side-5000,5002,5004, that is the memory being used up, eventually you'll get to $6FFF

Just before you do, you'd look at where in your IntyBASIC code you are and insert an ASM ORG $xxxxx where xxxxx is one of the other memory holes such as $D000 (with a range thru $DFFF)

in the new lst, you'll be able to find the reference to $D000, watch the numbers down the side until they get near $DFFF

Consult the 42k.bas file by groovybee included with the recent versions of IntyBASIC for a good guide.

                    ROMW 16
0x5000                          ORG $5000

                    ; This macro will 'eat' SRCFILE directives if the assembler doesn't support the directive.
                    IF ( DEFINED __FEATURE.SRCFILE ) = 0
                        MACRO SRCFILE x, y
                        ; macro must be non-empty, but a comment works fine.
                        ENDM
                    ENDI

                    ;
                    ; ROM header
                    ;
5000 000D 0050                 BIDECLE _ZERO           ; MOB picture base
5002 000D 0050                 BIDECLE _ZERO           ; Process table
5004 0027 0050                 BIDECLE _MAIN           ; Program start
5006 000D 0050                 BIDECLE _ZERO           ; Background base image
5008 000F 0050                 BIDECLE _ONES           ; GRAM
500A 001E 0050                 BIDECLE _TITLE          ; Cartridge title and date
500C 03C0                 DECLE   $03C0           ; No ECS title, jump to code after title,

[petit chou]

 

For newbies looking at the lst file can be overwhelming, here is a simple way of looking at it:

 

 

Okay, this is super helpful. Thank you so much!

[GroovyBee]

Do you happen to know what areas of memory are used by IntyBASIC?

Check out the 42k.bas example in the IntyBASIC contrib folder. The 42K memory map is detailed in its comments too.

[+DZ-Jay]

Check out the 42k.bas example in the IntyBASIC contrib folder. The 42K memory map is detailed in its comments too.

 

I wonder if we should create IntyBASIC macros to encapsulate the segment switches? If we consider that there's, say, five segments, we could have a macro that says "SwitchRomSegment(x)" and automatically insert the "ASM ORG $xxxx" appropriate for the argument.

 

It's not automatic, but at least it'll be easier to explain to programmers. Instead of Tarzilla's detailed but highly technical explanation above, you can say:

 

 

There are X number of ROM segments available for your program and code. The default is Segment #1. If your code and data exceed size of the segment, just call "SwitchRomSegment(x)" with next consecutive segment number somewhere in your code.

 

 

What do you think?

 

Of course, it would help if IntyBASIC gave an appropriate error when code exceeds the current segment block.

 

-dZ.

Edited May 12, 2016 by DZ-Jay

[GroovyBee]

I wonder if we should create IntyBASIC macros to encapsulate the segment switches? If we consider that there's, say, five segments, we could have a macro that says "SwitchRomSegment(x)" and automatically insert the "ASM ORG $xxxx" appropriate for the argument.

I think the DEF FN command would need to support some kind of preprocessing directives e.g. #if/#else/#endif, etc. for that to work because you don't want to inject an executable code stream when its used.

[+DZ-Jay]

I think the DEF FN command would need to support some kind of preprocessing directives e.g. #if/#else/#endif, etc. for that to work because you don't want to inject an executable code stream when its used.

Hmm, I don't understand what you mean? Which executable code?

[GroovyBee]

Hmm, I don't understand what you mean? Which executable code?

You wouldn't want CPU executable code to be emitted while you checked parameter x (from your example) to decide which asm org $xxxx directive to emit. You'd want that done by the equivalent of C/C++ preprocessor directives.