Using XL memory without breaking stuff?

[itaych]

I'm considering doing a little more work on Ice-T. Trouble is, I'm out of memory. The only remaining unused memory is that under the OS ROM and I'd like to make use of it. Why did I never use it before, you are surely asking? The reasons are several:

1. I didn't need it. XE banked memory provided far more than what was necessary and was the easiest to use.
2. I didn't know how to do it without instantly crashing the system.
3. Even if I did it, I wouldn't know what areas in memory would be safe to use without breaking disk, printer and serial port I/O, all of which must remain available. The OS VBI routine needs to keep running, the keyboard interrupts need to be serviced as usual. There are also probably a few things I've forgotten or that I'm not even aware of that need to keep working.
4. Some working environments (various DOSes, custom OSes, etc) themselves use the XL memory for their own purposes, making questions 2 and 3 even harder to answer. I also need to worry about not breaking them.

Well, issue 1 is no longer correct. For issue 2 I could probably do some research and figure it out (I'd still appreciate getting the answer here), but 3 and 4 have me stumped.

 

Of course if these issues have been discussed elsewhere I'd love a link. Cheers

[flashjazzcat]

When designing TLW (which uses RAM under the OS), I followed the example set by Turbo BASIC XL, which uses wrappers for interrupt handlers which switch the OS ROM in before jumping through the original vectors and then back out again before returning. Just an INC PORTB on the way in and DEC PORTB on the way out. If you need some example code I can probably dig some out, but the methodology is extremely simple, and allows use of $C000-$CFFF, $D800-$FFFF without any restrictions whatsoever.

 

Of course you rule out use of disk-based SpartaDOS using this method, but in my experience this hasn't been a major issue and the word processor remains in high esteem and broad usage. SDX can run in extended RAM, leaving the Shadow RAM untouched, so I see no good reason not to use RAM under the OS.

 

Only thing to remember is to assemble the code running in the Shadow RAM so it loads at - say - $4000 and gets copied to the target address during the load process after you've set up your interrupt wrappers.

[itaych]

This may make sense when the only interrupts in the system are the VBI and the keyboard, but in my situation I need maximal performance while there is constant I/O going on, and as far as I understand there is an interrupt for every single byte incoming over the serial port. This happens >1000 times/sec and I worry that adding even the slightest overhead will slow the system down noticeably. I'd have to see what the ROM does and implement the same thing myself. (it might just call the R: handler which is already in RAM so perhaps this is not so difficult.)

 

Also any IOCB call to disk/printer/serial/keyboard (gotta love those clicks) must also switch PORTB, and must itself not be located in the OS RAM - so this is definitely not a "set and forget" solution with zero restrictions.

 

As for SDX, well in my scenario the extended RAM is already taken so SDX gets completely ruled out. Not sure how happy the users would be about that.

[flashjazzcat]

Serial interrupts are no problem at all, since any IO running fast enough to cause issues with the overhead of writing to PORTB would probably eschew IRQs anyway and use polling.

 

You're correct about CIO and SIO calls as well (forgot to mention those), but again: the code is lightweight and works.

 

SDX requires at minimum one 16K bank in extended mode, so the only way you'd rule it out is if you used all four banks on a 130XE for your own purposes. Larger RAM capacities are extremely common these days. I got around the issue by making extended RAM usage scalable and "SDX aware". Mechanisms exist which allow you to a) sense that SDX is present and b) avoid any extended banks it's already using.

 

The wrapper approach is frankly the nearest thing you'll get to a "set and forget" solution. At least I can say I set it a decade ago and subsequently didn't have to give it a further thought. Any alternative approach seems to me burdened with complexity.

Edited July 20, 2018 by flashjazzcat

[itaych]

Serial interrupts are no problem at all, since any IO running fast enough to cause issues with the overhead of writing to PORTB would probably eschew IRQs anyway and use polling.

 

This might be possible if I hardwired Ice-T to run with a specific serial port type, but as it is there's no choice but to use the standard R: device, and there's no way to tell it to disable interrupts and work in polling mode.

[flashjazzcat]

Right, well I've told you how to use the Shadow RAM but if there are abundant reasons not to, you'll have to either reduce the size of your executable or start using overlays. Not sure what else to suggest. The OS IRQ handler is already slow to dispatch so an extra couple of PORTB writes and a JSR, etc, aren't going to make much difference.

Edited July 20, 2018 by flashjazzcat

[itaych]

Sorry if I sound like an idiot, what are these overlays you speak of?

 

And where did you find information on how TurboBASIC works? Did they publish example code for this technique? Thanks.

[flashjazzcat]

Not at all. By overlays, I just mean splitting up your application so that it pages chunks in and out from disk. Unworkable unless you insist that the program is launched from a fixed disk or RAMdisk, and not a solution I went near.

 

Unless you suddenly need an extra 14K of storage, code refactoring can yield good results. When I was updating the SIDE loader, for instance, I hit the wall at 16K but later released 600 bytes in ROM by using colum-based RLE compression on some of the bitmaps. In fact I spent about fifty per cent of the development cycle looking for extra space in ROM, and probably recovered well over 1KB via laborious iteration.

 

As for Turbo BASIC: I could find no information at all so just dissected it using (at the time) the Atari800WinPlus debugger and DIS6502. Here's a chunk of the TLW output listing:

  1135 86CB			inthand
  1136 86CB 2C 0F D4			bit nmist
  1137 86CE 10 03			bpl notdli
  1138 86D0 6C 00 02			jmp (vdslst)
  1139 86D3			notdli
  1140 86D3 48				pha
  1141 86D4 8A				txa
  1142 86D5 48				pha
  1143 86D6 A9 87			lda #> exitint
  1144 86D8 48				pha
  1145 86D9 A9 0E			lda #< exitint
  1146 86DB 48				pha
  1147 86DC BA				tsx
  1148 86DD BD 05 01			lda $0105,x
  1149 86E0 48				pha
  1150 86E1 D8				cld
  1151 86E2 48				pha
  1152 86E3 8A				txa
  1153 86E4 48				pha
  1154 86E5 98				tya
  1155 86E6 48				pha
  1156 86E7 AD 01 D3			lda portb
  1157 86EA 8D 0D 82			sta pbsave
  1158 86ED 09 01			ora #1
  1159 86EF 8D 01 D3			sta portb
  1160 86F2 8D 0F D4			sta nmires
  1161 86F5 6C 22 02			jmp (vvblki)
  1162 				;
  1163 86F8			irqhand
  1164 86F8 48				pha
  1165 86F9 A9 87			lda #> exitirq
  1166 86FB 48				pha
  1167 86FC A9 10			lda #< exitirq
  1168 86FE 48				pha
  1169 86FF 08				php
  1170 8700 AD 01 D3			lda portb
  1171 8703 8D 0D 82			sta pbsave
  1172 8706 09 01			ora #1
  1173 8708 8D 01 D3			sta portb
  1174 870B 6C 16 02			jmp (vimirq)
  1175 				;
  1176 870E			exitint
  1177 870E 68				pla
  1178 870F AA				tax
  1179 8710			exitirq
  1180 8710 AD 0D 82			lda pbsave
  1181 8713 8D 01 D3			sta portb
  1182 8716 68				pla
  1183 8717 40				rti
  1184 				;
  1185 8718			vbion
  1186 8718 AD 36 02 8D 46 87 + 	mwa $236 bk_vec
  1187 8724 A9 2F 8D 36 02 A9 + 	mwa #bkint $236
  1188 872E 60				rts
  1189 				;
  1190 872F			bkint ; break key interrupt
  1191 872F A9 00			lda #0
  1192 8731 85 D0			sta macflag
  1193 8733 8D 01 83			sta macsaveflag ; disable re-activation of macro
  1194 8736 85 E6			sta minpflag
  1195 8738 85 E7			sta markflag ; block off
  1196 873A 85 11			sta $11
  1197 873C 8D FF 02			sta $2ff
  1198 873F 8D F0 02			sta $2f0
  1199 8742 85 4D			sta $4d
  1200 8744 68				pla
  1201 8745 40				rti
  1202 				;
  1203 8746			bk_vec
  1204 8746 FF FF			.word $FFFF
  1205 				;
  1213
  1214 874F			ciov
  1215 874F 48				pha
  1216 8750 AD 01 D3			lda portb
  1217 8753 8D 9C 82			sta portbsav
  1218 8756 09 01			ora #1
  1219 8758 8D 01 D3			sta portb
  1220 875B 68				pla
  1221 875C 20 56 E4			jsr $e456
  1222 875F 48				pha
  1223 8760 08				php
  1224 8761 AD 9C 82			lda portbsav
  1225 8764 8D 01 D3			sta portb
  1226 8767 28				plp
  1227 8768 68				pla
  1228 8769 C0 00			cpy #0
  1229 876B 60				rts

This code was originally assembled using the native MA65 assembler I wrote in the 90s and then ported to DASM and then to MADS, so it's frankly a mess, but you should get the general idea. The initialisation code disables all interrupts, turns the OS off, writes IRQ and NMI machine vectors to RAM, then stuffs all the code underneath the OS.

[baktra]

And how much extra memory do you actually need?

If it is not that much, you can copy whole ROM to RAM and use for example:

 

Area for international character set CC00-CFFF (1 KB)

Floating point package D800-DFFF (2 KB)

 

If you don't need those, you can get 3 KB almost for free.

[itaych]

An extra 3 kb could be great, though I'm worried that (a) these addresses are not necessarily correct for all OS versions including third party ones, and (b) I'll break SDX.

[flashjazzcat]

Is there any initialisation code which can be completely jettisoned once the program's been set up? TLW has quite a large segment which does stuff like parsing the command line, getting environment variables, and figuring out how much RAM is installed, and that segment gets overwritten with resident code since it's not subsequently required. Of course you may already be doing this: I don't know.

[itaych]

There is a startup routine that gets overwritten but most of the init code must remain resident because Ice-T was designed to survive a push of the Reset button. But perhaps some of it is single use, I'll look into it. Thanks!

Edited July 20, 2018 by itaych

[flashjazzcat]

TLW is reset-proof as well but it's probably a bit heavier on the DOS and startup stuff. Hope you can eke out some space!

[+Stephen]

Can we ask what your plans for updating Ice-T are? I absolutely love this program, it's the only thing I use for BBSing on my Ataris.

[itaych]

Considering the fact that I'm out of memory, not much! I did want to iron out some things and make an 'official' (i.e. not alpha) release, but I haven't found time to deal with it in years and the memory issue has been frustrating me. I think in the alpha release Ymodem-G will hang because the popup notification window with the warning message, will cause a buffer oob check to fail - probably no one uses it so no one cares, but it's the sort of thing that I wouldn't let slip by in a real release.

@flashjazzcat, I've reviewed the sources, all single use init code is overwritten after running.

[baktra]

An extra 3 kb could be great, though I'm worried that (a) these addresses are not necessarily correct for all OS versions including third party ones, and (b) I'll break SDX.

 

The floating point package cannot go anywhere else, so it would be safe.

 

For sure, there are OS ROMs that sacrifice the international character set for something more useful. By using RAM under ROM, you possibly break everything that uses RAM under ROM. That includes SpartaDOS, of course.

[itaych]

This is really annoying. Did the authors of SpartaDOS just assume that no one will ever want to use that extra memory? Isn't there some system call that could tell DOS to release that memory, and reload that code from disk when the application exits (like most other DOSes do)?

[baktra]

This is really annoying. Did the authors of SpartaDOS just assume that no one will ever want to use that extra memory? Isn't there some system call that could tell DOS to release that memory, and reload that code from disk when the application exits (like most other DOSes do)?

 

Unfortunately, that might not be easy. SpartaDOS is an advanced DOS that consumes considerable amount of RAM. More sophisticated file management system requires more and longer control blocks, for example. The shell and built-in commands require more RAM too. So the SpartaDOS creators had to make a choice. Either torture the user with intensive disk access, or sacrifice RAM under ROM. The device drivers (useful feature) consume even more memory, but can be loaded and unloaded as needed.

 

A DOS that tried to address the problem is BW-DOS. A SpartaDOS clone that doesn't use RAM under ROM and keeps the same LOMEM as SpartaDOS. But lot of commands that were built-in were offloaded to standalone programs stored on disk. So doing complex operations required lot of beeping and two drives were preferred. Not so bad, BW-DOS was for users running serious tools and utilities, not gamers.

 

A bank-switching cartridge is of a great help in this case.

[flashjazzcat]

I imagine the SpartaDOS developer simply evaluated all the pros and cons, chose a solution and stuck with it, just as you will have to do. You can over think these things. Use the Shadow RAM and alienate disk-based Sparta and 800 users. Rely on >128K and alienate 130XE owners. If the software is good, people will compromise. I never received a single complaint about TLW's Shadow RAM usage once it boasted an 80 column editor.

[_The Doctor__]

depending on the terminal.... and who I am calling or playing with... I am completely silly... BBS geek

A Modem

X Modem (crc flavors)

Y Modem

F Modem

L Modem

Z Modem

Kermit

though most are incorporated into the same blob as they share most of the same pieces and parts to get the job done... XYFL windowed share quite a bit

Edited July 21, 2018 by _The Doctor__