A simple resistor and capacitor on the switch tied to the LOAD input would have provided some simple hardware debounce and the software loop would probably not be necessary.
The problem with this approach for Vorticon's purposes is that the LOAD is being driven by a microcontroller, so the input is not bouncing.
If that is the case then a timeout is not going to solve this, interrupts will continue to come. Several things come to mind:
1. Disable interrupts immediately upon entering the routine, and re-enable them when you are done. Keep in mind though, that if the interrupts are coming in fast and furious, it could easily overwhelm the 99/4A. As soon as you are done with the routine and re-enable interrupts, another could be right there waiting.
2. Modify the SmallyMouse code to limit the interrupt rate. The code is available, I had to compile it myself and load it on the microcontroller.
3. You could put a triggered one-shot between the interrupt and the 99/4A to limit the interrupt rate. Once an interrupt triggers the one-shot, any more interrupts are blocked until the one-shot timeout, which you could adjust to something sensible (once every 30ms or so would be a mouse update every two video frames).
You are absolutely correct Matt. I just thought the debouncing in software trick was neat.
Unfortunately, there is no way to disable the LOAD interrupt as it is umaskable. As for modifying the SmallyMouse code, it would be easier to just use an arduino with USB support like the Teensy++ and just decode the USB packets coming from the mouse (they have a standard format) and output direct positional data to a few pins without having to worry about quadrature encoding, phase shifts and all that messy stuff. That way a simple connection to the PIO port will work perfectly. This is actually what I want to experiment with next. The SmallyMouse was really designed to connect to vintage computers already equipped with a mouse port (Amiga, Atari ST, Electron etc...) and I have found that it's more pain than it's worth to try adapting it to mouseless systems. As to your third option, it requires additional hardware which would defeat the purpose...