Just throwing this out there. I’ve no idea if it will work for what you are doing.
There’s a higher spec’d (faster) 9900 built into the F18A video adapter. I’ve always wondered if it could be used as some sort of an accelerator.
Someone around here has the source for Myarc XB, I believe. I thought it was somewhat backwards compatible, and also written in assembly.
I don't know if it would be a good starting point, but maybe worth a look.
Are you using a TL866 or some variation of? There is a "Check ID" checkbox in the GUI that verifies that the chip is the correct type. Unchecking the box may allow you to program the 2764. From the manual