fujinet #FujiNet - a WIP SIO Network Adapter for the Atari 8-bit

[tschak909]

Any Ardiuno/esp hackers:

 

I have checked in a new SIO state machine design:

https://github.com/FujiNetWIFI/atariwifi/blob/master/esp32/tests/newstatemachine/newstatemachine.ino

 

It is different in that the whole command frame is read in at once, after command assertion, and the rest of the states are collapsed into an acknak and process pair of states.

 

Problem is, this seems to take long enough that the command line re-asserts itself again, and I'm not exactly sure how...

 

(not to mention that sio_checksum is always returning 0x1f, for whatever reason)

 

This, after a 6 hour blast to rewrite the SIO state machine to try and improve reliability and performance, while accomplishing neither.

 

Anybody have any thoughts?

-Thom

 

[ivop]

It's indeed weird that trying to shorten the paths through the state machine, actually made it slower.

 

One thing that made things faster on the atmega328 (albeit a slower CPU and less powerful instructionset) was to ditch the Arduino libraries. Do direct bit manipulations on the hardware registers. Just like you would do on the Atari

 

And previous embedded SIO implementations didn't use a state machine as such. The main loop is basically waiting for the command line to assert and then read stuff and calculate the checksum as fast as possible, and use a switch statement or jump table to react on it rather quickly. I'd assumed a 80MHz ARM would allow for your state machine to keep up with the SIO bus, but perhaps it does not? Or maybe it would if you skip the API layer?

 

 

 

[tschak909]

It's indeed weird that trying to shorten the paths through the state machine, actually made it slower.
 
One thing that made things faster on the atmega328 (albeit a slower CPU and less powerful instructionset) was to ditch the Arduino libraries. Do direct bit manipulations on the hardware registers. Just like you would do on the Atari [emoji4]
 
And previous embedded SIO implementations didn't use a state machine as such. The main loop is basically waiting for the command line to assert and then read stuff and calculate the checksum as fast as possible, and use a switch statement or jump table to react on it rather quickly. I'd assumed a 80MHz ARM would allow for your state machine to keep up with the SIO bus, but perhaps it does not? Or maybe it would if you skip the API layer?
 
 
 

I am going to try implementing the esp32 version via IDF, which is basically directly on top of freeRTOS.

The previous tests do work. I was trying to improve esp32 performance.

Sent from my SM-G920F using Tapatalk

[ivop]

freeRTOS is a whole new direction, but could be worth it I think. Probably a better and faster API to the bare bones. Looks like freeRTOS also runs on the ESP8266

 

[mozzwald]

2 hours ago, ivop said:

Yeah, the AMS1117 is pretty shitty

 

Have been looking for better schottky/blocking diodes, and this is what I came up with so far:

 

19TQ015

 

$0,70 from China (min. 10pcs) or $1,70 at Farnell (1pc)

 

Very low Vf for low currents!

This one should keep SIO's 5V input well above 4.6V, probably even above 4.8V, before entering the AMS1117.

That's pretty good, but the component package is a bit large for my tastes (I ❤️ SMD)  

[tschak909]

must be no larger than a grain of sand.

 

[ivop]

Found some more, but most of them aren't SMD, or are extremely expensive, like $30+ a piece (Microsemi LX2400).

 

This one has an SMD variant: SPV1001

 

Vf at 1A is around 50mV (!)

 

Cost is around $4-$5.

 

I keep looking when I'm bored

 

As for the TO-220 housing, couldn't it lay flat on his back underneath the NodeMCU board? Don't think it'll generate much heat at 1A max.

 

[tschak909]

For any ESP/Arduino hackers that would like to help:

 

We need to improve the speed of the SIO state machine. The marginal timing is causing subtle time-outs.

 

Right now, we are marginal when it comes to the initial ACK of a command frame, and it's different on both the 8266 and the ESP32, on the 8266, we seem to be hitting the receipt of the command frame and ACKing within one command frame cycle at 19200 baud, while on the ESP32, we seem to be missing the first command frame attempt. 

 

We are currently using an interrupt that triggers when CMD falls, which resets our state machine to get the ID.

 

My initial instinct on ACK'ing when we get something both valid and meant for us as soon as possible seems to be the right move, as grabbing the whole command frame and THEN acknowledging it seems to take too many cycles.

 

The current working state machine is here:

https://github.com/FujiNetWIFI/atariwifi/blob/master/esp32/tests/multilator/multilator.ino

 

I attempted a re-write to try and improve performance, here:

https://github.com/FujiNetWIFI/atariwifi/blob/master/esp32/tests/newstatemachine/newstatemachine.ino

and somehow mananged to make it infinitely worse.

 

Any insight that anyone can give on this, would be most welcome.

-Thom

[tschak909]

@48kRAM has been able to wire up an analyzer to the SIO port and start capturing some trace data from Fujinet, trying to see if we're marginal on timings.. this will take some time to sift through.

 

 

I have provided the files here, if anyone wants to sift through them with Sigrok.

 

To get the UART display, add a UART decoder, set baud rate to 19200, and set the RX and TX channels appropriately.

sigrok-captures.zip

[tschak909]

Measuring timing between end of COMMAND frame and resulting ACK response from #FUJINET

[manterola]

Maybe there are some garbage in the Serial buffer you need to get rid off.

You can try something like getting 5 bytes, if checksum fails, get another one. I tried this and it worked perfectly (at least at 19200) with a atr saved in the spiffs. 

BTW, newCmdReady is the volatile var that is changed in the ISR to TRUE  when command line is low.

void recCmdFrame() {
    static byte ndx = 0;
    byte rb;
    while (Serial.available()>0 && newCmdReady == false ) {
        rb = Serial.read();
        if (recInProgress == true) {
            cmdWord[ndx] = rb;
            ndx++;
            if (ndx >= 5) {
                if (checksum(cmdWord,4)==cmdWord[4]) {
                    recInProgress = false;
         digitalWrite(PIN_MCULED, HIGH);
                    ndx = 0;
                    newCmdReady = true;
          Debug_println("passed cksum");
                }
                else {  //bad checksum, let's try reading more bytes as long as cmd low
                    yield();
                    Debug_println("failed cksum");
                    ndx = 4;
                    cmdWord[0] = cmdWord[1];
                    cmdWord[1] = cmdWord[2];
                    cmdWord[2] = cmdWord[3];
                    cmdWord[3] = cmdWord[4];
                }
            }
        }
    }
 

Edited December 25, 2019 by manterola

[tschak909]

@manterola are you testing this with an 8266? Somehow I seem to be blowing right past the first send of the command frame before I ACK, causing a noticeable delay in loading, as can be heard here:

 

ESP32: slower beeps: (you can also hear a slight thpt, as the first command frame blows right past)

 

8266: fast beeps:

 

I love how the audio output can be used as an effective diagnostic tool. You can HEAR the timings and the transition between command and data frames...

-Thom

[manterola]

I'm using the nodemcu you recommended from Amazon.. Amica or something like that. 

As you can see I'm reading the 5 bytes first. After I read 5 bytes I see if checksum matches, if not  I keep reading an additional one as long as cmd line is low. If checksum works, then I tried to process the command frame by checking the device and command and responding ack if those are correct. If checksum does not match I answer nothing. 

Not sure about high speed yet, I would need to keep changing the baud rate and answering to the status command in the right way, I guess. 

But reading the 5 bytes first and then ack'ing works. 

 

[tschak909]

@manterola High speed seems to work for me, by accepting the $3F command, returning the desired POKEY divisor, $0A, and switching to the new baud rate, like in this code:

 

/**
   High Speed
*/
void sio_high_speed()
{
  byte ck;
  byte hsd=0x0A; // US Doubler
//  byte hsd=0x28; // Standard Speed (19200)

  ck = sio_checksum((byte *)&hsd, 1);

  delayMicroseconds(DELAY_T5); // t5 delay
  SIO_UART.write('C'); // Command always completes.
  SIO_UART.flush();
  delayMicroseconds(200);

  SIO_UART.write(hsd);

  // Write checksum
  SIO_UART.write(ck);
  SIO_UART.flush();
  delayMicroseconds(200);

  SIO_UART.begin(52640); // US Doubler
  // SIO_UART.begin(19200); // Standard
}

Ideally, I would keep track of the current baud rate, and flip it when only talking to the disk drives ($31-$38), while flipping back to 19200 for everything else. (High speed device $70 will come later I suspect).

 

Have you tried the Multilator test? (tests/esp32/multilator) ? It works on both 8266 and ESP32 devices.

 

-Thom

Edited December 25, 2019 by tschak909

[evilmoo]

12 hours ago, tschak909 said:

Measuring timing between end of COMMAND frame and resulting ACK response from #FUJINET

Maybe this is completely unrelated, but it seems odd to me that CLKIN keeps running when there's no data being received.

[mozzwald]

10 minutes ago, evilmoo said:

Maybe this is completely unrelated, but it seems odd to me that CLKIN keeps running when there's no data being received.

Weird. CLKIN is not defined in that sketch and should be controlled by the esp8266. 

 

11 hours ago, manterola said:

Maybe there are some garbage in the Serial buffer you need to get rid off.

You can try something like getting 5 bytes, if checksum fails, get another one. I tried this and it worked perfectly (at least at 19200) with a atr saved in the spiffs. 

BTW, newCmdReady is the volatile var that is changed in the ISR to TRUE  when command line is low.

Your function is confusing to me. What decides when to run recCmdFrame()? If ISR sets newCmdReady == TRUE when CMD is low, why are you reading serial when newCmdReady is FALSE? I'm not a seasoned programmer so perhaps I'm missing something. Can you post the complete sketch so this will make more sense?

[Gavin1968]

On 12/23/2019 at 4:11 PM, mozzwald said:

That's pretty good, but the component package is a bit large for my tastes (I ❤️ SMD)  

I prefer SMD as well, I would be surprised if it doesn't come in one?

Actually it seems to

http://images.100y.com.tw/pdf_file/19TQ015.pdf

 

 

Yep - https://www.amazon.com/19TQ015S-Schottky-Rectifying-Diode-D2Pak/dp/B0090BBR9I

Edited December 25, 2019 by Gavin1968

[manterola]

1 hour ago, mozzwald said:

Your function is confusing to me. What decides when to run recCmdFrame()? If ISR sets newCmdReady == TRUE when CMD is low, why are you reading serial when newCmdReady is FALSE? I'm not a seasoned programmer so perhaps I'm missing something. Can you post the complete sketch so this will make more sense?

Sorry my bad...

recInProgress is the volatile var that goes inside the ISR:

void ICACHE_RAM_ATTR isrCmd() {
  if (digitalRead(PIN_CMD) == LOW) 
    recInProgress = true;
  else
    recInProgress = false;    
}

The recCmdFrame() is called all the time, since it is inside the loop() function (Receive Cmd Frame and Process Cmd Frame):

I have re-posted the the function using the code style that I just discoved in this forum.. sorry for the previous post...

void recCmdFrame() {
    static byte ndx = 0;
    byte rb;
    while (Serial.available()>0 && newCmdReady == false ) {
        rb = Serial.read();
        if (recInProgress == true) {
            cmdWord[ndx] = rb;
            ndx++;
            if (ndx >= 5) {
                if (checksum(cmdWord,4)==cmdWord[4]) {
                    recInProgress = false;
         digitalWrite(PIN_MCULED, HIGH);
                    ndx = 0;
                    newCmdReady = true;
          Debug_println("passed cksum");
                }
                else {  //bad checksum, let's try reading more bytes as long as cmd low
                    yield();
                    Debug_println("failed cksum");
                    ndx = 4;
                    cmdWord[0] = cmdWord[1];
                    cmdWord[1] = cmdWord[2];
                    cmdWord[2] = cmdWord[3];
                    cmdWord[3] = cmdWord[4];
                }
            }
        }
    }

}

void loop() {
    if (genmode==MODEM) {
        todo();
    }
    else if (genmode==DISK) {
        recCmdFrame();
        procCmdFrame();
    }

}

Then I created two flags to move thru the machine states: newCmdReady which is initialized in FALSE and set to TRUE once  command frame is ready and validated and recInProgress (receiving process in progress) which starts in FALSE and it is updated by the ISR.

 

I am also attaching the INO , but it is just a simple starting and easy starting point idea of adding a short drive emulation to a modem program emulation, to boot the rverter handler and ICET or BOBterm, and then switch to modem mode for the rest of the time, until next boot. Nothing fancy.. 

I just like the idea of receiving and validating the cmd Frame first and also the idea of trying additional bytes in the serial buffer until the checksum works, instead to give up at the first try. Not sure how effective this strategy is in practical terms.. maybe not useful at all.

 

 

 

mm.ino

[tschak909]

Part of the work I am planning to do outside of the "D:" emulation, focuses on the following aspects:

 

* Creating a new N: device for TCP and UDP communication

* Creating an R: device that actually implements a proper device, instead of just dumping into R-Verter mode.

 

Since we have a large buffer, and the nature of the data we are dealing with is packet based, we do not have to ascribe to the unusual operating mode of traditional Atari RS-232 interfaces.

 

Concurrent mode can still be implemented as a way to keep an I/O channel open, and this will allow for existing comms programs to work, but it should not be absolutely required. The Proceed line can be used to indicate to the handler that there is data to be read, while the Interrupt line can be used to implement the required flow control. 

 

And when a proper telnet terminal is implemented, the interrupt driven nature + the new N: device will make something that can be a lot more sane than the work-arounds we have had to live with for the last 40 years.

-Thom

 

[ivop]

3 hours ago, Gavin1968 said:

I prefer SMD as well, I would be surprised if it doesn't come in one?

Actually it seems to

http://images.100y.com.tw/pdf_file/19TQ015.pdf

 

 

Yep - https://www.amazon.com/19TQ015S-Schottky-Rectifying-Diode-D2Pak/dp/B0090BBR9I

 

Very nice! The datasheet I had only listed the TO-220 and not the D2PAK version.

 

@tschak909 I'd really ditch the Arduino API and go to the ESP8266 API directly. See the ESP8266 Technical Reference Manual (not the usual datasheet).

 

The Arduino API is notoriously slow. Layers of classes on top of eachother. Seeing you also use interrupts instead of polling, and a state machine.... it all adds up. Hopefully I can wire-up my development board on the day after boxing day and start helping out.

 

Edited December 25, 2019 by ivop

[tschak909]

I am looking to see how I can make something that targets the 8266 and the ESP32. 

 

Understand, I am only using Arduino as a way to sketch out the desired functionality.

 

It allows me (and others) to focus on HOW something is going to work, while we gather all the necessary data from these experiments (as I did say before, there will be HUNDREDS), to make the production firmware.

 

There are lots of questions, writing them in Arduino will get these questions answered and doing optimization at this point is _VERY_ premature, and will ultimately harm the project.

 

As for my own opinion (of which I speak for no-one else), I really want the production of this device to be on the ESP32. It isn't that much more expensive, for an extreme leap in I/O capability.

 

-Thom

[ivop]

I understand you want to keep the code simple and readable without having to dig through hundreds of pages of manual to understand the code. But for example digitalWrite() is over 13 times slower than direct port access. And the serial object keeps its own buffers etc... which is unnecessary as the ESPs already have pretty large send and receive buffers on their own. AFAIK the ESP8266 API is the same as the ESP32 API, so code should run on both with no or minimal ifdeffery.

 

Edit: it's not that I want to rain on your parade in any way. I'm actually very excited about your plans! I even dreamt up a rudimentary X11R6 server a couple of nights ago

Edited December 25, 2019 by ivop

[tschak909]

It's not about that, but rather that I want to focus on figuring out the hundreds of questions that need answering. It will not stay in Arduino. But we will use Platform.io with the espressif IDF target.

Yes, this is a rewrite. But it allows me not to get bogged down with toolkit minutiae too early.

Sent from my SM-G920F using Tapatalk

[AtariGeezer]

4 hours ago, tschak909 said:

Part of the work I am planning to do outside of the "D:" emulation, focuses on the following aspects:

 

* Creating a new N: device for TCP and UDP communication

* Creating an R: device that actually implements a proper device, instead of just dumping into R-Verter mode.

 

-Thom

 

My sketch that has 850 Emulation and the TNFS Client now compiles and the 850 Emulation now loads the Booter and R: Handler perfectly

Shouldn't be too much longer before I call Kyle's BBS soon ?

[mozzwald]

20 minutes ago, AtariGeezer said:

My sketch that has 850 Emulation and the TNFS Client now compiles and the 850 Emulation now loads the Booter and R: Handler perfectly

Shouldn't be too much longer before I call Kyle's BBS soon ?

You have this working on the ESP32 yet or just the 8266? Care to share yet?

 

6 hours ago, manterola said:

Then I created two flags to move thru the machine states: newCmdReady which is initialized in FALSE and set to TRUE once  command frame is ready and validated and recInProgress (receiving process in progress) which starts in FALSE and it is updated by the ISR.

 

I am also attaching the INO , but it is just a simple starting and easy starting point idea of adding a short drive emulation to a modem program emulation, to boot the rverter handler and ICET or BOBterm, and then switch to modem mode for the rest of the time, until next boot. Nothing fancy.. 

I just like the idea of receiving and validating the cmd Frame first and also the idea of trying additional bytes in the serial buffer until the checksum works, instead to give up at the first try. Not sure how effective this strategy is in practical terms.. maybe not useful at all.

Thx for the test code, makes more sense now  I tried it on the ESP32 and couldn't get it to work. recInProgress never seemed to be true, or the timing is way off. I added debug messages to recCmdFrame() and never see "RCV IN PRGSS", yet I do see "RCV WHILE":

while (SIO_UART.available()>0 && newCmdReady == false ) {
  Debug_println("RCV WHILE");
  rb = SIO_UART.read();
  if (recInProgress == true) {
    Debug_println("RCV IN PRGSS");