Doom networking / UART bug

[Songbird]

Update: The UART bug doesn't seem to be what actually causes Doom over JagLink to lock up and reset the level periodically. Look at this post for my detailed explanation and the fix I implemented which appears to address the issue after several hours of testing. Otherwise, if you just want to see the code or try out the *.abs file on a Skunkboard or other developer card, it's attached to this post. Game on!

 

My original first post is quoted below:

 

 

Famously old topic here: how to fix Doom networking. Been scouring old posts, and here are some possible experiments:

1. Set parity to EVEN or ODD instead of NONE (EVEN seems to be transmitted on NONE according to Atari developer docs, page 74 of Software Reference) http://atariage.com/...-bug/?p=1318056
2. Send single $80 byte as header on each serial packet (Atari developer docs, page 6 of Hardware Bugs & Warnings)
3. Set speed to 38400 instead of 115200 http://atariage.com/...-bug/?p=1318056
4. Mute audio by writing zero to bit 8 of $F14000 before any serial transmit / receive (Atari developer docs, page 28 of Technical Reference)
5. Write back the ASICLK value after every byte received http://www.jagware.o...pic=451&p=13891

Did I get all of those right? Any other options I missed besides software UART emulation (already covered in this thread)?

 

Note that for #2, Doom streams 6 bytes at a time for each packet. However this is all done in a software loop which polls for TBE, plus they interleave sending and receiving each byte, so I'm doubtful this is fast enough to honor the stop bits requirement listed in the Atari docs to avoid the bug. It may be that you have to send $80 before each byte in the packet, which would effectively cut your link speed in half.

doom_n06.zip

Edited June 4, 2016 by Songbird

[TXG/MNX]

I think thats all known except we don't got the aircars/battlesphere networking code that seems to be stable aswell...

 

Sent from UMI hammer with Tapatalk

[+CyranoJ]

I think thats all known except we don't got the aircars/battlesphere networking code that seems to be stable aswell...

 

Sent from UMI hammer with Tapatalk

Seriously? We have had that since the day both roms were dumped. Anyone with the skills to fix doom could easily disassemble either rom.

[TXG/MNX]

Yep you're right ... :-)

 

Sent from UMI hammer with Tapatalk

[Songbird]

Ah, a challenging challenge. I like it! This is actually the first time I have seriously studied the networking code; I will try some experiments soon and post what I learn.

 

[+Stephen]

Very cool to see you are still active in the Jag scene Carl!

[Songbird]

Quick update: I got the Doom code compiling again. Even that took awhile, but fortunately DOSBox came through once I set up my environment path correctly. So, I've already made my first experiment and will try something this weekend hopefully.

[Songbird]

Dual dev station is a go!

 

Test #1: doom_n01.abs

Changes:

1. Changed serial packet size from 6 bytes to 7 bytes
2. Inserted 0x80 dummy byte as the first byte in the packet
3. Rearranged loops in I_NetTransfer to send all 7 bytes then receive all 7 bytes (or vice versa)

Result: Game could never establish a network successfully. I'm guessing the timing is off due to the changes in #3, and thus the network is getting out of sync. FAIL

[+Stephen]

Sweet - where'd you get the 3D printed case for the right Skunk?

[Songbird]

It was a gift.

[Songbird]

Test #2: doom_n02.abs

Changes:

1. Changed serial packet size from 6 bytes to 7 bytes
2. Inserted 0x80 dummy byte as the first byte in the packet
3. Reduced serial clock from 115200 bps to 38400 bps

Result: Game ran networked for over 2 hours without fail. I tried on 4 different levels, usually while moving just one controller but sometimes I put a clamp on the second controller so the second player went in a circle. Network appears to be rock solid, and 38400 should be more than enough speed to transmit such a small amount of data per frame. PASS

 

doom_n02.abs is attached for your testing pleasure. Assuming you already have the WAD installed on a Skunkboard or similar device, simply load doom_n02.abs to $4000 and execute from there. Enjoy, and definitely let me know if you see any network errors!

doom_n02.abs

[+CyranoJ]

Game ran networked for over 2 hours without fail. PASS

 

Fantastic work!

[neo_rg]

Good news, this might spur me into doing something now.

[Songbird]

It would be great if I could get a couple of people to test the change and provide feedback. I realize it's a very limited audience -- people with two Skunks and other hardware -- but your help would be appreciated. One thing I did was hook both Jags to the same TV, and just bounced back and forth between Jags manually to ensure everything was working. That way I could keep both Jags next to my PC for Skunk transfers.

 

To further clarify the test I ran, I actually left the game running deathmatch within a single level for over 120 minutes, then tested several more levels for about 15 minutes each. I also tested the game without the change and confirmed I saw a network error after about 5-10 minutes.

 

Furthermore, I did confirm Aircars is using the same 38400 rate. Given each player in Doom is only transmitting about 2000 bps, 38400 seems plenty fast.

 

Once I get some independent confirmation, I will edit the first post in the thread to contain both the doom_n02.abs file and the jagonly.c source file, so people can incorporate this into their builds.

 

Have fun!

[+CyranoJ]

I've got 2 Jags, I've got 2 skunkboards (and 13 Jagtopus boards)...

 

I don't have a link cable

 

Would love to test tho. Hope it all works out!

[+Sauron]

Might want to post about it in the main forum. Not sure that all Jag users regularly check the Programming forum, and you'd have better luck finding some people with the required hardware.

[JagChris]

It would be great if I could get a couple of people to test the change and provide feedback. I realize it's a very limited audience -- people with two Skunks and other hardware -- but your help would be appreciated. One thing I did was hook both Jags to the same TV, and just bounced back and forth between Jags manually to ensure everything was working. That way I could keep both Jags next to my PC for Skunk transfers.

 

To further clarify the test I ran, I actually left the game running deathmatch within a single level for over 120 minutes, then tested several more levels for about 15 minutes each. I also tested the game without the change and confirmed I saw a network error after about 5-10 minutes.

 

Furthermore, I did confirm Aircars is using the same 38400 rate. Given each player in Doom is only transmitting about 2000 bps, 38400 seems plenty fast.

 

Once I get some independent confirmation, I will edit the first post in the thread to contain both the doom_n02.abs file and the jagonly.c source file, so people can incorporate this into their builds.

 

Have fun!

 

Network errors were far less common in Deathmatch. Co-op is the one where the errors really popped up. To get the errors eliminated or cut down in co-op mode would be the real test I think.

[Songbird]

I'm running some tests today on Co-op, and unfortunately the network did crash on me a couple times:

 

First test: failed after 5 minutes

Second test: failed after 45 minutes

 

I was able to beat two levels (slowly) with no errors at all on the second test, then on the third level the network failed right about when I was ready to end the level. Frustrating. So, clearly not all the issues are gone, but it seems more stable than I remember from years ago. There must still be an issue with the stop bit alignment being off sometimes. I will attempt some additional code changes and see if I can make it better.

[Songbird]

I just reviewed the Aircars networking disassembly, which is not at all intuitive to follow because they don't just set the UART control regs to hard-coded values; they load them from values stored in RAM. It looks like they're only setting bits 5 and 6 in ASICTRL, which correspond to "enable receiver interrupts" and "clear error". They're also using 38400 for the clock rate. They are not setting parity at all. Doom doesn't use parity, either (side note: a parity bit is always transmitted anyway).

After multiple experiments tonight, including enabling parity, attempting to send the 0x80 "dummy" byte before every "real" byte, etc., I am still seeing network errors on co-op every 15-30 minutes. The Atari docs talk about being exact with the stop bits, and I'm wondering if the Doom code is just too simple, because it is pushing and pulling bytes manually without any concern for timing, other than checking for "buffer full" before sending another byte. It probably needs the receiver interrupt like in Aircars so it can suck down all the bytes precisely in time. Otherwise, if someone has other ideas on how to control the flow of the stop bits, I'd like to hear about that...

[Songbird]

I want to catch bad serial bytes sooner, so I changed the consistency value from:

consistancy = players[0].mo->x ^ players[0].mo->y ^ players[1].mo->x ^ players[1].mo->y;

to:

consistancy = outbytes[1] ^ outbytes[2] ^ outbytes[3] ^ outbytes[4];

then of course I changed the corresponding incoming consistency check to:

if (inbytes[5] != inbytes[1] ^ inbytes[2] ^ inbytes[3] ^ inbytes[4])

Strangely, this results in an instant network fail, which makes it seem no packets are ever sent without error. If that's true, then I am confused as to why the original consistency check doesn't fail sooner, because you would think the player X and Y values would drift pretty quickly...

 

Anyway, I think I will need a new approach. I really need a simpler UART program which would let me inspect the bytes as they are received. I don't have any visibility just watching Doom crash over and over again. So, that will probably slow down my progress until I get a suitable UART program developed.

 

[sh3-rg]

do you use the skunk console to spam stuff to the pc screen?

[walter_J64bit]

Now, I wish I had gotting two skunkboards. I'm Keeping an eye on this topic.

[Songbird]

do you use the skunk console to spam stuff to the pc screen?

I haven't been, but I suppose I could include skunk.s in the makefile and spam out the history of network packets to a PC file until it fails. That would give me a lot more visibility into what's happening.

[Welshworrier]

I've had a bit of a look at the source code and, even though I'm quite a novice at Jag coding I've written quite a bit of serial comms stuff over the years, there are a few issues that look suspect.

 

Doing the maths on the byte transfers:

6 bytes * 2 * 15 frames a second max = 180 bytes of data a second (approx 2k baud)

 

You could probably drop the xfer speed down to 19200 with negligible impact. As the uart only has a one byte buffer this would lessen the chance of overrun/error though as you are polling for the data there might be a game impact.

 

Looking at the Doom source shows how flaky the network comms are anyway - any single byte transmission error kicks the network game out. You may be better redoing the entire protocol

BTW:

consistancy = players[0].mo->x ^ players[0].mo->y ^ players[1].mo->x ^ players[1].mo->y;

consistancy = (consistancy>>8 ) ^ consistancy ^ (consistancy>>16);

As consistancy is defined as a byte earlier on in the function..............

When you tried the "if (inbytes[5] != inbytes[1] ^ inbytes[2] ^ inbytes[3] ^ inbytes[4])" you might want to ensure the precedence by bracketing the xors in case of compiler feature

i.e. if (inbytes[5] != (inbytes[1] ^ inbytes[2] ^ inbytes[3] ^ inbytes[4]))

Interesting to see how it all pans out

Edited May 31, 2016 by Welshworrier

[Songbird]

I already caught that the "consistancy = (consistancy>>8 ) ^ consistancy ^ (consistancy>>16);" isn't doing anything, but it's a harmless line of code and it was from the original source. However, good call on the order precedence -- I thought it would have done everything to the right of != but I was wrong. Thanks!

 

I previously dropped the speed to 19200, and while deathmatch seems rock solid, co-op is failing in less than 20 minutes in most cases. I am not sure why there is a difference, since it's the same network code with no additional transfers during co-op -- so it's possible it's simply a test artifact, meaning my deathmatch results are random good behavior instead of improved networking code.

 

I agree that, barring a simple UART work-around I'm missing, the protocol probably needs to be rewritten. There is no capacity for retries because there is no communication from the "other" console that the packet was received successfully. Some kind of ACK would go a long way in helping, but then that creates more network packets which also require protection to ensure they are received properly...