New MAME release

[+arcadeshopper]

linux not so much..  

 

all i did was call tipi after boot up.. and got this interesting crash ./mame ti99_4a -ioport peb -ioport:peb:slot2 samsmem -ioport:peb:slot7 tipi -conn rpi.tipipz2 -log

error.log

[+mizapf]

That is a heavy crash inside the emulation where something plowed through memory, first hitting the sound chip.

 

Although I am nearly exclusively using Linux (which should make the Linux build the most reliable one, as I am constantly testing it), I don't have this issue here, but it rings a faint bell. I may have seen that once, but I can't tell what was the problem.

 

You took the latest build from WHtech, I suppose. Please check all CRU bases, try with 32kmem instead of samsmem; check that you don't have the 16bit 32K expansion active.

[+mizapf]

This is my post #6000.

[+arcadeshopper]

45 minutes ago, mizapf said:

That is a heavy crash inside the emulation where something plowed through memory, first hitting the sound chip.

 

Although I am nearly exclusively using Linux (which should make the Linux build the most reliable one, as I am constantly testing it), I don't have this issue here, but it rings a faint bell. I may have seen that once, but I can't tell what was the problem.

 

You took the latest build from WHtech, I suppose. Please check all CRU bases, try with 32kmem instead of samsmem; check that you don't have the 16bit 32K expansion active.

k. yeah I grabbed it off whtech I recently rebuilt the laptop so that was easier than grabbing source and compiling used ooi to get it but it doesn't work right on the latest build of ubuntu so ran from command line 

cru is 1000 on tipi, will try 32k 

 

 

 

 

[+OLD CS1]

7 hours ago, mizapf said:

This is my post #6000.

Forever immortalized

 

[+mizapf]

10 hours ago, arcadeshopper said:

k. yeah I grabbed it off whtech I recently rebuilt the laptop so that was easier than grabbing source and compiling used ooi to get it but it doesn't work right on the latest build of ubuntu so ran from command line 

cru is 1000 on tipi, will try 32k

If you continue to see problems, try to build it by yourself after pulling the latest sources.

 

If you can build the binaries anyway, you should probably increase the log verbosity in the source files.

 

For example, open src/devices/bus/ti99/peb/tipi.cpp and add "LOG_RPI" to the #define VERBOSE line (within the parentheses). The more LOG_* flags you add in the VERBOSE line, the more output you get in error.log.

[+jedimatt42]

I cannot post technical information to Atariage without violating some cloudfront filter... so see attached message ( I had to zip the txt file too... cause even that wouldn't upload ?

 

CloudFrontFilterBypass.zip

 

In the message I show log details from the PI websocket and how to enable the websocket logging. And generally state that using my real TIPI PI, MAME emulation works only occasionally, if I get my request in real fast... maybe Qemu is so slow it doesn't trigger the websocket upgrade or something... IDK... 

 

I'm taking a break until @Albert has a solution for posting slashes. 

[+mizapf]

This is all a bit confusing. I can just say that the MAME TIPI emulation with QEmu works very stably here, each CALL TIPI, each access working, save, load, telnet, with TI-99/4A and Geneve. Not most, but all. I just cannot reproduce any of the issues you mentioned... ?

 

Should I probably try with a real Raspberry? Can I install the image on a normal Pi without the TIPI adapter card? I should have an empty memory card lying around. Raspi 4 - or lower?

[+jedimatt42]

44 minutes ago, mizapf said:

This is all a bit confusing. I can just say that the MAME TIPI emulation with QEmu works very stably here, each CALL TIPI, each access working, save, load, telnet, with TI-99/4A and Geneve. Not most, but all. I just cannot reproduce any of the issues you mentioned... ?

 

Should I probably try with a real Raspberry? Can I install the image on a normal Pi without the TIPI adapter card? I should have an empty memory card lying around. Raspi 4 - or lower?

 

I have tried a PI 3B+ and a Zero2W (as well as the service just running on x86) ... all of which worked with js99er, which of I imagine gets it's websocket support from the browser.

 

Yes, the websocket replaces the part where the service talks to the PI's gpio... so in emulation mode the PI does not interact with the physical TIPI board at all.   

 

https://github.com/jedimatt42/tipi/tree/main/emulation#using-a-real-pi

 

I'll still try Qemu...   

[+jedimatt42]

Quote

I'll still try Qemu...

Linux MAME -> Windows hosted Qemu ( cause that's where I had Qemu installed ) : Works repeatedly. 

 

A major difference between Qemu and all the environments that I've tried before, is that the PI 3B+, Zero2W, and docker/x86 are that the latter set are multi-core & fast by comparison to Qemu.

 

Qemu is super slow, and the arm emulation is taxing. So I would eventually want to deprecate it in favor of native cpu execution (x86 docker or virtual machine).

[+jedimatt42]

With the debugger open, MAME gets much slower than Qemu, and TIPI emulation success becomes intermittent. 

[+jedimatt42]

What tricks do people use to remember to set "UI controls disabled" after using the menu? If I don't remember to do this, any minor volume of typing into the 4A causes the emulation to hang and the screen dims... the menu / MAME isn't hung, but the emulation won't resume...   

 

Oh, wait, I finally figured it out... when I type 10 PRINT in BASIC, or TIPI or ALT-P to get a quote and then see it hang, it is because the P in PRINT paused. Pressing P again unpauses... In my head, I'm not pressing P, I'm typing a word... so I couldn't make the connection.

 

This has been frustrating me since 2015.  I have killed the process and started my debugging sessions from the beginning so many times due to this, until I retrain myself to carefully check the UI controls state after using the menu... 

 

Learning this is a game changer.

 

[HOME AUTOMATION]

I think ScrLk, used to be the shortcut key. I used to toggle it constantly.

[+arcadeshopper]

1 hour ago, HOME AUTOMATION said:

I think ScrLk, used to be the shortcut key. I used to toggle it constantly.

yeah you can change it to whatever you want in mame.ini

 

[+jedimatt42]

That isn't the problem. It was remembering to use it to disable the UI after anything like changing the cartridge or a floppy image... 

 

This isn't a Mizapf issue, it is just how MAME's global UI framework works. At least now I know I can disable the P option so if I forget to disable the UI I won't keep accidentally pausing. And at least now I know that it isn't crashed, and I can unpause.  

 

What is sad is that in the past when I have talked about this, others didn't understand what I was saying, and ended up affirming the idea that MAME just crashes when you type.. That was a misconception.  

[+mizapf]

Yes, the partial mode (now "UI control enabled") has always been a source of confusion. I remember well some situations, also here on this forum, where people report something like "MAME/MESS very unstable" ... "crashing every time I use it" ... and when I ask what and when it happened, you get something like "don't know, anytime ... window gets dimmed" ... and then I knew what was the problem: "Press P again". I understand it, if you don't know that a key press causes this behavior, it seems to you that this is an unexpected crash.

 

You should remember that MAME is originally an Arcade emulation framework, which still widely outnumber the computer systems. You usually have a very limited amount of inputs, like keys for the flippers, left/right/up/down/fire, coin insert, coin return. The P key was set as "Pause".

 

Things got complicated with the spin-off project MESS (later reunited with MAME) when they started to emulate systems with keyboards. Those systems simply need the keys that were assigned special functions for the arcade machines. So they set up two modes, the full mode where all inputs are under control of the emulated system, and the partial mode where the special function keys are working.

 

What I wonder about is why they never thought about a pop-up window "Game paused". Or maybe they did, but there are reasons against it that I don't know.

[+mizapf]

OK, I managed to put the TIPI image on an SD card for my Raspi 4. After some configuration work, I was able to activate the websocket mode. (By the way, apt-get update complains about repositories that changed name or so, from stable to oldstable.)

 

And I could run CALL TIPI in MAME, which ... yes, fails more often than it succeeds. I had some successful runs, but lots of hangs. I see some unexpected behaviors like closed connections; will check more closely with Wireshark. I believe this is a solvable problem.

 

I'll probably have to delay that until next weekend, still got to do some work (have to read a Master thesis of 120 pages; can't procrastinate any longer).

[+mizapf]

MAME 0.245 was released yesterday. The following changes may be interesting for you:

• Fixed ASYNC mode for TIPI (with QEmu Raspi).
• CRU 1800 is now default for TIPI. Visit the DIP switch menu for selecting another address.
• No more segfaults on exit when TIPI was used.
• DDCC-1 now running with Genmod since I added a AMA/B/C decoding (see configuration menu)

Working with the real Raspi is still not stable; I have to fight some evil race conditions, guess I need a bit more time.

Essentially, the reset by the DSR (setting CRU bit 1) breaks the connection setup for the websocket that happens in parallel. For the Raspi running in QEmu on localhost, the websocket setup finishes in time before the DSR reset occurs.

 

Edit: Windows and Linux binaries are on WHTech now, the Raspberry package will come later, as I currently use my Raspberry for TIPI debugging.

[+mizapf]

@jedimatt42I think I have gained some insight to what happens with the TIPI card emulation, connected to the real Raspberry Pi.

 

I spent hours during the last days, and in particular most of this Sunday on this problem.

 

1. All of my last hypotheses turned out to be wrong.

a) It was not a race condition between the websocket setup and the DSR reset.

b) It was not caused by different timings on the emulated vs. real Raspberry.

c) It was not caused by the difference between localhost communication vs. WLAN of the real Raspberry.

d) It was not caused by out-of-order TCP segments.

e) I cannot solve the problem on the MAME side, although a component in MAME may cause the issue.

 

2. The solution is by one of the following suggestions, applied to the file tipi/services/libtipi_web/websock.c:

a) The Nagle's algorithm should NOT be disabled in line 704, i.e. the setsockopt should be commented or removed.

b) There should be a small pause before the websocket_write(…"ASYNC",5) e.g. by usleep(1000), which is 1 ms.

 

I noticed that option b) is sufficient and probably safer than a).

 

3. The background of my suggestions (may be a bit technical).

MAME uses the epoll API for the asynchronous reception of websocket messages. The problem is that the websocket implementation in MAME seems to miss the ASYNC reception event if it follows the HTTP response too quickly. Wireshark shows the HTTP response being received, then the ASYNC message received, then the HTTP response being ACK'd, then the ASYNC being ACK'd, i.e. receive-receive-ack-ack.

 

As I saw in the code of the MAME websocket client, the reception of the HTTP response triggers an event - in fact, it is also possible that the ASYNC reception event supersedes the HTTP response reception event. The handler reads the incoming message header up to the blank line, searching for the "Sec-WebSocket-Accept" header. After that, it returns control to the scheduler, which then lets its thread run into the epoll_wait to wait for the next message. However, it seems as if the ASYNC message is already available together with the previous message. I'm not fully sure about that, but as a matter of fact, the epoll_wait keeps hanging. epoll_wait belongs to glibc, not to MAME, so I tend to believe it works correctly.

 

When we now choose to ignore that issue and do some operation like CALL TIPI, after the request is sent, another message is received, causing a reception event which wakes up the thread, and the following handling will see the (stale) ASYNC message as a response to CALL TIPI. This reproducibly makes the CALL TIPI command hang.

 

So now for my attempts to fix this. When I remove the setsockopt, the HTTP response is ACK'd, then the ASYNC message is ACK'd, and both raise their own events. This could be because the TIPI socket now delays the sending of the small ASYNC segment according to Nagle. So we have receive-ack-receive-ack, two separate reception events, and a properly received ASYNC.

 

I tested this by starting MAME about 30 times, and I only had 2 cases where the initialization failed, 28 times where it succeeded. So this is still not perfect.

 

Following my idea of the two messages being too close together, I tried to use a pause before the ASYNC message, first with 1 sec, then with 1 ms. It seems as if this also works, but I did not try it by 30 times yet. This may even be a bit safer, as I said above, because it does not depend on some background algorithm to delay a packet for an unknown period, but explicitly delays the packet.

 

I won't doubt that the behavior of the MAME websocket component is not fully correct; this may well be a bug. However, this is so deep inside the MAME library that I actually don't dare to play around with it. (Actually, the whole websocket implementation involves a whole bunch of files, and it took me hours to understand the structure, lots of asynchronous handling with callbacks and lambda expressions.)

 

Since I am quite sure that delaying the ASYNC should not be harmful to the other emulators, I'd ask you to test the above suggestions and pick one to modify websock.c.

[+arcadeshopper]

the way TIPI is designed is "wait for enable" "send"    delays will probably never work 100% .. you probably need to wait for it to establish before sending

 

[+mizapf]

The current sequence is this:

 

1) Client ---- HTTP request ----> TIPI service

2) Client <---- HTTP response / protocol switch ---- TIPI service

3) Client <---- websocket "ASYNC" message --- TIPI service

 

The problem, as I understood it, is that 3) comes too close after 2), and my suggestion is to put a small delay between them. As I said, this may be a glitch in the MAME websocket client which does not expect another message so close after the websocket setup.

[+jedimatt42]

I'll dig in. Not sure how quickly I can get back. I haven't studied the websocket side of TIPI in much detail yet.

[PeteE]

19 hours ago, mizapf said:

The current sequence is this:

 

1) Client ---- HTTP request ----> TIPI service

2) Client <---- HTTP response / protocol switch ---- TIPI service

3) Client <---- websocket "ASYNC" message --- TIPI service

 

The problem, as I understood it, is that 3) comes too close after 2), and my suggestion is to put a small delay between them. As I said, this may be a glitch in the MAME websocket client which does not expect another message so close after the websocket setup.

Imho, if the MAME websocket cannot handle that, it is an error.  There's no reason for the data to be dropped after the websocket upgrade.

[+mizapf]

I suspect not a bug, but a design glitch of the websocket client. The point is that the websocket client first awaits the HTTP response; it gets a reception event by the epoll_wait, but from the received message it only reads the header. Then the thread goes back into the epoll_wait. However, it should have continued reading, and then it would have found the ASYNC.

 

The ASYNC message is not dropped; it is still lying in the buffer, but it did not trigger a separate reception event, hence, the thread keeps hanging in the epoll_wait. I mean, obviously. I could be wrong again, but this sounds most logical to me, assuming that epoll_wait works flawlessly (and as I said, it is not part of MAME but of glibc).

 

So what happens is that the waiting thread is woken up when another message comes from the websocket server. This happens when a TIPI operation is performed, e.g. CALL TIPI. Since the ASYNC was not dropped nor consumed, it becomes part of the response, and this kills the DSR.

 

As said, the simplest way would be to insert a short pause before the ASYNC message. This seems to trigger a separate reception event, and the ASYNC is correctly sensed.

 

The alternative would be for me to try fixing this 3rd party library imported in MAME (asio), with the risk of breaking other stuff I don't even know the name of.

[PeteE]

I'm looking into this too.  I can reproduce the intermittent "call tipi" lockup using mame and tipi in docker.  How do I enable the output of mame messages that are LOGMASKED(LOG_WARN, "...") ?