RespeQt: Fork of AspeQt 1.0.0-preview7/r79

[Bryan]

Personally, I'd rather see a new name than a derivative one going forward.

[Kr0tki]

I have mailed him through Sourceforge, let's wait what we get from that.

Um, what I really wanted to say, is that I'm gonna wait for cyco130's eventual answer, and not that I'd require Joey Z to stop forking around and wait with me. Edited July 3, 2015 by Kr0tki

[+DrVenkman]

Alright, can we get over the arguing about forking, "de-Raying" and the rest of this nonsense and get back to the code? We can? Great, thanks!

 

Now here's where I stand with OS X builds of r1: In short, they suck. They're still plagued with the same exact incredibly high CPU usage, and worse, when actually transferring data to a real machine, the read operations stall out for seconds at a time. Even bringing up a DOS menu from BASIC can take 10 - 30 seconds, kind of randomly. I did a basic "Sample" from the OS X Activity Monitor and it looks like damn near 98% of the time the code is spinning around in an ioctl operation in the sioworker thread. I don't really grasp all that the code is doing there, nor do I know why the OS X builds seem to be the only ones affected by this kind of thing.

[Joey Z]

Alright, can we get over the arguing about forking, "de-Raying" and the rest of this nonsense and get back to the code? We can? Great, thanks!

 

Now here's where I stand with OS X builds of r1: In short, they suck. They're still plagued with the same exact incredibly high CPU usage, and worse, when actually transferring data to a real machine, the read operations stall out for seconds at a time. Even bringing up a DOS menu from BASIC can take 10 - 30 seconds, kind of randomly. I did a basic "Sample" from the OS X Activity Monitor and it looks like damn near 98% of the time the code is spinning around in an ioctl operation in the sioworker thread. I don't really grasp all that the code is doing there, nor do I know why the OS X builds seem to be the only ones affected by this kind of thing.

I'd like to give it a try on OSX at some point, but I don't own a Mac. Maybe I can get a Hackintosh setup going? Anyway, this is all dependent on time. From what I've heard, some Qt stuff tends to suffer on OSX because it ends up polling rather than waiting for events, which sucks up a ton of CPU. I don't know if this is just a problem with the Qt library itself, or if it's an issue with AspeQt/RespeQt.

[HiassofT]

I did a basic "Sample" from the OS X Activity Monitor and it looks like damn near 98% of the time the code is spinning around in an ioctl operation in the sioworker thread.

My guess would be that it's the TIOCMGET ioctls in readCommandFrame().

 

A similar thing happens on Linux when using a real serial port (onboard or PCI/PCIe 16550 card).

 

The code polls for changes of the command line in a loop and that needs, well, all the available CPU time.

 

For some reason this doesn't happen with USB-serial converters on Linux, IIRC the ioctl only returns every 1ms maybe sleeping until the next status-change packet arrives from the USB-serial chip.

 

The Windows code OTOH doesn't use busy-waiting but uses WaitCommEvent() which sleeps until the state of the command line changes.

 

In Linux there's the TIOCMIWAIT which does a similar thing to the Windows implementation - but I'm not sure if this ioctl is also available on OSX or if there's some equivalent ioctl.

 

Another thing you could try is to change the QThread::yieldCurrentThread() to QThread::usleep(1000). This should put the thread to sleep for 1ms instead of trying to transfer control to another thread (if no other thread is waiting for the CPU yieldCurrentTread will do nothing).

 

Depending on OS any Qt specifics usleep(1000) could fail - on Linux you'd have to use HPET timers or have the timer ticks every 1ms (HZ=1000) to get 1ms resolution. But you could also try sleeping for 3ms or slightly above - that should still work.

 

You could also have a look at the the CrossOver / Wine implementation and check what they are doing. Haven't looked at it yet and I'm also no OSX expert (actually I don't have any Apple products at all here).

 

so long,

 

Hias

[SS]

OK, I am apparently some sort of knee-biting moron but I cannot for the life of me figure out where the download link for the actual program is!!! Can somebody please help point me in the right direction here? I'm sure that the answer is string me right in the face; honestly I don't know how I can be so clueless sometimes ...

[Joey Z]

OK, I am apparently some sort of knee-biting moron but I cannot for the life of me figure out where the download link for the actual program is!!! Can somebody please help point me in the right direction here? I'm sure that the answer is string me right in the face; honestly I don't know how I can be so clueless sometimes ...

maybe you're not missing it. Are you looking for something precompiled? In that case there's nothing yet. You can however download a zip of the source which you can compile using Qt Creator or qmake then make on linux.

[SS]

maybe you're not missing it. Are you looking for something precompiled? In that case there's nothing yet. You can however download a zip of the source which you can compile using Qt Creator or qmake then make on linux.

 

Well, I knew that I was missing something​! Yes, what I thought that I was looking for was an already compiled build; I don't think that I have the skill set to do it myself.

 

What I DO have, however, is one of those SIO2PC/10502PC Dual USB devices. I would like to better be able to use it to write new floppies on my 1050DD but have had only limited successes thus far using the Aspeqt software. I am very much looking forward to giving this new version/build a try.

[Joey Z]

 

Well, I knew that I was missing something​! Yes, what I thought that I was looking for was an already compiled build; I don't think that I have the skill set to do it myself.

 

What I DO have, however, is one of those SIO2PC/10502PC Dual USB devices. I would like to better be able to use it to write new floppies on my 1050DD but have had only limited successes thus far using the Aspeqt software. I am very much looking forward to giving this new version/build a try.

Considering this software isn't very different from AspeQt yet, I doubt you'd have any better luck. I'm working on windows builds, but I don't really have a windows machine, so I've been trying to get things going in a VM. When I get a windows build done, I'll post it up and update this thread.

[SS]

Cool. Thanks.

[+DrVenkman]

Much thanks to HiassofT for the pointers! I spent some time digging into the code and identified several areas on sioworker.h and sioworker.cpp dealing with thread timing and priority. I didn't find any explicit statement that looks like the one HiassofT referred to, however ( QThread::yieldCurrentThread() ). I spent an hour or two yesterday tweaking some of those sleep values, and even tried explicitly changing thread priority (all the way from Lowest to RealTimePriority) but nothing I did seemed to change CPU usage at all.

 

Well, I'll take another look at things today and keep at it. Maybe I'll stumble across a solution. Even a bind pig finds an acorn once in awhile.

[+DrVenkman]

It's apparently too late to edit my earlier topic but there's been progress!

 

Apparently I was looking for the culprit in the wrong program module. I started digging around this morning in serialport-unix.cpp and found two explicit references to QThread::yieldCurrentThread() in the Mac-specifc code block on lines 345 and 355. These sections of code appear to control how RespeQt handles changes in the command lines for the SIO. Per the code comments, this first section is waiting for the command line to turn off, the second is waiting for it to turn on again. Changing those two code lines to HiassofT's suggested values of QThread::usleep(1000) gives me an application build that opens the SIO2USB device and idles at a nice, comfortable 4.5% - 8%. Fiddling with the options and restarting emulation gives me a peak of about 15% for a second or two. Typical prior builds open up the SIO2USB device and immediately jump to 95% - 105% CPU usage (this is on my 2.0 gHz Core i7 MacBook Pro). I'll try to boot an actual Atari from it in a little while, and do some file load/save operations from various applications as well, but for now at least, this looks like real progress.

 

So thanks HiassofT!

[+DrVenkman]

Update:

 

So the high cpu issue under OS X appears to be fixed with the changes I made above. Unfortunately, I've still got the same semi-random pauses and hesitations while doing serial reads. Just booting from a game disk, for instance, will start and read several sectors, pause, read another few or perhaps even a couple dozen, then pause again. Pauses might be a couple seconds or as long as maybe a minute. Repeat randomly until the boot process completes.

 

I took another process sample during one of the pauses but there's a LOT of data in there to parse. I need to see what thread is doing what and stalling out while that's going on ...

[phaeron]

The tightest portion of the SIO transaction is -COMMAND to ACK, which must happen within one frame or less. If you're getting a long delay, though, it means you've hit the device timeout instead of the command timeout. Sending the Complete or first data byte too quickly can trigger this as SIO needs some time to transition from command to data reads. The command timeout is 1-2 frames (16-33ms NTSC) whereas the device timeout is 960 frames (16 seconds NTSC). You could try adding some small delays in between ACK -> Complete -> Data to see if it makes the transfer more reliable.

[Joey Z]

Update:

 

So the high cpu issue under OS X appears to be fixed with the changes I made above. Unfortunately, I've still got the same semi-random pauses and hesitations while doing serial reads. Just booting from a game disk, for instance, will start and read several sectors, pause, read another few or perhaps even a couple dozen, then pause again. Pauses might be a couple seconds or as long as maybe a minute. Repeat randomly until the boot process completes.

 

I took another process sample during one of the pauses but there's a LOT of data in there to parse. I need to see what thread is doing what and stalling out while that's going on ...

I'm glad you're making progress, because I have no OSX machine to even try it on. If you come up with anything that works well, I'll definitely add it into RespeQt. I'd like to make this as cross-platform as it can be.

[+DrVenkman]

I'm glad you're making progress, because I have no OSX machine to even try it on. If you come up with anything that works well, I'll definitely add it into RespeQt. I'd like to make this as cross-platform as it can be.

 

Fortunately, the twiddling I'm doing looks to be entirely within chunks of OS X-specific blocks of code.

 

I haven't had a chance to dig much further today, though I've identified a what may be part of the code responsible for the SIO transitions phaeron is discussing above. I've got an unexpected day off work tomorrow so I'll experiment and see if I can make any progress.

[Joey Z]

 

Fortunately, the twiddling I'm doing looks to be entirely within chunks of OS X-specific blocks of code.

that's what I figured. conditional compilation is great

 

I haven't had a chance to dig much further today, though I've identified a what may be part of the code responsible for the SIO transitions phaeron is discussing above. I've got an unexpected day off work tomorrow so I'll experiment and see if I can make any progress.

When you get something working well enough, and you think it's ready to go into 'production', send me a copy of your source. I'll run a diff and merge the changes manually. Also, I'll need someone to do OSX builds when I get an official 'stable' release done, do you think you could help with that at all?

[+DrVenkman]

I'm happy to build OS X binaries to the best of my limited abilities. I've obviously already spent gigs of bandwidth downloading the latest Qt 5.4 libraries and currently have the latest Xcode as well thanks to a developer account I really don't need, lol. No need for anyone else to spend that time and effort unless they just want to.

 

Anyway, with HiassofT's help, it looks like the high CPU usage bug is licked. Now I just have to grok phaeron's advice and do the same for the SIO transitions. It looks like there's already a bunch of tiny delays inserted into several (but not all) segments of the OS X chunk in the serialport-unix.cpp code. Those values may be wrong or simply not large enough. It would be instructive to know why the prior author of that code section added those delays and what the significance is of the values chosen. Too bad the code isn't commented well.

[+DrVenkman]

Okay so progress after yesterday: I can reliably make the transfer process less reliable but I cannot yet improve the "stuttering." I'm working on it though.

 

The same .atr file will load quickly and perfectly using the old Windows AspeQt 1.0 preview 6 (under Crossover). Using a Mac-specific build will halt on anywhere from 4-10 sectors and wait several seconds. The transfer will then continue with the message "Sector xxxx [x2]" Again, same hardware, same .atr file will load perfectly with the Windows version. There are several possible areas of code that could be responsible but nothing is very obviously at fault.

 

Oh, also, I found one reference to AspeQt in sioworker.cpp I will get you a line number reference when I get home from work.

 

 

Sent from my iPhone using Tapatalk

[phaeron]

One thing I noticed is that serialport-win32.cpp's writeRawFrame() uses PurgeComm(PURGE_TXCLEAR), which discards pending transmitted data, while serialport-unix.cpp uses tcdrain(), which according to the man page waits for pending data to transmit. Might be worth checking the calls carefully to see which ones might not have been translated properly.

[Kr0tki]

I have mailed him through Sourceforge, let's wait what we get from that.

And cyco130 replied, and the outcome is, well, I'm gonna fill the role of project maintainer at SourceForge for a while.

 

I have already restored the website to the state from before the "crisis", using all of the SVN backups available, coupled with the magic of Archive.org. My next goal is to update the docs to reflect the current state of affairs ("de-Raying", if you will), restore various files missing from the repository (e.g. the source code of AspeCl, mentioned in this thread), and incorporate the fixes being developed in RespeQt, with proper credit of course.

 

Admittedly, I am not yet ready to partake in the actual development of AspeQt - the project is begging for help of an active developer, such as Joey Z. I'm going to focus on the mundane tasks for a while, such as building Windows and Linux binaries.

 

So the bottom line is, AspeQt at SourceForge is alive again, it needs participation of an active developer, and most importantly, it is not going to die as before - this time, I'm making backups.

Edited July 8, 2015 by Kr0tki

[+DrVenkman]

One thing I noticed is that serialport-win32.cpp's writeRawFrame() uses PurgeComm(PURGE_TXCLEAR), which discards pending transmitted data, while serialport-unix.cpp uses tcdrain(), which according to the man page waits for pending data to transmit. Might be worth checking the calls carefully to see which ones might not have been translated properly.

 

 

Your prior suggestion of inserting small delays in the SIO sequence to allow time for transitions appears to have helped a great deal. I managed to reduce loading times for my go-to test .atr from about 6 - 10 pauses (seven seconds each, by the way!) to about 2 - 3. But it's all empirical hit-or-miss - insert a short delay here, compile, test, repeat ... There's absolutely nothing scientific about it at all. It just seems like a nasty hack, you know? There ought to be a better way to go about this, especially as the two other OS X-compatible peripheral emulators aren't affected (SIO2OSX and SIOServer).

 

So thanks for that pointer. I'll look into that over the next few days and see what I can dig up.

 

Oh, and for Joey Z, that reference to AspeQt in sioworker.cpp is at line 217. I've already changed it in my working copy.

[HiassofT]

Another thing you could try is to change the QThread::yieldCurrentThread() to QThread::usleep(1000). This should put the thread to sleep for 1ms instead of trying to transfer control to another thread (if no other thread is waiting for the CPU yieldCurrentTread will do nothing).

...

But you could also try sleeping for 3ms or slightly above - that should still work.

I have to correct myself here - remembered the specs wrong - better use a usleep(500) here. t0 (time between lowering of /command and the first byte) is specced at 750-1600µs. Delaying for 1ms could be slightly too much and you might miss the first command byte(s).

 

As for testing: do you have a digital scope or logic analyzer you could hook up to the SIO bus (2 channels would be enough to see the timing relationship between /command and incoming bits).

 

so long,

 

Hias

[morelenmir]

Obviously looking far in to the future - I am not agitating for this feature, just asking if it is possible - would 'RespeQT' be able to load custom ROMs, such as the 'USDoubler', 'Happy', 'Speedy' upgrades and so forth? You would be effectively building a software version of the 'MegaSpeedy' board. Were that possible then an Android build supporting custom firmware would allow one to completely dispense with a real '1050' by simply using a £20 tablet from Amazon and a £12 DIY SIO2PC! That would be the best of all worlds!!!

 

In the past Phaeron has mentioned how time-consuming and tricky it would be to add this type of functionality to "Altirra". I wonder if it might be easier to implement with a dedicated floppy emulator like "RespeQT"?

[+DrVenkman]

One thing I noticed is that serialport-win32.cpp's writeRawFrame() uses PurgeComm(PURGE_TXCLEAR), which discards pending transmitted data, while serialport-unix.cpp uses tcdrain(), which according to the man page waits for pending data to transmit. Might be worth checking the calls carefully to see which ones might not have been translated properly.

 

Hmm. There's also a tcflush() which looks interesting, but I'll have to explore that further. Googling shows that at least some folks have reported that this doesn't work as expected (e.g., leaves data in the pipeline) and requires use of a delay. Since I'm busy inserting and removing delays all over the place to try to smooth out the transfers on an experimental basis, it may not make any difference. *shrug* I also see that there are some developer resources on Apple.com regarding device driver implementation and porting, as well as in implementing serial transfers in OS X. There's stuff in there about thread priority and making sure your I/O code isn't paged out of RAM while you need it. Some of those OS-specific flags might be worth looking into as well.

 

I wish I had about two weeks straight to bang my head against this stuff and just write an entire OS X-specific block of code from scratch; I've got my original copy of De Re Atari around here somewhere along with 15 years' worth of accumulated AtariAge wisdom to dig through ...

 

In the meantime, I'm back at it again tonight.