I have tried APE for windows with my Prolific and FTDI USB Serial Adapter on the mac using a Virtual Box VM, I cannot get it to successfully boot the Atari. The closest I can get is if I setup RI flow control, the Atari tries to boot when using the FTDI interface. But I end up with some beeps and pauses with BOOT ERROR's. I can only get this when I set the SIO to slow speed. Otherwise I get nothing. None of the other flow control options work. I also don't see an option in there to disable flow control. I can get RespeQt 4X for Windows to work properly on an XP VM using DSR flow control, using my adapter which has RTS and DTR lines removed. This is with the FTDI adapter. The Prolific Adapter partially works with NONE flow control, I can get programs to boot, but I get BOOT ERROR's though.
Using the Mac directly with RespeQt 4, using the FTDI Adapter with my custom DTR/RTS cut cable, I can get RespQt to work pretty reliably, but only with flow control set to NONE. No other option works. Using the Prolific Adapter with my customer DTR/RTS cut cable, RI and DSR flow control seems to work, although there are some SMALL pauses when booting. During the pauses you hear that 'farting' noise, but I am thinking that the Atari may be sending flow control because the MAC might be sending too quick. (It would be nice to know what the technical term is for that 'farting' sound). Looking at the logs on both the Windows and Mac version of RespeQt, when the pauses occur, you see messages such as "Serial port set to 19200" each time in-between the Read sector messages. With Flow Control set to NONE, you do not get any pauses, but I don't know if there are read errors occurring that I don't know about but I have timed booting and it is quicker with flow control set to NONE. Tapper for example takes 21 seconds to boot with RI flow control where as it takes 17 seconds to boot with flow control set to NONE.
I guess I am getting some successes, but overall this isn't working completely as expected.