FPGA Based Videogame System

[SnoopKatt]

I have read that some games do not poll the controller every frame. It is really up to the programmers to determine how fast their game needs to respond to input. There is nothing to prevent a programmer from reading the memory address for the controller port more than once during a 16ms period. However, given that the screen will only change every 16ms and that people cannot process changes on the screen that quickly, there is little reason to do so. Even the motions involved in a fighting game can be captured as one frame=one button press.

 

The latency reported for the AVS is very impressive, averaging around 1.4 frames behind a CRT. The Framemeister seems to be in the same ballpark and that costs much more than an AVS and you still need an RGB-modded console to get similar visual quality.

Yup, they could poll it quicker via software if they wanted to, but as you stated, there's little reason to! One reason I could think of using it for is breaking up ties in 2+ player fighting games, but that could tie up system resources.

 

I think the AVS is barely slower than an NES on composite (less than 1 ms). His total delay through the TV may be 1.4 frames, but I think that's the TV itself being 1.4 frames behind. That's a lot better than the Framemeister because that alone adds 1.5 frames before it even reaches the TV.

 

Also, here's a really good read on emulator input lag: https://retropie.org.uk/forum/topic/2019/an-input-lag-investigation

[Great Hierophant]

Yup, they could poll it quicker via software if they wanted to, but as you stated, there's little reason to! One reason I could think of using it for is breaking up ties in 2+ player fighting games, but that could tie up system resources.

 

I think the AVS is barely slower than an NES on composite (less than 1 ms). His total delay through the TV may be 1.4 frames, but I think that's the TV itself being 1.4 frames behind. That's a lot better than the Framemeister because that alone adds 1.5 frames before it even reaches the TV.

 

Also, here's a really good read on emulator input lag: https://retropie.org.uk/forum/topic/2019/an-input-lag-investigation

The link demonstrates how FPGA "emulation" can be superior to software emulation, even when the hardware on which the emulation is run is many orders of magnitude more powerful than the FPGA chip used in the AVS.

[narshe]

The claim about muscle memory is not entirely scratched off either. We need to find a game that exists in an old console that also exists non-emulated (for no lag) in an HD console.

I'd say MegaMan Legacy Collection, but I think since that's emulated (high level emulation if that makes a difference) it can't be used to compare true CRT to true HD.

Only having a WiiU game non emulated (like Shovel Knight, maybe?) in a CRT and then using the HDMI cable, thats the best way to know if one can detect this 9 ms of lag difference (32 ms?)

 

It would be interesting to find a good way to test this where we know exact numbers. I would love to find a setup that is known to be x milliseconds behind my Famicom and CRT setup to see how I can play.

 

MegaMan Legacy Collection isn't emulated. It was written to be the same though. They used the carts ROM to pull the sprites from, if I remember correctly, but no emulation. Either way, that game wouldn't be a good test.

 

I think the Wii U test would be good. You can use the Wii component cable on the Wii U I believe. The only problem with this is, you need to get good enough at a game where it would make a difference.

 

I have and play all the original console on a CRT because that's what I've known and am used to. Anything else I've tried doesn't feel responsive for those games. Call me crazy (again I guess).

[+phoenixdownita]

Disagree on polling only once per frame (no matter how the SNES does it).

Take SF II hadouken, or the fireball, the pad needs to register 3 directions + button and I believe it should be possible to perform it in one frame (I believe all SF II are performable in a single frame but I am not entirely sure).

If it takes 4 frames to perform an hadouken(1 per direction + button) we're talking 68ms .... seems a little off, but then again I have not checked the timing of the move.

[Andromeda Stardust]

Yup, they could poll it quicker via software if they wanted to, but as you stated, there's little reason to! One reason I could think of using it for is breaking up ties in 2+ player fighting games, but that could tie up system resources.

 

I think the AVS is barely slower than an NES on composite (less than 1 ms). His total delay through the TV may be 1.4 frames, but I think that's the TV itself being 1.4 frames behind. That's a lot better than the Framemeister because that alone adds 1.5 frames before it even reaches the TV.

 

Also, here's a really good read on emulator input lag: https://retropie.org.uk/forum/topic/2019/an-input-lag-investigation

Many games do poll the controller multiple times per frame. In fact it is required of games that use PCM sound due to a DMA bug where the interrupt with generate a phantom serial input on the NES controllers resulting in an incorrect controller read. This is most notorious for creating random Dpad right movements or even bit-shifting the entire controller read. As a workaround, early games except light gunners like Duck Hunt did not use the PCM sound, and later games that employed PCM ie drum synths in SMB2 and SMB3 would poll the controller repeatedly until two consecutive reads reported identical values. This can result in the unintended consequence of a turbo controller that swaps input on every poll (nominally ~30Hz) will crash the game if alternating inputs are combined with error correction.

 

As for skipping frames, if the NES CPU is so busy handling game physics or sprite placement that calculations overflow into the Vblank period, the PPU will fail to refresh the screen resulting in frame drop. Generally the CPU will not poll controller data during frame drop events either, but I'm not aware of any games that deliberately run at 30 or 20fps. Just sometimes when there is a ton of sprite activity onscreen, the game stutters resulting in slowdown. SNES is also known for slowdown in certain genres, being slower clock speed compared to competing Genesis/MD and Turbo/PCe systems, even though SNES had the most powerful CPU in terms of capabilities.

[keepdreamin]

Disagree on polling only once per frame (no matter how the SNES does it).

Take SF II hadouken, or the fireball, the pad needs to register 3 directions + button and I believe it should be possible to perform it in one frame (I believe all SF II are performable in a single frame but I am not entirely sure).

 

I don't think so. An eye blink takes at least 100ms. I don't see a 1/4 roll + button press being doing in 16ms. That seems implausible.

[Andromeda Stardust]

I don't think so. An eye blink takes at least 100ms. I don't see a 1/4 roll + button press being doing in 16ms. That seems implausible.

I agree with you; one frame is BS. Generally one button input per frame. The Hadouken Z-move on a Dpad or joystick would take a minimum four frames or 66.7 ms to excecute, and only a TAS-bot would be able to execute the commands that quickly. A human player would actually take longer than that to execute a diagonal or depress the button much less hit four positions.

 

Disagree on polling only once per frame (no matter how the SNES does it).

Take SF II hadouken, or the fireball, the pad needs to register 3 directions + button and I believe it should be possible to perform it in one frame (I believe all SF II are performable in a single frame but I am not entirely sure).

If it takes 4 frames to perform an hadouken(1 per direction + button) we're talking 68ms .... seems a little off, but then again I have not checked the timing of the move.

What he said.

[veelk55]

I love how in depth we got with the input lag thanks to 'narshe's' post. I feel enlightened.

I found this video just now:

Can anyone guess in milliseconds how long each one took?

Edited August 29, 2016 by veelk55

[+phoenixdownita]

I love how in depth we got with the input lag thanks to 'narshe's' post. I feel enlightened.

I found this video just now:

Can anyone guess in milliseconds how long each one took?

No need to guess, at 120fps it seems the nes version moves 4 frames ahead of the Analogue NT which at 60fps translates to 2 game frames (just go frame by frame)

The other 2 are even worse, but the fceux Legacy Collection being the most lagging.

The whole thing is 7 sec at 6fps of a 120fps capture -> 42 frames -> 0.35sec of real time, so 4 frames is 10% -> 35msec

 

In any case everyone seems to be thinking human hand-eye coordination cannot notice 16msec (1 frame at 60fps) .... if that was the case then why going 60fps and not instead just 30fps .... because you'll notice and we're talking about going from 17ms to 34ms. Look at the 1080i vs 1080p full -> the full is 60p (nothing else), why bother if you cannot see it, or react to it again because you can notice in terms of fluidity and reaction time.

 

Can I react that fast? Maybe not anymore or only occasionally. Can my younger me react that fast? Likely.

Would I still notice? Absolutely.

 

http://30vs60.com/

http://www.30vs60fps.com/

 

EDIT: I understand here we are talking a constant lag rather than a stutter or reduction in frames, still the lag is noticeable in some kind of games (turn based are not one of those, twitchy shoot'em up are). If the rhythm of the game is somewhat slow and tuned to give you a perfect response at 30fps then a lag of 17ms may not matter, if it has been tuned for a 60fps experience than adding an extra frame of lag can throw off your timing, adding 2 or more makes it aggravating.

There are ways to compensate for that, modern games may have to acknowledge the user TV would add 1 or 2 frames and compensate for that, but bitd CRT did not add any relevant lag.

[veelk55]

fceux being the most lagging... do you mean MegaMan Legacy Collection on the bottom right?

Cause fceux is the one on the bottom left.

 

Also, it just occured to me that the brain does take some time to feel the button press on my finger, probably happens after my eyes have seen the change on screen which is PROBABLY why I said earlier that the game looks like it's predicting my moves

[+phoenixdownita]

fceux being the most lagging... do you mean MegaMan Legacy Collection on the bottom right?

Cause fceux is the one on the bottom left.

 

Also, it just occured to me that the brain does take some time to feel the button press on my finger, probably happens after my eyes have seen the change on screen which is PROBABLY why I said earlier that the game looks like it's predicting my moves

Yes sorry.

 

BTW it's not just reaction time to a new situation (like a new bullet appearing on screen) as that can be in the order of 100msec-200mec or more, it's also the fluidity to know that once you engage muscle memory what you see happening on screen corresponds that what you are doing, that requires very little lag.

Then again it is very likely that very many games would feel just all-right as they were never coded to support such a high rate of change anyway, the few that did may feel a little odd, but 1 to 2 frames may still be in the realm of acceptable, not sure at what point it becomes just plain unbearable.

 

I can tell you that I had the Onlive mini console for a time and I couldn't play almost any of the Sega 16bit offering as I would die due to be off-time, my connection at the time added 14ms lag, not sure about the rest but likely around 150msec, I did not have the same issue for other games they had, those seemed to be more attuned to a sort of built-in lag.

 

EDIT: and regarding SF there is this

http://sf4answers.com/questions/117/what-does-frame-advantage-and-frames-mean

 

Which basically states that some moves do take 3 frames to start, but there are situations in which the game acknowledges a 1 frame advantage, for that to make sense it means that 1 frame can be enough to at least start another move. So I was off when I said fireball all in 1 frame, but still something may be counted against a single frame.

 

EDIT2: I was way off on the fireball

 

SF4 frame data

http://www.eventhubs.com/guides/2009/feb/17/how-read-frame-data-street-fighter-4/

SF4 Ryu frame data

http://www.eventhubs.com/guides/2008/nov/13/ryu-frame-data-street-fighter-4/

 

it seems the startup of a fireball is 13 frames, still there are moves at 3 frames, so lagging them by 2 or 3 frames would feel noticeable, don't you think?

[i reckon how that synchronizes with the actual pad moves and button presses I do not know but I expect there to be some sort of feedback as to not having to do everything upfront]

[TheRedEye]

Hey, I was the producer of Legacy Collection. It's not emulation, but it behaves exactly the same as an emulator would. For input lag purposes, you should treat it as if it were emulation.

[brentonius]

"It's not emulation, but it behaves exactly the same as an emulator would."

 

What does this mean exactly?

[brentonius]

"It's not emulation, but it behaves exactly the same as an emulator would."

 

What does this mean exactly?

[veelk55]

Oh wow the producer? That's awesome!

Well, I have another question as well, now that we're here.

How come on PS4 and PC, both USA and JAP modes play flawlessly, framerate wise, but in 3DS, MegaMan V (USA mode) has horrendous framerate?

I noticed that all games stutter every 5 seconds in 3DS as well, but MegaMan V curiously has it the worst.

 

Also I'm sad to hear that this type of porting won't be any advantage input lag wise. Isn't this called "High Level Emulation"?

 

Thanks!

Edited August 31, 2016 by veelk55

[Gentlegamer]

Hey, I was the producer of Legacy Collection. It's not emulation, but it behaves exactly the same as an emulator would. For input lag purposes, you should treat it as if it were emulation.

 

So each Mega Man game was reprogrammed to run natively on each platform?

[Andromeda Stardust]

 

So each Mega Man game was reprogrammed to run natively on each platform?

I'm skeptical. This requires a lot of effort and may introduce more bugs. That said, Nintendo ported a lot of NES and SNES games to GBA.

[brentonius]

Well if it was reprogrammed to run natively then they did a strange job of capturing audio. The diff. between in-game music between original console vs Legacy Collection is quite big and sounds emulated. I'd describe it as "cutty" audio if that makes sense. It would have been nice if it were properly sampled in wav format and played out without dropouts on certain channels while doing sfx.

[+phoenixdownita]

Did we at least settle that some amount of lag (be it 1 or 2 or maybe 3 frames) is actually noticeable in some of the twitchy games of yesteryears?

And anything more than that can throw timing off in certain classes of games?

 

Hence much effort in the FPGA recreation world should be put into striving to introduce as little lag as possible?

[Newsdee]

FPGA are inherently free of lag. This guy does quite a good job explaining why in his introduction to the MiST:

[+phoenixdownita]

If you stay ahead of the normal refresh rate of the original console by a couple of lines, yes.

But keep in mind that if you want HDMI you also need to stick to a pretty strict 60Hz, and that may require you to either buffer the full screen (lag) or change the work frequency of the original console (like kevtris has done for the NES, it runs a tiny little faster to make sure all is in sync at exactly 60Hz, he explained it here on this thread, he used the same tricks on all of his cores).

So told not all consoles/games are amenable to such strict 60Hz speed-up/slow-down as it may be, in specific some games were bitches and tweaked the VDP/VDU to produce out of sync and weird signals that a CRT can handle but a strict 60Hz framebuffer cannot (there's a damn game on CPC that I cannot play even over an XRGB Mini because it dynamically changes some timing on the 6845 CRTC mid picture [split screen where the top portion does one resolution the bottom a different one]).

 

So even on FPGA one can foresee a design where a full frame is generated fully buffered in memory (internal or external to the FPGA, at 320x200 pixels it's just 64K for 256 (8bit) color display, twice that for 64K colors) in digital form and then rendered to screen in a separate step, via HDMI encoder chip or otherwise.

Not saying many consoles would required to be like that (as kevtris cores attest) or that the speed-up would hamper the game (check the MVS refresh rate of 59.18 Mhz, either you speed up the CPU 1/60th when you push everything at 60Hz or there's less time during the VBL if you only force the VBL to be at 60Hz) but that some level of compromise would be required.

 

The Mist with being VGA only may get away by the fact there's some leeway in the VGA specs being still an analog RGBHV modulation and the monitors were usually able to compensate, a modern LCD over VGA likely does full framebuffer for a whole range of signals anyway ([email protected] anyone? or 1024x768 @43Hz?).

Let's not even talk about trying to replicate some full on arcade games: RType is a 384x256 at 55Hz ..... sure we can speed it up by 10% to reach 60Hz but then again it starts to get noticeable or you could duplicate 1 frame out of every 10 (that is if you have full framebuffer) it will introduce a little stutter (maybe noticeable maybe not, I don't even know anymore) and then compensate the extra half frame per second with a missed duplication that happens only once every 2 seconds .... but as you see it gets hickey.

 

Case in point I have a Sega Genesis connected to my LCD HDMI via XRGB Mini, the mini can be put in "pass-thru" mode where the HDMI timing is screwed up to follow the original timing within reason and as long as your TV can accept it (mine does not) you see no stutters, or it can be put in "locked" mode where the HDMI out is fixed 60Hz and then the original signal needs to be buffered in different ways at different stages (my TV likes this obviously) but you get the occasional stutter (either missing frame or duplicated). I have no choice so I play in locked mode and deal with the "stutter".

 

EDIT: just to be clear if kevtris gets his Z3K working by speed-up/slow-down whatever he's replicating I would not complain one bit and accept the occasional and rare game that may behave slightly different than on original hardware. I'd just reckon it's part of the compromise, or I need to put up with stutters and lags (as with the XRGB Mini btw http://junkerhq.net/xrgb/index.php/XRGB-mini_FRAMEMEISTER#Does_XRGB_Mini_add_input_lag_.3F)

[TheRedEye]

"It's not emulation, but it behaves exactly the same as an emulator would."

 

What does this mean exactly?

 

They were emulated internally but all the emulator commands were statically recompiled to C. It behaves exactly as an emulator would, but without there being any actual emulation code in the shipped product. We believe emulation is the best approach for republishing NES games, but clients kept telling us that we weren't allowed to emulate a Nintendo system (which is bunk), so we came up with a solution that did the exact same thing but with no "emulation" on the user end.

[Newsdee]

One thing to bear in mind is that lag isn't just a problem of one component but is introduced by the CPU juggling with independent systems, which themselves may be running at different frequencies. So they end up waiting for each other.

 

USB devices transmit at different rates, a low quality one might be running at 100hz which means it only updates every 10ms. A high quality one like a high-res mouse or the Atari-daptor runs at 1 Mhz so updates every 1ms.

 

But then the PC (and the emulated CPU inside it) has to be able to poll fast enough to catch updates, which can introduce additional lag. Then the same PC CPU needs to run a framebuffer to upscale the video which adds some ouput lag as well. And it can only do all of this sequentially. This is why you need a 3Ghz+ machine to accurately emulate a SNES with Higan.

 

How is an FPGA better? It's a parallel electrical system so everything is running at the same time. If the USB polls at 1ms the simulated NES CPU can see it right away, it doesn't need to wait for the host PC CPU to complete other tasks. For video it's the same, you might have a component running a framebuffer (which adds a bit of lag) but it gets the data straight from the NES CPU, again without having to wait for other stuff to complete.

 

TL;DR: FPGA implementations of old machines are awesome

Edited September 1, 2016 by Newsdee

[Newsdee]

Re: XRGB upscaling lag, yes but that display lag is only a tiny fraction of what the video above is showing (I'm sure there's some controller and emulation lag going on). MiST + XRGB should be closer to the Analogue NT part of the video.

 

I don't think we can avoid some upscaling lag if we only use displays that only accept exact 60hz or 50hz frequencies. As you say the MiST gets away with it outputing slightly non-standard VGA with various results that depend on the display (it works in general but sometimes you need to tweak LCD settings for best results).

 

To be fair though, this shortcoming is the same as the original machines. So it's authentic at least

Edited September 1, 2016 by Newsdee

[roland p]

Maybe something like v-sync can be used to address frame rate differences (instead of speeding up/down the emulated system to match the 60Hz at the output).