I think the overscan jitter changes are finished. The prior build was done using an older version of Stella's source. I've updated to the latest version and migrated my changes over to it.

TIA.hxx
Change currentFrameBuffer

    uInt8* currentFrameBuffer() const
    {
      return myCurrentFrameBuffer + myFramePointerOffset + myCurrentFrameJitter;
    }

Add two new variables

    // Derived from the difference between the scanline counts of the
    // current and prior frames.  If non-zero the next frame should jitter.
    Int32 myNextFrameJitter;
    
    // jitter amount for the current frame
    Int32 myCurrentFrameJitter;

TIA.cxx
Add new variables to the initialization list:

TIA::TIA(Console& console, Sound& sound, Settings& settings)
  : myConsole(console),
    mySound(sound),
    ...
    myCollisionsEnabled(true),
    myNextFrameJitter(0),
    myCurrentFrameJitter(0)

Locate this bit of code in endFrame()

  // Recalculate framerate. attempting to auto-correct for scanline 'jumps'
  if(myAutoFrameEnabled)

and add this directly above it

  // set the jitter amount for the current frame
  myCurrentFrameJitter = myNextFrameJitter * 160;
  
  // calculate the jitter amount for the next frame.
  // Jitter amount of a frame depends upon the difference
  // between the scanline counts of the prior two frames.
  myNextFrameJitter = myScanlineCountForLastFrame - previousCount;

  if (myNextFrameJitter < 0)
  {
  	myNextFrameJitter = --myNextFrameJitter / 2;
    
    // make sure currentFrameBuffer() doesn't return a pointer that
    // results in memory being accessed outside of the 160*320 bytes
    // allocated for the frame buffer
    if (myNextFrameJitter < -myFrameYStart)
    	myNextFrameJitter = myFrameYStart;
  }
  else if (myNextFrameJitter > 0)
  {
    myNextFrameJitter = ++myNextFrameJitter / 2;
    
    // make sure currentFrameBuffer() doesn't return a pointer that
    // results in memory being accessed outside of the 160*320 bytes
    // allocated for the frame buffer
    if (myNextFrameJitter > 320 - myFrameYStart - myFrameHeight)
      myNextFrameJitter = 320 - myFrameYStart - myFrameHeight;
  }  

CartDPCPlus.cxx
Besides the Overscan Jitter, there's also a DPC+ jitter going on if the DFxFRACINC registers are not re-initialized every frame. When this occurs the screen itself doesn't jitter (as the players/sprites don't jitter), just the playfield.


To fix this I changed this bit of code:

      case 0x00:
      	myFractionalCounters[index] = (myFractionalCounters[index] & 0x0F0000) | ((uInt16)value << ;
        break;

to this:

      case 0x00:
      	myFractionalCounters[index] = (myFractionalCounters[index] & 0x0F00FF) | ((uInt16)value << ;
        break;

Test build of Stella for OS X with the above changes applied:
Stella DPC+ Jitter.app.zip

Addendum: The three changed source files
emucore.zip