Maybe some of you already know this trick, but if there is anyone out there like me who want to see Vectrex games in color on MacMESS, but for one reason or another can't get the overlays to work correctly, you can get color vectors, even though it will be in monochrome.
Hit cmd-Return to get the main menu.
Scroll to the Dip Switches option.
Turn on the 3D Imager, then scroll down to the color options and change the color to what you want.
If you don't choose the same color for both eyes, the game will alternate between them, or choose COLOR for both Left and Right eyes, and vectors alternate from red to green to blue. A weird effect, but kinda distracting to play on some games.