Ok, for the good of the group, I think I know what's going on.... I read this document:
and I realized that I haven't paid any attention to the load capacitance value of the crystal I'm using. Since that value (load capacitance) can be all over the place, the schematics I referenced are all over the place for cap values, which is the root issue. The other stuff (using inductors or inverters or whatever) is probably ancillary (though perhaps a good idea for some other set of reasons I don't yet understand).
So, using the adafruit blog calculations, I get two different potential ranges for capacitors: 34pF to 40pF, or 48pF to 60pF (depending on stray capacitance, which I'm estimating to be between 2 and 8 pF). So, vaguely speaking, I should be using capacitors somewhere from 34pF to 60pF. I'm using 33pF caps... duh.
This also explains why I seem to be right on the edge of working (either no signal at all, or a luminance only signal). When I added jumpers from the caps to the crystal, I probably introduced more stray capacitance, which lowers the needed cap value slightly, which probably got me juuuuust inside the range where the VDP could put out a usable signal -- still not good enough to produce a proper colorburst after the front porch in the NTSC signal, but good enough to at least get the levels right for the luminance portion. Sort of makes me marvel at the ingenuity of NTSC actually. I suppose if I just kept randomly soldering junk onto the traces for the caps, I'd eventually get enough extra capacitance that 33pF would work, but that's hacky.
So, if I average the average of those two different approaches to calculating my capacitor values, I get 46pF, which is pretty close to a standard 47pF ceramic cap. So I'll start there. Since I don't have any of these caps on hand, I'm going to just order a couple of everything from 30pF to 60pF and work out from the middle until I get something nice and stable.
I do really wonder about the Byte magazine schematic though (the one I based part of my schematic on, derp)... it makes no sense based on what I now know. It's got two 33pF caps attached to the crystal. Assuming the load capacitance for the crystal they were using was something like 24pF, those 33pF caps would be about right. However, just jamming a 5-60pF variable cap in parallel to one side of the crystal makes no sense to me. Let's say you're trying to compensate for the fact that a 5% tolerance would allow a "33pF" cap to be anything from 31 to 35 pF. Okay, add a variable cap that ranges from 0 - 4pF to both legs. Adding it to one leg is just going to cause the crystal to not oscillate properly, right? I mean it's going to be additive on one side, so you'll have a cap on one side that ranges from 38pF to 90pF. What good is that? The article talks about using it to slightly alter the color signal, but I'm not sure I follow that at all. How does screwing around with the master crystal cause just the colorburst of the NTSC output to shift? To shift the colorburst, wouldn't you have to do some kind of post processing of the composite signal? I don't see how you could screw with the input to the VDP clock to get that result, at least not without having other potentially weird effects on the VDP.
Edited by calphool, Wed Aug 1, 2018 12:24 AM.