Several bits of learning and news to share here.
First, the mystery of why skunkbios_3.0.1-noverify.COF failed to flash when I was bootstrapping my first working board is solved, and it's rather silly. It's because it was stuck trying to connect to the jcp console. I realized this when I saw there's also a -noconsole variant in @Tursi's skunk_bios github repository here. I verified plugging in a USB cable while running bjlSkunkFlash and running "jcp -c" when it gets stuck gets the -noverify variant working.
Next, why was drag soldering so easy on my practice boards but super-hard on these Skunk PCBs? Because the Skunk PCBs don't have solder mask between the pads and the boards are very likely made using a low-quality substrate. Here's a close-up look at a fresh one (Thanks @Saturn for sending these over):
See how the boards are blue everywhere except around the gold pads where chips are mounted? That blue stuff is the solder mask, and it repels solder like oil repels water. However, it is difficult to manufacture boards with tiny solder mask bits between copper pads. The process that lays down the mask layer over the board needs to be precisely aligned or the mask will overlap the copper, giving you bumps that could cause your chip to not cleanly contact the pad. To account for that, you have to widen the "cut-out" for your pads in the solder mask when designing the PCB. Different manufacturers have different abilities, so if you're using a cheap manufacturer, you have to enlarge the opening more than for others. Further, if you try to put down too thin a sliver of solder mask, it might not result in a nice flat line of mask, but rather could splinter or bubble. This also varies, I believe mostly based on the quality of the solder mask material and process used by the manufacturer. Taken together, you basically get some minimum pad spacing for which you can lay down a solder mask between the pads. If the spacing is lower than that, you don't get solder mask between those pads. Some manufacturers will even fix this up for you and remove solder mask regions too small for them to manufacture. These days, good manufacturers can lay down a mask between pads with the relatively fine spacing (0.2mm) used here, but the skunkboard design files were very conservative by today's standards and had a very large clearance around their pads.
TL;DR: No solder mask here means solder is more likely to bridge across pins placed on these pads when drag soldering.
The "low quality substrate" bit relates to why pads were lifting off these things like crazy. Granted, I was pretty crap at soldering when I started off here and I'm still not great. What that means is I was keeping the iron on the board a lot longer than was ideal, especially when trying to use solder wick to remove solder bridges when I messed up, partially caused by the lack of solder mask. If you're using cheap substrate, such as TG-130/TG-140, it will start to burn pretty quickly, and then the pads will lift right off of it.
The owner of Osh Park goes into detail about these and other issues you'll run into manufacturing your boards at a cheap fab in this reddit post.
How did I learn all this? Well, I've been busy. I finished laying out my changes to the skunk PCBs and was looking into getting some prototypes built. I learned all of the above while researching and comparing the capabilities of the various PCB manufacturers members here recommended. Ultimately I decided to go with Osh Park for the initial prototype boards, as @grips03 recommended. It didn't hurt that all Osh's boards are purple. My younger daughter's favorite color is purple, and while I cycled through the various solder mask colors in a gerber viewer with her on my lap, she insisted the boards looked best in purple. Everywhere else, purple costs a little extra. At Osh, it's your only option!
I sent everything off about a week and a half ago, and got a package from Osh on Monday. Behold, Purple Skunks!
Note I couldn't resist tweaking the Skunk logo a little
The boards needed a little cleaning up. They had some nibs on them from where they were attached to the others in the panel, and they weren't beveled (Osh doesn't offer beveling). Nothing a little sanding followed by an alcohol wipe couldn't fix:
Note I did this outdoors with a respirator on. PCB dust is nasty!
As mentioned above, the key to my changes was routing some GPIOs from the EZHost USB controller chip through an AND gate to combine them with the Jaguar's GPIO line used to toggle the chip select line on the Serial EEPROM to allow enabling only one of two Serial EEPROM chips at a time. To fit this on the available sliver of PCB real estate left on the skunk PCBs, I had to use a tiny XSON8 package, and I was pretty nervous about being able to lay down tiny enough globs of solder paste by hand to get this working without shorting pins. However, I got it working. I did short at least one pin on all 3 boards, but I was able to easily clear it by covering the exposed portion of the pads with some flux and touching my soldering iron tip to them. This pulled enough solder out from under the chips to clear the shorts all but once, and in that case, I just applied more flux and hit it with the hot air gun again to get the solder to flow out onto the pads. Here's the chip sitting in a little-but-not-little-enough puddle of solder paste:
And after reflow + fixups:
As you can see, the whole IC package is about the size of an 0603 SMD component.
After soldering all the solder-paste components (This XSON8 IC, which is U7 on my layout, and U4, the USB power regulator IC with a heatsink pad on the bottom), I sprinted through the build of a single board to verify everything was working:
Here's a tip if you're doing some drag soldering: After you've tacked the chip in place, hit the pins and pads with the rework hot air gun on a relatively low setting for a few seconds to pre-warm them. Then quickly cover everything in flux and do the drag. Warming the pins and pads seems to prevent the solder from instantly solidifying when it touches them, allowing it to drag along with the iron as desired. This is probably all the more important when working with a relatively cheap iron like I am, since it can't compensate for encountering the thermal mass of a pad connected to a ground plane or even a handful of pins on an IC. If you don't have a hot air gun, don't despair if your first drag across the pins leaves a mess of bridged pins. Just quickly reapply flux if it all burnt up in the first pass, and drag the iron (very lightly tinned, but not naked) over it again to drag the solder away from the gaps. The pins & pads should have been warmed significantly by the first pass, achieving roughly the same affect as pre-warming them with the hot air gun, and you should get a much cleaner drag, with the solder flowing onto the pads and pins and away from the gaps. Repeat if needed.
After I got all the critical components on:
I stopped to test basic functionality. I was able to find the Cypress chip via USB, flash the CPLD, and flash the BIOS without any hitches, so I soldered on the remaining components:
And ran it through the test2.sh bringup test suite, which it passed. I ran it once more to be sure, and went to bed. It took me roughly 4.5 hours /(~2.5 @Shinto Jaguar game-by-game podcasts), of non-stop soldering and futzing with JTAG and stuff to get this far, and I was exhausted by the end of it. This was the first time I'd built an entire board (plus part of two others) in one sitting.
The next day, after a good cup of coffee, work, and family stuff, it was time to test the actual changes I'd made. I'd improved on the software I wrote above by integrating the EEPROM dumping and writing directly into a build of JCP, and merging in some code I found by @Songbird (Thanks again!) to support 93C86/2048 byte EEPROMs as well since the board above has one 93C46 (128 byte, standard for non-homebrew Jaguar carts) and one 93C86 (Many newer homebrew carts use this). The updated EEPROM code is already on my e2pmgr github repo, and I'll post the updated jcp code soon. It needs a little cleanup, as I still have to select between 128 and 2048 byte support at build time right now.
After a bit of debugging, it all worked flawlessly, so all the new HW functionality has been verified. I'm quite pleased with this, as is my older daughter, who is also fond of purple, and just shouted "PURPLE!" when I showed her the finished board yesterday morning. It's my first attempt at PCB layout and manufacturing, and it went off without a hitch.
I took a break to do some non-Jaguar stuff tonight and write this up, and I'm going to read up on all the local candidates and propositions and do my vote-by-mail thing tomorrow night, but the next thing I'm going to attack is the BIOS. I have a scheme worked out to stash EEPROM selection settings in an unused sector of the BIOS area of bank 2 (Or the top 2MB when using 6MB mode), but toggling the GPIOs from the skunk didn't go as smoothly as I'd hoped. You apparently can't write to the EZHost control registers using HPI DMA mode like you can the other on-chip memory addresses. Doing so just locks up the EZHost chip, as it appears someone else already learned, based on comments in usb/main.c in the skunkboard full release archive. Luckily, there's an HPI mailbox protocol you can use for this instead, and skunk CPLD appears to support it. I have all the code written up to do things that way, and it runs without reporting any errors, but... it's not actually toggling the GPIOs for some reason. Debugging needed.
Also, at some point, I'll build up the other two boards, and perhaps more if others are interested. If I order more, it'll have to be a relatively large batch to be economical (Basically, it costs about the same to order 50 as it does to order 5), so I'll certainly have spares, and if I can sell a handful or so, the whole project would be self-financed. However, we'll have to see if anyone still wants a skunkboard now that game drives are well and truly just around the corner.
Regardless, I'll be pushing my full PCB designs, BOM, updated JCP code, and hopefully, updated BIOS code (once it's working) to github soon as well, along with very, very verbose notes for anyone who really wants to dive into the details (I know, who knew it could get even more verbose than my posts here?), all public domain like the existing skunk files.