A couple of weeks ago was the Classic Game Fest in Austin. There was one corner block that had a bunch of assorted homebrewers, particularly Atari 800 and Sega Genesis. Then I realized I had my laptop and tried to see if I could get my old stuff running, as I knew my make files were smart enough to start an emulator.
The first thing that surprised me was getting a bunch of signed vs unsigned char warnings. It seems I had a newer gcc installed than when I had worked on that old code. After quickly changing them to plain (aka signed) char, my RPG demo wouldn't display the map tiles. I had a long time ago had troubles getting GCC to work with C++, in that the initialization code for C++ was emitting 68020 BSR.L instructions, which don't work on a plain 68000. I could never figure out where they came from. I had a script which built GCC 4.2.4 from scratch, and I thought it was something wrong with the way it built. I remembered that at one point I simply didn't put in the C++ initialization, and copied the m68000 libraries wholesale into some other target variants. I quickly wrote up a simple test of C++, and the initialization code worked. Hey, I like this newer compiler.
But it got me started playing around with the stuff again. I looked at my "SDK" libraries, tidied them up, used a few GNU assembler macros to clean up cartridge header stuff, and other tweaks. I decided that I should use this "mystery" compiler, and make basic C++ support work in my SDK. Then I was shocked to see how large my binaries were, so I created a lite printf. I saved about 30k over the "default" printf, easily more than half the size of the binary. I looked at my font loader, which was a big array of 0x11FFFF11 stuff, and used some macro tricks I had come up with since then to make "__XXXX__" macros where I could visibly see the graphics in source code. And then I cut out the blank 00-1F characters to save even more bytes in the simple demo I was working with.
Then I started trying to figure out where the hell that mystery compiler came from. I thought maybe it was a pre-built that I had downloaded from somewhere, but there was no evidence of it being from brew or macports, and nobody else was using 4.2.4. (The pre-builts that I had downloaded were for ARM.) Then I looked at the file dates, and found it was almost the same as the last change to my GCC build script, back in April 2016. That was apparently my previous attempt to play with Genesis code, but I had wanted to get C++ working first. This was actually the one where I had copied the libraries in an attempt to get C++ initialization to work properly. Then, not being able to get it working like I wanted, I gave up. My scripts used 4.2.4 because that was the highest version I could compile with my installed C compiler without having to install some other crap. So two days ago I started working on the build script on another computer (so I wouldn't mess up what I already had working) and tinkered around with it. Eventually I found what generated BSR.Ls that caused my original problems, in crtstuff.c, and was able to confirm which GCC library targets had them.
Then I realized my original problem was that I didn't understand some things about how GCC multi-target libraries worked. In particular, I was using "-m68000 -msoft-float" in my make files. The problem was that soft-float is only for 68020 and later. GCC already knew the 68000 had no floating point hardware, and would have compiled soft-float anyhow. Using "-msoft-float" (in either order) actually overrode the -m68000, selecting a library compiled for 68020, which used BSR.L instead of JSR. Note that the only thing that BSR.L would get me, aside from not working on a regular 68000, was position-independent code, which isn't very important when you're compiling to ROM. I also realized that the "default" target if you selected no CPU type was probably m68020. And then I realized that all this was merely the canary in the coal mine in that all the newlib libraries were compiled that way too, so LBSR would have been a problem later, but harder to notice than a problem with startup.
tl;dr: my problem was that using "-msoft-float" caused it to select 68020+ libraries.
And my old code was older than I thought, dating back to 2009 and 2010, before I got my big job in late 2010. I guess I had never tried them with the newer compiler in 2016, or I would have seen the signed char warnings back then. I was too busy trying to get support code working. Also, my current code formatting preferences must have started in 2010, so I've got to reformat that old stuff too.
So yesterday I started pushing my new library/SDK code to be my new default. I got Tubes to work, after finding a minor bug in my printf, but the RPG still won't show map tiles. And if I want to use a bit of C++ (I use C++ in embedded projects as "C with classes" aka "structs with functions"), now I can. I still need to clean up a lot more stuff, but I'll save that for when I actually have time to work on this in a couple of months. And that code savings with a lite printf? I looked at the old .bin files and that ~30K bloat was in the old stuff from 2009. Yeah, I really don't think I need printf to support floating point.