I think this is a game of pick your flaws.
I like the TI, but to even use bitmapped graphics requires a 3rd party utility and the system is extremely locked down.
It also uses ANSI BASIC which is much different than Microsoft. You can get used to it, but I'll never be thrilled with it.
There are also fewer hardware upgrades than for the the Atari or C64
The Atari has good hardware, and a good though non-standard BASIC. The string handling and file IO functions are the biggest limitation.
The standard BASIC is slow, but not there are faster upgrades if you really want them.
For all it's strengths and hardware, it lacks even basic support for the player / missile graphics which leaves you to use add on machine language.
It also requires understanding display lists if you want to do anything very colorful.
The C64 hardware sprites are easier but the BASIC has no Extended BASIC features. Simon's BASIC adds Extended BASIC features, but it's non-standard.
The CoCo and Plus/4 have very good Extended BASICs, but they don't have hardware sprites, but the CoCo does offer software sprites.
The CoCo doesn't have a sound chip to generate noise while you are doing something else, but it does have good sound and music support if you don't mind the computer stopping while it makes noise. It's possible to generate sound at the same time the machine is doing other things through software, but it requires machine language.
The Plus/4 has sound hardware, but it's pretty simple hardware. Every major sound chip out there is better.
The Plus/4's BASIC gives you access to more RAM from BASIC than any machine to come before it, so it's good for writing large programs.
GW BASIC on the PC is almost identical to the CoCo's BASIC.
The CoCo 3 has 16 color graphics that don't require messing with display lists. You can use any of the 16 colors from the selected palette on any pixel. This makes it more like the Atari or Amiga.
The 6502 lacks a lot of things that are required for efficient compiler support, but there are compilers for it.
The 6809 in the CoCo supports high level languages like C and Pascal quite well. If you want to graduate to a Unix like environment, OS-9 will let you do that and there are compilers for several languages that run underneath it. BASIC-09 is a bit of a cross between BASIC and PASCAL and offers a pretty good introduction to compilers. Several commercial games were written in it.
The C128 inherits the BASIC from the Plus/4 and adds new features as well has having more speed, C64 compatibility, and the option of CP/M to learn the Z80.
The Apple II has one of the largest selections of kids software out there. The M.E.C.C. series alone is worth having one if you have kids.
The CFFA 3000 board lets you run most things without copy protection direct from a USB thumb drive or CF card.
This is pretty comparable to what you'd find on the Atari, but you don't have to worry about multi-carts vs disk images, etc...
It has no hardware sprites, and you are left dealing with shape tables or machine language to draw them.
There is no sound hardware and you are left with third party addons or machine language for that.
The Apple IIgs is a different beast than the older Apple IIs. It is clocked faster than any 6502 machine outside of the Apple II line, it has some of the best sound hardware out there, excellent graphics, a GUI, and can support megabytes of RAM.
But to take advantage of that requires some sort of 3rd party BASIC. The 65816 does support high level languages better than the 6502.
The large color palette and mode where you can have a different palette every few lines makes it one of the easiest machines to display a lot of colors on.
Despite the nice new graphics modes, the CPU can't access the screen at full speed.
Some people will claim it's a 16 bit CPU, but the original opcodes take the same amount of time, so... it's kind of a moot point.
If you go with an Apple, you might as well get the IIgs IMHO.
I would also suggest the Amiga. You can do a lot with it from AmigaBASIC, but AmigaBASIC doesn't like extended RAM.
My company Designing Minds used Absoft BASIC to develop for the Amiga. I wrote a C/Assembly library to load IFF pictures, load and play IFF sounds, and to perform color cycling.
The Talking Storybook, Great States II, and World Tour series were all developed with Absoft BASIC and that library.
The library also works from regular AmigaBASIC.
It's pretty cool being able to draw a picture in Deluxe Paint, record some sound samples and just call the LoadILBM, etc... functions and it's just there in your program.
You can find pics of most of them off the following page, and download them from the 2nd link.
If you go with a PC, you can get used thin clients that offer more speed, better features, and better reliability than old systems for around $30 for the machine, and they can attach to the back of an LCD monitor.
They should run any old software a current PC can run, and they can run a lot of new software as well. They are certainly fast enough to emulate any of the 8 bit machines.
You also won't have to mess with a bunch of boards from different manufacturers. You'll probably just have to download the system drivers and create a CF card with everything on it.
You don't say where you are from, but there are some other good options if you are in Europe.
The Spectrum, no hardware sprites, no sound chip on the early models, but it has a decent though slow BASIC.
I hate the keyword entry technique, but it might be better for a child. You can replace the ROM to allow typing keywords out which I find much easier.
The BBC has an odd BASIC, so I'm not so fond of it. But it is fast. It has sound hardware, but no sprite hardware. Running at 2MHz, it's one of the fastest 6502 machines.
The Oric has a good extended BASIC. No sprites, but it has built in sound hardware and there are built in commands for certain sounds which can make programming easy.
The graphics are... unique though. You can get very good results if you know what you are doing, but simple it's not.
The Amstrad CPC has an excellent Extended BASIC from what I've seen. It has sound hardware but lacks sprite hardware on all but the plus machines.
The plus machines are some of the best 8 bits ever made hardware wise IMHO. But they did come out about when other 8 bits started dying off.
People with some real experience on those would be able to tell you more than I can.