It certainly can get expensive, but you have to really analyze what "expensive" really is to you. I did have high end equipment in the 90's. I was a single guy making a good salary in the computer field and I had skills that many (at that time) didn't. Companies saw it and I was paid well....so I splurged on myself quite often. I purchased the AWE64 Gold for like $300. Two 3DFX 12MB cards for SLI must have run me over $600. I know the Pentium 166Mhz ran me serious coin, and I remember purchasing a 9GB Fujitsu SCSI hard drive for like $900 at some point.
That said, when I see someone selling a Soundblaster Pro for fifty bucks or a Voodoo 2 for $75, I think of it as a steal. Is it a lot for "today's money"? Well, I am not using it in a "today's machine"...it is being purchased for a different purpose and that is to relive what was a fond memory of a good time in my life. I think a lot of people who spend serious coin on Atari games are doing it for that reason (as well as other reasons) and they are spending a whole boatload more than what I am on some vintage PC stuff.
Yes, I agree that you can have a lot of fun with retro PCs for a fraction of what a collection of minor, unfun consoles can cost. I've been selling off my Arcadias, APFs, Nuons, etc. because they just sit there collecting dust.
The best rule of thumb I've ever heard for whether retro equipment is expensive is whether or not it costs more now (say, on eBay) than when it was new at retail. There are not many retro PC components that you can say that about, while there's a fair amount of retro console stuff that does meet that standard (although not as much as you would think).
Here's my favorite build:
CASE: Generic Baby AT with full three-digit LED display. I bought a complete system just for this case. It is very clean and in excellent shape. I used to have a case with an LED display but not one that could display any number past "199." It turns out that this case is the perfect one for my motherboard/CPU combo.
MOTHERBOARD: FIC VA-503+ rev. 1.2. This is a classic SS7 motherboard with the MVP3 chipset and 1MB onboard cache. I use the Turbo switch on the case to change the FSB speed from 66MHz to 100MHz. Because the case has an LED whose display responds to the Turbo switch, there are two three-wire connectors coming from the Turbo switch, both of which are needed for setting the FSB on the motherboard.
MEMORY: 128MB generic PC100 SDRAM.
CPU: AMD K6-3+ 450 ACZ. This one overclocks to 550 MHz with no trouble. I put a Scythe fan on the heatsink, and it is very quiet. The "plus" versions of the K6-2 and K6-3 allow you to set the clock multiplier and disable caches via software (SETMUL).
VIDEO: Geforce 2 MX AGP (about as high as I could go with this SS7) and 3DFX Voodoo 3 3000 PCI (switch via BIOS).
AUDIO: Turtle Beach Montego 2 PCI (Vortex 2) / Creative AWE32 CT3900 with 32MB RAM / GUS Ace
DRIVES: 40GB WD Caviar / Mitsumi FDD with flash reader (SD card super convenient) / LG DVD-RW (DVD function used only for making Ghost 2003 images on DVD-R)
PS: Delta 300W ATX (need to upgrade this).
By disabling turbo and/or caches, I can run this rig at 486-33 MHz speeds, or run it full speed at 550 MHz. Super convenient.