Well, for me, "back in the day" was the early 2000s since that's when I got my first Atari.
I used one of those white Radio Shack 9V power supplies with a 45-degree tip adapter on it since mine didn't come with a power supply. My first nine games were borrowed from a friend who had them in the back of a closet at his house: Combat, Pac-Man, Chopper Command, Donkey Kong, Warlords, Space Invaders (red label!), M*A*S*H, Breakout, and Super Breakout. From there I started picking up cartridges when I found them at thrifts (and back then, you still could find them at thrifts); Missile Command, Football, and Grand Prix were among the first I remember. Somewhere around this time I had a gift certificate for the local mall that could be used at any store--I used it at Radio Shack to buy Pitfall from the in-store master catalog they used to have then (it was amazing to me that I could still order new Atari games in, like, 2001!).
I would get new games from Telegames for birthdays and Christmases as well. Defender II, Desert Falcon, Home Run, Street Racer, Pitfall II (I actually had this before I had the first one) Venture, Solaris, Gravitar...man, that was awesome. And a funny juxtaposition--my brother would get Phantasy Star Online for GameCube and I'd get Gravitar, and later we'd run downstairs and play them on the dedicated TVs that were set up near each other, with equal excitement. (This was kind of a thing we did; we played games together, but not the same games, or even the same machine. )
At first I played it on whatever 27" CRT we had in the living room downstairs, which was shared with a PlayStation, and later a PS2 and GameCube. Soon I bought an old 15"(?) TV at a rummage sale and used that next to the "big TV" until I bought a brand new 13" Orion with some money I got for my high school graduation (still have the TV!). I mostly played alone since not too many of my friends were that interested in it, although a few were. My brother played Breakout once, but that's as much as I ever saw him play it. It was actually my girlfriend (to whom I am now married) and kid sister who played the old games with me the most. Lots of warm, fuzzy memories there.