I'm also a recent convert; picked up a system about a year ago in Japan along with about 30 mostly Japanese games, because you can get them for a buck a pop there if you know where to look. It's at the point in its post-market life when it's just outdated but not yet "retro", so it's a buyer's market right now. More people are trying to dump systems and games than are trying to buy them, so there are a lot of great games out there for low prices.
It's also famous for being a really great emulation system. Get yourself a big memory stick (or an SD card with an adapter) and you can walk around with full collections for almost any classic system that play perfectly. There are a few exceptions, and it doesn't handle anything after the 16 bit era very well, but most systems from that era or earlier run great.
I've also played a lot of Outrun 2006, GTA: Vice City Stories, Jet de Go, and recently I've gotten the English subbed version of one of the Idolmaster games and it's just really cute and fun. I always wanted to try those games but they're usually inaccessible to westerners; one of the great things about the PSP is that it's very easy to flash custom firmware so you can run any homebrews or hacked versions of games (you need to do the same to run emulators anyway).
A lot of people don't even bother with UMD's anymore, although I personally prefer to have games on UMD because it's really got the best packaging of any handheld system I can think of. All the games I have come with full color manuals, and the game cases are nice and thick, not flimsy like either the PS Vita or Switch cases (neither of which usually come with any manuals either). I also just love UMD's themselves; they're just so weird! Even having owned a system for a while now, I just think it's so strange to run a handheld off these tiny little optical disks. The system's definitely a curiosity for that reason.