Yes, there is a popular misconception that any old stamp is valuable... It is wrong, but as someone who recently restarted collecting, that is part of what makes stamp collecting very accessible.
The internet has only made it more so. I'm at the stage of filling in my collections with inexpensive stuff... Right now, I can go on
ebay or wherever and buy enough to entertain me for hours for a few bucks

When I find a stamp about a subject I don't know about, I can look it up on Wikipedia and learn even more than I ever would have otherwise.
Try that with any other hobby... how much model railroad equipment can you get for a few bucks? Maybe a couple plastic trees? How much does a single part for that classic car cost you (been there done that

)?
Yes, as I start needing to complete sets with higher value stamps, I will probably need to spend more per stamp but I can do it at my own pace and a very reasonable budget. In the meantime, I am learning which dealers I can trust to sell me a nice stamp or two along with a history lesson at a fair price - and which to avoid like the plague.