Bird watchers don't expect a return from their hobby. I know people that have spent thousands of dollars 'Pimping" their cars, which they never get back. My son's hobby is fishing; he thinks it's a 'good' investment to spend $1200 on a new fishing rod, even though he has quite a few already.
So, why do we 'expect' that our hobby will be a good investment? I would guess that it is because stamps are closely tied to a nation's currency. Hence they are never separated from their original face value. Also the fact that almost all stamp catalogues are in fact dealer's price lists can be confusing and misleading to their true value.
Go to a dealer tomorrow to buy stamps and he will quote you non-stop the CV's and how much better his prices are. Return to the same dealer the day after to sell your stamps and CV's will be scorned and mostly ignored.
I love
ebay sellers that quote a CV and then their price at a tenth or a twentieth of that CV, they don't see the immediate implication that what they are selling cannot be worth the stated CV!
If you collect stamps as an investment, then you are likely to be disappointed. If you collect them for fun, you will never be disappointed. Either way, your inheritors will be glad you did not spend that money collecting and emptying whiskey bottles! :)