I've collected stamps since the 60's and have never hidden the fact that I collect stamps. I don't proselytize but if it comes up in conversation, I'm happy to discuss it.
It has paid off a number of times when a non-collector would save some interesting "stuff" for my collection.
I'm not a closet collector but my collection is in a closet.
It's the closet in a spare bedroom. No light and temperature/humidity controlled. Great place to store a collection!