I would think it depends on what you think is OK personally as stamp collecting is such a personalized hobby.
If it is really bad you might hide it from general view but know it is there and bring it out at times to enjoy it.

The expense of the idividual item would affect me in this area also. Sometimes I would be happy just to have a copy of the darn thing in any shape. Stamps / items in bad shape can still be used as coloyur guide or have great cancels or paper or perf guides to help you or others judge what a real one of those would be like. I have seen damaged sets of stamps sold on
ebay for just such a reason. Study items. Items had just for study purposes. But not everyone is into studying everything too.
Cancels are another reason to keep a nad condition stamp. Might be a rare cancel, a good example of a cancel, a real cancel as apposed to a possible fake or vise versa.