I cannot speak for this at all medically, but I've often wondered about covers and other mail that has been retained from say, TB Sanitariums, Smallpox Quarantine hospitals, etc., though the years. I seem to remember stories about trainloads of mail destined to/from these locations that were doused with disinfectant so as not to spread disease to others.
More recently, following the 9/11 disaster, wasn't there some modern version of an electronic means to neutralize any potential contaminants on mail being delivered to US households, along with the recommendation that people wash their hands after handling that mail?
As I said, I cannot speak about it medically, but I think it's reasonably accurate to suggest that after a given amount of time, any disease or germs on the stamps or covers would eventually die and therefore pose little threat to collectors.
However, the idea to wash one's hands after handling such items are certainly a good idea anyway. In reality, anything we touch, especially things like currency and coins, that the general public uses on a daily basis, can carry disease. There have also been reports of garbage-hunting collectors that have no concerns about rummaging through trash or dumpsters to retrieve collectible stamps.
I suppose one could carry this idea to the extreme and not collect anything!

However, as a practical matter, I am not aware of any collectors becoming sick because of their hobby, so I suppose while it can be an issue, it's not a very serious one or we would hear much more about the dangers associated with it.