Hey Big Texx.
I've just recently gotten into cover collecting as well. I often am annoyed at notations added to covers by collectors or dealers. I do erase pencil notations such as prices or added dates, as long as they are not too big or heavy. I use a good white gum eraser and rub very gently to remove the pencil.
I'd stop short of removing addressee's names or anything else to due with the origins of the cover, though. The idea of cover collecting, to my mind, postal history. Where that cover has been and to whom is all part of saving them in the first place.
And again, I wouldn't attempt to erase anything if I thought it might damage the cover. Sometimes you just have to live with it.
Here's an example. This stampless cover I've posted on a couple of other threads, has a notation of the year it was written, 1846, in pencil in the lower left corner. I don't know who wrote it. It's obviously not original to the cover, on the other hand the script looks like it might possibly be old.

In the end I decided to leave it because of the age of the paper. It's watermarked J WHATMAN TURKEY MILL 1843, so this paper is almost 170 years old. Better to leave it than risk damaging the paper.