Although it's not something you can hold in your hand, much less collect, antimatter is actually the most valuable substance by weight. Its cost has been estimated between tens of billions and trillions of dollars per gram. (
https://en.wikipedia.org/wiki/Antimatter#Cost ). A concentrated gram of it in one place would be exceedingly dangerous, though, and usually it's made and studied a few particles and atoms at a time.
Rare stamps like the penny magenta and treskilling yellow are indeed the most valuable
objects by weight, though, as antimatter wouldn't really count as an object as there's never been enough of it in one place to even been seen with the human eye.