Sounds logical. I think the reason they don't get a specialist about stamps is because most stamp collections are boring to a general TV audience.
Unless the item has a unique historical story, like a "B. Free Franklin" stampless cover, there is not a lot that Antique's Roadshow could say to make an interesting appraisal.
Another reason could be that they don't want to waste their time telling people that their wheelbarrow full of their grandfather's lifetime investment in Postal Commemorative Society First Day Covers is not going to pay for their retirement.
photo from e-bay, from an estate sale, in Florida.