To a very fine powder (or, worse, a very fine flake), paper is a very coarse surface, with lots of tendrils to cling to.
Try to imagine a large kite coming to rest in a forest canopy.
In most cases, brushing will only make things worse, rather like pushing the kite into a tree.
I suggest you try static electricity.
http://www.wikihow.com/Make-Static-Electricity has UN-VETTED suggestions on how to generate static electricity.
I'd rub a rubber ball on a nearby furry animal, child, or life partner, but that's me.
However it is done, rest the charged item on your stamp for a moment, or vice versa.
I think you'll find the talc leaving the stamp.
Ground the stamp when you are done, eg, touch a large metal object with one hand while you cover the stamp with the other hand.
Rinse (the rubber ball), and repeat.
For extra credit, try generating that static electricity on a plastic rolling pin, and see if you can do pages instead of stamps.
Cheers,
/s/ ikeyPikey