There used to be a stamp shop near Bryggen in Bergen, but it seemed to have moved away from that area and joined forces with Skanfil.
https://www.skanfil.no/info?id=339&Shop=FalseUnless things have changed since I lived in Bergen there should be plenty of buses going to the area, although you might have to walk a bit from the main route of the buses.
Depending on when you are going the stamp club in Bergen could be an option as well. Never been there myself, so have no idea if the club is worth a visit.
http://bergens.filatelist-klub.no/