Hello EgCollector, welcome.
Well, you have decided the big question, what do you like.
There are catalogue books printed that list most of the stamps issued from 1840 or so to now. The stamps are arranged by country and then by date issued, with each stamp having it's own number.
Scott cataloge is used in North America mostly.
Stanley Gibbons catalogue is used many places.
Michel is from Germany, Yvert et Tellier is from France, Yakura is Japan.
These can be specialized catalogues that go into great detail about stamps and there varieties of paper type, perforations (perfs) 9the serrated edges around most stamps), colour, printing changes, etc etc.
Unitrade is one specialized catalogue just for Canada and it's Provinces.
Scott makes on for the US.
and so on.
These are the numbers you might see online, say on eBay or elsewhere.
Scott = Sc, Stanley Gibbons = SG, Michel = Mi, Yvert = YT, etc.
Ask phil is a good web site that has lots of info and terminology for beginning stamp collectors.
Always remember, do it your way. Other's ways may be good for them but you may want to branch out or follow your own pathway.
You can display stamps in a paper paged album, in cardboard stock cards, on plastic black or white pages that have clear strips to hold the stamps in, there are plastic mounts similar to the plastic pages but you cut them to size, and the big one used by people who do not have tons of money to spend on display, is little hinges, made of a type of waxy paper with gum on that you link or moisten and stick one part to the stamps and the other to a paper page of some sort, some better than others.
Lots of different ways.
www.subway.com is a stamp and supplies shop that is good. There may be local shops closer to you. You can buy from eBay or online or other auction sites or go to a local stamp club or bourse (sale) or maybe there is a stamp shop near you.
have a headache yet? Lots of information to read.
You can look through the forum here and there are lots of topics and threads and I always see stamps I have never seen before and that are amazing.
If you are good at drawing or art you can even make your own stamps from different countries around the world. Like my avatar stamp, they are personalized.
This is what I collect but seeing and reading on here many other stamps and ideas has made me expand my horizons and collecting interests.