Is There A Need For A Philatelic Auction/Sales Search Engine?

I've grown tired of spending hours searching through auction lots looking for specific stamps to add to my collection. If these sites have a way to search for specific stamps, they often don't return what I'm trying to find. eBay has a decent search capability, but many other auction and sales sites do not.

To address this issue, I've been thinking of creating a web crawler (read search bot) to index stamp auction and sales sites and make the information searchable.

Do you, members of the SCF community, see any value in such a web-based tool?

Thanks in advance for your thoughts.

Something along the lines of what Stamp Auction Network (with mixed reviews) already does?

For a brief period several years ago, I scraped the big German auctionhouses while attempting to build a searchable database showing the sales history of certain items, but the inconsistencies in listing were too large of a hurdle. You don't realize how many different ways an auctionhouse can list a single catalog number until you try to automate a search for it.

The problem with scraping sites, is the end result of a sites' html is that it is not consistently structured data, but rather a mix of formatting, scripts, surrounding the content. While you can have moderate success, one slight modification of a web table and your parser will return bad data. Many sites are now dynamically generated and the sites themselves don't care about scrapers, and often try to prevent them or discourage circumventing through complex redirectors, frames and other script tricks.

The ideal way to do this, if you are serious, is leverage APIs of the sites and try to get at the data directly. Whether or not this is available is probably something you'd have to arrange with each house and talk about a mutual relationship of sorts. An exampl might be, Free API access in return for traffic generation, but that is probably a tough sell before you are established...

I'm pretty new to SAN and I can say that the way the dealers have the info listed caused me frustration on the site alone. Some dealer use SG numbers, some didn't. Some used "1d", some "One Penny", etc,etc. So in using their SAN search function I would get many more emails that did not interest me due to faulty results. I finally turned off the email notifications. I can only imagine the problem a crawler would have.

The issue with what Walkman is considering are the legal considerations. Here in the US, I think recent decisions are that scraping, in and of itself, is not illegal since much of the content is already public.

But the rub comes in at least two forms; international laws may vary and no matter what country you have to read and consider the websites terms of service.

Often, a website might also require permission before hand and frankly no matter what the website terms of service might read I would get permissions in writing anyway. First, this is just the considerate and professional thing to do. Second, I see little sense in doing the development work, getting the tool public, and then having to go back in and start disabling certain auction houses when they send you cease and desist demands.

Keep in mind that many auction houses will view this as a potential loss of income/web traffic. So you will probably have to do some 'selling' of the idea including offering links directly back to their own websites. It will also help greatly if your website is totally non-profit and non-commercial.

Getting and gathering written permissions is probably a bigger scope of work than the coding in this kind of project.
Don