Stamp Community Family of Web Sites
Thousands of stamps, consistently graded, competitively priced and hundreds of in-depth blog posts to read
Stamp Community Forum
 
Username:
Password:
Save Password
Forgot your Password?

Welcome Guest! Need help? Got a question? Inherit some stamps?
Our stamp forum is completely free! Register Now!

Is There A Need For A Philatelic Auction/Sales Search Engine?

 
To participate in the forum you must log in or register.
Author Previous TopicReplies: 5 / Views: 341Next Topic  
Valued Member
United States
414 Posts
Posted 11/22/2022   7:48 pm  Show Profile Bookmark this topic Add Walkman82 to your friends list Get a Link to this Message
I've grown tired of spending hours searching through auction lots looking for specific stamps to add to my collection. If these sites have a way to search for specific stamps, they often don't return what I'm trying to find. eBay has a decent search capability, but many other auction and sales sites do not.

To address this issue, I've been thinking of creating a web crawler (read search bot) to index stamp auction and sales sites and make the information searchable.

Do you, members of the SCF community, see any value in such a web-based tool?

Thanks in advance for your thoughts.
Send note to Staff
Member APS #174069, PNC3 #2386, AFDCS #29532O, PSS #8418

Visit my website @ www.scottsstampcollection.com

Pillar Of The Community
United States
795 Posts
Posted 11/22/2022   7:53 pm  Show Profile Bookmark this reply Add classic_paper to your friends list  Get a Link to this Reply
Something along the lines of what Stamp Auction Network (with mixed reviews) already does?
Send note to Staff  Go to Top of Page
Pillar Of The Community
United States
2657 Posts
Posted 11/22/2022   7:59 pm  Show Profile Bookmark this reply Add PostmasterGS to your friends list  Get a Link to this Reply
For a brief period several years ago, I scraped the big German auctionhouses while attempting to build a searchable database showing the sales history of certain items, but the inconsistencies in listing were too large of a hurdle. You don't realize how many different ways an auctionhouse can list a single catalog number until you try to automate a search for it.
Send note to Staff  Go to Top of Page
Presenting the GermanStamps.net Collection - Germany, Colonies, & Occupied Territories, 1872-1945
Pillar Of The Community
Learn More...
656 Posts
Posted 11/22/2022   8:24 pm  Show Profile Bookmark this reply Add rismoney to your friends list  Get a Link to this Reply
The problem with scraping sites, is the end result of a sites' html is that it is not consistently structured data, but rather a mix of formatting, scripts, surrounding the content. While you can have moderate success, one slight modification of a web table and your parser will return bad data. Many sites are now dynamically generated and the sites themselves don't care about scrapers, and often try to prevent them or discourage circumventing through complex redirectors, frames and other script tricks.

The ideal way to do this, if you are serious, is leverage APIs of the sites and try to get at the data directly. Whether or not this is available is probably something you'd have to arrange with each house and talk about a mutual relationship of sorts. An exampl might be, Free API access in return for traffic generation, but that is probably a tough sell before you are established...

Send note to Staff  Go to Top of Page
Edited by rismoney - 11/22/2022 8:26 pm
Pillar Of The Community
Learn More...
643 Posts
Posted 11/26/2022   10:52 am  Show Profile Bookmark this reply Add Stamps4Life to your friends list  Get a Link to this Reply
I'm pretty new to SAN and I can say that the way the dealers have the info listed caused me frustration on the site alone. Some dealer use SG numbers, some didn't. Some used "1d", some "One Penny", etc,etc. So in using their SAN search function I would get many more emails that did not interest me due to faulty results. I finally turned off the email notifications. I can only imagine the problem a crawler would have.
Send note to Staff  Go to Top of Page
Moderator
Learn More...
United States
10990 Posts
Posted 11/26/2022   11:28 am  Show Profile Check 51studebaker's eBay Listings Bookmark this reply Add 51studebaker to your friends list  Get a Link to this Reply
The issue with what Walkman is considering are the legal considerations. Here in the US, I think recent decisions are that scraping, in and of itself, is not illegal since much of the content is already public.

But the rub comes in at least two forms; international laws may vary and no matter what country you have to read and consider the websites terms of service.

Often, a website might also require permission before hand and frankly no matter what the website terms of service might read I would get permissions in writing anyway. First, this is just the considerate and professional thing to do. Second, I see little sense in doing the development work, getting the tool public, and then having to go back in and start disabling certain auction houses when they send you cease and desist demands.

Keep in mind that many auction houses will view this as a potential loss of income/web traffic. So you will probably have to do some 'selling' of the idea including offering links directly back to their own websites. It will also help greatly if your website is totally non-profit and non-commercial.

Getting and gathering written permissions is probably a bigger scope of work than the coding in this kind of project.
Don
Send note to Staff  Go to Top of Page
  Previous TopicReplies: 5 / Views: 341Next Topic  
 
To participate in the forum you must log in or register.


Go to Top of Page
Disclaimer: While a tremendous amount of effort goes into ensuring the accuracy of the information contained in this site, Stamp Community assumes no liability for errors. Copyright 2005 - 2022 Stamp Community Family - All rights reserved worldwide. Use of any images or content on this website without prior written permission of Stamp Community or the original lender is strictly prohibited.
Privacy Policy / Terms of Use    Advertise Here
Stamp Community Forum © 2007 - 2022 Stamp Community Forums
It took 0.14 seconds to lick this stamp. Powered By: Snitz Forums 2000 Version 3.4.05