A few years back, a typical online buyer looking to buy apparel or an electronic gadget would scour the web to compare prices from various e-commerce sites and look for the best deals available. Prices of the same commodity would vary vastly over different sites which in turn prompted sites to monitor price changes in real-time using analysts. In today’s hyper-competitive e-commerce scenario, retail players grab every opportunity to attract potential customers. Price comparison websites are one of the opportunities e-retailers leverage to attract customers.
Price comparison websites are a win-win for both a business owner and a customer. The customers benefit by gaining good deals, convenient shopping experience, a better coverage over major e-commerce sites, and display of greater variety for the same product. On the other hand, business owners benefit by gaining greater no of leads, better conversion rates, and enhanced customer services.
How do Price Comparison Websites work?
Price comparison websites skim through multiple e-commerce sites to gather data about products and services like prices, descriptions, features, reviews etc. This information is then combined on a price comparison website and results are tailored according to a visitor’s search request.
This way when a buyer searches for a product on a price comparison website, the site compares and displays listings of the same product from multiple sellers. The buyer can then compare the listings based on price deals, features, shipping costs etc to find the best deal.
The algorithms governing the price comparison functions use data as input. Extracting and updating this data in real time is a challenging task for price comparison engines. Moreover, e-commerce sites use dynamic pricing which changes frequently and need to be updated with the price comparison engines immediately. For example, Amazon takes exactly two minutes to adapt to a price change and that’s 43000 minutes faster than the industry average.
The challenges in getting and using data for price comparison include:
- Comparison technology: The data from different retailers will be having different structures. Building a comparison engine will be a tough task
- The volume of data: You will be handling a large volume of data making it a complex task
A lot of startups work with Datahut to obtain data for their price comparison engines. Product data from e-commerce sites is collected using the following ways:
Feeds from merchants
Price comparison websites enter into a collaboration with stores or company to gain data directly from a retailer’s web feeds. Traffic from comparison shopping websites is a major source of revenue for internet retailers. If you can build a partnership with them – they can give you access to their API. However – you will need to pay a premium to actually use the API at higher volume. Merchants set up either an API or use FTP to directly deliver data.
Note: One of the common ways of monetising a price comparison sites is by becoming an affiliate partner and getting a referral commission.
While this method of data collection provides direct access to data, it has its cons. Oftentimes getting real-time updates to changes becomes difficult. Moreover, different merchants provide data in multiple formats, making it difficult to integrate onto one platform.
Product feeds from third-party API
Retail product data can also be obtained through third-party APIs. When integrated with shopping carts, some services provide e-commerce data via an API request. In this case, third-party services charge for the volume of data extracted. These companies actively crawl the most common retailer websites and provide this data in a database for easy access by price comparison websites.
This method saves a lot of development time and allows easy integration of price comparison engines with multiple shopping carts. However, the biggest challenge in API integration is that it is a difficult and expensive process when developing multiple integrations adhering to specifications of every shopping platform. Shopping giants like Magento, WooCommerce, OpenCart, Shopify etc can be integrated via API2Cart which provides a unified API to connect all shopping carts simultaneously.
In most cases, options 1 and 2 might not be available when you are just starting up and have fewer funds. In such cases, Web scraping is one of the reliable and efficient ways of getting product data from target sites according to your requirements.
You can acquire pricing data by either building your own in-house web scraping set up or employing a data-as-a-service (DaaS) provider who’ll provide with the requisite data as per your requirement. For efficient results in your price comparison site, the quality of the data has to be top-grade and having updated datasets with latency becomes an essential requirement. Since managing a price comparison website is itself a cumbersome task, we recommend you go with a web scraping service provider for pricing data extraction.
Web scraping services would employ crawling bots to regularly gather information from websites for further processing. This instantly provides the price comparison engine with accurate and updated information from target sites without having to depend on merchants to get data. However, many websites have special anti-crawling mechanisms, which block users that make too many requests.
We recommend you read our post on How to Choose a Web Scraping Service