Even though Amazon is the leader in e-commerce marketplaces – eBay still has its fair share in the online retail industry. Brands selling online should be monitoring prices on eBay as well to gain a competitive advantage.
To scrape product data from eBay at a huge scale regularly is a challenging problem for data scientists. Here is an example of scraping eBay using python to identify the prices of mobile phones.
Lets us imagine a use case where you need to monitor the pricing of a product, say a mobile phone from eBay. Also, you want to visualize the range of price offering available on the mobile phone which you want to monitor. Moreover, you have other mobile phones under consideration so you may also want to compare their prices as well.
In this blog, we will be scraping eBay to collect the prices of phones and find out the difference between their offerings on the eBay website.
Also Read: Busting 8 Myths About Web Scraping
Scraping eBay product data step by step
In this section, we will walk you through the step by step process of scraping eBay for products and their prices.
1. Selecting the required information
The very first task in web scraping is to identify the target web page. It is the web page from which you need to extract all the required information.
We will be scraping eBay for the product listings so we can just open the eBay website and type our product in their search bar and hit enter. Once page loads with all the product listing of that product, all you need to do is pull that URL out from the browser. This URL will be our target URL. In our case, the URL will be,
“https://www.ebay.com/sch/i.html_from=R40&_nkw=galaxy+note+8&_sacat=0&_pgn=1“
Notice the two parameters in this URL i.e. “nkw” (new keyword) and “pgn” (page number) parameter. These parameters in the URL defines the search query. If we change “pgn” parameter to 2, then it will open the second page of the product listings for galaxy note 8 phone and if we were to change “nkw” to iPhone X then eBay will search for iPhone X and will show you the corresponding results.
2. Finalizing the tags for extraction
Once we have finalized the target web page, we need to understand its HTML layout to scrape the results out. This is the most important and critical part of web scraping and basic HTML knowledge is a pre-requisite for this step.
When on the target web page, do “inspect element” and open the developer tools window or just do CTRL+SHIFT+I. In the new window, you will find the source code of the target web page. In our case, all the products are mentioned as list elements so we have to grab all these lists.
In order to grab an HTML element, we need to have an identifier associated with it. It can be an id of that element or any class name or any other HTML attribute of the particular element. We are using the class name as the identifier. All the lists have the same class name i.e. s-item.
On further inspection, we got the class names for the product name and product price which are “s-item__title” and “s-item__price” respectively. With this information, we have successfully completed step 2!
3. Putting the scraped data in a structured format
After having our extractors/identifiers we only need to extract specific portions out from the HTML content. Once this is done, we need to organize this data into a suitable structured format. We will be creating a table where we will have all the product names in one column and their prices in the other.
4. Visualizing the results (optional)
Required libraries and Installation
To implement web scraping for this use case, you will need python, pip (package installer for python), and BeautifulSoup library in python for web scraping. You will also need pandas and numpy library to organize the collected data into a structured format.
Installing Python and PIP Depending upon your operating system, you can follow this blog link to setup python and Pip in your system.
Installing Beautiful soup library
apt-get install python-bs4 pip install beautifulsoup4 3. Installing pandas and numpy
pip install pandas pip install numpy We are done setting up our environment and now can begin with the scraping implementation using python. The implementation consists of the steps discussed in the earlier section.
Python implementation for scraping eBay
In this section, we will perform two scraping operations i.e. one for the iPhone 8 and another for the galaxy note 8 mobile phones. Implementation has been repeated for the two mobile phones for easier comprehension. A more optimized version can have two separate scrapping activities combined into one which is not required right now though.
Scrapping eBay for Galaxy Note 8 products
item_name = []
prices = []
for i in range(1,10):
ebayUrl = "https://www.ebay.com/sch/i.html?_from=R40&_nkw=note+8&_sacat=0&_pgn="+str(i)
r= requests.get(ebayUrl)
data=r.text
soup=BeautifulSoup(data)
listings = soup.find_all('li', attrs={'class': 's-item'})
for listing in listings:
prod_name=" "
prod_price = " "
for name in listing.find_all('h3', attrs={'class':"s-item__title"}):
if(str(name.find(text=True, recursive=False))!="None"):
prod_name=str(name.find(text=True, recursive=False))
item_name.append(prod_name)
if(prod_name!=" "):
price = listing.find('span', attrs={'class':"s-item__price"})
prod_price = str(price.find(text=True, recursive=False))
prod_price = int(sub(",","",prod_price.split("INR")[1].split(".")[0]))
prices.append(prod_price)
from scipy import stats
import numpy as np
data_note_8 = pd.DataFrame({"Name":item_name, "Prices": prices})
data_note_8 = data_note_8.iloc[np.abs(stats.zscore(data_note_8["Prices"]))< 3,]
Collected Data for Galaxy note 8
scraping eBay | Note 8 data
Scrapping eBay for iPhone 8
item_name = []
prices = []
for i in range(1,10):
ebayUrl = "https://www.ebay.com/sch/i.html?_from=R40&_nkw=iphone+8_sacat=0_pgn="+str(i)
r= requests.get(ebayUrl)
data=r.text
soup=BeautifulSoup(data)
listings = soup.find_all('li', attrs={'class': 's-item'})
for listing in listings:
prod_name=" "
prod_price = " "
for name in listing.find_all('h3', attrs={'class':"s-item__title"}):
if(str(name.find(text=True, recursive=False))!="None"):
prod_name=str(name.find(text=True, recursive=False))
item_name.append(prod_name)
if(prod_name!=" "):
price = listing.find('span', attrs={'class':"s-item__price"})
prod_price = str(price.find(text=True, recursive=False))
prod_price = int(sub(",","",prod_price.split("INR")[1].split(".")[0]))
prices.append(prod_price)
from scipy import stats
import numpy as np
data_note_8 = pd.DataFrame({"Name":item_name, "Prices": prices})
data_note_8 = data_note_8.iloc[np.abs(stats.zscore(data_note_8["Prices"])) < 3,]
Collected data for iPhone 8
scraping eBay | Iphone data
Visualizing the price of the products
Now is the time to visualize the scraped results. We will be using the boxplots to visualize the distribution of prices of mobile phones.
Box plot helps us in visualizing a trend in numerical values. The green line is the median of the collected price data. The box extends from the Q1 to Q3 quartile values of the data, with a line at the median (Q2). The whiskers extend from the edges of the box to show the range of the data.
scraping eBay | Price Comparison
For iPhone 8, most of the prices lie between INR 25k-35k whereas most of the galaxy note 8 phones are available in the price range of 25k-30k.
However, variation in prices of the iPhone 8 is much more than galaxy note 8. iPhone 8 is available for minimum INR 15k on eBay whereas the minimum cost of galaxy note 8 on eBay is around
22-23K INR!
Datahut as a reliable scraping partner
There are a lot of tools that can help you scrape data yourself. However, if you need professional assistance with minimal technical know-how, Datahut can help you. We have a well-structured and transparent process for extracting data from the web in real-time and provide in the desired format. We have helped enterprises across various industrial verticals. From assistance to the recruitment industry python/ to retail solutions, Datahut has designed sophisticated solutions for most of these use-cases.
You should join the bandwagon of using data-scraping in your operations before it is too late. It will help you boost the performance of your organization. Furthermore, it will help you derive insights that you might not know currently. This will enable informed decision-making in your business processes.
Conclusion
In this blog, we successfully used python for scraping eBay for two different products and their pricing. We also compared the available prices for galaxy note 8 and iPhone 8 to make a better purchase decision. Web scraping coupled with data science can be leveraged for smart decision making be it in the fortune 500 companies or in your day to day life.
Wish to avail web scraping services for your data needs? Contact Datahut to know more.