Search⌘ K
AI Features
Home
Courses
Predictive Data Analysis with Python

Scraping Domains

Explore how to automate web scraping of domain sales data from namebio.com using Python and Selenium. Learn to navigate multiple tables by clicking buttons, extract and save domain, price, and sale date information efficiently into a CSV file for predictive analysis.

We'll cover the following...

The website namebio.com will be scraped in this lesson. This site contains the domain name information like which domain is sold when, for what price, and where it is parked. In free mode, we can look at one-hundred data points with four columns of data which include the domain name, domain price, date it is sold, and the site where the domain is parked. The data is displayed and available in the form of a table like in the following image.

Writing the scraper

As can be seen in the above image, to get to the next table of domains, the next number button needs to be clicked and this is where the real power of selenium comes into play. Each and every button on the webpage can be clicked simply by finding its xpath and clicking the button using the click() function. Let’s see this in action.

Python 3.5
# Import the require packages
from selenium import webdriver
import time
import csv
# Initialize the column names list
col = ['Name','Price','Date','Parked_At']
# Write the column name list in the csv file
with open('Domain.csv', 'w') as csvFile:
        writer = csv.writer(csvFile)
        writer.writerow(col)
chrome_options = webdriver.ChromeOptions()
#chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
# Initialize the browser object
driver = webdriver.Chrome('<Provide the path of chromedriver>', options=chrome_options)
# URL to go to
url = "https://namebio.com/"
# Maximize the browser window
driver.maximize_window()
# Opens the URL in the browser
driver.get(url)
# Counter to traverse through the 10 tables
i = 0
while i < 10:
  #  Get all elements with the table row tag using the xpath method
  trs = driver.find_elements_by_xpath('//*[@id="search-results"]/tbody/tr')
  # Traverse through the rows
  for tr in trs:
    # Split each row to get four values
    tr = str(tr.text).split(' ')
    del tr[2:3]
    # Write the list of values to the csv file
    with open('Domain.csv', 'a') as csvFile:
        writer = csv.writer(csvFile)
        writer.writerow(tr)
    # Close the csv file
    csvFile.close()
  
  # When all rows are read then click the next arrow button to move to next table
  driver.find_element_by_xpath('//*[@id="search-results_next"]/a').click()
  # Give a 2 sec delay for the table to get loaded
  time.sleep(2)
  # Increment the counter
  i+=1
# Close the browser
driver.close()

The above code automates the ...