Scrapy
Explore how to use the Scrapy library to systematically scrape data from websites. Understand the process of selecting elements via CSS selectors and implement a practical example to extract question data from a live site.
We'll cover the following...
Scrapy
Scrapy is one of the other tools that we can use to extract data from sites. It has a more organized structure and works in a systematic manner. We won’t be going into too much detail though. Instead, we will look at a real-time example of scraping data using Scrapy to understand how it works.
Exercise
Task
We will visit a site that displays questions/queries asked by people all over the world. A screenshot of the image is shown below.
We will be scraping the top ten questions being asked at the time the site is requested by our code. Everything scraped will include four aspects of a question:
- Summary
- Votes
- Views
- Number of answers
Single question’s HTML
Go to the site ...