Extract All Books Under a Category

In this lesson, we’ll implement a process to scrape the details of all the books in a category. When we visit a category page, we can see all the books in that category. However, if many books are listed under the category, we might need to paginate. Let’s use the Mystery category in this lesson for scraping.

Approach

Before jumping into the implementation, let’s consider a solution. Since we know the category, we can navigate to the category page. On the category page, we see many books paginated, and each book has a separate link associated with it, pointing to the details of that particular book.

If we can track these links for each book while paginating until the pagination ends, we’ll have the links to get the details of every book under that category. Then, we can use those links to navigate to see the details of each page and scrape them.

Implementing the approach

Now, let’s try to implement the approach we discussed. Before that, we need to decide two selectors. We need one selector to locate the element that contains the book details page link. The ...