Updating the Web Spider
Explore how to update a web spider by splitting tasks and leveraging TaskQueue to control concurrency. Understand how to manage asynchronous flow using promises and async/await, and avoid deadlocks while efficiently downloading multiple links.
We'll cover the following...
We'll cover the following...
Web spider v4
Now it’s time to adapt our web spider to implement a limited parallel execution flow using the TaskQueue class we’ve just created.
First, we need to split the spider() function into two functions, one simply initializing a new TaskQueue ...