πŸ›£οΈ Week 04 - Web Scraping in Python using the requests and scrapy libraries

Lab Roadmap (90 min)

Author

Dr. Jon Cardoso-Silva

Today you will encounter a new type of lab. We will start using Jupyter Notebooks as a way to present the lab material.

The focus of today is to get some practice with web scraping. You will learn about the requests and scrapy libraries and how to use them to extract data from the web. Then, you will use your Python skills to make sense of the data you have collected.

This lab is part of the GENIAL project.

If you have never accessed ChatGPT, you must create an account.

Click on chat.openai.com and sign up with your email address (it doesn’t have to be your LSE email address).

When you reach Part 3 of this lab, read the specific instructions for GENIAL participants.

🎯 Learning Objectives

  • Learn how to send a request to a web server and get a response
  • Learn how to extract the entire content of a web page
  • Learn to do more targeted web scraping using CSS Selectors
  • Practice using Python to manipulate data

πŸ“‹ Lab Tasks

🎯 ACTION POINTS

  1. Download the Jupyter Notebook below, save it to your DS105 folder and open it in VS Code

  1. Follow your class teacher’s instructions to complete the lab tasks.