🖥️ Week 03, Day 01 - Lecture
Demos and Recipes: Web Scraping and HTML Parsing
By the end of today’s lecture, you should be able to: i) Explain when web scraping is useful and when an API is the better choice, ii) Inspect a web page’s HTML structure using browser developer tools, iii) Fetch a page with requests and parse it with BeautifulSoup, iv) Extract structured data (text, links, table rows) from parsed HTML, v) Identify ethical and legal considerations before scraping a site.
⏰ Monday, 27 July 2026 | 10:00am - 1:00pm 📍 CBG.2.01 (see LSE’s 🗺️ campus map)
Last week you pulled structured JSON from APIs and untangled nested records with pd.json_normalize. But not every data source hands you a clean endpoint. Today you learn what to do when the data you need is on a web page and there is no API: fetch the HTML, read its structure, and extract what you need with Python.
🗣️ Lecture Overview
- Mentimeter check-in: your midterm experience, whether the feedback was actionable, and your early ideas for the final project
- Final project support: work on your project with help from me and your class teacher
- Coffee break
- When there is no API: scraping vs APIs, ethics and
robots.txt, fetching raw HTML withrequests - Reading HTML the way a browser does: browser developer tools, HTML tags, attributes, and selectors, first steps with
BeautifulSoup - Extracting structured data: live demo scraping the Wikipedia London boroughs table,
findandfind_all, CSS selectors, building a DataFrame from scraped rows - Wrap-up: what the lab covers this afternoon and the final project deadline
🎬 Lecture Slides
Use keyboard arrows to navigate. Select the slides below or view fullscreen.
After the Lecture
This afternoon you practise web scraping with Jonas, and he can also help you with your final project.
Questions?
🔗 Web Scraping Resources