Demos and Recipes LSE ME204 · Data Engineering Principles for the Social Sciences
🖥️ Week 03 Day 01 Lecture

Web Scraping and HTML Parsing

Jon Cardoso-Silva

Dr Jon Cardoso-Silva

LSE Data Science Institute

🗓️ 27 Jul 2026

📋 Mentimeter Check-in

10:00 – 10:15

Reflections on midterm feedback

📷 Mentimeter QR code

Scan the QR code or go to menti.com

Quick polls:

  • Was the midterm feedback actionable? Will it help you with the final one?
  • What is your final project about? Do you have a data source picked?
  • Have you started writing code for the final project?

🛠️ Final Project Support

10:15 – 11:15

I’ll be going around the room to help you. Use this hour to make progress on your project.

What you could be doing right now:

  • Picking an API and making your first request
  • Browsing websites that might have data you want but no API (today’s demo will show you how to get it)
  • Flattening nested JSON with pd.json_normalize if you have code started
  • Cleaning and reshaping your collected data into a DataFrame
  • Come talk to me about scope, feasibility, or a technical problem

💡 Deadline reminder: submit your final project by 5pm Friday 31 July.

☕ Coffee Break

11:15 – 11:30

When we come back:

  • What to do when there is no API
  • Reading HTML with browser developer tools
  • Live demo: scraping the Wikipedia London boroughs table into a DataFrame

1️⃣ When There Is No API

11:30 – 11:45

APIs hand you structured data

So far, every dataset you collected came from an API.

import requests

url = "https://api.open-meteo.com/v1/forecast"
params = {"latitude": 51.51, "longitude": -0.13,
          "daily": "temperature_2m_max"}
response = requests.get(url, params=params)
data = response.json()
{
  "daily": {
    "time": ["2026-07-27", ...],
    "temperature_2m_max": [24.1, ...]
  }
}

The server organised the data for you. You called .json() and got a Python dictionary.

What if there is no API?

The data you need is on a web page, embedded in HTML. No endpoint, no JSON.

A developer built this page so humans could read it in a browser.

You want to work backwards: inspect the HTML, find the patterns the developer used, and write code to extract the data.

This is web scraping.

Scraping vs APIs

APIs

  • Server hands you structured data (JSON, CSV)
  • Documented endpoints and parameters
  • Rate limits are published
  • Stable across page redesigns
  • The type of data collection we’ve been doing 🖥️ Week 01-02

Web scraping

  • You extract data from the page HTML yourself
  • No documentation on the data layout
  • No rate limit published (you set your own)
  • Breaks when the site redesigns
  • Works when no API exists

💡 Use an API when one exists. Scrape only when the data you need has no API and the site permits it.

Before you scrape: ethics and law

Check robots.txt first

Most websites publish a file at /robots.txt that tells crawlers which parts of the site they prefer not to be scraped.

User-agent: *
Disallow: /private/
Disallow: /api/

User-agent: GPTBot
Disallow: /

This is a request, not a technical barrier. Your scraper can ignore it, but whether you should is an ethical question.

Our approach in ME204

  • Check robots.txt before scraping
  • Add time.sleep() between requests
  • Scrape public data for educational purposes only
  • Do not scrape personal data
  • Do not redistribute scraped data commercially

⚠️ If a site says no, stop. Terms of service override your interest in the data.

Fetching a page with requests

Same library, different content. Instead of JSON, you get HTML.

import requests

url = "https://en.wikipedia.org/wiki/London_boroughs"
response = requests.get(url)

print(response.status_code)
# What does it look like?
print(response.text[:500])
200
<!DOCTYPE html>
<html class="client-nojs vector-feature-language-in-header-
enabled" lang="en" dir="ltr">
<head>
<meta charset="UTF-8">
<title>London boroughs - Wikipedia</title>
...

response.text is a single long string of HTML. To find the data inside it, you need to understand the structure of that HTML.

2️⃣ Reading HTML the Way a Browser Does

11:45 – 12:15

Well, sort of…

What is HTML?

A web page is a tree of nested tags. Each tag has a name, optional attributes, and content.

<html>
  <head>
    <title>London boroughs</title>
  </head>
  <body>
    <h1>London boroughs</h1>
    <p>London has 32 boroughs.</p>
    <table class="wikitable">
      <tr><th>Borough</th></tr>
      <tr><td>Camden</td></tr>
    </table>
  </body>
</html>

html_tree html html head head html->head body body html->body title title "London boroughs" head->title h1 h1 "London boroughs" body->h1 p p "London has 32..." body->p table table class="wikitable" body->table tr1 tr table->tr1 tr2 tr table->tr2 th th "Borough" tr1->th td td "Camden" tr2->td

Markdown and HTML

You will find several similarities between Markdown and HTML. It follows a similar idea but with angle brackets instead of symbols.

What you want Markdown HTML
Bold text **Bold** <b>Bold</b>
A heading # Heading <h1>Heading</h1>
A list item - Item <ul><li>Item</li></ul>
A link [text](url) <a href="url">text</a>

When you write Markdown and render it in VS Code or on a website, software converts it to HTML. The browser only reads HTML.

Browser developer tools

Right-click any element on a web page, then click Inspect.

Three steps:

  1. Right-click the element you want (a table cell, a heading, a link)
  2. Click Inspect to open the Elements panel
  3. Hover over tags in the panel to highlight them on the page

This is how you find the tag name and class you need for your scraping code.

Tags, attributes, and selectors

An HTML tag up close:

<table class="wikitable sortable">
  <tr>
    <td>
      <a href="/wiki/Camden">Camden</a>
    </td>
  </tr>
</table>
  • table is the tag name
  • class="wikitable sortable" is an attribute
  • Camden is the text content

CSS selectors you will use today:

Selector Finds
table All <table> elements
.wikitable Elements with class="wikitable"
#content The element with id="content"
table.wikitable <table> elements with class wikitable

The same selector syntax that CSS uses to style elements is what BeautifulSoup uses to find them.

First steps with BeautifulSoup

BeautifulSoup turns the raw HTML string into a tree you can search.

from bs4 import BeautifulSoup

soup = BeautifulSoup(response.text, "html.parser")

# The page title
print(soup.title.text)

# The first <h1> heading
print(soup.find("h1").text)
London boroughs - Wikipedia
London boroughs

soup is the whole tree. .find() walks the tree and returns the first tag that matches.

find vs find_all

find stops at the first match. find_all collects every match.

find: one result

table = soup.find("table", class_="wikitable")
print(type(table))
<class 'bs4.element.Tag'>

Returns the first <table> with class="wikitable", or None if no match.

This is a Tag object. Things you can do with it:

tag.name          # "table"
tag["class"]      # ["wikitable", "sortable"]
tag.text          # all text inside
tag.find("tr")    # first <tr> inside
tag.find_all("td") # all <td> inside

find_all: a list

rows = table.find_all("tr")
print(len(rows))
34

Returns every <tr> inside that table, as a list you can loop over.

CSS selectors in BeautifulSoup

An alternative to chaining find and find_all: write a CSS selector in one string.

Using soup.select():

# Same result as: table.find_all("tr")
rows = soup.select("table.wikitable tr")
print(len(rows))
34
# First <td> inside a wikitable
cell = soup.select_one("table.wikitable td")
print(cell.text.strip())
Barking and Dagenham

CSS selector mini reference:

Selector Meaning
table any <table>
.wikitable class wikitable
#content id content
table.wikitable <table> with class
table.wikitable tr <tr> inside that table
td a <a> inside a <td>

💡 Use select for short paths. Use find / find_all when you need keyword arguments like class_=.

3️⃣ Extracting Structured Data

12:15 – 12:50

How to turn a web page into a DataFrame.

The Wikipedia London boroughs table

The page is at:

en.wikipedia.org/wiki/London_boroughs

Step 1: fetch and parse

Two lines you have seen before, plus one new one.

import requests
from bs4 import BeautifulSoup

url = "https://en.wikipedia.org/wiki/London_boroughs"
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")

print(soup.title.text)
London boroughs - Wikipedia

Step 2: find the table

The page has several tables. Use Inspect to find the right class, then find it.

table = soup.find("table", class_="wikitable sortable")

# Preview the first 300 characters of the table HTML
print(table.prettify()[:300])
<table class="wikitable sortable" ...>
 <tbody>
  <tr>
   <th>Borough</th>
   <th>Inner</th>
   <th>Status</th>
   <th>Area (sq mi)</th>
   ...
  </tr>
  <tr>
   <td><a href="/wiki/Barking_and_Dagenham">Barking and Dagenham</a></td>
   ...

Step 3: extract the header row

The first <tr> contains <th> (header) cells. Extract them with a list comprehension.

header_row = table.find("tr")
headers = [th.text.strip() for th in header_row.find_all("th")]
print(headers)
['Borough', 'Inner', 'Status', 'Area (sq mi)',
 'Population (2021)', 'Co-ordinates', ...]

With a raw for loop, the same thing would have been:

headers = []
for th in header_row.find_all("th"):
    headers.append(th.text.strip())

🔔 Remember: From now on, prefer list comprehensions or pandas functions over for loops when building a list.

Step 4: extract data rows

Same pattern, row by row. Skip the header row with [1:].

With a for loop:

rows = []

for tr in table.find_all("tr")[1:]:
    cells = [td.text.strip()
             for td in tr.find_all("td")]
    if cells:
        rows.append(cells)

As a list comprehension:

rows = [
    [td.text.strip()
    for td in tr.find_all("td")]
    for tr in table.find_all("tr")[1:]
    if tr.find_all("td")
]
print(f"{len(rows)} rows extracted")
print(rows[0])
33 rows extracted
['Barking and Dagenham', '', 'Borough', '13.93', '218110', ...]

Step 5: build a DataFrame

import pandas as pd

df = pd.DataFrame(rows, columns=headers)
df.head()
Borough Inner Status Area (sq mi) Population (2021)
0 Barking and Dagenham Borough 13.93 218110
1 Barnet Borough 33.49 389344
2 Bexley Borough 23.38 248287
3 Brent Borough 16.70 339800
4 Bromley Borough 57.97 330000

💡 Compare this to the CSV you loaded in 💻 Week 01 Day 02 Lab. Same boroughs, different source. The CSV was prepared by someone else. Here you built the table yourself from a web page.

What about pd.read_html?

One line instead of five steps.

Step 1: fetch all tables

dfs = pd.read_html(url)
print(type(dfs))
print(f"{len(dfs)} tables found")
<class 'list'>
5 tables found

dfs is a list of DataFrames, one per <table> on the page. You need to pick the right one by index.

Step 2: inspect and pick

dfs[0].head()
Borough Inner Status …
0 Barking and Dagenham Borough …
1 Barnet Borough …
2 Bexley Borough …

5 rows x 6 columns

Step 3: give it a proper name

df_london_boroughs = dfs[0]

The trade-off:

  • Quick for exploration
  • You cannot control which table it picks (you get all of them and choose by index)
  • You cannot control how it cleans each cell
  • Use it when you want a fast look. Use BeautifulSoup when you need precision.

💡 Both approaches are valid. pd.read_html is a good first check. If the result is messy or you need a specific table, switch to BeautifulSoup.

🏁 Thanks!

12:50 – 13:00

This afternoon

You will practise web scraping with Jonas, and he can also help you with your final project.

💻 Today’s Lab

Questions? Come find us during the lab or email me.

Where you want to be by tonight

  • You have picked your data source: an API, a web page to scrape, or both
  • You have made at least one successful request and saved the raw response
  • You know what your collected data looks like
  • Rough code is fine. Polish it tomorrow. The important thing is that data is coming in.

⚠️ Submit by 5pm Friday 31 July. Assessment details

LSE Summer School 2026 | ME204 Week 03 Day 01