LSE ME204 · Data Engineering Principles for the Social Sciences
🖥️ Week 01 Day 03 Lecture

Collecting Data from the Web (APIs) with Python

Dr Jon Cardoso-Silva

LSE Data Science Institute

🗓️ 15 Jul 2026

1️⃣ ONS Data Sources and File Downloads

10:00 – 10:30

The CSV file you used yesterday in the lab, data/london_boroughs.csv, came from the ONS Census 2021 dataset TS001. Let’s see how you would download it yourself.

ONS Census Dataset TS001

ONS page for TS001, showing the dataset title, summary, variables, and CSV download option.

Office for National Statistics (ONS)

• The United Kingdom’s national statistical institute
• Publishes official statistics about the population, economy, and society
• Runs the census for England and Wales

Dataset reference TS001

• TS means Topic Summary
• TS001 is the reference code assigned to this table
• Title: Number of usual residents in households and communal establishments
• Census date: 21 March 2021

Open the ONS dataset page

Download the CSV with curl

Say you have the actual address where the CSV file is stored on the website:

https://static.ons.gov.uk/datasets/TS001-2021-3.csv

You can give that address to curl and choose the name of the local copy:

curl --location \
  "https://static.ons.gov.uk/datasets/TS001-2021-3.csv" \
  --output TS001-2021-3.csv

ls -lh TS001-2021-3.csv

• --location follows another address if the website redirects the download.
• --output chooses the name of the saved file.

The \ at the end of a line tells the terminal the command continues on the next line. You can type it as one long line instead.

The browser and curl can download the same file. Run curl --help in the terminal to see all options.

2️⃣ HTTP Requests in the Terminal and Python

10:30 – 11:00

The browser and curl both asked an ONS server for the CSV file. On the web, that exchange follows a set of rules called HTTP.

HTTP Requests and Responses

HTTP stands for Hypertext Transfer Protocol.

• Hypertext means documents with links. Tim Berners-Lee proposed the idea at CERN in 1989: pages that point to other pages.
• Transfer means moving those documents between computers.
• Protocol means agreed rules for the exchange.

HTTP defines how a client and server exchange messages on the web.

The ONS Download in the Network Panel

A browser records its HTTP requests and responses under Developer Tools > Network.

The ONS CSV download beside Chrome Developer Tools. The Network panel records three document requests ending with TS001-2021-3.csv and status 200.

One click on Download produced three requests and three responses.

The ONS Download Sequence

curl --location follows those redirects until it receives the CSV.

Python’s Requests Library

Requests documentation page showing the project name, current release, example code, and links to its guides.

Python comes with a standard library: modules installed as part of Python itself.

Requests is a third-party library. It is developed and released separately from Python, so it must be installed before a program can import it.

Requests is installed on the Linux computer you have on Nuvolos.

import requests

This line makes the installed library available inside your program.

HTTP Requests with curl and requests

Both tools received the same response content. curl writes it to standard output; Python keeps a response object in memory until we inspect or save it.

In the terminal

curl "https://static.ons.gov.uk/datasets/TS001-2021-3.csv"

In Python

import requests

url = "https://static.ons.gov.uk/datasets/TS001-2021-3.csv"
response = requests.get(url, timeout=60)
print(response.text)

Response content written to the terminal:

Lower Tier Local Authorities Code,...,Observation
E06000001,Hartlepool,1,Lives in a household,91471

Response content printed from memory:

Lower Tier Local Authorities Code,...,Observation
E06000001,Hartlepool,1,Lives in a household,91471

Request and Response Headers

Headers are key-value metadata sent with an HTTP request or response. Request headers describe the client and what it can accept. Response headers describe the returned content.

In the terminal

curl --verbose \
  "https://static.ons.gov.uk/datasets/TS001-2021-3.csv"

In Python

print(response.request.headers["User-Agent"])
print(response.headers["Content-Type"])
print(response.headers["Content-Length"])

Selected lines (> request, < response):

> GET /datasets/TS001-2021-3.csv HTTP/2
> Host: static.ons.gov.uk
> User-Agent: curl/...
< HTTP/2 200
< content-type: binary/octet-stream
< content-length: 37995

Selected values:

python-requests/...
binary/octet-stream
37995

Check the Response Status

print(response.status_code)

200

200

Success

The server returned the requested content.

301 / 302

Redirect

The client should send another request to a different address.

404

Not found

The server could not find the requested address.

response.raise_for_status()

raise_for_status() stops the program for unsuccessful 4xx or 5xx responses.

🔍 Your turn: what would you type?

The London Datastore publishes a spreadsheet of daily Santander Cycle Hire counts since 2010:

https://data.london.gov.uk/download/2r84d/ac29363e-e0cb-47cc-a97a-e216d900a6b0/tfl-daily-cycle-hires.xlsx

You want to save this file as cycle-hires.xlsx. What do you type in the terminal?

curl --location \
  "https://data.london.gov.uk/download/2r84d/ac29363e-e0cb-47cc-a97a-e216d900a6b0/tfl-daily-cycle-hires.xlsx" \
  --output cycle-hires.xlsx

An XLSX file stores spreadsheet data as compressed XML inside a ZIP archive. Unlike CSV, it can hold formatting, formulas, and multiple sheets. cat cannot read it because the bytes are not plain text, but curl downloads it the same way it downloads a CSV.

The full HTTP status code reference is in the appendix slides at the end of this deck.

Inspect the Response Text

Step by step

What type is the response body?

type(response.text)

str

What does the first line say?

response.text.splitlines()[0]

Lower Tier Local Authorities Code,...,Observation

The first line is the CSV header you saw in W01D02.

Shortcut (once you are comfortable)

print(type(response.text))
print(response.text.splitlines()[0])

<class 'str'>
Lower Tier Local Authorities Code,...,Observation

You can combine checks into one cell once the pattern is familiar.

Save the CSV with curl and Python

Both commands create the same local file: TS001-2021-3.csv.

In the terminal

curl \
  "https://static.ons.gov.uk/datasets/TS001-2021-3.csv" \
  --output TS001-2021-3.csv

--output writes the response content to the named file.

In Python

with open("TS001-2021-3.csv", mode="w", encoding="utf-8") as f:
    f.write(response.text)

f.write() writes the response text to the named file.

3️⃣ REST APIs, Endpoints, and Parameters

11:00 – 11:15

Let’s talk about weather. On Monday, Open-Meteo supplied the data behind our heatwave demonstration. We will now inspect how a client asks Open-Meteo for weather data.

One Browser, Two Kinds of Page

🌐 A webpage for people

Open-Meteo Forecast API documentation

• The server returns HTML.
• The browser renders headings, forms, and links.
• We use the page to learn what requests the service accepts.

{} An API endpoint

Open a London rainfall request for three days

• The server returns JSON.
• The browser displays that JSON as text.
• curl and Python can request the same address.

All three tools send the same HTTP request. The difference is what they do with the response: the browser renders HTML as a page and displays JSON as text.

API and REST

API stands for Application Programming Interface. It defines how one piece of software can request data or actions from another.

REST stands for Representational State Transfer. For our work, a REST API means:

• a resource has an endpoint, which is its URL
• the client sends an HTTP method such as GET
• parameters specify the requested place, variable, and time span
• the server returns a representation of the resource, usually JSON
• each request contains what the server needs to handle that request

REST describes the request pattern. It does not name a different kind of client.

Open-Meteo Endpoint and Parameters

The endpoint (the base address):

https://api.open-meteo.com/v1/forecast

The query string (after the ?, joined by &):

?latitude=51.5085&longitude=-0.1257&daily=rain_sum
 &timezone=Europe/London&forecast_days=3

Everything before ? is the endpoint. Everything after is a query parameter, joined by &.

Parameter	Value	Asks for
latitude	51.5085	London’s latitude
longitude	-0.1257	London’s longitude
daily	rain_sum	daily rainfall total
timezone	Europe/London	dates in London time
forecast_days	3	three forecast days

Open-Meteo does not require an API key for this classroom request. Some APIs require authentication, which we will handle when a task needs it.

The Same Request in a Browser and curl

In a browser:

https://api.open-meteo.com/v1/forecast?latitude=51.5085&longitude=-0.1257&daily=rain_sum&timezone=Europe%2FLondon&forecast_days=3

In the terminal:

curl "https://api.open-meteo.com/v1/forecast?latitude=51.5085&longitude=-0.1257&daily=rain_sum&timezone=Europe%2FLondon&forecast_days=3"

Both clients receive the same JSON shape:

{
  "daily_units": {"time": "iso8601", "rain_sum": "mm"},
  "daily": {
    "time": ["YYYY-MM-DD", "YYYY-MM-DD", "YYYY-MM-DD"],
    "rain_sum": [number, number, number]
  }
}

🔍 Historical Weather vs Historical Forecast

(Activity)

11:15 – 11:30

Open-Meteo publishes two historical endpoints: one for reanalysis¹ data, one for old forecasts. Can you find both endpoints in the docs and build a request for each?

¹ Reanalysis combines historical observations with a weather model to fill gaps. Think of it as the best available estimate of what the weather was, not a direct measurement.

Find the endpoint, build the URL, compare

In groups of 3–4. Open open-meteo.com/en/docs and look at the sidebar.

Pair A

Find the Historical Weather API.

Pair B

Find the Historical Forecast API.

1. Both pairs: build a request for London (51.5085, -0.1257), daily temperature_2m_max and temperature_2m_min, 1–14 July 2025.

2. Compare what the two endpoints returned. Same numbers or different?
   (Eyeballing it is fine)

The two endpoints serve different data products for the same location and period. Historical Weather returns reanalysis (a model fitted to observations). Historical Forecast returns what the forecast said at the time.

Post your group’s finding in one sentence to the Discussion Forum.

Discussion Forum on Moodle

☕ Coffee Break

11:30 – 11:45

After the break:

• Fetch weather data with Python requests
• Turn JSON into a table (on paper, then with pandas)
• Save and reload JSON
• A preview of VS Code and Jupyter for the lab

4️⃣ Fetching Open-Meteo Data with requests

11:45 – 12:25

Build a Rainfall Request

import requests

url = "https://api.open-meteo.com/v1/forecast"

params = {
    "latitude": 51.5085,
    "longitude": -0.1257,
    "daily": "rain_sum",
    "timezone": "Europe/London",
    "forecast_days": 3
}

response = requests.get(url, params=params)

The dictionary keeps the parameters readable. Requests adds them to the endpoint URL and sends an HTTP GET request.

requests.get() returns a single value called a response. Like a dictionary, it bundles several pieces of information under one name: .status_code, .text, .headers, and .json().

Run this in ME204_W01D03_Lecture.py or in ipython. No output appears unless you ask for one. The response object is now in memory.

Check the Response

print(response.status_code)
print(response.url)

200
https://api.open-meteo.com/v1/forecast?latitude=51.5085&longitude=-0.1257&daily=rain_sum&timezone=Europe%2FLondon&forecast_days=3

• 200 means the request succeeded.
• response.url shows the endpoint and encoded query parameters that Requests sent.

200
https://api.open-meteo.com/v1/forecast?latitude=51.5085&longitude=-0.1257
  &daily=rain_sum&timezone=Europe%2FLondon&forecast_days=3

Turn the Response into Python Objects

data = response.json()
type(data)

dict

The HTTP response contains JSON text. response.json() decodes that text into Python objects.

JSON	Python
object	dictionary
array	list
string	string
number	integer or float

Explore the JSON Structure

Code

print(type(data))

print(data.keys())

print(type(data["daily"]))

print(data["daily"].keys())

Output

<class 'dict'>

dict_keys(['latitude', ..., 'daily_units', 'daily'])

<class 'dict'>

dict_keys(['time', 'rain_sum'])

Use keys to move through nested dictionaries:

daily = data["daily"]

Lists Inside the daily Dictionary

Code

print(type(daily["time"]))

print(type(daily["rain_sum"]))

print(len(daily["time"]))

print(len(daily["rain_sum"]))

Output

<class 'list'>

<class 'list'>

3

3

The matching positions describe the same day:

print(daily["time"][0])
print(daily["rain_sum"][0])

The values can change when the forecast changes. The structure and units tell us how to read them.

🔍 From JSON to Rows and Columns

(Activity)

12:25 – 12:37

You have seen the parallel lists inside daily. Now draw the table they become.

Draw the table

This JSON has four lists under daily and three keys outside it.

{
  "latitude": 51.51, "longitude": -0.13, "elevation": 27.0,
  "daily_units": {"time": "iso8601", "temperature_2m_max": "°C",
                  "temperature_2m_min": "°C", "rain_sum": "mm"},
  "daily": {
    "time": ["2025-07-01","2025-07-02","2025-07-03","2025-07-04","2025-07-05","2025-07-06","2025-07-07"],
    "temperature_2m_max": [24.4, 19.8, 19.1, 22.0, 25.3, 27.1, 23.6],
    "temperature_2m_min": [13.7, 14.4, 12.6, 11.9, 14.1, 16.2, 15.0],
    "rain_sum":           [0.0,  2.1,  5.4,  0.0,  0.0,  0.0,  1.3]
  }
}

In your groups (5 min). On paper or a shared doc:

1. Draw the table this data would become. How many rows? How many columns?
2. What is the column header for each? What value goes in row 3, column 2?
3. What would you do with latitude, longitude, and elevation? Drop them or add them to the table?

Save and Reload the JSON

Save the Python objects as JSON:

import json

file_path = "../data/weather/london_forecast_3_days.json"

with open(file_path, mode="w", encoding="utf-8") as f:
    json.dump(data, f, indent=4)

Reload the file into a new variable:

with open(file_path, mode="r", encoding="utf-8") as f:
    saved_data = json.load(f)

print(type(saved_data))
print(saved_data["daily"].keys())

Saved to ../data/weather/london_forecast_3_days.json
<class 'dict'>
dict_keys(['time', 'rain_sum'])

Read the Data Before You Analyse It

API access tells you how you received the data. It does not tell you how the values were produced.

Before analysis, check:

• Source: which organisation or model produced the values?
• Data type: observation, estimate, forecast, or model output?
• Spatial unit: point, grid cell, station, borough, or country?
• Temporal unit: hourly, daily, monthly, or annual?
• Variable definition: what does rain_sum include, and in which unit?

For example, Open-Meteo also distributes historical weather derived from ERA5-Land, a reanalysis model organised on a spatial grid. That description matters when deciding what a value can support.

The endpoint is the access route. The model or measurement system is the data source.

5️⃣ What you will use this afternoon

12:37 – 12:48

The afternoon lab runs in a Jupyter notebook inside VS Code on Nuvolos. Here is what you will see when you open it.

VS Code Is an Application

Visual Studio Code, usually called VS Code, is an application for working with code and files. It is an integrated development environment, or IDE, because several tools appear in one window.

Explorer

Browse folders and open files.

Editor

Read and change the selected file.

Terminal

Run the same Linux commands you used on Monday and Tuesday.

On Nuvolos, VS Code runs inside your ME204-2026 space and works with the files under /files/.

Open the Starter Notebook and Run a Cell

A Jupyter notebook is a document that keeps code, written explanation, and results together.

1. In Nuvolos, open Applications and start VSCode.
2. In the Explorer, open /files/notebooks/.
3. Select ME204_W01D03_Lab.ipynb.
4. Select a code cell and press the triangular Run Cell button.
5. Read the output directly below the cell.

message = "The notebook is running Python"
print(message)

The notebook is running Python

Run cells from top to bottom. A later cell may depend on variables or imports created earlier.

Open the Integrated Terminal

Select Terminal > New Terminal from the VS Code menu. The terminal panel opens at the bottom of the window, running the same Linux shell you used on Monday.

This terminal starts inside VS Code, but it runs the same commands as the separate Terminal application.

6️⃣ Sneak peek: from JSON to a pandas table

12:48 – 12:55

You just drew the table a JSON response becomes. pandas builds it in one line. This is a preview of 🖥️ Week 01 Day 04 Lecture and Lab, where you will learn pandas properly.

JSON to DataFrame

pandas is a third-party Python library for working with tables. It is installed on Nuvolos.

import pandas as pd

daily_data = data["daily"]
weather = pd.DataFrame(daily_data)
weather

Each dictionary key becomes a column. Values at the same list position become one row.

        time  rain_sum
  2026-07-15       0.0
  2026-07-16       1.4
  2026-07-17       0.2

Run ME204_W01D03_Lecture.py before class for today’s values.

Preview to_csv

One method writes the table as a CSV file:

weather.to_csv(
    "../data/weather/london_forecast_3_days.csv",
    index=False
)

• to_csv() writes the DataFrame to the named path.
• index=False prevents pandas from adding the row numbers as another column.

This is a preview. 🖥️ Week 01 Day 04 Lecture teaches how to inspect, select, and transform data with pandas.

Thanks!

12:55

This afternoon, open ME204_W01D03_Lab.ipynb and use the same request pattern to collect and inspect weather data.

💬 Post questions and observations to the Discussion Forum on Moodle. Your class teacher is also there to help during the afternoon lab.

References: Data and Download Tools

• ONS Census 2021 dataset TS001: the official page for the population dataset used in the download demonstration.
• curl command-line manual: the official reference for every curl option, including --location and --output.
• Chrome DevTools: Inspect network activity: instructions for opening the Network panel and examining requests made by a webpage.
• Everything curl: Redirects: an explanation of HTTP redirects and why curl needs --location to follow them.

References: HTTP and Python

• MDN: HTTP messages: a guide to what HTTP requests and responses contain.
• MDN: HTTP response status codes: a reference for codes such as 200, 301, 302, and 404.
• Requests: Quickstart: the official beginner’s guide to requests.get(), response content, status codes, and redirects in Python.

References: HTTP Response Status Codes

The first digit tells you the broad result: 1xx information, 2xx success, 3xx redirection, 4xx a problem with the request, and 5xx a problem on the server.

1xx: Information

Code	Meaning
100 Continue	Keep sending the request
101 Switching Protocols	Change to another protocol

2xx: Success

Code	Meaning
200 OK	Request succeeded
201 Created	A new resource was created
202 Accepted	Request accepted for later processing
204 No Content	Request succeeded with no response body

3xx: Redirection

Code	Meaning
301 Moved Permanently	Use the new address permanently
302 Found	Use another address for now
304 Not Modified	Use the cached copy
307 Temporary Redirect	Repeat the same request temporarily elsewhere
308 Permanent Redirect	Repeat the same request permanently elsewhere

4xx: Problem with the request

Code	Meaning
400 Bad Request	Server could not understand the request
401 Unauthorized	Authentication is required
403 Forbidden	Server refuses access
404 Not Found	Requested address was not found
405 Method Not Allowed	HTTP method is not allowed here
408 Request Timeout	Request took too long
409 Conflict	Request conflicts with the current state
429 Too Many Requests	Client has sent requests too quickly

5xx: Problem on the server

Code	Meaning
500 Internal Server Error	Server encountered an unexpected problem
501 Not Implemented	Server does not support the requested action
502 Bad Gateway	Another server returned a bad response
503 Service Unavailable	Server is temporarily unavailable
504 Gateway Timeout	Another server took too long to respond

References: APIs and Open-Meteo

• MDN: Application Programming Interface: a concise definition of an API and how software uses one.
• MDN: Representational State Transfer: a short explanation of REST and web requests for resources.
• Open-Meteo Forecast API: the official endpoint builder and parameter reference for forecast requests.
• Open-Meteo Historical Weather API data sources: descriptions of the models and reanalysis datasets that Open-Meteo distributes.

References: VS Code and Jupyter

• Visual Studio Code documentation: the official guide to the editor, Explorer, extensions, and other parts of VS Code.
• VS Code: Jupyter notebooks: instructions for opening, running, and editing notebook cells in VS Code.
• VS Code: Integrated terminal: instructions for opening and using a terminal inside VS Code.
• Jupyter Notebook documentation: an explanation of notebook documents, cells, kernels, and outputs.
• ME204: Using Nuvolos: the course guide for opening the ME204-2026 space and its applications.

References: Python and pandas

• Python: json: the standard-library reference for saving JSON with json.dump() and loading it with json.load().
• pandas: DataFrame: the official reference for constructing a table from a dictionary.
• pandas: to_datetime: the official reference for converting strings into date and time values.
• pandas: DataFrame.to_csv: the official reference for writing a DataFrame to CSV.

These slides were built with Quarto.

LSE Summer School 2026 | ME204 Week 01 Day 03