LSE ME204 · Data Engineering Principles for the Social Sciences
🖥️ Week 01 Day 04 Lecture

Why We Like Our Data in Tables

Jon Cardoso-Silva

Dr Jon Cardoso-Silva

LSE Data Science Institute

🗓️ 16 Jul 2026

1️⃣ What You Built This Week

10:00 – 10:35

Building Blocks So Far

🖥️ Terminal Navigate, create, read

cd, ls, cat, nano, mkdir

🐍 Python types Store values

str, int, float, list, dict

🔁 Control flow Decide and repeat

for, range(), if

🌐 Requests Fetch data from an API

requests.get(), .json()

📦 JSON Save and reload

json.dump(), json.load()

📂 File I/O Open, read, write

open(), .readlines(), .split(), print(f"...")

What you do with these is only limited by your creativity.

Yesterday’s Notebook

Let me walk you through what you built yesterday.

Your W01D03 notebook: fetching weather data, navigating JSON, writing CSV by hand.

🔍 Find the Heavy Rain Days

Find every day in your london_2025.csv where rainfall was above 10mm.
For each one, print the date, the rain total, and the precipitation total.

To load the file:

with open("../data/weather/london_2025.csv") as f:
    lines = f.readlines()

Use whatever you prefer: a .py script, ipython, or a Jupyter notebook.

Expected output:

2025-01-01  rain 10.5 mm  precip 10.5 mm
2025-01-05  rain 18.8 mm  precip 22.3 mm
2025-01-26  rain 11.0 mm  precip 11.0 mm
2025-02-24  rain 13.2 mm  precip 13.2 mm
2025-06-07  rain 13.2 mm  precip 13.2 mm
2025-07-19  rain 16.2 mm  precip 16.2 mm
2025-07-31  rain 26.7 mm  precip 26.7 mm
2025-08-29  rain 19.5 mm  precip 19.5 mm
2025-09-03  rain 25.6 mm  precip 25.6 mm
2025-10-03  rain 12.4 mm  precip 12.4 mm
2025-10-23  rain 14.2 mm  precip 14.2 mm
2025-11-14  rain 15.2 mm  precip 15.2 mm
2025-11-22  rain 10.4 mm  precip 10.4 mm
2025-12-05  rain 10.6 mm  precip 10.6 mm
2025-12-18  rain 17.7 mm  precip 17.7 mm

Post your code to the Discussion Forum.

Discussion Forum on Moodle

The Pure Python Solution

Let me show you in the notebook.

After the walkthrough, I will also show you Part IV from yesterday’s solutions.

2️⃣ What pandas Gives You

10:35 – 11:00

Once your data is in a table, every common question has a short answer. You do not write a new loop each time.

The Activity, with pandas

Step by step

Imagine you had these lines earlier in your code:

import pandas as pd

df = pd.read_csv("../data/weather/london_2025.csv")

Then:

is_heavy = df["rain_sum"] > 10

heavy_days = df[is_heavy]

heavy_days

Or, as a chain

(
    pd.read_csv("../data/weather/london_2025.csv")
    .query("rain_sum > 10")
)

Same 15 rows. The same answer you printed with “pure Python”

pd.read_csv()

import pandas as pd

df = pd.read_csv("../data/weather/london_2025.csv")
df
date rain_sum precipitation_sum
0 2025-01-01 10.5 10.5
1 2025-01-02 0.0 0.0
2 2025-01-03 0.0 0.0
3 2025-01-04 0.8 3.5
4 2025-01-05 18.8 22.3
… … … …
360 2025-12-27 0.0 0.0
361 2025-12-28 0.0 0.0
362 2025-12-29 0.0 0.0
363 2025-12-30 0.0 0.0
364 2025-12-31 0.0 0.0

365 rows x 3 columns

One call read the file, found the header, split the rows, and cast the types.

The Inspection Toolkit

df.head()
date rain_sum precipitation_sum
0 2025-01-01 10.5 10.5
1 2025-01-02 0.0 0.0
2 2025-01-03 0.0 0.0
3 2025-01-04 0.8 3.5
4 2025-01-05 18.8 22.3 
df.shape
(365, 3)
df.columns
Index(['date', 'rain_sum', 'precipitation_sum'], dtype='object')
df.dtypes
date                  object
rain_sum             float64
precipitation_sum    float64
dtype: object

These four are the first thing you run after loading any table.

What df["rain_sum"] Returns

df["rain_sum"]
0      10.5
1       0.0
2       0.0
3       0.8
4      18.8
       ... 
360     0.0
361     0.0
362     0.0
363     0.0
364     0.0
Name: rain_sum, Length: 365, dtype: float64

A single column from the table. pandas calls it a Series: one value per row, with an index on the left.

What df["rain_sum"] > 0 Returns

df["rain_sum"] > 0
0       True
1      False
2      False
3       True
4       True
       ... 
360    False
361    False
362    False
363    False
364    False
Name: rain_sum, Length: 365, dtype: bool

Same length, same index, but every value is True or False. The comparison runs on every row at once, without a loop.

Filtering Rows

df[df["rain_sum"] > 0]

Wrap the True/False column in df[...] and pandas keeps only the True rows.

original df date rain_sum mask Jan 01 10.5 True Jan 02 0.0 False Jan 03 0.0 False Jan 04 0.8 True Jan 05 18.8 True filtered df[mask] date rain_sum Jan 01 10.5 Jan 04 0.8 Jan 05 18.8 original->filtered  keep True rows  

Selecting Columns

df[["date", "rain_sum"]]

Name the columns you want. No position indexing.

original df date rain_sum precipitation_sum 0 2025-01-01 10.5 10.5 1 2025-01-02 0.0 0.0 2 2025-01-03 0.0 0.0 3 2025-01-04 0.8 3.5 selected df[["date", "rain_sum"]] date rain_sum 0 2025-01-01 10.5 1 2025-01-02 0.0 2 2025-01-03 0.0 3 2025-01-04 0.8 original->selected  keep named columns  

Method Chaining

(
    pd.read_csv("../data/weather/london_2025.csv")
    .query("rain_sum > 0")
    .filter(["date", "rain_sum"])
)

Read, query rows, filter columns: three verbs, top to bottom.

Why chain? Each step reads in order, with no intermediate variable names to track, closer to how you would describe the task in English.

The trade-off. You cannot inspect intermediate steps. When debugging, break the chain into separate variables and check each one.

(
    pd.read_csv("../data/weather/london_2025.csv")
    # .query("rain_sum > 0")
    # .filter(["date", "rain_sum"])
)
365 rows x 3 columns

Stop here. Did the file load? Right number of rows?

(
    pd.read_csv("../data/weather/london_2025.csv")
    .query("rain_sum > 0")
    # .filter(["date", "rain_sum"])
)
151 rows x 3 columns

Uncomment the next line. Did the filter keep the right rows?

(
    pd.read_csv("../data/weather/london_2025.csv")
    .query("rain_sum > 0")
    .filter(["date", "rain_sum"])
)
151 rows x 2 columns

Full chain. Are the right columns left?

3️⃣ Filter, Summarise, Group

11:00 – 11:25

One Number from a Column

Yesterday (loop)

total = 0
for i in range(len(rain)):
    total += rain[i]
print(total)
601.1

Four lines to get one number.

Today (pandas)

df["rain_sum"].sum()
601.1
df["rain_sum"].mean()
1.647
df["rain_sum"].max()
26.7

One line each.

.describe()

df.describe()
rain_sum precipitation_sum
count 365.0 365.0
mean 1.65 1.77
std 4.13 4.21
min 0.0 0.0
25% 0.0 0.0
50% 0.1 0.2
75% 1.5 1.6
max 26.7 26.7

count: how many values are not missing

mean: the average

std: how spread out the values are

min / max: the extremes

25% 50% 75%: the quartiles (50% is the median)

Boolean Filter + Count

rainy = df[df["rain_sum"] > 0]

len(rainy)

Does that number match what you got in yesterday’s lab?

Adding a Column

df["month"] = df["date"].str[:7]

df.head()

df["date"].str[:7] slices the first seven characters of every date string at once. No loop.

The DataFrame did not change until you stored the result in df["month"].

Group by Month

Yesterday (Part IV loop)

rainy_by_month = {}
for i in range(len(dates)):
    month = dates[i][:7]
    if month not in rainy_by_month:
        rainy_by_month[month] = 0
    if rain[i] > 0:
        rainy_by_month[month] += 1

Today (pandas)

(
    df[df["rain_sum"] > 0]
    .groupby("month")
    .size()
)

The dictionary, the loop, the two if statements, and the counter are all replaced by one chain that returns the same answer.

Split, Apply, Combine

  1. Split the rows into groups by the values in a column
  2. Apply a function (count, sum, mean) to each group
  3. Combine the results into a new table

pandas calls this pattern .groupby(). The name comes from SQL.

Reference: pandas, Group by: split-apply-combine

The Generic Pattern

df.groupby("grouping_column")["value_column"].function()
Part What it does
.groupby("month") Split the rows into groups by the values in that column
["rain_sum"] Select the column to compute on
.sum() / .mean() / .size() Apply the function to each group

Read more examples: pandas Group by documentation and Cookbook: Grouping

☕ Coffee Break

11:25 – 11:40

After the break:

  • Using AI to style a table, then checking it against the documentation

4️⃣ The Documentation Is the Ground Truth

11:40 – 12:15

🔍 Style a Table with AI

Open the lecture notebook. You have a groupby result from the section before the break.

  1. Copy the AI prompt from the notebook cell and paste it into your AI tool.
  2. Paste the code the AI gives you back into the next notebook cell.
  3. Run the cell. Does it work? Does the table look right?
  4. Open the pandas Styler documentation.
  5. Find the method the AI used. Is it real? Are the parameters correct?
  6. Change one thing (the colour, the format, or which column gets highlighted) using the documentation, not the AI.

The Principle

Do the DataFrame transformation yourself. That is where your analytical thinking is.

Once you have the table you want to present, ask the AI for the styling code. Formatting is where AI tools save you time, while the table design stays yours.

Then check the output against the docs. AI tools sometimes invent method names or use parameters that do not or no longer exist.

The pandas Styler documentation is the ground truth.

Three Reference Pages

When an AI gives you pandas code you do not understand, start here:

When you want to know… Go here
What is a Series or DataFrame? Intro to data structures
What does .groupby() do? Group by: split-apply-combine
Is this a known pattern? Cookbook: Grouping

The Technique Rule

If you use a technique that was not covered in lectures or labs, include a markdown cell that:

(a) explains why you chose it, and

(b) walks through how you tested and validated each step, as if you were writing a short tutorial for yourself.

This applies to techniques from the pandas documentation, from Stack Overflow, or from an AI tool.

This is one way to show us you did not blindly copy-paste from an AI but steered it to show you something new and relevant. We will check how coherent it is with the course material and with your own writing and coding style elsewhere in your submission.

5️⃣ What Comes Next

12:15 – 13:00

This Afternoon’s Lab

Start your midterm: sign up for OpenWeather, set up your project folder, store your API key in a .env file, and make your first authenticated request.

The lab page, Authenticating to APIs, has the step-by-step instructions. Your class teacher is there for technical setup.

Saving a DataFrame

df.to_csv("london_2025_monthly.csv", index=False)

.to_csv() writes your DataFrame to a CSV file.

The lab has a surprise about this method. See if you can spot what goes wrong.

Dates Are Strings (For Now)

The dates in your DataFrame are strings right now.

"2025-01-01" is text, not a date. pandas cannot sort by month or filter by season until it knows the column holds dates.

Next week you will meet pd.to_datetime(), which turns those strings into proper dates. The lab has a <details> block that previews it.

✍️ Midterm Project

⏳ Deadline Tuesday, 21 July 2026 at 8 pm UK time
💎 Weight 25% of your final grade
📂 Submission Nuvolos assignment hand-in
🤖 AI policy Position 3: full authorised use

The Question

“Do coastal cities cool down faster at night than inland cities?”

You will use the OpenWeather API, a different API from the Open-Meteo service you used in the labs.

This afternoon in the lab: sign up for a free API key at openweathermap.org and start reading their documentation.

Thanks!

13:00

This afternoon, open the lab page and get started on your midterm project setup. The midterm brief is on the course site.

💬 Post questions and observations to the Discussion Forum on Moodle. Your class teacher is also there to help during the afternoon lab.

LSE Summer School 2026 | ME204 Week 01 Day 04