🗓️ Week 01 | Day 02 Working with Data Types and APIs
ME204 – Data Engineering for the Social World
Dr Jon Cardoso-Silva
15 July 2025
Recap & Today’s Agenda
Yesterday: We mapped a data workflow and thought about the process of data pipelines at a conceptual and high-level. In the afternoon, you familiarised yourself with the Nuvolos environment and started to run some Python code.
Today: We start to look at the technical details of how to handle data effectively in a computer. It will be our formal introduction to the concept of API (using the requests library to demonstrate), and to the concept of DataFrames (using the pandas library).
Can’t an AI do the analysis for us? 10:05 – 10:30
One interesting question from yesterday’s lecture:
“could we get an AI to do the whole data processing and data analysis for us?”
It’s likely that temperatures of 40°C (in Farenheits: 104°F) will become the new normal in the UK.
I came up with this exploratory question which we could subject to data analysis:
How has the frequency of heatwaves in the UK changed over the last 10 years?
An Initial Prompt
Let’s build this together.
I will use Claude to help me with this, just because it’s the one we have access to at LSE (you can try to mirror what I’m doing, live, with any other AI of your choice).
Would the following prompt work? Why or why not?
Create a full visual data science analysis report on the frequency of heatwaves in the UK over the last 10 years.
Let’s try it, think about the output and then tweak it. We will force it to use a specific data source and a specific definition of a “heatwave” and see how it changes the output.
The “Human in the Loop”
Based on my tests ahead of the live demo, the results might vary when I repeat the prompt live but I think we will always find that for a reliable, reproducible answer, we need to be in control.
We need to choose a trustworthy data source.
We need to use a clear, specific robust definition of a “heatwave”.
We need to build a workflow that is reproducible and anyone can re-run and verify.
This is why we need to learn what’s “under the hood”.
Under the Hood: Data Types 10:30 – 11:00
Before we discuss how to collect data, let’s go even deeper into the computer’s memory.
How is data actually stored in a computer?
Computers only understand 0s and 1s
Numbers, text, images, and sounds are all stored as sequences of 0s and 1s in your computer’s memory. Each 0 or 1 is called a bit.
Think of a bit as a tiny box:
\[
\require{color}
\fcolorbox{black}{white}{$\phantom{0}$} \phantom{\leftarrow \text{a bit can have a value of $0$}}
\]
Computers only understand 0s and 1s
Numbers, text, images, and sounds are all stored as sequences of 0s and 1s in your computer’s memory. Each 0 or 1 is called a bit.
Think of a bit as a tiny box:
\[
\require{color}
\begin{array}{ccc}
\fcolorbox{black}{#eeeeee}{0} & \leftarrow & \text{a bit can have a value of $0$}
\end{array}
\]
Computers only understand 0s and 1s
Numbers, text, images, and sounds are all stored as sequences of 0s and 1s in your computer’s memory. Each 0 or 1 is called a bit.
Think of a bit as a tiny box:
\[
\begin{array}{ccc}
\fcolorbox{black}{#111111}{$\textcolor{white}{1}$} & \leftarrow & \text{OR it can have a value of $1$}
\end{array}
\]
but nothing else!
Boolean data type (aka bool)
For everything that has a ‘Yes’ or ‘No’ answer, we can use a single bit.
\[
\textcolor{#9753b8}{\texttt{is_it_raining}} = \begin{cases}
\fcolorbox{black}{#eeeeee}{$\textcolor{black}{0}$} & \text{if it is not raining} \\
\fcolorbox{black}{#111111}{$\textcolor{white}{1}$} & \text{if it is raining}
\end{cases}
\]
In Python:
# if it is rainingis_it_raining =True# if it is not rainingis_it_raining =False
What about numbers?
Suppose we want to represent positive numbers (0 included). We can’t do that with just a single bit!
With \(2\) bits, we can represent \(4\) different numbers:
💡 You won’t need to do this type of calculation manually in this course!
However, understanding the theory behind binary numbers is crucial for:
Using libraries like numpy and pandas efficiently:
Choosing the correct data types (e.g., int32, float64) saves memory and makes your code faster.
It helps you avoid common pitfalls like overflow errors.
Working with real-world datasets:
Some data is stored in binary formats.
Understanding the fundamentals helps you debug weird data issues.
Enjoy the simplicity of Python but remember that understanding what’s under the hood matters!
What If I need a decimal number?
Decimal numbers are represented using the floating-point (float) data type.
\[
\textcolor{#9753b8}{\texttt{pi}} = 3.14159
\]
It’s conceptually similar: a certain number of bits are used to store the number, split between:
the sign,
the main digits (mantissa), and
the position of the decimal point (exponent).
And what about text?
Text is stored by mapping each character to a number. UTF-8 is the modern standard that includes characters from most of the world’s languages, and even emojis!
The letter ‘A’ is 65
The letter ‘a’ is 97
The emoji ‘😊’ is 128522
Each of those numbers is then stored in binary, just like we saw before.
The Problem with Separate Lists
Think about this code, which you might have seen yesterday.
# From yesterday's labdates = ['2025-07-14', '2025-07-15', ...]max_temps = [22, 25, ...]conditions = ['Cloudy', 'Sunny', ...]# To get info for the first day...msg =f"Date: {dates[0]}"msg +=f", Temp: {max_temps[0]}"msg +=f", Condition: {conditions[0]}"print(msg)
Coordination Problem: The data is disconnected. If you sort one list, but not the others, you’ve corrupted your data.
No Type Enforcement: A list can hold anything. You could accidentally put a string in max_temps, leading to all sorts of errors down the line.
Inefficient: Python lists are flexible but slow for numerical calculations.
The Solution: numpy and pandas
Enter numpy and pandas
numpy is the foundational library for numerical computing in Python. It provides an efficient* array object that stores data of the same type, using the binary representations we just discussed. This makes it incredibly fast.
pandas is built on top of numpy. It provides the DataFrame, a two-dimensional table that can hold columns of different types.
Figure 1: Numpy and Pandas logos.
The NumPy Array: A Grid of Values
A numpy array is a grid of values, all of the same type.
import numpy as np# A regular Python listpython_list = [22, 25, 21, 19]# A numpy array where each value# is stored as a single byte (8 bits)# 8 bits can represent only 256 different valuesnumpy_array = np.array(python_list, dtype=np.int8)print(numpy_array)
Output:array([22, 25, 21, 19], dtype=int8)The formatting of the output is just an explicit signal to the reader that this is not a simple list, but a numpy array.
Choosing the right dtype is a trade-off between memory and precision. Using a smaller dtype can save a lot of memory, which is crucial for large datasets.
Data Type
Signed/Unsigned
Bits
Range (Approximate)
np.int8
Signed
8
-128 to 127
np.uint8
Unsigned
8
0 to 255
np.int16
Signed
16
-32,768 to 32,767
np.uint16
Unsigned
16
0 to 65,535
np.int32
Signed
32
-2,147,483,648 to 2,147,483,647
np.uint32
Unsigned
32
0 to 4,294,967,295
np.int64
Signed
64
-9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
np.uint64
Unsigned
64
0 to 18,446,744,073,709,551,615
np.float16
Signed
16
-65,504 to 65,504 (~3 digits precision)
np.float32
Signed
32
-3.4e+38 to 3.4e+38 (~7 digits precision)
np.float64
Signed
64
-1.8e+308 to 1.8e+308 (~16 digits precision)
The DataFrame: A Collection of Arrays
A pandas DataFrame is essentially a dictionary where the keys are column names and the values are numpy arrays (wrapped inside a pandas object called a Series).
We can convert our messy lists into a clean, structured DataFrame with pandas.
Code:
import pandas as pd# Combine our lists into a dictionaryweather_data = {'date': ['2025-07-14', '2025-07-15'],'max_temp': [22, 25],'condition': ['Cloudy', 'Sunny']}# Create a DataFramedf = pd.DataFrame(weather_data)print(df)
Output:
date max_temp condition02025-07-1422 Cloudy12025-07-1525 Sunny
Data is now organised, linked, and typed.
max_temp will be an integer type (int64), making it fast for calculations.
date and condition will be ‘object’ types (strings).
The CSV Connection
DataFrames are brilliant for working with data and one of the most common formats that mirrors the structure of a DataFrame very closely is the Comma-Separated Values (CSV) file.
DataFrames can be easily saved to a CSV file:
df.to_csv('weather_data.csv', index=False)
If you open a CSV file in a text editor, you’ll see that it’s just a plain text file with a few commas separating the columns:
Data doesn’t always come in a tabular format. Sometimes you will inevitably have to work with JSON data.
Somewhat formally:
JSON stands for JavaScript Object Notation. It’s a lightweight data-interchange format that is easy for humans to read and write, and easy for machines to parse and generate.*
But I like to think about it as a mixture of lists and dictionaries.
It’s a website that provides weather data for a given location.
It’s a free service (with some limits on the number of requests).
Collecting Data from APIs 11:00 – 11:30
What’s an API?
API stands for Application Programming Interface.
Think of it as a waiter at a restaurant.
You (the “client”) have a menu of requests you can make.
You give your order to the waiter (the API), who goes to the kitchen (the “server”) and brings back the food (the “data”) you asked for.
It’s a structured way for computer programs to talk to each other and for us to to talk to them.
Using the requests library
In Python, we can use the requests library to make HTTP requests to APIs.
It does not come pre-installed with Python, so we need to install it:
pip install requests
Then, after importing it, we can use it to make a request to the API:
import requestsbase_url ="https://api.open-meteo.com/v1/forecast"params = {"latitude": 51.5085,"longitude": -0.1257,"daily": "temperature_2m_max,temperature_2m_min,weather_code","forecast_days": 3}response = requests.get(base_url, params=params)# Check the response status codeprint(response.status_code) # 200 means success!# Get the content in JSON formatdata = response.json()
From API to DataFrame
Assuming the request was successful, you will get something like this:
Nuvolos environment and started to run some Python code.
Claude to help me with this, just because it’s the one we have access to at LSE 
