What We Mean by Data Engineering, and What You’ll Build
🎞️ Slides
Course overview, the tools you’ll use, where you’ll be by week 3, how AI can help you learn, and the Day 1 can’t AI do all of this for us? activity.
📓 Syllabus
LSE ME204 (2026) – Data Engineering Principles for the Social Sciences · Session 2 · 13–31 Jul 2026
Check this page daily: content, room details, and assessment dates can change as the course runs.
Lectures run in the morning at CBG.2.01 and labs run in the afternoon at LRB.5.02, Monday to Thursday each week. There is no class on Fridays.
Note from Jon: The syllabus on this page has a sequence that differs from the one you saw in the course outline when you signed up for ME204. The learning outcomes of the course are the same. Only the order of topics has changed. I always edit syllabi when I get closer to delivering my courses, because I have new, better ideas of how things will flow based on most recent teaching experience and things that happened in industry, etc.
Lectures and labs in Week 03 have a different rhythm. Most of the material covered is not required to appear in your final project. Treat it as a toolkit you can draw on if it fits your data and your question. We also reserve around one to one and a half hours each day for you to work on your final project and get help from us.
🗓️ Week 01 13 Jul 2026
-
16 Jul 2026
Day 01 (Mon 13 Jul)
What We Mean by Data Engineering, and What You’ll Build
🖥️ Lecture
💻 Lab
Where Am I? Finding Your Way and Writing Code
🛣️ Roadmap No code editor yet, terminal only
Find your way around the filesystem with the zone-hop game, then write and run your first script in nano.
Day 02 (Tue 14 Jul)
Reading Files the Way a Machine Does
🖥️ Lecture
Reading Files the Way a Machine Does
🎞️ Slides
How the filesystem is organised, Markdown source against its rendered form in glow, CSV and JSON as plain text, browsing structured data with fx, a first look at a binary file, and your first steps in Python (variables, lists, dictionaries, and reading a file).
💻 Lab
Teaching Python to Read Your Files
🛣️ Roadmap
Practice variables and lists in Python, then open and read a CSV into lists (the matching JSON file is there for reference only).
Day 03 (Wed 15 Jul)
Collecting Data from the Web (APIs) with Python
🖥️ Lecture
Collecting Data from the Web (APIs) with Python
REST APIs, fetching data with curl in the terminal, then doing the same with the requests library in a Jupyter notebook, pulling OpenMeteo weather data.
💻 Lab
Getting Weather Data with Python requests
🛣️ Roadmap Your first day in VS Code and Jupyter notebooks
Open VS Code for the first time, work in a Jupyter notebook there, then fetch, parse, and save the API data.
Day 04 (Thu 16 Jul)
Why We Like Our Data in Tables
🖥️ Lecture
Why We Like Our Data in Tables
An introduction to pandas and the DataFrame, why tabular data opens up more tools, and weighing AI against the official pandas documentation, so you check its worked examples instead of trusting code just because it runs.
💻 Lab
Authenticating to APIs
🛣️ Roadmap Midterm problem set released today
Sign up for OpenWeather, store your API key in a .env file, set up your midterm project folder on Nuvolos, and make your first authenticated request.
Important
Midterm problem set released today. See Assessment Structure for the brief, criteria, and deadline.
💂 Enjoy! (Fri - Sun)
No classes on Friday. Many students at the LSE Summer School are visiting London for the first time. If that is your case, here is a suggestion for a day out:
- Take the 15 double-decker bus at its starting point in Trafalgar Square.
- Enjoy views of some iconic London landmarks: the Royal Courts of Justice (near LSE), St Paul’s Cathedral, the Tower of London, and Tower Bridge.
- Get off the bus at the Aldgate East stop.
- Walk up to Brick Lane. If you are hungry, try the Upmarket Food Hall, or queue for the famous Beigel Shop.
🗓️ Week 02 20 Jul 2026
-
23 Jul 2026
Day 05 (Mon 20 Jul)
Making Charts Worth Looking At
🖥️ Lecture
Making Charts Worth Looking At
Interactive charts with plotly, and titles that convey the takeaway.
💻 Lab
Make a Chart That Makes Your Point
Plot your summarised data in plotly, then title it to state the finding. Midterm work period.
Day 06 (Tue 21 Jul)
Code Makeovers with pandas
🖥️ Lecture
Code Makeovers with pandas
Two code makeovers and a few optional pandas tools (.assign(), to_datetime, Boolean filtering, concat). Loops still get you to Good–Very Good tonight. For those using AI: tips on how to check whether output matches what you understand and want from the code.
💻 Lab
Super Tech Support
No new exercise. Your class teacher will assist with general queries that help you make progress on the midterm (due tonight at 8 pm).
Important
Your midterm problem set is due today. See Assessment Structure.
Day 07 (Wed 22 Jul)
Keeping a History of Your Work with Git
🖥️ Lecture
Keeping a History of Your Work with Git
Git basics, tracking changes, writing your own commit messages to think back on each change, and publishing Markdown as a site with GitHub Pages.
💻 Lab
Working on a Shared Repository
Git from here on: every project goes in a repo
In pairs or trios, create one shared GitHub repository, practise parallel commits, then create and resolve a merge conflict on purpose.
Day 08 (Thu 23 Jul)
Untangling Nested JSON: the pandas way
🖥️ Lecture
Untangling Nested JSON: the pandas way
Python functions and list comprehensions, list-of-dicts vs dict-of-lists, pd.json_normalize with record_path and meta, and pd.concat when a nested list still needs stitching.
💻 Lab
Nested JSON with PokéAPI
Fetch nested JSON from PokéAPI (no API key), collect images, build a key-fields table and flatten stats with pd.json_normalize, then move the functions into utils.py and import after a kernel restart.
Important
Final project brief announced today. Midterm grades are due to LSE tomorrow. See Assessment Structure.
💂 Enjoy! (Fri - Sun)
No classes on Friday. This is a great opportunity to make progress on your final project, but don’t forget to take a break and explore the city. Another suggestion: a visit to Borough Market, one of the largest and oldest food markets in London.
🗓️ Week 03 27 Jul 2026
-
30 Jul 2026
Day 09 (Mon 27 Jul)
Demos and Recipes: Web Scraping and HTML Parsing
🖥️ Lecture
Demos and Recipes: Web Scraping and HTML Parsing
Scraping vs APIs, parsing HTML when there is no API, and extracting structured data from web pages.
💻 Lab
Scrape and Parse HTML
Structured recap and final project work with support from the teaching team.
Day 10 (Tue 28 Jul)
Demos and Recipes: SQL and Joining Data
🖥️ Lecture
Demos and Recipes: SQL and Joining Data
Tables, keys, SELECT, GROUP BY, ORDER BY, joins and merges across sources, and comparing SQL queries with pandas method chains.
💻 Lab
Query and Join Data in SQLite
Write and run SQL queries, then answer the same questions with pandas method chains and compare the two approaches. Final project work period.
Day 11 (Wed 29 Jul)
Demos and Recipes: Dashboards, Responsible Reporting, and Quarto
🖥️ Lecture
Demos and Recipes: Dashboards, Responsible Reporting, and Quarto
Rich visualisation and dashboards, not claiming beyond what the data shows (e.g. correlation is not causation), and publishing analysis and narrative with Quarto.
💻 Lab
Structured Recap and Final Project Work
Teaching staff check your understanding of the key topics and how the work should be done, then you work on the final with support.
Day 12 (Thu 30 Jul)
Demos and Recipes: Data Governance, Good Documentation, and Agentic Coding
🖥️ Lecture
Demos and Recipes: Data Governance, Good Documentation, and Agentic Coding
Data governance, documentation others can follow, and working with agentic coding tools while staying in control of what they change.
💻 Lab
Structured Recap and Final Project Work
Teaching staff check your understanding of the key topics and how the work should be done, then you work on the final with support.
Important
Submit your final project by 5pm tomorrow, Fri 31 Jul. See Assessment Structure.
Assessment
See Assessment Structure on the Course Info page for the full breakdown, criteria, and deadlines.