Demos and Recipes LSE ME204 · Data Engineering Principles for the Social Sciences
🖥️ Week 03 Day 04 Lecture

Governance, Documentation, and Agentic Coding

Dr Jon Cardoso-Silva

LSE Data Science Institute

🗓️ 30 Jul 2026

🛠️ Final Project Support

10:00 – 11:00

I’ll be going around the room to help you. Use this hour to make progress on your project.

What you could be doing right now:

• Working in NB03: improving your charts, and deciding whether what you found is worth reporting
• Writing down the dead ends, the filters and approaches you tried that took you nowhere
• Polishing an insight you have, and working out how to tell the story around it
• Come talk to me about scope, feasibility, or a technical problem. If you are thinking about collecting more data or applying a different filter, we can work out together whether it is worth the time you have left

💡 Deadline reminder: submit your final project by 5pm tomorrow, Friday 31 July.

☕ Coffee Break

11:00 – 11:15

When we come back:

• What makes a set of steps a data pipeline, taking LinkedIn’s engineering blog as the case study
• How I use agentic tools on my own work
• Setting up your agentic coding environment, with Jonas

1️⃣ Professional Data Pipelines

11:15 – 11:35

The medallion architecture

We split your project across three notebooks so you practise a small version of what a professional pipeline does.

Bronze

Store the data in the format you collected it in, or close to it. It might be structured, like a CSV or JSON, or unstructured, like scraped HTML. Leave the cleaning for a later script.

In your project: whatever NB01 saved.

Silver

Fix the types, drop the rows you cannot use, and join the sources on a shared column like a date or a station name. Append to the same tables whenever you collect more data.

In your project: what NB02 wrote to the database.

Gold

Write the aggregation once, as a table or a VIEW, then query it with a SELECT. A notebook, a scheduled report, a dashboard, and a forecasting model can all read from it.

In your project: each plot_df you build in pandas for the charts in NB03.

A gold table is worth building when you will run the same query repeatedly on data you keep collecting. Think of the Covid case numbers: the same chart, rebuilt every day for two years. Nobody wants to re-run a notebook by hand for that, so you save the aggregation as a table or a VIEW and point the dashboard at it. Over three weeks we cannot know which of their plot_dfs would be the recurring one, so building them in pandas each time is right for this project.

What is a medallion architecture?, Databricks glossary

💡 Keep your bronze files in data/raw/. If you find a mistake in your cleaning on Friday morning, you can redo NB02 and NB03 without collecting the data a second time.

Case Study: LinkedIn’s data pipelines

• The Engineering team at LinkedIn maintain a very good technical blog, where they write about the data systems they build
• They have many moving pieces on their platform and, of course, they store data about everything (job posts, applications, recruitment, feed posts, etc)
• This is how they depict some of their different components and databases →

Figure 1 from Driving data enhancement and recruitment success with LinkedIn’s unified integrations, LinkedIn Engineering Blog, 10 March 2026

Case Study: pulling data into LinkedIn Recruiter

• Here is one of their data pipelines: the one behind LinkedIn Recruiter, pulling in hiring data from the employers’ own applicant tracking software (they call it BuildOut)
• This technical pipeline involves:
  • asking another company’s API for the data
  • saving the response exactly as it came back
  • converting it into the format LinkedIn’s own tools expect
  • saving that converted version separately
  • deciding how often to go back and ask for updates

Figure 3 from Driving data enhancement and recruitment success with LinkedIn’s unified integrations, LinkedIn Engineering Blog, 10 March 2026

The partner entity is stored before anything is converted, which is the same reason we keep the original responses under data/raw/. If the conversion turns out to be wrong, they replay from what was stored, with no second request to the partner.

2️⃣ How I Use Agentic Tools

11:35 – 12:00

• I use agentic tools as THE interface to nearly everything on my computer
• I experiment with several of them, Claude Code, Cursor, GitHub Copilot, and others, but what I keep are the workflows I have written for the tasks I repeat
• I also have skills for exploratory data analysis, for strict software engineering practices, and for writing. The writing one is not going well, because AI loves to write like AI
• I have integrations for calendar, e-mail and other stuff, but I don’t grant write access to any of those tools!

To search my own data / repeated tasks

I keep four years of course material in one folder. Before writing a new deck I want to know what I have taught before, so I ask the agent, and it runs the searches and reads the files.

I say what I am about to teach. It picks the folders and the search terms.

Five previous versions, with how each one framed the topic.

Tell them the first search was wrong. I used rg -ril, where -r is ripgrep’s replace flag, so it took il as a replacement string, printed the same wrong text on every line, and reported a count of 2061 files that meant nothing. I caught it by reading the output. The agent runs a search in seconds, and it runs the wrong search in seconds too.

To enhance what I can do

I often have to iterate on it a few more times, of course

3️⃣ Setting up agentic coding

Jonas Weinert

• Init an agent in an existing project, then add a CLAUDE.md / AGENTS.md so the next prompt follows your stack
• Plan mode: a multi-step task with messy data, where the agent drafts an editable plan before it runs
• Loops and agent teams: set the checks up front, then let the agent run with less watching

12:00 – 13:00

🏁 Thanks!

12:55 – 13:00

This afternoon

The last support session before you submit. Jonas and I will both be there, so bring us anything that is still not working.

💻 Final Project brief

HUGE THANKS indeed to your attention and dedication through this intense course! 🙇 🎉

And congrats on making it to the end with cool new hacking skills on your toolbox.

LSE Summer School 2026 | ME204 Week 03 Day 04