ME204 2026 Icon

✍️ Midterm Project (25%)

ME204 (2026) – Data Engineering Principles for the Social Sciences

Author

Dr Jon Cardoso-Silva

Last updated

16 July 2026

This is your first graded assessment, worth 25% of your final grade for ME204. It is an individual project where you apply the skills and principles we have covered so far to a new data question using a new API.

📝 Key Info

⏳ Deadline Tuesday, 21 July 2026 at 8 pm UK time
💎 Weight 25% of final grade
📂 Submission Nuvolos assignment hand-in (guide; multiple submissions allowed, latest counts)
🤖 AI policy Position 3: full authorised use (details)

📚 The Commission

You have been commissioned by the fictitiously famous Office of Quirky Inquiries (OQI) to answer a question about weather and geography. A secret source tells us that the OQI is not that interested in the actual answer, weirdly, but in the process you go through to answer it.

“Do coastal cities cool down faster at night than inland cities?”

Your goal is to collect weather data for several cities, define what “nighttime cooling” means, measure it, and compare coastal and inland locations. You will use the OpenWeather API, which requires a free API key, so every request is linked to your own account rather than shared across the class.

⚠️ Update (17 July): two data paths.

Some of OpenWeather’s endpoints require a credit card on file, even on the free tier. I do not want that to be a barrier for anyone, so the brief now offers two paths to the same question, graded with the same rubric:

  • Path A: historical hourly data. You subscribe to One Call API 3.0 or 4.0 (credit card on file, but free within the daily limit).
  • Path B: forecast data. You use a forecast endpoint that is part of the free plan, no credit card needed.

The question, the rubric, and the bonus all apply to both. Read The Data Source below, pick one path, and go with it.

🤔 Key Decisions

📌 DECISIONS, DECISIONS, DECISIONS

The OQI requires you to use the OpenWeather API but leaves the rest up to you. You must decide:

  • Cities: Choose 3-4 cities, at least one coastal and at least one inland. Your choice must make for a fair comparison. Explain what your cities share and why the coastal/inland difference is the most likely explanation for any patterns you find. Back up your claims with data from reputable sources (geographic databases, national meteorological services, or similar). We will grade you on the quality of this reasoning.
  • Time period: If you use historical data, how many weeks will you collect, and why? If you use the forecast endpoint, you get 5 days of data at 3 hour intervals. Either way, explain what your time window gives you and what it leaves out.
  • Variables: Which fields from the API response will you use? You may want more than just temperature.
  • Nighttime definition: This may sound like a silly question, but what hours count as “night”? It is not obvious, and it may depend on the season and latitude of your cities.
  • Cooling rate: How will you define and measure “cooling”? The drop from the evening peak to the morning low? An average hourly decline? Something else?
  • Aggregation: Will you compare nightly averages, weekly patterns, or something else? What grouping makes sense given your time window?

In your notebooks or scripts, document each decision and explain your reasoning. If you consulted external sources (search engines, documentation, AI tools), say what you found and how it shaped your choice.

📡 The Data Source

This project uses the OpenWeather API, a different API from the Open-Meteo service you used in the labs.

You will need a free API key. Sign up at openweathermap.org, then find your key on the API keys page. The NB01 template walks you through storing the key in a .env file using python-dotenv.

The template shows a working call to the current weather endpoint. For this project you need weather data across multiple time points for your chosen cities. OpenWeather offers two paths.

Path A: Historical data (requires a separate subscription with a credit card on file). The One Call API 3.0 or 4.0 provides hourly data going back years. You subscribe, set your daily call limit to 1,000 (the free allowance), and will not be charged if you stay within the limit. Read the OpenWeather docs for details on the timemachine endpoint.

Path B: forecast (free plan, no credit card). OpenWeather’s free tier includes a forecast endpoint that returns data every 3 hours for the next 5 days. Find it in the API documentation. One call per city gives you around 40 timestamps. The response has the same nested JSON structure as the current-weather call, including temp, feels_like, humidity, and wind. You get roughly 5 nights of data per city, which is enough to define and compare nighttime cooling. In your documentation, note that this is forecast (model-predicted) data, not observed measurements, and discuss what that means for your conclusions.

Both paths are valid and graded with the same rubric. Pick one and commit to it.

A heads-up on timestamps: depending on when you start working on this, you will notice the API returns timestamps in a format we may not have covered in the lectures and labs yet. The NB01 template includes one example of converting this format to a readable date. Can you find it? From there, you will need to apply the same idea to the rest of your data.

📂 What You Submit

Your submission is a single folder, handed in through the Nuvolos assignment. Follow Hand in an Assignment on Nuvolos for the clicks. You choose one format for your code: Jupyter notebooks or Python scripts. Do not mix them.

Notebook path

midterm/
├── .env
├── README.md
├── NB01-Data-Collection.ipynb    ← template
├── NB02-Data-Transformation.ipynb
├── NB03-Data-Analysis.ipynb
├── NB04-Bonus.ipynb              ← optional
├── data/
│   ├── *.json   (raw API responses)
│   └── *.csv    (transformed data)
└── figures/
    └── *.png    (saved charts)

Script path

midterm/
├── .env
├── README.md
├── 01-collection.py                ← template
├── 02-transformation.py
├── 03-analysis.py
├── 04-bonus.py                     ← optional
├── data/
│   ├── *.json   (raw API responses)
│   └── *.csv    (transformed data)
└── figures/
    └── *.png    (saved charts)
What the template gives you (NB01 or the collection script)

The template includes: the import block, .env loading with python-dotenv, a working API call to the OpenWeather current weather endpoint for London, a status-code check, a printout of the JSON response, and one example of converting the timestamp to a readable date. You can run it and see a 200 before writing any code of your own.

From there, you switch to the endpoint that matches your chosen path (Path A or Path B above), adapt the request, loop across your chosen cities, and save each response as a JSON file in data/.

If you choose Python scripts

Your scripts must include commentary as structured comments (lines starting with #) that contain the same decision annotations and reasoning the rubric requires. Every figure must be saved to figures/ using plt.savefig() or equivalent. Your code must run top to bottom with python 01-collection.py, then python 02-transformation.py, and so on.

README.md

Your README must include:

  • Your LSE ID on the first line (for example **LSE ID:** 2026xxxxxxxx).
  • A brief project overview and the research question.
  • Your methodology: which cities, which variables, how you defined cooling rate, and why.
  • Reproduction steps: what packages are needed, where the .env file goes.
  • A summary of your findings with the final chart embedded as an image.

Keep it concise. The README is not a rewrite of your notebooks.

✍️ Format and Quality Standards

Code: as short as possible, but no shorter than to hurt readability. If you can replace a block with a shorter expression that a Week 1 student can still follow, the shorter version is better. A block of 40 lines that does what 15 lines could do is padding. A single expression compressed past the point of readability is obscurity.

Documentation: as short as possible, but no shorter than to hurt clarity of argument. Every sentence must tell the reader something they cannot see from the code. If it restates what the next line does (“Here I load the CSV”), delete it. We want to know what you decided and why, not the full deliberation.

  • Reproducibility: your notebooks or scripts must run top to bottom in your Nuvolos instance without errors. When we mark, we open your environment and re-run everything.
  • Figures: all charts must be saved to figures/ as .png files. The README embeds the final chart from this folder.

🤖 Generative AI Policy

This course uses Position 3: full authorised use of generative AI. See the Course Info § AI Policy and the Day 1 slides for the full details. The marking-relevant points are below.

We use a mix of Python scripts and LLM routines to build an inventory of what each submission contains. This produces evidence, not a grade. A human marker reads your work first, then checks the inventory, and decides the mark.

✔️ How We Will Grade Your Work

We assess your thinking and decision-making process, not just whether the code runs. A submission where everything works but shows no evidence of intentional choices will pass and no more.

From the Really Good band (70%) upward, your decision annotations must be grounded in specific observations from your own data: actual values, patterns, or comparisons you encountered. Telling us that a variable is conceptually a good measure of cooling is an argument anyone could make before running the code. Telling us that when you compared two variables for your cities, the data showed a specific pattern that shaped your analysis is evidence of genuine engagement.

If you used approaches beyond what was taught in the course, document why. Unexplained use of unfamiliar patterns will be read as uncritical AI use.

A note on errors: we distinguish between errors that reflect misunderstanding and errors that reflect an oversight in otherwise competent work. A genuine mistake in a well-documented submission is penalised proportionately in the criterion where it occurred. It will not cascade across your entire grade if the rest of your work shows understanding.

📡 Data Collection and API Work (30 marks)
Level Description
Pass (50%) Code fetches data from the OpenWeather API with a working key. JSON files saved in data/. Cities are stated.
Good (60%) Code follows the API interaction patterns taught in W01D03 (adapted for a new API). Cities and variables chosen with a stated comparative purpose. Functions or loops used to fetch multiple cities. Justification present and within scope.
Really Good (70%) City selection makes for a fair comparison, backed by reputable sources. The justification explains what the cities share and why the coastal/inland difference is the most likely driver. Code handles the timestamp format and multi-city collection without unnecessary complexity. Specific data observations support the reasoning, not just general arguments.
WOW The comparative framework is inventive. The student names trade-offs they identified through inspecting their data. Collection code is notably clean and well-adapted to the API’s structure.
📊 Analysis and Visualisation (35 marks)
Level Description
Pass (50%) Data loaded from CSV. Some analysis present. A chart exists using plotly, matplotlib, or seaborn.
Good (60%) A cooling-rate measure is defined and applied. Pandas operations used (groupby, filter, new columns). Chart has a title and labels. Conclusions follow from the analysis shown.
Really Good (70%) Decision annotations in the code reference specific values or patterns from the data to explain analytical choices. The cooling-rate definition is justified with data. Chart title states a finding, not a description of axes. Chart type serves the comparison. The analysis shows the student responded to what the data contained, not just applied a recipe.
WOW The student finds a pattern in the data and names it. In the decision annotations, the student weighs alternatives using evidence. The chart states the finding so clearly that a reader could understand it without the surrounding text.
📝 Documentation and Communication (35 marks)
Level Description
Pass (50%) README exists and explains the project. Code has some commentary. Chart embedded in README. Notebooks/scripts run.
Good (60%) README clearly explains methodology and reproduction steps. Commentary in the code explains analytical steps, not just what the code does. Submission is well-organised with data and figures in the right folders.
Really Good (70%) Documentation explains why, not just what. Commentary is concise: every sentence belongs. README methodology reflects the student’s own reasoning about this specific project. Specific data values are cited in decision annotations. The student’s voice is recognisable (not generic AI output). If the student refactored after W02D02, a short note explains what changed and why.
WOW Documentation across the whole submission tells a coherent story. Commentary is precise and economical. README could serve as a reference for how to present a short data project. The student’s reasoning is visible throughout and could not have been produced without doing the work.
🏆 Bonus: feels_like vs temp (+5 marks)

Both temp and feels_like are in the same API response you already collected. In NB04 or 04-bonus.py, compare whether the humidity-adjusted measure (feels_like) tells a different story about coastal vs inland nighttime cooling. Does a humid coastal city “feel” like it cools less, even if the thermometer drops faster?

The bonus is outside the 100-mark total. Maximum achievable: 105, capped at 100. No penalty for skipping it.

Level Description
1-2 marks Comparison is present but shallow. Chart shows both variables without much interpretation.
3-4 marks The student shows whether the pattern changes and explains why or why not, grounded in the data.
5 marks The comparison adds a genuine insight the main analysis missed. The student identifies what feels_like captures that raw temperature does not, with specific values from their cities.

🔗 Useful Resources

📊 API and Data

💻 Course Materials

  • 💻 W01D03 Lab: API fetching with requests
  • 💻 W01D04 Lab: Authenticating to APIs (OpenWeather signup, .env, project setup)
  • 💻 W02D01 Lecture (Mon 20 Jul): plotly charts and narrative titles
  • 💻 W02D02 Lecture (Tue 21 Jul): vectorised pandas and pd.to_datetime

🌐 External Resources

🆘 Getting Help

  • Moodle forum: post questions any time
  • Email:

📎 Appendix