# DS105W (2025/2026) - Mini Project 2

**Report date:** 24 March 2026 | **Data collection date:** 18 March 2026 | **Author:** James Weiru Lu

This project answers the question: "How does travelling between Inner London and Outer London change when it is done at peak versus off-peak hours?" using data from the [TfL Journey Planner API](https://api.tfl.gov.uk) and the [ONS Postcode Directory](https://data.london.gov.uk/dataset/postcode-directory-for-london-exp5p/).

## 📌 DECISIONS, DECISIONS, DECISIONS

### 1. Reference point (Inner London origin)

I used LSE (WC2A 2AE) as the fixed destination for all inbound journeys. The assignment brief suggests LSE as a reasonable default, and it sits within the Greater London Authority's [Central Activities Zone](https://www.london.gov.uk/programmes-strategies/planning/implementing-london-plan/london-plan-guidance-and-spgs/central-activities-zone), described as "London's vibrant centre." Fixing one destination controls the comparison: any observed peak/off-peak differences can be attributed to time-of-day effects on the route, not to differences in destination accessibility. The W07 Methodology Scoping exercise (W07 Slide 4) required specific, evidence-backed definitions, so I anchored my choice in the GLA's formal designation rather than intuition.

Other locations I considered but rejected were Charing Cross (the official centre of London, but adjacent to a mainline station that would bias routes toward the Southeast) and Holborn (directly on the Central and Piccadilly lines, biasing results toward those corridors).

### 2. Outer London origins

I selected ten Outer London postcodes to create analytical contrast across directional corridors, deprivation levels, and dominant transport modes:

| Origin | Postcode | Direction | Corridor | IMD context |
|---|---|---|---|---|
| Barking | IG11 7LU | East | District / H&C | Lower |
| Romford | RM1 1AU | Northeast | Elizabeth line | Lower |
| Croydon | CR0 1LF | South | Southern / Thameslink | Lower |
| Harrow | HA1 1AR | Northwest | Metropolitan line | Mid |
| Barnet | EN5 5XY | North | Northern line | Higher |
| Bromley | BR1 1AA | Southeast | National Rail | Higher |
| Sutton | SM1 1EA | South-southwest | Thameslink / Southern | Mid |
| Woolwich | SE18 6HQ | Southeast | Elizabeth line | Lower |
| Hounslow | TW3 1HA | West | Piccadilly line | Mid |
| Walthamstow | E17 7JN | Northeast | Victoria line | Lower |

The set spans all cardinal directions around London, includes three corridor types (Tube-dominated, Elizabeth line, National Rail), and varies from IMD rank ~5,300 (Woolwich, more deprived) to ~27,900 (Barnet, less deprived). This deliberate variation follows the principle of choosing destinations to generate contrast, as the W07 Methodology Scoping notebook warned that representing each route with only one observation makes the comparison fragile. All ten postcodes were confirmed active (no termination date) and located in Outer London boroughs via the ONS Open Geography Portal. Four of the ten (Barking, Walthamstow, Woolwich, Sutton) are classified as `usertype = 1` ("large user") in the ONS directory, which reflects high mail volume at town-centre postcodes near stations rather than an absence of residents. This validation is documented in NB02 Section 8.2. The ONS merge itself follows the left-join pattern from the W08 Lecture and W08 Lab for enriching API data with geographic context.

### 3. Time periods

| Period | Times used | Justification |
|---|---|---|
| Peak | 07:30, 08:00 | Falls within TfL's [AM peak definition](https://content.tfl.gov.uk/connectivity-assessment-guide.pdf) (06:30–09:30). Two times reduce snapshot dependence. |
| Off-peak | 11:30, 12:00 | Falls within TfL's inter-peak window (09:30–16:00). Midday avoids school-run effects and lunchtime spikes. |

I collected all journeys for Wednesday 18 March 2026, a regular weekday with no planned TfL engineering works at the time of collection. The direction is inbound (Outer London → LSE), simulating the morning commute. I used two departure times per period rather than one because the W07 Methodology Scoping exercise made me realise that a single timestamp makes the comparison fragile. Two times per window yield approximately six journey observations per origin-period group (TfL returns ~3 options per request), which is sufficient for median comparison even if not for robust statistical inference.

### 4. Metrics

My primary metric is duration (minutes), the most intuitive measure of journey quality. Secondary metrics include fare (£), transfer count, walking minutes, leg count, and mode sequence. I deliberately collected multiple dimensions because the research question asks how travel "changes" — not only whether it slows down. I entered NB03 expecting duration to be the main finding, but the data showed fare was the only metric that consistently separated peak from off-peak. This pivot is documented throughout NB03's reflections and follows the principle from the SAMPLE WORK SKILL 2 document: let the data change your claims.

## 🤖 AI usage

I used AI at several points during this project:

- **Claude (DS105W course bot):** I used the course bot for debugging `pd.json_normalize()` parameter combinations when the TfL response nesting was deeper than expected. I also used it to think through the sign test methodology — specifically, whether the binomial sign test was appropriate for n=10 origin-level medians versus the paired t-test or Wilcoxon alternatives.
- **General Claude:** I used it for structuring the analysis pipeline (NB01 → NB02 → NB03 flow), proofreading notebook reflections, and generating the report planning strategy.
- **Not used for:** Final insight selection, narrative title wording, or the interpretation paragraphs — those reflect my own analytical judgement shaped by the data exploration in NB03.

<div style="font-family: system-ui; padding: 20px 30px 20px 20px; background-color: #FFFFFF; border-left: 8px solid #ED9255; border-radius: 8px; box-shadow: 0 4px 12px rgba(0, 0, 0, 0.1);max-width:600px;color:#212121;">

<span style="color: #ED9255;font-size:1.15em;font-weight:bold;">NOTE FROM JON:</span> James was very deliberate about keeping his own voice throughout the notebooks and this report, using AI as a tool for debugging and structuring rather than for generating final prose. His AI disclosure is one of the clearest examples in this cohort of explaining both what AI did and what it didn't do.

</div>

## 🔩 Technical Implementation

![](./figures/report_data_science_workflow_summary.jpg)

This project began as a question-driven start — the assignment set the research question, and every subsequent decision served it. The diagram above traces the pipeline and the three feedback loops that shaped the final analysis.

The main spine runs through three notebooks. NB01 collects 40 API responses (10 origins × 4 departure times) from the TfL Journey Planner and saves them as a single combined JSON in `data/raw/`. NB02 flattens the nested JSON using `pd.json_normalize()` with `record_path` and `meta` parameters (W07 Lecture), extracts secondary metrics from the `legs` array using `explode()` and vectorised `groupby().agg()` aggregation (W08 Lecture), merges with the ONS Postcode Directory via a left join on standardised postcode keys (W08 Lab), and outputs one cleaned CSV (`data/processed/journey_options.csv`, 120 rows × 34 columns). NB03 loads the processed data and explores it through 12 sections covering completeness, distribution shape, metric-by-metric comparison, geographic context, and a final consistency test.

The workflow was not linear — the feedback loops in the diagram fired at three points. First, the **need more data** loop: I initially collected data for only one origin (Barking) and two times to test the pipeline end-to-end, following the W07 lecture's advice to "start very, very small" before scaling. After confirming the NB02 transformation worked, I went back to NB01 and expanded to ten origins. Second, the **structure change** loop: during NB03's investigation phase, I needed distance and deprivation context to rule out competing explanations for the fare pattern — this sent me back to NB02 to add the ONS merge, which was not in the original pipeline. Third, the **pivot** between Explore and Investigate: I entered NB03 expecting duration to dominate, but Sections 4–6 showed it was flat; the analysis redirected to fare, which the sign test in Section 9 confirmed as the only consistent signal. In total, NB03 produced 11 exploratory charts; only two were selected for this report — a visible compression from exploration to communication that the sample report and SAMPLE WORK SKILL 2 document both demonstrate.

All analytical operations use vectorised pandas/NumPy: `groupby().agg()`, `groupby().median()`, boolean counting via `(shift_table > 0).sum()`, and `np.where()` for corridor classification. The only non-vectorised calls are `df.apply(..., axis=1)` for matplotlib annotation text, which is a presentation task with no vectorised API, affecting only 10 data points.

## 📊 Insights & Visualisations

### Insight 1: Peak hours added cost, not time

![](./figures/insight_1_cost_not_time.png)

Across all ten Outer London origins, median journey durations were nearly identical between peak (07:30 and 08:00) and off-peak (11:30 and 12:00), with differences ranging from −1.0 minutes (Harrow, Woolwich) to +3.0 minutes (Sutton). Even the largest shift represents less than 5% of total journey time. Fares told a completely different story. Nine of ten origins showed higher median peak fares, with premiums ranging from £0.60 (Walthamstow) to £2.65 (Bromley). The sole exception was Harrow (−£0.25), where the TfL planner found a cheaper route combination at peak.

The upper panel makes this contrast visible using the three-layer strip+box+median-point pattern from the W08 Lecture: arrows between off-peak and peak medians point consistently upward for fare, while the percentage view in the lower panel shows fare premiums of 18–47% dwarfing duration shifts of ±5%. I avoided a dual y-axis design here, following the W09 Seven Sins checklist (Slide 24), and instead split the comparison into two panels. I chose median over mean because fare distributions show slight right skew and n=6 per group makes the mean sensitive to a single outlier — following the W09 Lecture principle (Slide 17) of computing both and reporting the gap. Both the `plot_df` filtering pattern (W05 Lecture, reinforced W09 Lecture) and seaborn were used for all visualisations.

**Limitations:** Each origin-period group has approximately six journey observations (3 API options × 2 departure times), so medians are coarse at this sample size. The data cover a single date (Wednesday), so day-to-day fare variation is not captured. TfL fares are deterministic — set by zone crossings and time of day — so within-group fare variation reflects which route the planner selected, not random pricing. These are planned journeys, not observed travel times; actual peak delays may differ from the API's estimates.

### Insight 2: Fare is the only metric that consistently worsens at peak

![](./figures/insight_2_fare_only_consistent.png)

The first insight established that fare increases at peak. This second insight asks: is that pattern unique, or do other journey dimensions also worsen? I tested four metrics across all ten origins using a consistency count — how many origins show a positive peak-minus-off-peak median shift? — computed using the vectorised boolean-to-integer trick `(shift_table > 0).sum()` from the W04 Lecture on Vectorisation.

Fare was positive at 9 of 10 origins. Duration was positive at only 5 (essentially a coin flip), transfers at 1, and walking at 2. To assess whether these counts could arise by chance, I used a binomial sign test. Under the null hypothesis that peak is equally likely to be higher or lower (p = 0.5), the probability of 9 or more positive outcomes out of 10 is approximately 0.021 (two-sided) — below the 0.05 threshold. No other metric came close to rejecting the null (all p = 1.000). I chose the sign test over a paired t-test (which requires normality, uncheckable at n=10) and the Wilcoxon signed-rank test (which requires symmetric differences, unlikely for skewed fare premiums). The sign test's sole assumption — independent binary outcomes — is defensible at this sample size.

This is not a null result. It is a positive finding about the *structure* of the peak penalty: it operates through pricing, not through service degradation. Commuters travelling from Outer London to LSE at 08:00 instead of 12:00 paid more but did not travel slower, transfer more often, or walk further. The W07 Methodology Scoping exercise said to let the data shape the claims — I entered expecting duration to dominate, but the consistency test redirected the narrative to fare. As the W09 lecture reinforced: "the insight can be something like nothing changes… that is a valid insight on its own."

**Limitations:** The sign test discards magnitude — it tests only directional consistency. Ten origins is a small cross-section, and the pattern may not generalise to all Outer London locations. The 50% threshold (dashed line in the figure) is the null expectation, not an analytical benchmark. All findings are observational — "fare was higher at peak" is a factual observation, not a causal claim about pricing policy.

## 📝 Reflection

<div style="font-family: system-ui; padding: 20px 30px 20px 20px; background-color: #FFFFFF; border-left: 8px solid #ED9255; border-radius: 8px; box-shadow: 0 4px 12px rgba(0, 0, 0, 0.1);max-width:600px;color:#212121;">

<span style="color: #ED9255;font-size:1.15em;font-weight:bold;">NOTE FROM JON:</span> What follows is James's own reflection on the project. He traces specific decisions back to lecture exercises, names where the data surprised him, and says clearly what he would do differently.

</div>

This project taught me to treat methodology as analytical work, not administrative overhead. The W07 Methodology Scoping exercise was where I first drafted my scope — origin, destinations, time windows, metrics — and its criteria of specific definitions, feasible scope, and evidence-backed decisions directly shaped every choice in the Decisions section. I originally planned a single departure time per period. The scoping exercise made me realise that a single timestamp makes the comparison fragile, so I expanded to two times per window. This small change tripled the per-group sample size from ~3 to ~6 observations, which made the median comparisons in NB03 meaningfully more stable.

I entered NB03 expecting duration to be the main finding — after all, the research question is about how travel "changes" and the most intuitive change is time. The data showed otherwise: duration differences were negligible (−1 to +3 minutes) while fare premiums were substantial (up to £2.65). Rather than forcing a duration story, I restructured the analysis around fare. This pivot was the most important analytical decision in the project, and it came directly from the SAMPLE WORK SKILL 2 principle of letting the data change the claims. The W09 lecture reinforced this when it noted that finding nothing changes is itself a valid insight.

Working with the TfL API's deeply nested JSON was the most technically demanding part of the project. The W07 Lecture introduced `pd.json_normalize()` with `record_path` and `meta`, and the W07 Lab let me practise it on the TfL endpoint specifically. In NB02, I used these parameters to flatten the journey-level data in one call, then extracted leg-level metrics (walking minutes, mode sequence) using `explode()` and vectorised `groupby()` aggregation from the W08 Lecture. The ONS merge followed the left-join pattern from the W08 Lab. That merge enabled three analyses that would have been impossible without it: testing distance versus fare premium, testing IMD versus fare premium, and classifying transport corridors. All three produced null or weak results — but those nulls are analytically valuable because they rule out competing explanations for the fare pattern.

With more time, I would add a third departure time per period (08:30 for peak, 11:00 for off-peak) to reduce snapshot dependence further, collect across at least two weekdays to capture day-to-day variation, and add origins specifically chosen to hold distance constant while varying corridor type — this would give a cleaner test of whether transport mode drives the fare premium independently of geography. I would also explore the evening peak (17:00–18:30) to test whether the fare premium pattern is symmetric.

One practical lesson: when I attempted to re-run NB01's collection cell on 24 March to verify reproducibility, the API returned a 400 error for the 18 March date. Through trial and error with progressively more recent dates, I discovered that the TfL Journey Planner only serves journey plans within approximately seven days of the request — a constraint not documented in the API portal. This means the raw JSON file is the sole reproducibility artifact; the API call itself cannot be replicated after the lookback window expires.