## NOTE FROM JON

I selected this as a model solution because James communicates his reasoning with unusual clarity. The figures carry the argument clearly, the titles state findings with proper conditions, and the statistical choices fit the data shape and sample size. He started with one expectation, tested it, found a different pattern, and showed that change in a way a reader can follow. His reflections are specific and honest at each step. He also documents AI use clearly, including what AI helped with and what remained his own judgement and writing.

James shared a full AI chatlog with his submission and documented AI usage clearly in each notebook. He states what AI helped with and what remained his own work, including final insight selection, narrative wording, and interpretation.

What follows is James's own README.

---

# DS105W (2025/2026) - ✍️ Mini Project 2

**Author:** James Weiru Lu

This project is part of the [LSE DS105W Winter Term 2025/2026](https://lse-dsi.github.io/DS105/2025-2026/winter-term/course-info.qmd) course and aims to answer the question: **"How does travelling between Inner London and Outer London change when it is done at peak versus off-peak hours?"** with data from the [TfL Journey Planner API](https://api.tfl.gov.uk) and the [ONS Postcode Directory](https://data.london.gov.uk/dataset/postcode-directory-for-london-exp5p/).

**Approach:** I fixed LSE (WC2A 2AE) as the Inner London destination and selected ten Outer London origins spanning all cardinal directions, three transport corridor types (Tube-dominated, Elizabeth line, National Rail), and a range of deprivation levels (IMD ~5,300 to ~27,900). I collected journey data for two peak times (07:30, 08:00) and two off-peak times (11:30, 12:00) on a single weekday (Wednesday 18 March 2026), yielding 120 journey observations across 40 API calls. I then tested which journey dimensions — duration, fare, transfers, walking — consistently separate peak from off-peak.

**Key finding:** The peak/off-peak change is almost exclusively financial, not temporal. Median journey durations were nearly identical between peak and off-peak (−1 to +3 minutes), but fares were consistently higher at peak for 9 of 10 origins (premiums of £0.60 to £2.65). A binomial sign test confirmed fare as the only metric that consistently worsens at peak (p = 0.021, two-sided). See the full analysis in [REPORT.md](REPORT.md).


## 📂 Repository Structure

```
/
├── notebooks/
│   ├── NB01-Data-Collection.ipynb       # API requests → raw JSON
│   ├── NB02-Data-Transformation.ipynb   # Flatten, engineer, merge ONS → CSV
│   └── NB03-Exploratory-Data-Analysis.ipynb  # 12-section EDA → 2 report insights
├── data/
│   ├── raw/
│   │   └── journey_collection_combined.json
│   └── processed/
│       └── journey_options.csv          # 120 rows × 34 columns
├── figures/
│   ├── report_data_science_workflow_summary.jpg
│   ├── insight_1_cost_not_time.png
│   └── insight_2_fare_only_consistent.png
├── README.md
├── REPORT.md
├── .env                 # Not tracked — you create this yourself
├── .gitignore
└── requirements.txt
```


## 📡 Data Sources

| Source | Type | Usage |
|---|---|---|
| [TfL Journey Planner API](https://api-portal.tfl.gov.uk/api-details#api=Journey) | Live API (requires free key) | Journey duration, fare, legs, mode sequence for each origin × departure time |
| [ONS Postcode Directory (Feb 2022)](https://data.london.gov.uk/dataset/postcode-directory-for-london-exp5p/) | Static CSV (122 MB, gitignored) | Borough codes, LSOA/MSOA, IMD rank, lat/long for geographic enrichment |


## 🔧 How to Run

1. **Register for a TfL API key** at [api-portal.tfl.gov.uk](https://api-portal.tfl.gov.uk/). Subscribe to the *500 Requests per min* product.

2. **Clone this repository:**

    ```bash
    git clone <github-repo-ssh-url>
    ```

3. **Create a `.env` file** inside the `notebooks/` directory:

    ```text
    API_KEY=your_tfl_api_key_here
    ```

    This file is excluded from version control via `.gitignore`. The notebooks use `load_dotenv()` to read it.

4. **Download the ONS Postcode Directory** from the [London Datastore](https://data.london.gov.uk/dataset/postcode-directory-for-london-exp5p/). Save the file `london_postcodes-ons-postcodes-directory-feb22.csv` to `data/raw/`. Do not rename it. Confirm it is correctly gitignored:

    ```bash
    git status   # the CSV should NOT appear in the output
    ```

5. **Install dependencies:**

    ```bash
    pip install -r requirements.txt
    ```

6. **Run the notebooks in order:**

    | Notebook | What it does | Output |
    |---|---|---|
    | **NB01** | Collects 40 API responses (10 origins × 4 departure times) with `time.sleep(2)` between requests | `data/raw/journey_collection_combined.json` |
    | **NB02** | Flattens nested JSON via `pd.json_normalize()`, engineers leg-level features, merges with ONS Postcode Directory | `data/processed/journey_options.csv` (120 × 34) |
    | **NB03** | 12-section EDA: completeness, distributions, metric comparisons, geographic context, sign test, polished report visuals | `figures/insight_1_*.png`, `figures/insight_2_*.png` |

    **⚠️ Note:** The TfL Journey Planner API only serves journey plans for dates within approximately seven days of the request (discovered empirically — not documented in the API portal). NB01's collection cell was executed on 18 March 2026 and will return a 400 error if re-run after that window. The saved cell output and `data/raw/journey_collection_combined.json` are the reproducibility artifacts. NB02 and NB03 load from the JSON file and run without any API dependency.


## 🤖 AI Usage

I used AI at several points during this project, documented in detail within each notebook's personal reflections.

- **Claude (DS105W course bot):** Debugging `pd.json_normalize()` parameter combinations when the TfL response nesting was deeper than expected. Thinking through sign test methodology — specifically whether the binomial sign test was appropriate versus paired t-test or Wilcoxon alternatives at n = 10.
- **General Claude:** Structuring the NB01 → NB02 → NB03 pipeline, proofreading notebook reflections, and generating the report planning strategy.
- **Not used for:** Final insight selection, narrative title wording, or the interpretation paragraphs — those reflect my own analytical judgement shaped by the data exploration in NB03.
