---
title: "Airbnb Edinburgh Data Preparation"
subtitle: "Preparing Data for Causal Inference & Fairness (Week 11)"
author: "Dr Ghita Berrada"
format:
  html:
    self-contained: true
    toc: true
    toc-depth: 3
    theme: cosmo
execute:
  warning: false
  message: false
---

# 🧭 Overview

In Week 11, we introduce **causal inference** and **algorithmic fairness**.  
To make the examples intuitive, we use real Airbnb data from **Edinburgh**.

This script prepares the dataset we will use in class.

Specifically, it creates:

1. **Treatment** → whether a listing is an **entire home** (vs. private or shared room)  
2. **Outcome** → estimated **booking_rate** (bookings per month)  
3. **Confounders** → price, bedrooms, host experience, neighbourhood, etc.  
4. **Protected attribute for fairness** → **neighbourhood clusters** (a data-driven grouping of Edinburgh areas)  
5. **Text features** → 50-dimensional **LSA embeddings** summarising listing descriptions  

The goal is a dataset that:

- Demonstrates **selection bias**  
- Has enough structure for **causal inference**  
- Contains a meaningful grouping for **fairness analysis**  
- Is small and clean enough for students to analyse

We’ll keep explanations gentle and focus on the ideas, not the code mechanics.


# 🧰 Step 0: Load Packages

We use standard tidyverse tools + quanteda for text.

```{r setup}
library(tidyverse)
library(quanteda)
library(quanteda.textmodels)
library(lubridate)
library(janitor)

set.seed(42)
```


# 📥 Step 1: Load the Raw Data

InsideAirbnb provides:

* `listings.csv` → one row per listing, with details like price, location, host info
* `reviews.csv` → one row per *review*, with dates

```{r}
listings_raw <- read_csv("data/airbnb_edinburgh_listings.csv") %>%
clean_names()

reviews_raw <- read_csv("data/airbnb_edinburgh_reviews.csv") %>%
clean_names()

glimpse(listings_raw)
```

**Why two files?**

* Bookings aren't directly reported.
* But reviews often follow a stay → we can *estimate* bookings using review counts.

This is common in empirical research.


# 🧪 Step 2: Treatment — `entire_home`

Edinburgh listings do **not** include `instant_book`, which is often used in causal inference exercises.

Instead, we define:

## **Treatment variable = “Is this listing an ENTIRE home/apartment?”**

This is realistic and policy-relevant:

* Entire homes tend to book more often
* But hosts who offer entire homes differ systematically from those offering rooms
* → This creates **selection bias**, which is perfect for teaching causal inference

```{r}
airbnb <- listings_raw %>%
mutate(
entire_home = room_type == "Entire home/apt"
)

airbnb %>% count(entire_home)
```

We will later examine:

> Do entire-home listings get *more bookings because they are entire homes,*
> or because the hosts/locations/pricing differ?

::: {.callout-note}

You might wonder: **Why choose `entire_home` as the treatment variable?**

Here’s why it’s ideal for teaching causal inference:

### ✅ 1. Strong, realistic *selection bias*

Hosts don’t choose their room type randomly.
Entire homes differ systematically from private rooms:

* They are usually larger
* Often located in different parts of the city
* Tend to be more expensive
* Attract different types of guests
* Hosts of entire homes may be more professional

This creates **exactly the kind of confounding** that causal inference needs to tackle.

### ✅ 2. Clear causal question

It lets us ask a meaningful, intuitive “what if?”:

> *Would a listing get more bookings if it were an entire home rather than a private room?*

### ✅ 3. Transparent for fairness analysis

Room type availability varies across neighbourhoods.
This lets us explore whether:

* causal effects differ across areas
* a policy recommendation (e.g., “more entire homes get more bookings”)
  has **unequal impacts** on different socioeconomic clusters

### ✅ 4. No missing data, no complex imputation

This makes the dataset cleaner and avoids distracting preprocessing steps.

**In short:**
`entire_home` gives us a *real*, *interpretable*, and *non-random* treatment — perfect for illustrating confounding, causal estimation, and fairness.

:::

# 📈 Step 3: Outcome — Estimated Booking Rate

Airbnb does **not** give us booking counts.

Instead, we use:

## **number_of_reviews_ltm** (“reviews in the last 12 months”)

Industry research estimates:

* Roughly **50%** of guests leave reviews
* So: **bookings ≈ 2 × reviews**

```{r}
airbnb <- airbnb %>%
mutate(
bookings_last_year = number_of_reviews_ltm * 2,
booking_rate = bookings_last_year / 12
)

summary(airbnb$booking_rate)
```

This gives a realistic, interpretable outcome variable.

::: {.callout-note}

Airbnb does **not** provide booking counts directly.
However, reviews provide a **reliable proxy**:

* Not every guest leaves a review
* Industry studies (and Airbnb itself) report that **roughly 40–60%** of guests leave one
* A common assumption in research is therefore:

  > **bookings ≈ 2 × reviews**

This is **not exact**, but it gives a consistent and interpretable approximation of listing demand.

We then convert annual bookings into a **monthly booking rate**, which is:

* easier to understand
* less sensitive to outliers
* directly usable for causal analysis

**Important:**

The **exact number** of bookings is less important than the fact that **everyone is measured consistently**.
Causal inference relies on *comparisons*, not perfect measurement.

:::



# 🧱 Step 4: Confounders

In causal inference we try to **“block” selection bias** by controlling for confounders — variables that affect both treatment and outcome.

Below we construct several key confounders students will use later.


## 4.1 Price

Price is usually right-skewed, so we use both:

* raw price
* log-price (for smoother modelling)

```{r}
airbnb <- airbnb %>%
mutate(
log_price = log(price + 1)
) %>%
filter(between(price, 20, 600))   # remove extreme or erroneous prices
```



## **4.2 Host Characteristics**

In some cities, Airbnb provides rich host-level metadata (e.g., when the host joined, response rate, superhost status).
However, **the Edinburgh dataset does not include these fields**.

It *does* include one important host feature:

**`calculated_host_listings_count`**

This tells us how many properties the same host manages on Airbnb.

Why it matters:

* Hosts with **many listings** tend to be more professional
* Professional hosts are **more likely to rent entire homes** (our treatment)
* They also tend to receive **more bookings**

This makes `host_listings` a **confounder**:
it influences *both* treatment assignment and outcomes.

We clean and rename this variable for clarity.

```{r}
airbnb <- airbnb %>%
  mutate(
    # Replace missing host listing counts with 1 (most hosts have exactly 1 listing)
    host_listings = replace_na(calculated_host_listings_count, 1)
  )
```

:::callout-note

### A note on missing values

In a fully rigorous analysis, we would carefully impute missing values in `calculated_host_listings_count` using methods like mean/median imputation or predictive models.

However, in practice:

- most missing values correspond to hosts with exactly one listing,
- Airbnb’s own methodology treats one-listing hosts as the baseline category, and
- the variable has a highly skewed distribution (most hosts = 1 listing).

So, for simplicity and speed, we assume missing values = 1 listing, which is reasonable for teaching purposes and does not meaningfully affect causal or fairness results.
:::

That’s all we need for host characteristics in the Edinburgh dataset.


> Edinburgh’s Airbnb data does not contain host history (`host_since`) or response metrics.
> However, it *does* give us the number of listings managed by each host.
> This is actually a very useful confounder: hosts with multiple properties act more like businesses, and they systematically differ from casual hosts.
> We include this variable in our causal models to adjust for differences in professionalism.





# 🗺️ Step 5: Neighbourhood Clusters — Fairness Attribute

Airbnb listings contain a neighbourhood name, but Edinburgh has **111 unique neighbourhood labels**, many extremely small.

Instead, we form **5 spatial clusters** using latitude & longitude:

## Why use clusters?

* Neighbourhoods differ in opportunity and demand → realistic fairness differences
* Clustering gives **balanced**, **interpretable**, **data-driven** geographic groups
* Avoids sensitive attributes like race, which should not be inferred from names in the UK context
* Clusters remain stable across model runs
* 
```{r}
set.seed(123)
km <- kmeans(select(airbnb, latitude, longitude), centers = 5)

centroids <- as.data.frame(km$centers)
centroids
```

This yields 5 clusters centered roughly on:

1. **Central Edinburgh**
2. **South (Newington/Marchmont)**
3. **West (Corstorphine/Murrayfield)**
4. **Leith / Broughton area**
5. **East (Portobello etc.)**

## Assign labels (based on centroid inspection)

```{r}
labels <- c("Central", "South", "West", "Leith", "East")

airbnb <- airbnb %>%
mutate(
neighbourhood_cluster = factor(km$cluster, labels = labels)
)

airbnb %>% count(neighbourhood_cluster)
```

This becomes our **fairness group**.

::: {.callout-note}

Real fairness analysis often examines whether **different social groups** receive different outcomes from algorithms or policies.

But in the Edinburgh Airbnb dataset:

* We **do not** observe sensitive attributes such as race, age, gender
* We **do** observe detailed geographic information (latitude/longitude)

To study fairness responsibly and realistically, we use **neighbourhood clusters** derived from geographic coordinates.

### ✔ Why neighbourhood makes sense as a fairness attribute

Neighbourhoods in cities like Edinburgh often differ in:

* socioeconomic status
* property conditions
* access to transport and amenities
* typical guest demand
* tourism density

These differences can affect:

* how many bookings hosts receive
* whether certain hosts benefit less from a policy like “switch to entire home”
* how the platform’s recommendations distribute visibility or revenue

In other words:

> **Neighbourhood captures structural inequalities that can influence algorithmic outcomes, even without individual-level demographics.**

### ✔ Using neighbourhood clusters avoids ethical issues

* We’re not inferring race or gender
* We’re not guessing sensitive traits
* We’re using real, observed geographic variation in a transparent and non-intrusive way

### ✔ Ideal for teaching fairness concepts

Neighbourhood-based fairness lets us examine:

* disparities in average booking rates
* differences in causal effects across clusters
* whether a policy recommendation benefits some areas more than others

—all without requiring sensitive personal data.

**This is a safe, realistic, and pedagogically meaningful fairness attribute.**

:::


# 📝 **Step 6: Review-Based Behavioural Features (No Text Available)**

Edinburgh’s *Inside Airbnb* dataset does **not** include written review text—only **dates** of reviews.
Even without text, we can extract **useful behavioural signals** from review patterns:

## ✔ Why use review-based numerical features?

They proxy for:

* listing **popularity**
* recency and volume of demand
* guest engagement
* host responsiveness/exposure

These variables help reduce **selection bias** by controlling for differences in listing quality and demand.


## 6.1 **Compute reviews in the last year**

```{r}
reviews_last_year <- reviews_raw %>%
  mutate(date = ymd(date)) %>%
  filter(date >= today() - years(1)) %>%
  count(listing_id, name = "reviews_last_year")

airbnb <- airbnb %>%
  left_join(reviews_last_year, by = c("id" = "listing_id")) %>%
  mutate(
    reviews_last_year = replace_na(reviews_last_year, 0)
  )
```

::: {.callout-note}

**Why do we set `reviews_last_year = 0` when a listing has no matching review activity?**

Because in the raw data:

* Listings with **no reviews in the past 12 months** simply never appear in `reviews_raw`
* After joining, these listings receive `NA` for `reviews_last_year`

But an `NA` does **not** mean “unknown” — it means the listing **had zero recent reviews**.

Replacing missing values with zero is therefore the **correct interpretation**:

* It preserves the meaning of “no recent guest activity”
* It avoids accidentally dropping valid listings from the dataset
* It keeps the feature numeric and compatible with causal ML pipelines

In short:

> **NA means 0, not missing data.**

:::


This gives us a **clean numeric feature** reflecting recent booking activity.


# 🧩 **Step 7: Combine All Features (Structured Dataset Only)**

Because Edinburgh listings provide **no description text**, we build a final dataset using only **structured features** suitable for causal and fairness analysis.

These include:

## **Treatment**

* `entire_home` (TRUE/FALSE)

## **Outcome**

* `booking_rate` (bookings per month)

## **Confounders**

* `price`, `log_price`
* `minimum_nights`, `availability_365`
* `number_of_reviews`, `reviews_last_year`
* `host_listings`
* `latitude`, `longitude`
* `neighbourhood_cluster` (fairness group)

```{r}
airbnb_final <- airbnb %>%
  select(
    id,
    
    # Treatment
    entire_home,
    
    # Outcome
    booking_rate,
    
    # Confounders
    price, log_price,
    minimum_nights,
    availability_365,
    number_of_reviews,
    reviews_last_year,
    host_listings,
    
    # Location and fairness
    latitude, longitude,
    neighbourhood_cluster
  )
```

This produces a **clean, compact dataset** with everything needed for Week 11.

# 💾 **Step 8: Save Final Lecture Datasets**

We create two versions:


## 8.1 **Simple dataset**

For introductory examples (selection bias, naive comparisons):

```{r}
airbnb_simple <- airbnb_final %>%
  select(
    entire_home,
    booking_rate,
    price,
    minimum_nights,
    number_of_reviews,
    neighbourhood_cluster
  )

saveRDS(airbnb_simple, "data/airbnb_simple.rds")
```

## 8.2 **Full dataset**

For DoubleML, causal forests, fairness:

```{r}
saveRDS(airbnb_final, "data/airbnb_full.rds")
```


# ✔ **Summary**

After preprocessing, the dataset contains:

## **1. Treatment**

* `entire_home`
  → Whether a listing is an entire home/apartment
  → Great example of a *self-selected* treatment (hosts choose)

## **2. Outcome**

* `booking_rate`
  → Estimated monthly bookings using review counts
  → A realistic, interpretable outcome for causal analysis

## **3. Confounders**

Structured features describing:

* price & minimum nights
* availability & popularity (reviews)
* host scale (number of listings)
* geography (latitude/longitude)
* neighbourhood socioeconomic cluster

These variables help adjust for differences between listings that choose vs. do not choose to be entire homes.

## **4. Fairness Attribute**

* `neighbourhood_cluster`
  → Data-driven socioeconomic grouping of areas in Edinburgh
  → Enables fairness evaluation across urban stratification

## **5. Output Datasets**

* **`airbnb_simple.rds`** → small, interpretable subset
* **`airbnb_full.rds`** → full feature set for advanced methods

**🎉 Our dataset is now perfectly tailored for:**

* causal diagrams
* selection bias demonstrations
* Double ML (high-dimensional residualization)
* heterogeneous treatment effects
* fairness metrics across neighbourhoods





