---
title: "LSE ME204 Week 02 Day 03 - Lecture Material (with solutions)"
author: "Dr. Jon Cardoso-Silva"
date: "2023-07-19"
date-format: "DD MMMM YYYY" # See https://quarto.org/docs/reference/dates.html#using-a-date-format
format: 
    html:
        self-contained: true
        link-external-icon: true
        toc: true
---

# ⚙️ Setup

```{r setup, message=FALSE, warning=FALSE}
library(dplyr)
library(ggplot2)
library(jsonlite)
library(httr2)

library(pbapply)
```

Note: if the above throws an error, it's likely because you don't have some of the packages installed. Run `install.packages()` to install them.

## Replicate yesterday's code

Let's recreate the `df` we obtained yesterday morning.

First, we get the raw JSON data:

```{r}
# This line takes a few minutes to run
camden_crime <- fromJSON("../../data/On_Street_Crime_In_Camden_Map.json")
```

Then, we define constants and create a useful function:

```{r}
columns <- camden_crime$meta$view$columns$fieldName
selected_cols <- c("id", "epoch", 
                   "street_id", "street_name", "ward_name",
                   "location_subtype", "category", "service", 
                   "outcome_category", "outcome_date", 
                   "last_uploaded")

get_row <- function(list_item){

  # Name the items and filter to just the columns we care about
  row <- setNames(list_item, columns) 
  row <- row[selected_cols]
  
  # Treat the NULL cases
  null_elements <- sapply(row, is.null)
  row[null_elements] <- NA
  
  df <- as_tibble(row)
  return(df)
}

```

Now we're ready to create the data frame:

```{r}
# using pbapply to get a progress bar
df <- pblapply(camden_crime$data, get_row ) %>% bind_rows()
```
Let me handle some data types (I'll refrain from convert `character` columns to `factor` for now, for reasons that will become apparent later).

```{r}
#Convert relevant data types to integer and Date.
# Note:
#   the original date columns comes as datetime but the HH::MM::SS part is empty
#   Therefore, I'm parsing and then later re-defining these columns as just date (YYYY-MM-DD)
df <- 
  df %>% 
  mutate(across(c(id, street_id), as.integer), 
         across(c(epoch, outcome_date, last_uploaded), lubridate::ymd_hms),
         across(c(epoch, outcome_date, last_uploaded), lubridate::as_date))
```


::: callout-tip
Since the `camden_crime` data frame is consuming a substantial amount of memory (1.4GB) in your computer/RStudio, it is recommended to delete it from memory using the `rm()` function:
:::

```{r}
rm(camden_crime)
```

# Part 1. 🪆 The essential layers of a graph

Today we will be working with [ggplot2](https://ggplot2.tidyverse.org). Many of you might have already used ggplot to plot bar plots, scatter plots and etc., but maybe you don't know why the code looks the way it does. The code below will accompany my explanation of the [Grammar of Graphics](http://vita.had.co.nz/papers/layered-grammar.pdf)

**⬛️ TEACHING MOMENT**

## Step 1.  `ggplot()` produces an object (remember OOP?), that we can later alter and modify:

```{r}
## This produces a plot, albeit an empty one
g <- ggplot()
plot(g)
```

```{r}
class(g)
```

```{r}
# ggplot adheres to the S3 class specification, not S4
isS4(g)
```

These are all the different attributes of this object:

```{r}
names(g)
```

## Step 2.  Let's start by thinking about `data` and `mapping`. Since we will be plotting the crime data we collected, we can pass `df` to ggplot:

```{r}
# There is no data in the ggplot object we created
g$data
```

```{r}
# Let's recreate g with some data
g <- ggplot(data=df)

# Now this should return the same data we have in df
g$data
```

But just passing data doesn't mean we have anything to plot. Our plot is still empty:

```{r}
plot(g)
```

The first thing we do is map particular columns of our data frame to the **axis** of our plot. Say we want to plot the number of incidents that were registered per **ward_name**. We could **map** the `ward_code` column to the X-axis (horizontal) like this:

```{r}
g <- ggplot(data=df, mapping=aes(x=ward_name))
```

The mapping occurs within a special function called `aes()`, which stands for aesthetics. `aes()` simplifies our life, as we can use the name of the columns directly, we don't need to wrap them within quotes (that is, we don't need to do `x="ward_code"`).

Let's check all DATA \<-\> AXIS mappings:

```{r}
g$mapping
```

Great! This should give us a plot, right?

```{r}
plot(g)
```

**🗣️CLASSROOM-DISCUSSION:** What happened here?

``` markdown
Write down your notes and observations here
```

💡 What would happen if you were to map `ward_code` to the Y-axis (vertical) instead?

## Step 3. We need a `geom`. There are many **layers** a components to a plot; the X and Y axis are essential components but we still need to bring some form of **geometric objects** to the plot: typically, a rectangle, a point, or a line.

Let's start with a simple bar plot (rectangle):

```{r}
# Note that we use the symbol + to add layers to `g`, not %>%
g + geom_bar()
```

Ok, this looks more like a plot! We can't read the labels in the X axis though, so maybe it's best if we map `ward_name` to the Y axis instead:

```{r}
ggplot(data=df, mapping=aes(y=ward_name)) + geom_bar()
```


## Step 4: 🕵 Put Your Detective Skills to Work

The solution to the following task may not come to you naturally. You might need to use external resources like Google, StackOverflow, or an AI assistant to help you figure out how to accomplish it.

**🎯 ACTION POINT** (20 minutes)

This task can be done individually or in pairs/groups.

Your task is to modify the code below so that the bars in the resulting plot are ordered from tallest to smallest. You can choose to modify the creation of the `g` object or add lines before or after the creation of the `g` object to achieve the desired result -- your call. Just don't modify the original `df`!


```{r}
# Modify this code
g <- ggplot(data=df, aes(y=ward_name)) + geom_bar()
g
```

(A VERY MANUAL SOLUTION)

This works but it's not a good practice!!!

```{r}
# A simple, but inefficient approach would be to manually curate a vector with the
# order in which Y labels should appear.
y_labels <- c("Frognal", "Kentish Town North", "Camden Square", "South Hampstead", 
              "Primrose Hill", "Kentish Town South", "Regents Park", "Kings Cross", 
              "Camden Town", "Frognal and Fitzjohns", "Swiss Cottage", "Gospel Oak", 
              "Hampstead Town", "Fortune Green", "Highgate", "Belsize", "Cantelowes", 
              "West Hampstead", "Haverstock", "Kentish Town", "Kilburn", "King's Cross", 
              "Regent's Park", "St Pancras and Somers Town", "Camden Town with Primrose Hill", 
              "Bloomsbury", "Holborn and Covent Garden")

# If we possess such labels, we can use it with scale_x_discrete()
g <- 
  ggplot(data=df, aes(y=ward_name)) + 
  geom_bar() + 
  scale_y_discrete(limits=y_labels)
g
```

(A BETTER SOLUTION)

```{r}
# We could write R code to calculate the order of X labels
y_labels <- df %>% select(ward_name) %>% count(ward_name) %>% arrange(n) %>% pull(ward_name)

# Then just re-construct g the same way
# If we possess such labels, we can use it with scale_x_discrete()
g <- 
  ggplot(data=df, aes(y=ward_name)) + 
  geom_bar() + 
  scale_y_discrete(limits=y_labels)
g
```

(AN OPTIMAL SOLUTION)

Using the forcats package

```{r}
library(forcats)

# Then just re-construct g the same way
# If we possess such labels, we can use it with scale_x_discrete()
g <- 
  ggplot(data=df, aes(y=fct_rev(fct_infreq(ward_name)))) + 
  geom_bar() 
g
```



# Part 2. Other `geom_` functions and some customisation

## Step 1. `geom_col`

The `geom_bar()` function we used earlier is typically used to represent counts of data. While we were able to use it with just the **x** or **y** aesthetics, it actually requires both **x** and **y** aesthetics (as stated in the documentation). The function automatically fills in the missing axis with a count() to generate the plot.

But what if I wanted to have full control over the height of the bars? I would have to map both x and y and use another geom, `geom_col()`. 

To replicate the same plot above, I can't just simply use `df` as it is. I'd have to **summarise** my data first. I like to use a temporary data frame for this purpose and I typically call it `plot_df`:

```{r}
plot_df <- df %>% group_by(ward_name) %>% count() %>% arrange(n)
plot_df
```

The data frame above is a summarisation of the original data, it is much smaller but it has everything I need for my plot:

```{r}
g <- ggplot(data=plot_df, aes(y=fct_inorder(ward_name), x=n)) + geom_col()
g
```

## Step 2. `geom_point`

Instead of rectangles, I might want to use points to represent the same thing.

```{r}
g <- ggplot(data=plot_df, aes(y=fct_inorder(ward_name), x=n)) + geom_point()
g
```

## Step 3. Add other elements and customise your plot

What if we wanted to accentuate the disparity in the number of records per region by changing the **size** of the dots?

```{r}
g <- ggplot(data=plot_df, 
            aes(y=fct_inorder(ward_name), x=n, size=n)) 
g <- g + geom_point()
g
```

You know what? Let's go overboard and also add a gradient of colours to represent `n`:

```{r}
g <- ggplot(data=plot_df, aes(y=fct_inorder(ward_name), x=n, size=n, color=n)) 
g <- g + geom_point()
g
```

I don't think these legends are actually necessary here. If you agree with me, we can get rid of them by modifying the **scale** of the size and colour attributes, as well as customise several other elements:

```{r}
# Definition
g <- ggplot(data=plot_df, aes(y=fct_inorder(ward_name), x=n, size=n, color=n)) 

# Geoms
g <- g + geom_point()

# Customise scales
g <- g + 
  scale_size_area(guide=NULL) + 
  scale_color_gradient(guide=NULL) +
  # This one avoids cropping the Holborn dot
  expand_limits(y=nrow(plot_df) + 1) 


g
```

Let's give our plot a stylish makeover to demonstrate the impressive customisation capabilities of ggplot2:

```{r fig.width=8, fig.retina=1.7}
# Definition
g <- ggplot(data=plot_df, aes(y=fct_inorder(ward_name), x=n, size=n, fill=n)) 

# Geoms
g <- g +
  # See options of shapes here: http://www.sthda.com/english/wiki/ggplot2-point-shapes
  # `alpha` adds transparency and `stroke` adds border to the geom
  geom_point(stroke=1.05, alpha=0.6, shape=23)

# Customise scales
g <- g + 
  scale_size_area(guide=NULL) + 
  scale_fill_gradient(guide=NULL) +
  scale_x_continuous(limits=c(0, 60000), 
                     breaks=seq(0, 60000, by=10000),
                     labels=scales::unit_format(unit = "K", scale = 1e-3)) +
  expand_limits(y=nrow(plot_df) + 1) 

# Titles
g <- g + labs(title="Nearly 30% of the records in Camden come from just two wards",
              subtitle = "'Holborn and Covent Garden' and 'Bloomsbury' lead the crime records",
              x="Count",
              y="Ward Name")

# Themes
g <- g + theme_bw()

g
```


## Step 4. Aren't there better colour schemes?

This blue palette doesn't seem really suitable. Check the documentation for `scale_colour_gradient` and tweak the colour scheme. Try to find a palette that somehow communicates that large numbers are worse than small ones.

```{r}
# Your code goes here
```

(SOLUTION)

Although this might be a matter of choice and taste, I'd recommend a couple of cool R packages that you can play with to make your plots look more professional and ready for academic publication: [ggsci](https://cran.r-project.org/web/packages/ggsci/vignettes/ggsci.html) and [ggthemes](https://jrnold.github.io/ggthemes).

```{r fig.retina=1.7, fig.width=10, warning=FALSE, message=FALSE}
library(ggthemes)
library(ggsci)
g + scale_fill_material("red", guide=NULL) + theme_pander()

```


# Part 3. Handling date and other stuff

## Step 1. A more complex summarisation

Let's expand our `group_by()` to calculate the number of occurrences per ward **per day**:

```{r}
# epoch seems to represente the DAY in which the incident happened/was reported
plot_df <- df %>% group_by(epoch, ward_name) %>% count()
plot_df
```

👉 Notice how the dataset is much larger this time. It is still a small dataset, but now it has individual records for each ward per day since 2015 until early 2023.

```{r}
# What is the latest outcome_date in the data frame?
max(plot_df$epoch)
```
## Step 2. Daily reports for 'Holborn and Covent Garden'

Let's focus on this particular ward:

```{r}
plot_df %>% filter(ward_name == 'Holborn and Covent Garden')
```

How do you create an **area plot** that varies over time according to the number of records in this region?

```{r}
g <-
  ggplot(plot_df %>% filter(ward_name == 'Holborn and Covent Garden'),
            aes(x=epoch, y=n)) +
  geom_area()

# Customise this plot later if you want to practice
g
```

Let's stack the 'Bloomsbury' ward to the plot:

```{r}
g <-
  ggplot(plot_df %>% filter(ward_name == 'Holborn and Covent Garden' | ward_name == 'Bloomsbury'),
            aes(x=epoch, y=n, fill=ward_name)) +
  geom_area()

# Customise this plot later if you want to practice
g
```

## Step 3. Play detective again

**🎯 ACTION POINT**

We can easily replace `geom_area()` with `geom_line()` to see the two lines representing both wards. But something is off.. Why are the lines not coloured properly? Can you figure out what's going on? 

Once you figure out how to fix the line plot, customise the plot:
- Add a title and subtitle
- Give the axis titles that are more meaningful
- Increase the thickness of the line a little and give each ward a single colour

```{r}
# Modify this cell of code to fix the plot

g <-
  ggplot(plot_df %>% filter(ward_name == 'Holborn and Covent Garden' | ward_name == 'Bloomsbury'),
            aes(x=epoch, y=n, fill=ward_name)) +
  geom_line()

# Customise this plot later if you want to practice
g
```

(SOLUTION)

```{r fig.retina=1.5, fig.width=10}
# Recreate plot_df here for replicability reasons
plot_df <- 
  df %>% 
  filter(ward_name == 'Holborn and Covent Garden' |
         ward_name == 'Bloomsbury') %>%
  group_by(epoch, ward_name) %>% 
  count()


# Data and Mapping
g <- ggplot(plot_df, aes(x=epoch, y=n, colour=ward_name, linetype=ward_name))
  
# Geoms
# I collected a brownish red shade from https://www.color-hex.com
g <- g + geom_line(size=0.8)

# Customise scales
g <- g + 
  scale_x_date(date_breaks = "12 months",
                      date_minor_breaks = "6 months",
                      date_labels = "%Y",
                      limits=c(lubridate::ymd("2014-07-01"), lubridate::ymd("2023-07-01"))) +
  scale_colour_manual(name="Ward", 
                      values=c("Holborn and Covent Garden"="#d95f02",
                               "Bloomsbury"="#7570b3")) +
  scale_linetype_manual(name="Ward", 
                        values=c("Holborn and Covent Garden"="dashed",
                                 "Bloomsbury"="dotdash"))

# Titles
g <- g + labs(title="Crime patterns (ups and downs) follow a similar trend in both wards",
              subtitle = "But crimes in 'Bloomsbury' seem to have increased considerably since around mid-2021",
              y="Number of Records per Day",
              x="Date")

# Themes
g <- g + 
  theme_minimal() +
  theme(axis.title = element_text(size=rel(1.3)),
        axis.text = element_text(size=rel(1.2)),
        axis.title.x = element_text(margin=margin(t=15)),
        axis.title.y = element_text(margin=margin(r=15)),
        plot.title = element_text(size=rel(1.7)),
        legend.position = c(0.21, 0.8))


g
```

