---
title: "DS202A Week 02 - Source Code Part II"
author: "[Dr. Jon Cardoso-Silva](https://jonjoncardoso.github.io)"
date: "2023-10-10"
format: html
self-contained: true
---

This `.qmd` is an expanded version of what I taught in the Week 02 lecture. It contains some of the code I used in the lecture, plus some extra explanations and examples.

# Context

In Week 02's labs, you tackled a problem set centred around `tidyverse `in R, where the goal was to transform a given dataset onto a desired output. It made me happy to see how much you all engaged with those exercises, prompting you to realise you had some knowledge gaps about R programming in general that needed to be addressed. This is a great way of learning, and I am truly glad you took the time to ask us questions about what was puzzling you in that lab.

This is Part II of the source code for the Week 02 lecture. You can read Part I [here](https://moodle.lse.ac.uk/mod/page/view.php?id=1210403). The point of this document is to talk more about `tidyverse`.

## FAQ

**Q:** What really is the `tidyverse`?

> `tidyverse` is not one but a set of packages that, together, help you tackle the most common data manipulation tasks. You can see the name of the packages involved on [their website](https://www.tidyverse.org/packages/).

**Q:** Where do I get a list of all the functions available in `tidyverse`?

> You have to go to the documentation of the specific package you are interested in. For example, if you want to see all the functions available in `dplyr`, you can go to [its documentation](https://dplyr.tidyverse.org/reference/index.html). Click on the name of a function to see an explanation as well as examples of how to use it.

💡 You might also want to download some **cheatsheets**. Here are the links to those I shared in the lecture:

- [`dplyr` cheatsheet](https://github.com/rstudio/cheatsheets/blob/main/data-transformation.pdf) (general data manipulation)
- [`ggplot2` cheatsheet](https://github.com/rstudio/cheatsheets/blob/main/data-visualization-2.1.pdf) (data visualisation)
- [`tidyr` cheatsheet](https://github.com/rstudio/cheatsheets/blob/main/tidyr.pdf) (tidying up data)
- [`lubridate` cheatsheet](https://github.com/rstudio/cheatsheets/blob/main/lubridate.pdf) (dealing with dates)

**Q:** How do I install `tidyverse`?

> Just run `install.packages("tidyverse")` in your R console. This will install all the packages that are part of the `tidyverse` ecosystem. 
>
> DO NOT leave an `install.packages()` command in your .qmd file. Always do this in the R Console. Otherwise, you won't be able to render your markdown file as an HTML later.

# How to write code from scratch

I am using the 💻 [Week 02 Lab](./lab.qmd) as an example.

## 1. Dedicate the first chunk to loading packages

In your `.qmd` file, make it a habit to create and dedicate the first chunk of code to loading all the packages you'll use using the `library()` function. True, you may not know all the packages you will need when starting a new file. But making the first chunk a reserved space for this task creates a neat, centralised hub. In my case, I often begin with just `library(dplyr)`. I keep writing in my notebook, adding markdown text and code as I go along, but as soon as I feel the need for additional R packages like `readr`, I simply circle back to that first chunk to add them. You have to rerun the chunk for the packages to be loaded.

Following this iterative process, this is the list of packages I ended up using in the Week 02 lab:

```{r setup, include=TRUE, message=FALSE, warning=FALSE}
library(dplyr)       # for data manipulation
library(tidyr)       # also for data manipulation
library(readr)       # for reading data
library(lubridate)   # for working with dates
library(tidyselect)  # for selecting columns
```


```{r include=FALSE}
# This assumes you are running this 
here::i_am("2023/autumn-term/src/solutions/LSE_DS202A_2023_W02_lab_solutions.qmd")
filepath <- here::here("data/UK-HPI-full-file-2023-06.csv")
uk_hpi <- read_csv(filepath)
```

### Why bother?

Best practices exist for a reason. The main benefit of keeping a tidy list of packages at the start of your file is so that, when rendering your `.qmd` file to a nice and beautiful HTML webpage, the program will run it from scratch, from top to bottom. If you reference a function in the middle of your file but you haven't loaded the package that contains that function, the program will throw an error.

**Q:** Do I need to include every single package in there?

> Not really. As mentioned in the lecture, if you use a particular package only once or twice, you don't need all of its functions. Instead, you can just call the function directly. For example, suppose you want to use lubridate's `dmy()` function but you won't need any other date-related functions. You can just call `lubridate::dmy()` directly without having to load the whole package (`library(lubridate)` is not necessary).

## 2. Read your data early on

This one was easy in W02 labs because we gave you the code:

```r
filepath <- "data/UK-HPI-full-file-2023-06.csv"
uk_hpi <- read_csv(filepath)
```

But what if we hadn't? You would have to be able to locate where your data file is located and then read it. Even with the code above, it is possible you would get an error message. A **No such file** error or similar should prompt you to think about **relative paths**. And also, is the data truly inside the `data/` folder? Or, more fundamentally, _is there_ a `data/` folder? Then, go about fixing these problems one by one.

## 3. Do things step-by-step

When you saw the final solution to Part III, you probably understood what was going on but didn't feel confident enough (yet) to write this pipe chain yourself from scratch:

```r
selected_cols    <- c("Date", "RegionName", "12m%Change")
selected_regions <- c("United Kingdom", "England", "Scotland", "Wales", "Northern Ireland") 

df <-
    uk_hpi %>%
    select(all_of(selected_cols)) %>%
    rename(region = RegionName, date = Date, yearly_change = `12m%Change`) %>%
    filter(region %in% selected_regions) %>%
    drop_na(yearly_change) %>% 
    mutate(date = dmy(date)) %>% 
    arrange(desc(date))

df %>% head(8)
```

Some of it is my fault! By showing you just the result, I didn't help you reason about it step-by-step. Here is how to do it:

### 3.1 The `select()` line

The end goal is to arrive at a data frame with the three columns:

|date      |region           | yearly_change|

This should prompt you to glimpse at the original data frame to see if you can find these columns there:

```{r}
glimpse(uk_hpi)
```

OK, I can't find these exact columns, but I can see that there are columns called `Date`, `RegionName` and `12m%Change`. 

Only now would I start thinking about the `select()` function. If I was new to dplyr, I would go to the website, more specifically to the [select()](https://dplyr.tidyverse.org/reference/select.html) documentation. There, I would see that the function takes two arguments: `data` and `...`. The first one is the data frame you want to select columns from. The second one is a set of column names you want to select. This might not be super clear at first, but then I could scroll down to see examples.

Putting it all together, I would arrive at:

```{r}
select(uk_hpi, Date, RegionName, `12m%Change`)
```

Yes! That's what I need, and it should be fine; you could stop here and move on. Let's do that.

### 3.2 The `rename()` line

We are asked to use dplyr's [rename()](https://dplyr.tidyverse.org/reference/rename.html) function. Again, the examples are perhaps the best way to get a sense of how to use it. I could conclude that what I need is something like:

```{r}
rename(uk_hpi, region = RegionName, date = Date, yearly_change = `12m%Change`)
```

This is OK. But wait! What's with all these additional columns I don't need?

This would help me realise that I need to connect this with the previous step. By running rename on the original `uk_hpi` data frame, I have not made use at all of the previous `select()` line. 

Putting the two things together, I could arrive at the following:

```{r}
rename(select(uk_hpi, Date, RegionName, `12m%Change`), region = RegionName, date = Date, yearly_change = `12m%Change`)
```

While the above works, it is not very pleasant to read. Sure, alternatively, the following would also work:

```{r}
temporary_df <- select(uk_hpi, Date, RegionName, `12m%Change`)
rename(temporary_df, region = RegionName, date = Date, yearly_change = `12m%Change`)
```

But why create a temporary data frame if I don't need to?

Here is where I start to see the need for a pipe operator, `%>%`. What I want is to take the output of the `select()` line and feed it to the `rename()` line. This is precisely what the pipe is for. I can rewrite the above as:

```{r}
# It is good to have one pipe per line
uk_hpi %>%
    select(Date, RegionName, `12m%Change`) %>%
    rename(region = RegionName, date = Date, yearly_change = `12m%Change`)
```

Note that I am not saving the output of this whole pipe chain anywhere. That is because I am still in **prototype mode**. I am still building my code step-by-step, and I don't want to save anything until I am sure it works.

Let's go to the next step:

### 3.3 The `filter()` line

Remember: the [filter()](https://dplyr.tidyverse.org/reference/filter.html) function restricts the rows of a data frame to a subset of observations that satisfy a particular condition. By browsing the documentation and looking at examples, I realised that I had to use a bunch of conditions together. 

> I need to keep all rows for which the `region` contains the word "United Kingdom" **OR** "England" **OR** "Scotland" **OR** "Wales" **OR** "Northern Ireland". 

Some programming languages use the `or` word, but R uses the `|` symbol for this.

💡 Read more about logical operators in R [here](https://rdrr.io/r/base/Logic.html). 

💡 Read more about relational operators in R [here](https://rdrr.io/r/base/Comparison.html)

After battling with the syntax, I arrived at:

```{r}
uk_hpi %>%
    select(Date, RegionName, `12m%Change`) %>%
    rename(region = RegionName, date = Date, yearly_change = `12m%Change`) %>%
    filter(region == "United Kingdom" | region == "England" | region == "Scotland" | region == "Wales" | region == "Northern Ireland")
```

This one works! If I want to be extra concise, maybe I could use the `%in%` operator (you would probably only know about it with practice):

```{r}
uk_hpi %>%
    select(Date, RegionName, `12m%Change`) %>%
    rename(region = RegionName, date = Date, yearly_change = `12m%Change`) %>%
    filter(region %in% c("United Kingdom", "England", "Scotland", "Wales", "Northern Ireland"))
```

Nice!

### 3.4 The `drop_na()` line

You probably noticed the number of NOT AVAILABLEs we got in the previous step. We need to get rid of them. The [tidyr::drop_na()](https://tidyr.tidyverse.org/reference/drop_na.html) function does exactly that.

You would probably get stuck here until you discover that `drop_na()` exists and is not part of `dplyr` but rather `tidyr`. This is why I loaded `tidyr` at the start of the file. How would you arrive at this discovery? This one requires some googling skills! You could google something like "how to remove NA values using tidyverse" and you would probably find the answer. Searching online is perhaps the most useful skill professional programmers have - try practising it too!

Eventually, you would arrive at:

```{r}
uk_hpi %>%
    select(Date, RegionName, `12m%Change`) %>%
    rename(region = RegionName, date = Date, yearly_change = `12m%Change`) %>%
    filter(region %in% c("United Kingdom", "England", "Scotland", "Wales", "Northern Ireland")) %>%
    drop_na(yearly_change)
```


### 3.5 Dealing with dates

The last step is to convert the `date` column to a proper date format. This is where `lubridate` comes in. I would google something like "how to convert a string to a date in tidyverse" and I would probably find the answer.

Without going too much into the details, I would arrive at:

```{r}
uk_hpi %>%
    select(Date, RegionName, `12m%Change`) %>%
    rename(region = RegionName, date = Date, yearly_change = `12m%Change`) %>%
    filter(region %in% c("United Kingdom", "England", "Scotland", "Wales", "Northern Ireland")) %>%
    drop_na(yearly_change) %>% 
    mutate(date = dmy(date))
```

I can use the `dmy` function directly because I had already imported `lubridate` at the start of the file. Otherwise, since I'm only using it once, I could also have done `mutate(date = lubridate::dmy(date))` to the same effect.

The final step would be to sort the data frame by date, in descending order:

```{r}
uk_hpi %>%
    select(Date, RegionName, `12m%Change`) %>%
    rename(region = RegionName, date = Date, yearly_change = `12m%Change`) %>%
    filter(region %in% c("United Kingdom", "England", "Scotland", "Wales", "Northern Ireland")) %>%
    drop_na(yearly_change) %>% 
    mutate(date = dmy(date)) %>% 
    arrange(desc(date))
```

💡 Pro-tip: I could only trust that my sorting was correct because I first used the `dmy()` function to convert the column `date` from a string to a date-type column. If I hadn't converted it, the sorting could have been wrong. This is because the column `date` was a string, and strings are sorted alphabetically. This means that "10/01/2021" would come before "02/01/2021", which is not what we want.

# What's Next?

Did I miss any details? Send me a message so I can amend and release an updated version of this file/webpage!

I hope this helps you in your learning journey. See you in the next lecture!