---
title: "LSE ME204 Week 02 Day 02 - Lecture Material"
author: "Dr. Jon Cardoso-Silva"
date: "2023-07-18"
date-format: "DD MMMM YYYY" # See https://quarto.org/docs/reference/dates.html#using-a-date-format
format: 
    html:
        self-contained: true
        link-external-icon: true
        toc: true
---

# ⚙️ Setup

```{r setup, message=FALSE, warning=FALSE}
library(tidyverse)
library(jsonify)
library(jsonlite)
library(rvest)
library(httr2)
```

Note: if the above throws an error, it's likely because you don't have some of the packages installed. Run `install.packages()` to install them.

# Part 1. 🗃️ Handling JSON files

Head back to the [Camden Crime data](https://opendata.camden.gov.uk/Crime-and-Criminal-Justice/On-Street-Crime-In-Camden-Map/893b-tp33) we used last week. This time, export the dataset as a JSON file. 

You should end up with a `On_Street_Crime_In_Camden_Map.json` file somewhere in your computer. If your computer can handle it, try to open it in a text editor just to observe its structure.

**🧑‍🏫 TEACHING MOMENT**

1. Let's read the JSON into R using the function `fromJSON` from the jsonlite package:

```{r}
# It will take a little while. It is a huge file!
camden_crime <- fromJSON("On_Street_Crime_In_Camden_Map.json")
```

2. Now let's **inspect** the JSON object. I don't want to use `str()` like we did with other data structures because it will be too long. Instead, let's use `names()` to see the names of the elements in the JSON object:

```{r}
names(camden_crime)
``` 

which should return:

```text
[1] "meta" "data"
```

The XML file also had these two elements. One for metadata and one for the actual data. Let's inspect the metadata first:

```{r}
names(camden_crime$meta)
```

We see that it only has one element, called `view`. Let's go one level deeper:

```{r}
names(camden_crime$meta$view)
```

Now this is likely to be useful! You should see the following output:

```text
 [1] "id"                       "name"                     "assetType"
 [4] "attribution"              "attributionLink"          "averageRating"
 [7] "category"                 "createdAt"                "description"
[10] "displayType"              "downloadCount"            "hideFromCatalog"
[13] "hideFromDataJson"         "indexUpdatedAt"           "licenseId"
[16] "newBackend"               "numberOfComments"         "oid"
[19] "previewImageId"           "provenance"               "publicationAppendEnabled"
[22] "publicationDate"          "publicationGroup"         "publicationStage"
[25] "rowsUpdatedAt"            "rowsUpdatedBy"            "tableId"
[28] "totalTimesRated"          "viewCount"                "viewLastModified"
[31] "viewType"                 "approvals"                "clientContext"
[34] "columns"                  "displayFormat"            "grants"
[37] "license"                  "metadata"                 "owner"
[40] "query"                    "rights"                   "tableAuthor"
[43] "tags"                     "flags"
```

This is a lot of information! 

One particular element that I find useful is `columns`. Let's inspect it:

```{r}
camden_crime$meta$view$columns
```

A data frame! 

```{r}
# Let's use the glimpse() function to inspect it
# glimpse() is another alternative to str() that we saw on the first day of class
camden_crime$meta$view$columns %>% as_tibble() %>% glimpse()
```

**👥 IN GROUPS OF 3-4:**

3. Using your best investigative powers, try to guess -- yep, guess, we don't have a data dictionary -- what each column means. Write down your guesses in the space below. I will give you 5 minutes to do this and then I'll share the solution.

```markdown
# My favourite trick here is

camden_crime$meta$view$columns %>% as_tibble() %>% count(dataTypeName)
```

4. Let's move over to inspect the `data` element. Running `names(camden_crime$data)` returns NULL. This means that the `data` element is not a **named list**. When I reach NULL, I usually try to use `class()` to see what type of object it is:

```{r}
class(camden_crime$data)
```

which returns:

```text
[1] "list"
```

How big is this list? Let's use `length()` to find out:

```{r}
length(camden_crime$data)
```

More than 370k elements! Let's take a look at just first element:

5. Let's inspect the first element of the list. We can use `camden_crime$data[[1]]` to access the first element of the list. This is also a list, just not a named list. We see that it has 30 elements...

```{r}
length(camden_crime$data[[1]])
```


**👥 IN GROUPS OF 3-4:**

You already know this dataset. So, using what you already know about it and what we've learned from inspecting its JSON version, write code to create a data frame with the following columns from the JSON file:

- `id`
- `street_id`
- `street_name`
- `location_subtype`
- `category`
- `service`
- `outcome_category`
- `outcome_date`
- `ward_code`


```r
# Your code goes here

```


# Part 2. 📩 Using an API

## 2.1. What is an API?

API stands for Application Programming Interface. It is a set of rules that allows one piece of software to talk to another.

In the context of data science, APIs are a way to access data from a remote server. It's different from web scraping in that the data is not embedded in a web page. Instead, it is stored in a database and the developers who manage the database have created an API to allow users to access the data.

Developers can request that you authenticate yourself before you can access the data. This is usually done by providing an API key. This is a unique identifier that is associated with your account. It is used to track your usage of the API and to limit the amount of data you can access. This is typically imposed to prevent you from overloading the server with requests.

Let's take a look at some [Public Open APIs](https://github.com/public-apis/public-apis).

A good example:
- 🐲 [Game of Thrones API](https://anapioficeandfire.com/)

## 2.2. Using an API in R

`rvest` won't work as it is too high level. Instead, we will use the [httr2](https://httr2.r-lib.org/reference/index.html) package. Let's get some data from the [Game of Thrones API](https://anapioficeandfire.com/):

```{r}
# The API endpoint is the URL where the API is located
url <- "https://anapioficeandfire.com/api/books/1"

# Let's make the request
library(httr2)

# We start by building a request object
# then we perform the request
request(url) %>% req_perform()
```

We get a **response** with a status code of 200. This means that the request was successful. Click [here](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) for a list of HTTP status codes.

Let's inspect the response:

```{r}
response <- request(url) %>% req_perform() 

# Check the httr2 reference page for resp_* functions
response %>% resp_content_type()
```

It's a JSON file! Let's parse it:

```{r}
book1_json <- response %>% resp_body_json()
```

**👥 IN GROUPS OF 3-4:**

1. Spend some time inspecting the JSON you just parsed. Use the same techniques we used in the first part of this lecture. 

2. Now, just as a warm up, write some code to create a very simple data frame `df_book1` with the following columns:

- `bookName`
- `numberOfPages`
- `released`

3. Now here's the tricky part. Create a `df_characters` data frame. This data frame should have one row for each character in the book. It should have the following columns:

- `bookName`
- `characterName`
- `characterGender`
- `characterCulture`
- `characterBorn`
- `characterDied`

```{r}
# Your code goes here

```

# Extra

If time allows, we will look at [Wikimedia APIs](https://api.wikimedia.org/wiki/Getting_started_with_Wikimedia_APIs) -- the API you will be using later in the lab. This is more of a real-world scenario, and the API requires using the httr2 functions that handle authentication. 