---
title: "LSE ME204 Week 01 Day 04 - Lecture Material"
author: "Dr. Jon Cardoso-Silva"
date: "2023-07-13"
date-format: "DD MMMM YYYY" # See https://quarto.org/docs/reference/dates.html#using-a-date-format
format: 
    html:
        link-external-icon: true
        toc: true
---

# ⚙️ Setup

```{r setup, message=FALSE, warning=FALSE}
library(tidyverse)
library(rvest)
```

Note: if the above throws an error, it's likely because you don't have the `rvest` package installed. You can install it by running the following code:

``` r
install.packages("rvest")
```

# Step 1. 🕵🏻 Inspect the website

We will be scraping [Books to Scrape](http://books.toscrape.com/), a website that was created for the sole purpose of practising web scraping. All of the prices, products and ratings are fictitious.

![](figures/books_to_scrape.png)

To start, open the website in your browser and inspect the HTML code. You can do this by right-clicking on the page and selecting "Inspect" (or a similar option depending on your browser).

Now, let's suppose our goal is to collect data from the homepage and create a <mark>[tidy dataframe](https://lse-dsi.github.io/ME204/2023/weeks/week01/day02/slides.html?q=tidy%20data#/section-1)</mark> with the following columns:

-   `title`: the full title of the book
-   `price`: the price of the book
-   `rating`: the rating of the book

**👥 IN GROUPS OF 3-4:**

Using what you learned in the previous lecture, work on the following task:

-   In the space below, write down the HTML tag(s) and/or HTML attributes you would use to select the necessary data. Be as specific as possible with your answers.


(SOLUTION)

``` markdown
# TITLE

`html -> body -> div -> div -> div(.row) -> div -> section -> div -> ol -> li -> article -> h3 -> a(title)`

# PRICE

`body -> div -> div -> div(.row) -> div -> section -> div -> ol -> li -> article -> div -> div -> p(class="price_color")`

# RATING

`#default > div > div > div > div > section > div:nth-child(2) > ol > li:nth-child(1) > article > h3 > a`

```

Note that this is just an example. We could have chosen to go more directly to the tags, classes and attributes that we need. 

# Step 2. ⛏️ Scrape the first book

Now let me show you how to use `rvest` to navigate step by step through the following HTML elements that you found above:

## 2.1. Read the HTML code

First, we need to use the `read_html()` function to read the HTML code of the website:

```{r}
url <- "http://books.toscrape.com/"
html <- rvest::read_html(url)
html
```

Does the output look familiar?

```{r}
html_elements(html, "body")

html_elements(html, "head")
```

## 2.2. Navigate through the HTML elements

Now, we can use the `html_elements()` function to navigate through the HTML elements. This function takes two arguments:

-   `x`: the HTML code
-   `css`: the CSS selector or tag name

**Let's start with `body`**

```{r}
html_elements(x = html, css = "body")
```

or, simply:

```{r}
html_elements(html, "body")
```

or even a third alternative, using the pipe operator:

```{r}
html %>% html_elements("body")
```

As usual, the first thing to do when you don't know what a function does is to check the documentation. You can do this by running `?html_elements` in the console, or even by browsing the [rvest documentation](https://rvest.tidyverse.org/reference/html_elements.html) online.

**What's *immediately* inside `body`?** (Emphasis on *immediately*)

```{r}

html %>% html_elements("body") %>% html_children()
```

**How to go one level deeper?**

Just add a space to the CSS selector:

```{r}
html %>% html_elements("body div")
```

**But what if I want just the first `<div>` element?**

You can use another, similarly named function: `html_element()`. This function is similar to `html_elements()`, but it only returns the first element that matches the CSS selector.

```{r}
html %>% html_element("body div")
```

**How to access elements with a specific class?**

```{r}

html %>% html_elements(".row")
```

🎯 **ACTION POINT**

Now, apply what you've learned above and navigate through the HTML elements until you find the **title** of *the first* book. Work in small groups of 3-4 people to collaborate and share your findings. I will give you 15 minutes to do this and then I'll share the solution.

```{r}
# Your code goes here
```

(SOLUTION)

We don't have to specify the entire set of tags to get to the title. We can work on specifying the most specific tag first. For example, to get the first title, we can do:

```{r}
html %>% html_elements("h3 a") %>% html_attr("title")
```

That doesn't look very tidy, though. Let's learn a new function (`html_attr()`) that allows us to extract the value of an HTML attribute. In our case, we want to extract the `title` attribute of the `<a>` tag:

```{r}
book_title <- html %>% html_element("h3 a") %>% html_attr("title")
book_title
```

Now there is our title! 🎉

🎯 **ACTION POINT**

Can you find out equally simple ways to get the **price** and **rating** of the first book? Work in small groups of 3-4 people to collaborate and share your findings. I will give you 10 minutes to do this and then I'll share the solution.

(SOLUTION)

```{r}
book_price <- 
    html %>% 
    html_element("p.price_color") %>% 
    html_text()
book_price
```

```{r}
book_rating <- 
    html %>% 
    html_element("p.star-rating") %>% 
    html_attr("class") %>% 
    word(2)
book_rating
```


# Step 3. ⛏️ Scrape all books on the homepage

Now that we know how to scrape the first book, it is easier to adapt to scrape all books on the homepage. We just need to use the `html_elements()` function instead of `html_element()`.

```{r}
book_titles <- 
  html %>% 
  html_elements("h3 a") %>% 
  html_attr("title")
book_titles
```

```{r}
book_prices <- html %>% html_elements("p.price_color") %>% html_text()
book_prices
```

```{r}
book_ratings <- 
    html %>% 
    html_elements("p.star-rating") %>% 
    html_attr("class") %>%
    word(2)
book_ratings
```


🎯 **ACTION POINT**

How would you create a dataframe with the title, price and rating of all books on the homepage? Try to work out a solution in 5 minutes and then I'll share the solution.

(SOLUTION)

```{r}
library(tibble)

df <- data.frame(
    title = book_titles,
    price = book_prices,
    rating = book_ratings
)
df
```

# Step 4. 📦 Create a function

Now, let's create a function that takes a URL as input and returns a dataframe with the title, price and rating of all books on that page.

```{r}
library(tibble)
library(purrr)
library(dplyr)
library(rvest)

scrape_books <- function(url) {
    
    # Read the HTML code
    html <- read_html(url)
    
    # Scrape the title
    book_titles <- 
      html %>% 
      html_elements("h3 a") %>% 
      html_attr("title")
    
    # Scrape the price
    book_prices <- 
      html %>% 
      html_elements("p.price_color") %>% 
      html_text()
    
    # Scrape the rating
    book_ratings <- 
        html %>% 
        html_elements("p.star-rating") %>% 
        html_attr("class") %>% 
        word(2)
    
    # Create a dataframe
    df <- tibble(
        title = book_titles,
        price = book_prices,
        rating = book_ratings
    )
    
    # Return the dataframe
    return(df)
}
```

Let's test our function:

```{r}
scrape_books("http://books.toscrape.com/")
```

Test that it works for other subpages that have the same structure:

```{r}
url <- "http://books.toscrape.com/catalogue/category/books/historical-fiction_4/index.html"
df_historical_fiction <- scrape_books(url)
```

# Step 5: 👥 Group Exercise

Now, let's put your hyperlink navigation skills to the test!

When you click on a book cover, it will take you to a webpage with additional information about the book. For example, click [here](http://books.toscrape.com/catalogue/the-red-tent_273/index.html) to view the page for the book "The Red Tent".

🎯 **ACTION POINT** (45 min)

In your groups, modify the `scrape_books()` function to extract additional information and include it in the final data frame. The new columns to include are:

-   `description`: the description of the book
-   `category`: the category of the book
-   All the columns under "Product Information" (e.g., `UPC`, `Product Type`, `Price (excl. tax)`, etc.)
-   The URL of the book cover image

(SOLUTION USING A FOR LOOP AND SPECIFIC `table.*` CSS SELECTORS)

```r
library(tibble)
library(purrr)
library(dplyr)
library(rvest)

scrape_books <- function(url) {
    
    # Read the HTML code
    html <- read_html(url)
    
    # Scrape the title
    book_pages <- paste0(url, html %>% html_elements("h3 a") %>% html_attr("href"))
    book_titles <- html %>% html_elements("h3 a") %>% html_attr("title")

    # Scrape the price
    book_prices <- html %>% html_elements("p.price_color") %>% html_text()
    
    # Scrape the rating
    book_ratings <- 
        html %>% 
        html_elements("p.star-rating") %>% 
        html_attr("class") %>% 
        word(2)
    
    # Create a dataframe
    df <- tibble(
        title = book_titles,
        price = book_prices,
        rating = book_ratings,
        page = book_pages
    )

    # Scrape the description using a for loop
    book_descriptions <- vector("character", length(book_pages))
    book_categories <- vector("character", length(book_pages))
    book_upc <- vector("character", length(book_pages))
    product_type <- vector("character", length(book_pages)) 

    for (i in 1:length(book_pages)) {
        book_page_html <- read_html(book_pages[i])
        book_descriptions[i] <- book_page_html %>% html_element("p:not([class])") %>% html_text()
        book_categories[i] <- book_page_html %>% html_element("ul.breadcrumb li:nth-child(3) a") %>% html_text()
        book_upc[i] <- book_page_html %>% html_element("table.table-striped tr:nth-child(1) td") %>% html_text()
        product_type[i] <- book_page_html %>% html_element("table.table-striped tr:nth-child(2) td") %>% html_text()
    }

    df$description <- book_descriptions
    df$category <- book_categories
    df$upc <- book_upc
    df$product_type <- product_type

    # Return the dataframe
    return(df)
}
```

# Step 6: 🧹 Tidy up the data

If time allows, take the opportunity to practice modifying the resulting data frame using the principles you've learned in earlier lectures. Make sure that the data types are appropriate and optimised for storage efficiency. For the sake of this exercise, we will assume that all columns are important and should be retained in the final data frame.

**No solution is provided here, but it should be straightforward to apply what you've learned in 🗓️ [Week 01 -- Day 02](/2023/weeks/week01/day02/page.qmd).**