---
title: "🗓️ Week 10 - The Basics of Text Mining"
subtitle: "Theme: Applications"
author: "Dr. Ghita Berrada"
format: 
  html:
    from: markdown+emoji
    page-layout: full
    toc: true
    toc-depth: 2
self-contained: true
---

Welcome to Week 10!

Until now, we'd worked with structured (tabular) data. It's time to see how we can deal with unstructured data. And the example of unstructured we'll see in this course is text.

**:gear: A bit of setup before starting**

As usual, let's install/load the libraries we'll need throughout this notebook before proceeding.

You might need to install the following libraries:

``` r
install.packages("quanteda")
install.packages("quanteda.textplots")
install.packages("quanteda.textstats")
install.packages("tidytext")
install.packages("stm")
install.packages("lubridate")
```

You'll need to load the following libraries:

```{r, echo=FALSE, message=FALSE, warning=FALSE}
library(tidyverse)
library(tidymodels)
library(quanteda)
library(quanteda.textplots)
library(quanteda.textstats)
library(tidytext)
library(stm)
library(skimr)
library(scales)
library(lubridate)
library(stringi)
library(plotly)
```

# Part 1: Representing text as structured data

In today's lecture, we'll take a look at the lyrics of Taylor Swift's songs across albums and compare them to Beyonce's lyrics.

```{r}
beyonce_lyrics <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-09-29/beyonce_lyrics.csv')
beyonce_songs <- readr::read_csv('data/beyonce_songs.csv')
taylor_swift_lyrics <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-09-29/taylor_swift_lyrics.csv')
sales <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-09-29/sales.csv')
charts <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-09-29/charts.csv')
```

Let's have a closer look at the data:

```{r}
head(taylor_swift_lyrics)
```

```{r}
skim(taylor_swift_lyrics)
```

```{r}
head(beyonce_lyrics)
```

The Beyonce lyics dataset lists song lyrics line by line. Let's transform it so that each row contains all the lyrics for a single song (i.e same format as the Taylor Swift dataset).

```{r}
beyonce_merged <- beyonce_lyrics %>%
  group_by(song_id) %>%
  summarise(
    across(c(song_name, artist_id, artist_name), first),
    line = paste(line, collapse = "\n")
    )
```

Let's have a very quick look at the sales and charts data:

```{r}
sales %>%
  filter(country == "UK") %>%
  mutate(title = fct_reorder(title, sales)) %>%
  ggplot(aes(sales, title, fill = artist)) +
  geom_col() +
  scale_x_continuous(labels = label_dollar(prefix = "£", big.mark = ",")) +
  labs(x = "Sales (UK)")
```

```{r}
charts %>%
  filter(chart == "UK")
```

```{r}
sales %>%
  filter(country %in% c("World", "WW")) %>%
  mutate(title = fct_reorder(title, sales)) %>%
  ggplot(aes(sales, title, fill = artist)) +
  geom_col() +
  scale_x_continuous(labels = label_dollar(prefix = "£", big.mark = ",")) +
  labs(x = "Sales Worldwide")
```

Beyonce outperforms Taylor Swift on the British market but worldwide it's another story. We'll mainly use the sales/chart data to provide album date but this data could be used for other types of analysis (e.g what are the song/album characteristics that make them top the charts/sell well?).

Let's add album release dates to our Taylor Swift data frames first:

```{r}
# create album, release date dataframe from charts dataframe
released_dates <- charts %>%
  distinct(album = title, released) %>% # Select unique pairs of album (title) and released date
  mutate(released = str_remove(released, " \\(.*")) %>% # Remove anything in parentheses from the 'released' column, e.g., "October 22 (Deluxe)" -> "October 22"
  mutate(released = mdy(released)) # Convert the 'released' column to a date format (using lubridate's `mdy` function)


dated_swift_lyrics <- taylor_swift_lyrics %>%
  rename(album = Album) %>%  # Rename to match `released_dates` column
  left_join(released_dates, by = "album") %>%
  mutate(featuring_artists = "") %>%
  mutate(track_id = paste0("S", stri_rand_strings(n(), 10, pattern = "[0-9]")))
```

For the Beyonce data, things are a tad more complicated:

```{r}
# we need to ensure to remove any featuring artist from song names (they'll be saved in a separate column)
cleaned_beyonce_lyrics <- beyonce_merged %>%
  mutate(
    # Extract featured artists (if any)
    featuring_artists = str_extract(song_name, "(?<=\\(Ft\\. ).*(?=\\))"),
    
    # Clean song name by removing "(Ft. ...)" part for matching
    clean_song_name = song_name %>%
      str_remove(" \\(Ft\\. .*\\)") %>%  # Remove the featured artists part
      str_remove(" \\(feat\\. .*\\)") %>% # Remove feat. variant if exists
      str_to_lower() # Convert to lowercase for matching
  )

# we do the same for the dataset that contains album information
cleaned_beyonce_tracks <- beyonce_songs %>%
  mutate(
    # Clean track name by removing "(Ft. ...)" or "(feat. ...)" part
    clean_track_name = track_name %>%
      str_remove(" \\(Ft\\. .*\\)") %>% # Remove "(Ft. ...)" part
      str_remove(" \\(feat\\. .*\\)") %>% # Remove "(feat. ...)" variant if exists
      str_to_lower(),  # Convert to lowercase for matching
    album_name = album_name,  # Ensure album name is preserved
    release_date = release_date # Ensure release date is preserved
  )


# finally we join datasets so that songs are associated with an album and a release date
dated_beyonce_lyrics <- cleaned_beyonce_lyrics %>%
  inner_join(
    cleaned_beyonce_tracks,
    by = c("clean_song_name" = "clean_track_name", "artist_name" = "artist_name")
  ) %>% mutate(clean_song_name = str_to_upper(str_sub(clean_song_name, 1, 1)) %>%
           str_c(str_sub(clean_song_name, 2, nchar(clean_song_name)))) %>%
  rename(album=album_name,Lyrics=line,released=release_date,Artist=artist_name,Title=clean_song_name) %>%
  mutate(featuring_artists = replace(featuring_artists, is.na(featuring_artists), "")) %>%
  select(-c("song_name","artist_id","track_name","song_id"))
```

We finally merge the dataframes for both Taylor Swift and Beyonce together:

```{r}
# Standardize column names in both data frames
dated_beyonce_lyrics <- dated_beyonce_lyrics %>%
  rename(Artist = Artist, album = album, Title = Title, Lyrics = Lyrics, released = released, featuring_artists = featuring_artists, track_id = track_id)

dated_swift_lyrics <- dated_swift_lyrics %>%
  rename(Artist = Artist, album = album, Title = Title, Lyrics = Lyrics, released = released, featuring_artists = featuring_artists, track_id = track_id)

# Now combine the data frames
combined_lyrics <- bind_rows(dated_beyonce_lyrics, dated_swift_lyrics)
```

## 1.1 The notion of a **corpus**:

```{r}
corp_songs <- quanteda::corpus(combined_lyrics, text_field="Lyrics")
quanteda::docnames(corp_songs) <- combined_lyrics$`Title`
```

```{r}
# Live demo
#head(corp_songs)
#corp_songs
summary(corp_songs, n = ndoc(corp_songs))
```

The notion of a **token**:

Let's look at the total number of tokens in all the songs.

```{r}
plot_df <- summary(corp_songs, n = ndoc(corp_songs))

max_x_break <- round(max(plot_df$Tokens), -3)

g <- (
  ggplot(plot_df, aes(x=Tokens))
  + geom_histogram(binwidth=30, fill="#C63C4A")
  + theme_minimal()
  
  + scale_x_continuous(breaks = seq(0, max_x_break, max_x_break/10))
  
  + labs(x="Number of Tokens",
         y="Count",
         title="Typically, how many tokens are there in those songs?",
         subtitle = "Corpus: songs of Beyonce and Taylor Swift")
  
  + # Prettify plot a bit
  theme_bw() +
  theme(plot.title=element_text(size=rel(1.5)),
        plot.subtitle = element_text(size=rel(1)),
        axis.title=element_text(size=rel(1.3)),
        axis.title.x=element_text(margin=margin(t=10)),
        axis.title.y=element_text(margin=margin(r=10)),
        axis.text=element_text(size=rel(1.2)))
)

g
```

Now let's look at the total number of tokens per song per artist:

```{r}
plot_df <- summary(corp_songs, n = ndoc(corp_songs)) %>%
  mutate(Artist = combined_lyrics$Artist)  # Add Artist column to summary data

g_facet <- ggplot(plot_df, aes(x = Tokens, fill = Artist)) +
  geom_histogram(binwidth = 30, alpha = 0.8, position = "identity") +
  facet_wrap(~ Artist, scales = "free") +
  labs(
    x = "Number of Tokens",
    y = "Count",
    title = "Token Distribution by Artist",
    subtitle = "Separate histograms for Beyoncé and Taylor Swift"
  ) +
  theme_minimal() +
  scale_fill_manual(values = c("Beyoncé" = "#237c9d", "Taylor Swift" = "#17c3b2")) +
  theme(
    plot.title = element_text(size = rel(1.5)),
    plot.subtitle = element_text(size = rel(1)),
    axis.title = element_text(size = rel(1.3)),
    strip.text = element_text(size = rel(1.3)),
    legend.position = "none"
  )

g_facet

```

How many unique tokens are there in the songs?

```{r}
plot_df <- summary(corp_songs, n = ndoc(corp_songs))

max_x_break <- round(max(plot_df$Types), -2)

g <- (
  ggplot(plot_df, aes(x=Types))
  + geom_histogram(binwidth=30, fill="#C63C4A")
  + theme_minimal()
  
  + scale_x_continuous(breaks = seq(0, max_x_break+25,by=25))
  
  + labs(x="Number of Types (unique tokens)",
         y="Count",
         title="Typically, how many unique tokens are there in those songs?",
         subtitle = "Corpus: songs of Beyonce and Taylor Swift")
  
  + # Prettify plot a bit
  theme_bw() +
  theme(plot.title=element_text(size=rel(1.5)),
        plot.subtitle = element_text(size=rel(1)),
        axis.title=element_text(size=rel(1.3)),
        axis.title.x=element_text(margin=margin(t=10)),
        axis.title.y=element_text(margin=margin(r=10)),
        axis.text=element_text(size=rel(1.2)))
)

g
```

And how many unique tokens are there per artist?

```{r}

plot_df <- summary(corp_songs, n = ndoc(corp_songs)) %>%
  mutate(Artist = combined_lyrics$Artist)  # Add Artist column to summary data

g_facet <- ggplot(plot_df, aes(x = Types, fill = Artist)) +
  geom_histogram(binwidth = 30, alpha = 0.8, position = "identity") +
  facet_wrap(~ Artist, scales = "free") +
  labs(
    x = "Number of Types (unique tokens)",
    y = "Count",
    title = "Unique Tokens Distribution by Artist",
    subtitle = "Separate histograms for Beyoncé and Taylor Swift"
  ) +
  theme_minimal() +
  scale_fill_manual(values = c("Beyoncé" = "#237c9d", "Taylor Swift" = "#17c3b2")) +
  theme(
    plot.title = element_text(size = rel(1.5)),
    plot.subtitle = element_text(size = rel(1)),
    axis.title = element_text(size = rel(1.3)),
    strip.text = element_text(size = rel(1.3)),
    legend.position = "none"
  )

g_facet
```

## 1.2 More on tokens

```{r}
tokens_songs <- quanteda::tokens(corp_songs)
```

Exploring:

```{r}
# Live demo of types
tokens_songs
```

**The notion of n-grams**

```{r}
tokens_ngrams(tokens_songs,n=4)[[1]]
#tokens_ngrams(tokens_songs,n=2)[[1]]
```

## 1.3 The Document-Term Feature Matrix

```{r}
dfm_songs <- quanteda::dfm(tokens_songs)
dfm_songs
```

```{r}
# To bring it to a familiar object
dfm_as_data_frame <- quanteda::convert(dfm_songs, to="data.frame")
dim(dfm_as_data_frame)
```

Let's create a data frame that'll be useful in future analysis

```{r}

songs_df<-bind_cols(docvars(dfm_songs), dfm_as_data_frame) %>% mutate(released = ymd(released)) %>% rename(album=album...4)

```

```{r}
dfm_songs %>% quanteda::topfeatures()

#docfreq(dfm_speeches)
```

There will be an explanation about a technique called TF-IDF, then we'll discuss what the following resulting data frame represents:

```{r}
dfm_tfidf(dfm_songs, scheme_tf = "prop")
```

------------------------------------------------------------------------

**TF-IDF (Term Frequency - Inverse Document Frequency) very briefly**

-   Text vectorizer method that transforms the text into a usable vector i.e assigns a number to words in a text
-   Each number is a function of the word (term) frequency and inverse document frequency:
    -   term frequency (TF) is how often a word appears in a document, divided by how many words there are. Term frequency measures how common a word is.
    -   inverse document frequency (IDF) is how unique or rare a word is and is: $IDF(t) = log_e(\textrm{Total number of documents} / \textrm{Number of documents with term t in it})$

**Example:**

-   Consider a document containing 100 words where the word *cooker* appears 5 times. The term frequency (i.e., TF) for *cooker* is then (5 / 100) = 0.05.

-   Now, assume we have 10 million documents and the word *cooker* appears in one thousand of these. Then, the inverse document frequency (i.e., IDF) is calculated as log(10,000,000 / 1,000) = 4.

-   Therefore, the TF-IDF weight is the product of these quantities: 0.05 \* 4 = 0.20.

To learn more about TF-IDF, see [this link](https://towardsdatascience.com/tf-idf-simplified-aba19d5f5530) or [this one](https://towardsdatascience.com/text-summarization-using-tf-idf-e64a0644ace3)

# Part 2: Feature Engineering for Machine Learning

## 2.1 Rebuilding

Let's re-build the corpus more carefully:

```{r}
# Get rid of punctuation or stopwords

tokens_songs <- 
  # Get rid of punctuations
  quanteda::tokens(corp_songs,remove_punct = TRUE, remove_symbols= TRUE) %>% 
  # Get rid of stopwords
  quanteda::tokens_remove(pattern = quanteda::stopwords("en")) %>%
  quanteda::tokens_ngrams(n=1:2)

dfm_songs <- quanteda::dfm(tokens_songs)
quanteda.textplots::textplot_wordcloud(dfm_songs)
```

Let's try and differentiate between artists now:

Here's the wordcloud for Taylor Swift:

```{r}

#create the subset for Taylor Swift
corp_songs_swift <- quanteda::corpus_subset(corp_songs, quanteda::docvars(corp_songs, "Artist") == "Taylor Swift")

# Get rid of punctuation or stopwords
tokens_songs_swift <- 
  # Get rid of punctuations
  quanteda::tokens(corp_songs_swift,remove_punct = TRUE, remove_symbols= TRUE) %>% 
  # Get rid of stopwords
  quanteda::tokens_remove(pattern = quanteda::stopwords("en")) %>%
  quanteda::tokens_ngrams(n=1:2)

dfm_songs_swift <- quanteda::dfm(tokens_songs_swift)
quanteda.textplots::textplot_wordcloud(dfm_songs_swift)
```

And here's the one for Beyoncé:

```{r}

#create the subset for Beyoncé
corp_songs_beyonce <- quanteda::corpus_subset(corp_songs, quanteda::docvars(corp_songs, "Artist") == "Beyoncé")

# Get rid of punctuation or stopwords
tokens_songs_beyonce <- 
  # Get rid of punctuations
  quanteda::tokens(corp_songs_beyonce,remove_punct = TRUE, remove_symbols= TRUE) %>% 
  # Get rid of stopwords
  quanteda::tokens_remove(pattern = quanteda::stopwords("en")) %>%
  quanteda::tokens_ngrams(n=1:2)

dfm_songs_beyonce <- quanteda::dfm(tokens_songs_beyonce)
quanteda.textplots::textplot_wordcloud(dfm_songs_beyonce)
```

How about I determine the top features for Taylor Swift and Beyoncé?

```{r}
dfm_songs_swift %>% quanteda::topfeatures()
```

```{r}
dfm_songs_beyonce %>% quanteda::topfeatures()
```

I can now construct the TF-IDF matrix for both Taylor Swift songs and Beyoncé songs:

```{r}
dfm_tfidf_swift <- dfm_tfidf(dfm_songs_swift, scheme_tf = "prop")
dfm_tfidf_beyonce <- dfm_tfidf(dfm_songs_beyonce, scheme_tf = "prop")
```

Let's convert the TF-IDF matrices into dataframes:

```{r}
tfidf_df_swift<-bind_cols(docvars(dfm_tfidf_swift), quanteda::convert(dfm_tfidf_swift, to="data.frame")) %>% rename(album=album...4) %>% mutate(released=ymd(released)) 
tfidf_df_beyonce<-bind_cols(docvars(dfm_tfidf_beyonce), quanteda::convert(dfm_tfidf_beyonce, to="data.frame")) %>% mutate(released=ymd(released))
```

## 2.2 LSA

You know about PCA - we’ve been playing with this linear dimensionality reduction technique for a while now. We want to show you an alternative method called **Latent Semantic Analysis (LSA)** this time. The linear algebra behind it is a bit more complex, but the idea is the same: we want to reduce the number of features (words) in our dataset to just a few dimensions - even if that comes with the cost of losing some interpretability.

One of the quanteda packages has a function called `textmodel_lsa` that does this for us. We will use it to reduce the number of features to 3 dimensions:

```{r}
df_lsa_swift <- quanteda.textmodels::textmodel_lsa(dfm_tfidf_swift, nd=3)$docs %>% as.data.frame()


```

Let's visualise the result

```{r}
# Assuming df_lsa_swift contains the LSA result with V1, V2, V3 columns.
# Rename the columns for clarity
df_lsa_swift <- df_lsa_swift %>%
  rename(Dim1 = V1, Dim2 = V2, Dim3 = V3)

# Add the document names (song titles)
df_lsa_swift$document <- rownames(df_lsa_swift)

# Create a 3D scatter plot using plotly

plot_ly(df_lsa_swift, 
        x = ~Dim1,  # First dimension of LSA
        y = ~Dim2,  # Second dimension of LSA
        z = ~Dim3,  # Third dimension of LSA
        text = ~document,  # Display document (song) names on hover
        type = "scatter3d", 
        mode = "markers",
        marker = list(size = 3, color = 'blue', opacity = 0.6)) %>%
  layout(title = "3D LSA Visualization of Taylor Swift Songs",
         scene = list(xaxis = list(title = "Dimension 1"),
                      yaxis = list(title = "Dimension 2"),
                      zaxis = list(title = "Dimension 3") ))
```

Each LSA dimension represents a topic. Here's a plot to help you interpret each of the LSA dimensions:

```{r}
# Get the LSA term weights (V matrix from SVD) and convert to a data frame
lsa_terms <- quanteda.textmodels::textmodel_lsa(dfm_tfidf_swift, nd = 3)$features %>% as.data.frame()

# Add term names as a column
lsa_terms <- tibble::rownames_to_column(lsa_terms, "term")

# Rename columns for readability
colnames(lsa_terms) <- c("term", "Dim1", "Dim2", "Dim3")

# Reshape data for visualization
lsa_terms_long <- lsa_terms %>%
  pivot_longer(cols = starts_with("Dim"), names_to = "Dimension", values_to = "Weight")

# Select top 10 terms per dimension
top_terms <- lsa_terms_long %>%
  group_by(Dimension) %>%
  slice_max(order_by = abs(Weight), n = 10) %>%
  ungroup()

# Visualize the top terms for each dimension
ggplot(top_terms, aes(x = reorder(term, Weight), y = Weight, fill = Dimension)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ Dimension, scales = "free") +
  coord_flip() +
  labs(
    title = "Top 10 Terms per LSA Dimension",
    x = "Terms",
    y = "Weight",
    subtitle = "Each LSA dimension represents a latent concept (topic)"
  ) +
  theme_minimal(base_size = 11)
```

You can read more about LSA [here](https://spotintelligence.com/2023/08/28/latent-semantic-analysis/) or [here](https://blog.marketmuse.com/glossary/latent-semantic-analysis-definition/).

## 2.3 Explaining clusters with textstat_keyness!

Using `textstat_keyness()`, you can compare frequencies of words between target and reference documents. Here we want to compare what makes Taylor Swift songs (our "target") different from Beyoncé songs!

```{r}
dfm_filtered <- dfm_songs %>% dfm_subset(dfm_songs$Artist %in% c("Taylor Swift","Beyoncé"))

tstats <- textstat_keyness(dfm_filtered %>% dfm_group(Artist), target="Taylor Swift", measure="lr")
textplot_keyness(tstats, labelsize=2)
```

## 2.4 Topic Modelling

There are other algorithms (e.g Latent Dirichlet Allocation) you can use! Find out more about this in [here](https://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-allocation/).

Instead of k-means, we will use another unsupervised technique called topic modelling. As the name implies, this clustering technique is more suitable for text data. The most popular of such families of models is called **Latent Dirichlet Allocation (LDA)**. We will not go into how it works, but you can read more about it [here](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation).

```{r}
tmod_lda <- topicmodels::LDA(dfm_songs %>% dfm_subset(docvars(corp_songs, "Artist") == "Taylor Swift"), k = 4)

## What are the topics mostly associated with each cluster?
tmod_lda %>% topicmodels::terms(10)
```

```{r}
tmod_lda <- topicmodels::LDA(dfm_songs %>% dfm_subset(docvars(corp_songs, "Artist") == "Beyoncé"), k = 4)

## What are the topics mostly associated with each cluster?
tmod_lda %>% topicmodels::terms(10)
```

If you want to see more on the analysis of Taylor Swift and Beyoncé songs, have a look [here](https://rpubs.com/RosieB/642806) or [here](https://medium.com/@bearnest/a-fun-text-analysis-of-taylor-swifts-songs-502e74fed2b4) or [here](https://www.promptcloud.com/blog/data-visualization-text-mining-taylor-swift-song-lyrics/). For more of Taylor Swift, check [this page](https://www.kaggle.com/code/fajarkhaswara/comparing-taylor-swift-and-ed-sheeran) too.
