---
title: "📆 Week 10: Basics of text mining"
subtitle : "Theme: Applications"
author: "Dr. Ghita Berrada"
format: 
  html:
    from: markdown+emoji
    page-layout: full
    toc: true
    toc-depth: 2
self-contained: true
---

## Introduction: Why Text Mining Matters

Until now, we've worked with structured (tabular) data. But consider this: **80-90% of organizational data is unstructured**, with text being the most common form. Social media posts, customer reviews, legal documents, medical records—all contain valuable insights that traditional data analysis methods cannot capture.

Today, we'll analyze discourse on **Truth Social**, the social media platform launched in 2022. You can find the full dataset [here](https://zenodo.org/records/7531625). This case study will introduce you to the fundamental concepts and techniques of text mining, from representing text as data to discovering hidden themes in millions of posts.

### Learning Objectives

By the end of this lecture, you should be able to:

1. Transform unstructured text into structured, analyzable data
2. Choose appropriate preprocessing strategies based on your analytical goals
3. Understand the mathematical foundations of text similarity and dimensionality reduction
4. Compare traditional and modern approaches to topic discovery
5. Apply critical thinking about what these methods reveal—and obscure

---

## Setup

```{r}
library(tidyverse)
library(quanteda)
library(quanteda.textplots)
library(quanteda.textstats)
library(quanteda.textmodels)
library(tidytext)
library(tidymodels)
library(lubridate)
library(topicmodels)
library(R.utils)
library(reticulate)
library(text2vec)
library(plotly)
library(spacyr)
#library(igraph)
library(bertopicr)
library(word2vec)
library(udpipe)
```

## Loading the Truth Social Dataset

### Dataset Overview

In this lecture, we will work with two files from the Truth Social dataset:

#### **1. `truths.tsv`**

Contains individual posts ("Truths") from the platform, including:

* post ID
* timestamp
* author ID
* engagement metrics
* *free-form text*
* URL information

#### **2. `users.tsv`**

Contains metadata for each user, such as:

* user ID
* username
* profile details

#### **Why these two files?**

Because they allow us to demonstrate two essential skills in text mining:

1. **Diagnosing and cleaning messy real-world text data**
2. **Joining post-level and user-level datasets**

In practice, social media exports are often imperfect:
character encoding issues, unescaped special characters, malformed rows, etc.
This lecture walks through how to detect and fix such issues before analysis.

These two files also contain information that will be relevant to our downstream analysis.

---

### Step 1 — Inspect the header of `truths.tsv`

We begin by examining only the **header row**, because it tells us the number of **columns the dataset intends to have**.

We do *not* assume the dataset is malformed — we simply inspect its declared structure.

```{r}
truths_file <- "../data/truth_social/truths.tsv"
users_file  <- "../data/truth_social/users.tsv"

# Read the header (first line only)
header_line <- readr::read_lines(truths_file, n_max = 1)

header_line
```

#### Count the number of declared fields in the header

```{r}
header_fields <- header_line %>%
  stringr::str_split("\t", simplify = TRUE) %>%
  length()

header_fields
```

#### Interpretation

This tells us how many columns every row **should** contain.
For this dataset, the header contains **13 fields**, representing the schema for a Truth Social post.

We use this number as the baseline for evaluating the rest of the file.

---

### Step 2 — Inspect the structure of *all* rows

Now we ask the question:

> Do all rows in the file actually match the header’s expected structure?

To check this, we count how many **tab-separated fields** each row contains.

A TSV row with *N* fields will contain *N−1* tab characters.

```{r}
field_counts <- readr::read_lines(truths_file) %>%
  stringr::str_count("\t") %>%
  `+`(1)   # fields = number of tabs + 1
```

#### Display the distribution

```{r}
field_counts %>% table()
```

#### Interpretation

* The header declares **13** fields
* Most rows have **13 fields** → these are **well-formed**
* Some rows have **too many** fields (14, 15, 22) →
  these almost always result from **unescaped tab characters inside the text column**,
  or stray markup such as `<emoji: fire>`

This demonstrates the existence of **structurally inconsistent rows**.
We now must decide how to handle them.

---

### Step 3 — Why keeping only 13-field rows is appropriate here

Because this is a **teaching dataset**, our goal is:

* to have a clean, rectangular dataset
* to focus on text mining techniques
* to avoid complicated heuristics for reconstructing malformed lines

The vast majority of rows (823,928 of them) are perfectly well-formed.
These rows support the header schema and can be used safely.

Malformed rows cannot be reliably repaired because:

* the text may contain arbitrary tabs
* extra fields may be mistaken for metadata
* information may be lost or shifted
* there is no guaranteed pattern to reconstruct the original structure

Therefore:

> For teaching purposes, we keep only the rows that match the header’s structure
> and discard those that cannot be read reliably.

In real research, we might attempt sophisticated reconstruction.
But for instructional clarity, *reducing noise helps us focus on core NLP techniques*.

---

### Step 4 — Create a cleaned version of `truths.tsv`

This pipeline:

* reads all rows
* counts their fields
* keeps only header + rows with 13 fields
* writes out a clean TSV

```{r}
clean_lines <- readr::read_lines(truths_file) %>%
  tibble(raw = .) %>%
  mutate(
    field_count = stringr::str_count(raw, "\t") + 1,
    is_header = row_number() == 1             # always keep header
  ) %>%
  filter(is_header | field_count == header_fields) %>%
  pull(raw)

clean_file <- "data/truth_social/truths_clean.tsv"
readr::write_lines(clean_lines, clean_file)
```

---

### Step 5 — Read the cleaned Truth Social data

Now that every row has the same number of fields, we can safely load the dataset.

```{r}
truths <- readr::read_tsv(clean_file, show_col_types = FALSE)
glimpse(truths)
```

This dataset is now ready for text mining.

---

### Step 6 — Load the users dataset

```{r}
users <- readr::read_tsv(users_file, show_col_types = FALSE)
glimpse(users)
```

---

### Step 7 — Merge posts (`truths.tsv`) with user metadata (`users.tsv`)

The posts file (`truths.tsv`) identifies authors using a column named `author`.
The users file (`users.tsv`) contains user IDs in a column named `id`.

We join them to enrich the posts with user information.

```{r}
merged_data <- truths %>%
  left_join(users, by = c("author" = "id"))

glimpse(merged_data)
```

---

### Final Outcome

We now have:

* A **clean**, **rectangular**, **consistent** dataset of Truth Social posts
* A table of user metadata
* A merged dataset ready for **text mining analysis**

The dataset contains:
- **Post text**: The actual content users shared
- **Timestamps**: When posts were created
- **User information**: Verified status, follower counts
- **Engagement metrics**: Likes, shares, replies

From here, we can proceed with our analysis.

This workflow also demonstrates valuable real-world skills:

* diagnosing malformed files
* understanding delimiter problems
* cleaning corrupted text data
* merging relational datasets


**Analytical Question #1:** What topics dominate Truth Social discourse, and how do they evolve over time?

To answer this, we first need to understand how to represent text mathematically.

---

## Part 1: From Text to Numbers—The Representation Problem

### 1.1 The Fundamental Challenge

Computers don't understand "words"—they process numbers. The central challenge of text mining is: **How do we convert text into numerical representations while preserving meaning?**

Consider these two posts:

- Post A: "The election was rigged and stolen"
- Post B: "Stolen election, totally rigged"

Humans immediately recognize these express similar sentiments. But how do we make this mathematically explicit?

### 1.2 Building a Corpus

```{r}
# -------------------------------------------------------------
# 1. Create the corpus from Truth Social posts
# -------------------------------------------------------------
# We specify that the text comes from the 'text' column
corp_truth <- corpus(merged_data, text_field = "text")

# -------------------------------------------------------------
# 2. Add document-level variables (docvars)
# -------------------------------------------------------------

# --- (A) Add post date ---
# merged_data contains two timestamps: timestamp.x and timestamp.y.
# timestamp.x refers to when the post was created, which is what we want.
docvars(corp_truth, "post_date") <- ymd_hms(merged_data$timestamp.x)

# --- (B) Add engagement metric ---
# We create a simple engagement score using available fields.
docvars(corp_truth, "engagement") <-
  merged_data$like_count +
  merged_data$retruth_count +
  merged_data$reply_count

# --- (C) Add user-level metadata ---
# These come from users.tsv after merging.
docvars(corp_truth, "username")       <- merged_data$username
docvars(corp_truth, "follower_count") <- merged_data$follower_count
docvars(corp_truth, "following_count")<- merged_data$following_count

# Optional: add scrape indicators
docvars(corp_truth, "finished_truth_scrape") <- merged_data$finished_truth_scrape

# -------------------------------------------------------------
# 3. Examine the corpus
# -------------------------------------------------------------
summary(corp_truth, n = 10)

```



#### Code + Explanations

```{r}

# -------------------------------------------------------------
# 1. Create the corpus from Truth Social posts
# -------------------------------------------------------------
# We specify that the text comes from the 'text' column
corp_truth <- corpus(merged_data, text_field = "text")

# -------------------------------------------------------------
# 2. Add document-level variables (docvars)
# -------------------------------------------------------------

# --- (A) Add post date ---
# merged_data contains two timestamps: timestamp.x and timestamp.y.
# timestamp.x refers to when the post was created, which is what we want.
docvars(corp_truth, "post_date") <- ymd_hms(merged_data$timestamp.x)

# --- (B) Add engagement metric ---
# We create a simple engagement score using available fields.
docvars(corp_truth, "engagement") <-
  merged_data$like_count +
  merged_data$retruth_count +
  merged_data$reply_count

# --- (C) Add user-level metadata ---
# These come from users.tsv after merging.
docvars(corp_truth, "username")       <- merged_data$username
docvars(corp_truth, "follower_count") <- merged_data$follower_count
docvars(corp_truth, "following_count")<- merged_data$following_count

# Optional: add scrape indicators
docvars(corp_truth, "finished_truth_scrape") <- merged_data$finished_truth_scrape

# -------------------------------------------------------------
# 3. Examine the corpus
# -------------------------------------------------------------
summary(corp_truth, n = 10)
```


> After merging the `truths.tsv` and `users.tsv` files, we construct a text corpus using the `quanteda` package.
>
> A corpus is a structured collection of text documents, and `quanteda` allows us to attach metadata (“document variables” or docvars) to each document.
>
> These docvars are crucial because they enable us to group, filter, or model texts based on:
>
> * when the post was made (`post_date`)
> * how much engagement it received (`engagement`)
> * information about the author (`username`, `follower_count`, `following_count`)
>


**Concept: What is a Corpus?**

A **corpus** (plural: corpora) is a structured collection of texts. Think of it as a specialized data structure that:

- Stores documents (our Truth Social posts)
- Preserves metadata (user info, timestamps, engagement)
- Enables systematic analysis across thousands or millions of documents

The corpus is our "dataset" for text analysis—just as you'd load a CSV for tabular analysis.

### 1.3 Tokenization—Breaking Text into Units

```{r}
# Create corpus
corp_truth <- corpus(merged_data, text_field = "text")

# ---- Add document-level metadata BEFORE tokenization ----
median_followers <- median(merged_data$follower_count, na.rm = TRUE)

docvars(corp_truth, "high_follower") <-
  merged_data$follower_count >= median_followers

docvars(corp_truth, "engagement") <-
  merged_data$like_count +
  merged_data$retruth_count +
  merged_data$reply_count

docvars(corp_truth, "post_date") <- merged_data$timestamp.x
# ----------------------------------------------------------

# Create tokens
tokens_truth <- tokens(
  corp_truth,
  remove_punct   = TRUE,
  remove_symbols = TRUE,
  remove_numbers = FALSE,  # Keep numbers for now
  remove_url     = TRUE
)

# Examine tokens from first post
tokens_truth[[1]]
```

**Concept: Tokens and the Granularity Decision**

A **token** is the basic unit of analysis. Usually, tokens are words, but they could also be:
- **Characters**: Useful for language detection, authorship analysis
- **Words**: Most common choice, balances interpretability and information
- **Sentences**: For discourse analysis, argument mining
- **Paragraphs**: For document structure analysis

**Critical thinking question:** What do we lose when we tokenize? 

- Word order: "I didn't love it" vs "I loved it, didn't I?"
- Punctuation as meaning: "Great!" vs "Great."
- Sentence boundaries: Context

The choice of tokenization level depends on your research question. For topic discovery on Truth Social, word-level tokens work well because:

1. We want to capture vocabulary patterns
2. Posts are short (like tweets)
3. We're less concerned with nuanced sentiment

### 1.4 The Token vs. Type Distinction

```{r}
# Count tokens and types
token_stats <- summary(corp_truth, n = ndoc(corp_truth))

ggplot(token_stats, aes(x = Tokens)) +
  geom_histogram(binwidth = 5, fill = "#C63C4A") +
  labs(title = "Distribution of Post Lengths on Truth Social",
       subtitle = "Most posts are short, but some are lengthy rants",
       x = "Number of Tokens", y = "Count") +
  theme_minimal()
```

Let's visualise things better (we remove extreme outliers):

```{r}
token_stats_filtered <- token_stats %>% filter(Tokens < 200)

ggplot(token_stats_filtered, aes(x = Tokens)) +
  geom_histogram(binwidth = 1, fill = "#C63C4A") +
  labs(
    title = "Distribution of Post Lengths on Truth Social (Outliers Removed)",
    subtitle = "Posts longer than 200 tokens (~0.3%) removed for clarity",
    x = "Number of Tokens", y = "Count"
  ) +
  theme_minimal()

```

**Concept: Tokens vs. Types**

- **Token**: Each occurrence of a word. The sentence "the cat saw the dog" has 5 tokens.
- **Type**: Each unique word. The same sentence has 4 types (the, cat, saw, dog).

The **type-token ratio (TTR)** measures vocabulary diversity:
$$\text{TTR} = \frac{\text{Number of Types}}{\text{Number of Tokens}}$$

**On Truth Social, what might TTR tell us?**
- Low TTR (0.3-0.5): Repetitive messaging, possibly coordinated or bot-like
- High TTR (0.7-0.9): Diverse vocabulary, possibly more authentic individual voices
- Very high TTR (>0.95): Likely short posts or random text


#### Comparing Vocabulary Diversity Across User Groups

Now that we understand what the Type–Token Ratio (TTR) measures, we can use it to explore a practical question in our Truth Social dataset:

> **Do users with many followers write differently from users with few followers?**

We do not have a “verified” or “influencer” label, but we *do* have a meaningful real variable:
  
- **`follower_count`** — the number of followers each user has.

Follower count is a reasonable proxy for user influence or visibility on the platform.  
If linguistic style varies with influence, we might see differences in TTR across these groups.

##### Why compare follower groups?

- **High-follower users** may be public figures or highly visible accounts.  
  Their posts might use more consistent, message-oriented language.

- **Low-follower users** may behave more like ordinary individuals, sometimes producing more varied or conversational vocabulary.

To examine this, we divide users into **high-follower** and **low-follower** groups using the median follower count in the dataset, and then compare the TTR of their posts.

##### 👉 **Code: Create follower groups and compute TTR**

```{r}

# Determine the median follower count
median_followers <- median(merged_data$follower_count, na.rm = TRUE)

# Add follower-group information to the corpus
docvars(corp_truth, "high_follower") <-
  merged_data$follower_count >= median_followers

# Add TTR and the group variable to token_stats
token_stats <- token_stats %>%
  mutate(
    TTR = Types / Tokens,
    high_follower = docvars(corp_truth, "high_follower")
  )
```

##### 👉 **Code: Visualize TTR differences**

```{r}
ggplot(token_stats, aes(x = high_follower, y = TTR, fill = high_follower)) +
  geom_boxplot() +
  labs(
    title = "Do High-Follower Users Use More Diverse Vocabulary?",
    subtitle = "Comparing type–token ratio (TTR) across influence levels",
    x = "High-Follower User?",
    y = "Type–Token Ratio"
  ) +
  scale_fill_manual(values = c("#C63C4A", "#5B8E7D")) +
  theme_minimal()
```

##### Interpretation

The boxplot compares the type–token ratios (TTR) of posts written by high-follower and low-follower users.  
Two observations stand out:

1. **High- and low-follower users have very similar TTR distributions.**  
   Both groups show:
   - very high median TTR values (close to 1),  
   - wide ranges caused by short posts (where TTR is artificially high),  
   - substantial overlap between groups.

   This suggests that **follower count does not strongly predict vocabulary diversity** on Truth Social.  
   Influential users do not appear to use a meaningfully broader or narrower range of vocabulary than typical users.

2. **The NA group consists of posts where follower data is missing.**  
   These posts show extreme TTR values, often equal to 1.  
   This typically occurs when:
   - the post is extremely short, or  
   - the associated user could not be matched in the users dataset.

   Because these cases arise from missing metadata rather than a meaningful user group,  
   they should generally be **excluded from interpretation**.

Overall, the analysis suggests that **TTR is not strongly associated with user influence** on this platform.  
Vocabulary diversity appears relatively stable across users regardless of how many followers they have.


---

## Part 2: Preprocessing—What to Keep and What to Discard

Before we can analyze topics, we need to decide: **Which words matter?**

### 2.1 The Stopword Problem

💭 Which words do you expect to be the most frequent?

```{r}
# Most frequent words without removing stopwords
tokens_raw <- tokens(corp_truth)
dfm_raw <- dfm(tokens_raw)
topfeatures(dfm_raw, 20)
```

We see: "the", "a", "and", "is", "to" (and lots of punctuations)... Is this useful at all?

**Concept: Stopwords**

**Stopwords** are high-frequency words that carry little semantic content. The logic: if "the" appears in 95% of documents, it doesn't help distinguish documents.

**But here's the critical question:** What counts as a stopword depends on your task!

Standard stopword lists (like SMART or Snowball) were designed for **information retrieval**. But:

- For **authorship analysis**: Function words like "the", "is", "very" are actually discriminative (personal style markers)
- For **sentiment analysis**: "not", "no", "never" are crucial—remove them and "not good" becomes "good"!
- For **topic modeling**: Stopwords add noise without adding meaning

Now, let's try and remove stopwords and see how things change:

```{r}
# Remove stopwords for topic analysis
tokens_truth_clean <- tokens_truth %>%
  tokens_remove(pattern = stopwords("en"))

# Custom stopwords for Truth Social
custom_stops <- c("amp", "https", "rt", "truthsocial")  # Platform artifacts
tokens_truth_clean <- tokens_remove(tokens_truth_clean, pattern = custom_stops)

# Compare top features
dfm_clean <- dfm(tokens_truth_clean)
topfeatures(dfm_clean, 20)
```

How does it look now?

The most frequent tokens in the cleaned corpus are a mix of expressive elements (e.g., “emoji”, “joy”, “rolling_on_the_floor_laughing”) and common conversational terms such as “us,” “people,” “just,” and “like.”  
Political keywords like **“trump,” “biden,”** and **“president”** also appear prominently, reflecting the platform’s strong focus on political discussion.  
Overall, the vocabulary suggests a blend of emotional expression, everyday conversation, and high-frequency political messaging.

**Analytical principle:** Always examine your top features *after* preprocessing. Do they make sense? Are there domain-specific stopwords you missed?

### 2.2 Stemming vs. Lemmatization: A Critical Choice

Look at these (example) Truth Social posts:

- "Biden is destroying America"
- "Biden destroyed the economy"  
- "Biden's destruction of our country"

These use different forms of "destroy," but express similar ideas. Should we treat them as the same or different?

**Option 1: Stemming**

```{r}
# Apply stemming (crude but fast)
tokens_stemmed <- tokens_wordstem(tokens_truth_clean, language = "en")

# Example: what happens to our words?
# destroying → destroy
# destroyed → destroy
# destruction → destruct  (imperfect!)
```

Let's have a look at our tokens after stemming:

```{r}
dfm(tokens_stemmed) |> topfeatures(20)
```

After stemming, the most frequent terms in the corpus reflect three dominant themes: emotional expression (“emoji,” “joy,” “pray,” “thank”), everyday conversational language (“just,” “like,” “one,” “know,” “time”), and political discourse (“trump,” “biden,” “presid”).  
Stemming also groups related forms together (e.g., “people/peoples” → “peopl”), making the high-level patterns clearer while preserving the strong presence of political and affective vocabulary typical of Truth Social posts.

**How stemming works:** 

Algorithmic removal of common suffixes using rules:

- "ing" → remove
- "ed" → remove  
- "ation" → remove

**Porter Stemmer rules (simplified):**

1. SSES → SS (caresses → caress)
2. IES → I (ponies → poni)
3. S → (blank) (cats → cat)

**Advantages:**

- Fast (rule-based)
- Language-agnostic (once rules are defined)
- Reduces vocabulary size significantly

**Disadvantages:**

- Crude: "university" → "univers", "studies" → "studi"
- Creates non-words
- Can merge unrelated words: "organ" and "organization" → "organ"
- Can separate related words: "Europe", "European" → different stems

**Option 2: Lemmatization**

Lemmatization reduces words to their *dictionary base form* using part-of-speech (POS) information.  
Unlike stemming, lemmatization uses **linguistic knowledge** to choose the correct base word:

- "destroyed" → "destroy" (verb infinitive)  
- "better" → "good" (adjective base form)  
- "destruction" → "destruction" (noun; correctly *not* reduced to "destroy")  

This makes lemmatization more accurate and interpretable than stemming.

For this lecture, we use **UDPipe**, a fast and lightweight NLP toolkit that runs entirely inside R.  
It provides lemmatization, POS tagging, and syntactic information without requiring Python (unlike `spacyr`) or external models.

#### 👉 **Code: Lemmatize the dataset using UDPipe**

```{r}
# 1. Download and load an English model (only downloaded once)
model <- udpipe_download_model(language = "english-ewt")
ud_model <- udpipe_load_model(model$file_model)

# NOTE:
# Lemmatizing the full dataset would take a long time.
# For demonstration purposes, we use a small slice of the data.
text_slice <- merged_data$text[1:2000]   # take the first 2000 posts

# 2. Lemmatize the slice
ud_out <- udpipe(text_slice, ud_model)

# ud_out contains one row per token:
# token, lemma, upos (universal POS), xpos (language-specific POS), etc.
head(ud_out)
```



---

#### 👉 **Reconstruct each document using lemmas**

```{r}
lemma_texts <- ud_out %>%
  group_by(doc_id) %>%
  summarise(text = paste(lemma, collapse = " "),
            .groups = "drop")

# Build a quanteda corpus from lemmatized text
corp_lemma <- corpus(lemma_texts, text_field = "text")

# Tokenize and clean
toks_lemma <- tokens(corp_lemma,
                     remove_punct = TRUE,
                     remove_numbers = TRUE,
                     remove_symbols = TRUE) %>%
  tokens_remove(stopwords("en"))

# Create a dfm
dfm_lemma <- dfm(toks_lemma)
dfm_lemma |> topfeatures(20)
```

The most frequent lemmas in the lemmatized corpus illustrate three characteristic themes of Truth Social posts:

1. **Emotional and expressive content**  
   
   Many top items are emojis or emotion-related lemmas (e.g., “emoji,” “fire,” “heart,” “smile,” “rage”).  
   This reflects a communication style that relies heavily on emotional signaling and visual emphasis.

2. **General conversational vocabulary**  
   
   Common verbs and function words such as “go,” “one,” “time,” “just,” “get,” “know,” and “can” show that everyday, informal language dominates much of the platform’s discourse.

3. **Religious or value-oriented language**  
   
   Lemmas like “god” and “latin_cross” indicate the presence of religious framing, which appears frequently enough to rise into the top features even in a small sample.

Notably, political terms (e.g., “trump,” “biden”) do **not** appear among the top lemmas here, which likely reflects two factors:  

(1) we analyzed only a small slice of the dataset for computational reasons, and  
(2) lemmatization groups many inflected forms together, reducing the apparent frequency of individual political terms.  

Overall, the lemmatized top features emphasize emotional expression, simple conversational structure, and recurring symbolic or value-laden language.

####  ✔ Advantages of UDPipe

* Very fast — suitable for large datasets
* Works entirely inside R (no conda, no Python)
* Provides POS tags and lemmas
* Simple, reliable, and portable for teaching

#### ✖ Disadvantages

* Less accurate than spaCy or Stanza
* Lemmatization may occasionally produce unexpected forms
* Not ideal for high-precision linguistic research


#### Interpretation


UDPipe demonstrates the core *principle* of lemmatization: different surface word forms are mapped to a single, meaningful base form.  
This reduces vocabulary size, improves interpretability, and helps topic modeling or clustering focus on meaning rather than inflection.

For teaching purposes, UDPipe strikes an excellent balance between speed, simplicity, and linguistic structure.

**Advantages of lemmatization:**

* Produces **real words** that are easy to interpret
* Uses POS tags for better accuracy
* Keeps noun/verb distinctions when appropriate

**Disadvantages:**

* Slower (requires linguistic parsing)
* Requires a language model (e.g., spaCy English)
* Sometimes preserves distinctions you might prefer to merge


#### **Decision Framework**


| Task                                        | Stemming | Lemmatization | Neither |
| ------------------------------------------- | -------- | ------------- | ------- |
| Topic modeling (broad themes)               | ✓        | ✓             |         |
| Sentiment analysis                          |          | ✓             |         |
| Information retrieval/search                | ✓        |               |         |
| Authorship attribution                      |          |               | ✓       |
| Legal text analysis (exact wording matters) |          |               | ✓       |
| Casual/exploratory analysis                 | ✓        |               |         |
| Academic research (precision matters)       |          | ✓             |         |


#### Quick clarification

Lemmatization is most useful when your goal is to preserve **meaning** and **grammatical relationships**.
Stemming is better when speed matters or when you want to collapse many forms aggressively.
Neither is appropriate when *exact wording* is important (e.g., legal documents or stylometric analysis).


**For Truth Social analysis:** We'll use stemming for exploratory topic modeling (speed + reduces vocabulary), but lemmatization if we were doing sentiment analysis (preserves crucial negations).

```{r}
# Let's continue with stemmed tokens
tokens_final <- tokens_stemmed
```

### 2.3 N-grams—Capturing Context

Single words (unigrams) lose context:

Post: "not good"
Unigrams: ["not", "good"] 

If "not" is a stopword, we get ["good"]—completely wrong!

**Concept: N-grams**

An **n-gram** is a sequence of n consecutive tokens:

- **Unigram** (n=1): "election"
- **Bigram** (n=2): "election fraud", "fake news"  
- **Trigram** (n=3): "make america great"

```{r}
# Create bigrams
tokens_ngrams <- tokens_ngrams(tokens_final, n = 1:2)

# Most common bigrams
dfm_ngrams <- dfm(tokens_ngrams)
topfeatures(dfm_ngrams, 20)
```

### Unigrams and Bigrams After Stemming

Once we stem the text and generate both unigrams and bigrams, the most frequent features reveal how users combine words and emojis in Truth Social posts.

Several patterns stand out:

1. **Emoji–word combinations are extremely common.**  
   
   Bigrams such as *emoji_us* and *us_emoji* appear among the top features.  
   This reflects the platform’s heavy use of emoji as both emotional markers and rhetorical devices, often combined with simple pronouns (“us”).

   *Note*: "us" could also refer to the US!

2. **High-frequency stems match the earlier unigram analysis.**  
   
   Common conversational stems (*just, get, peopl, go, like*) remain dominant.  
   These represent everyday, informal language typical of short social media posts.

3. **Political terms still appear, but bigrams dilute them.**  
   
   Words like *trump* and *biden* rank highly as unigrams, but do not appear in the top bigrams—  
   suggesting these names often occur alone rather than in stable two-word expressions.

4. **Emotion and value framing persist.**  
   
   Stems like *joy*, *truth*, and various emoji combinations emphasize the platform’s expressive and affective tone.

Overall, the n-gram results show that Truth Social discourse is characterized by:

- heavy emoji use,  
- simple conversational phrasing, and  
- recurring political and emotional keywords.  

Bigrams are especially useful for revealing which expressions tend to co-occur, and in this case they highlight how emoji frequently interact with pronouns and emotional terms in the platform’s communication style.



**Analytical question:** On Truth Social, are certain bigrams disproportionately used by **high-follower** versus **low-follower** users?  
This could reveal differences in communication style or messaging strategies between more and less influential accounts.

---

## Part 3: The Document-Term Matrix—Text as Geometry

### 3.1 From Tokens to Matrix

```{r}
# Create document-term matrix (DTM)
dfm_truth <- dfm(tokens_final)

# Dimensions
dim(dfm_truth)  # e.g., [50000 documents × 25000 features]

# Convert to viewable data frame (first 10 docs, 10 features)
dfm_truth[1:10, 1:10] %>% 
  convert(to = "data.frame")
```

**Concept: The Document-Feature Matrix (DFM)**

The DFM is a **mathematical representation** of our corpus:

- **Rows**: Documents (Truth Social posts)
- **Columns**: Features (unique words/n-grams)
- **Values**: Counts (how often word j appears in document i)

**Example:**

| Document | election | fraud | biden | trump | country |
|----------|----------|-------|-------|-------|---------|
| Post 1 | 2 | 1 | 0 | 3 | 1 |
| Post 2 | 0 | 0 | 4 | 0 | 2 |
| Post 3 | 1 | 1 | 2 | 1 | 0 |

**Why this matters:** By representing text as numbers, we can now:

- Calculate distances between documents
- Apply linear algebra (PCA, SVD)
- Use machine learning algorithms
- Compute similarity metrics

**Key property:** Most DFMs are **sparse** (90-99% zeros). Why?

- Most words don't appear in most documents
- This is why specialized sparse matrix formats are used (memory efficient)

#### What the DFM Output Shows

The small preview of the DFM (first 10 documents × first 10 features) provides a concrete illustration of the principles we just discussed:

- The **full matrix** has 823,927 documents and 138,175 features — consistent with a very large, high-dimensional representation.
- The excerpt shows that **most entries are zero**, confirming the expected sparsity of real-world DFMs.
- Only a few posts contain any of the sampled terms; for example, `text1` contains several features (“anons,” “public,” “awakening,” “emoji”), while most of the other posts contain none.

This small slice demonstrates exactly why DFMs are sparse and why sparse matrix techniques are essential: despite the large vocabulary, any individual post uses only a tiny fraction of all possible terms.


### 3.2 Document Similarity—Finding Related Posts

**Analytical question:** Can we find duplicate or near-duplicate posts (possible bot activity)?

**Concept: Text Similarity Measures**

Treat each document as a **vector in high-dimensional space**. Post 1 above is:
$$\vec{d_1} = [2, 1, 0, 3, 1]$$

**Cosine Similarity** measures the angle between vectors:

$$\text{cosine}(\vec{d_1}, \vec{d_2}) = \frac{\vec{d_1} \cdot \vec{d_2}}{||\vec{d_1}|| \cdot ||\vec{d_2}||} = \frac{\sum_{i=1}^{n} d_{1i} \times d_{2i}}{\sqrt{\sum_{i=1}^{n} d_{1i}^2} \times \sqrt{\sum_{i=1}^{n} d_{2i}^2}}$$

**Intuition:**

- 1.0 = Identical direction (same content)
- 0.0 = Orthogonal (no shared words)
- Close to 1 = Very similar

**Why cosine instead of Euclidean distance?**

- Cosine ignores magnitude (document length)
- Long post with same words as short post → still similar
- Euclidean would say "very different" just because one is longer

```{r}
# Calculate pairwise cosine similarity
dfm_sample <- dfm_sample(dfm_truth, size = 1000)  # Sample for speed
cos_sim <- textstat_simil(dfm_sample, 
                          method = "cosine",
                          margin = "documents")

# Find highly similar pairs (possible duplicates/bots)
cos_matrix <- as.matrix(cos_sim)
high_sim <- which(cos_matrix > 0.95 & cos_matrix < 1, arr.ind = TRUE)

# Examine suspicious pairs
merged_data$text[high_sim[1,]]
```

#### Interpreting High Cosine Similarity Between Posts

Cosine similarity measures how similar two documents are based on the words they contain.  
Values above **0.95** usually indicate near-duplicate content, coordinated messaging, or automated posting.

However, the two posts flagged in our sample are **not semantically similar** at all.  
This happens because social media text is extremely **short and sparse**: each post contains only a few stems after preprocessing.  
When two documents share even a small set of common stems (e.g., *get, will, america*), their sparse vectors can point in almost the same direction, producing an artificially high cosine score.

This illustrates an important limitation:  

> **In sparse, short-form text, cosine similarity can signal structural similarity even when the messages differ in meaning.**

For robust duplicate detection, cosine similarity should therefore be combined with additional checks (e.g., n-gram comparison, character-level similarity, or tf–idf weighting).


**Other similarity measures:**

**Jaccard Similarity** (for binary presence/absence):
$$J(A, B) = \frac{|A \cap B|}{|A \cup B|}$$

Good for: "What proportion of words do these documents share?"

```{r}
jaccard_sim <- textstat_simil(dfm_sample, method = "jaccard")
```

**Use cases on Truth Social:**

1. **Duplicate detection**: Cosine > 0.98 → likely copy-paste or bot
2. **Community detection**: Cluster users by posting similarity
3. **Trending topics**: Posts with high pairwise similarity appearing in same time window
4. **Influence tracking**: Find "seed" posts that get copied/paraphrased

### 3.3 TF-IDF—Not All Words Are Equal

Look at the top features in our DFM:

```{r}
topfeatures(dfm_truth, 20)
```

What we see: "emoji", "us","trump", "america", "peopl" (stemmed), "like"...

**Problem:** These words appear in *many* documents. Do they really distinguish posts?

**Concept: TF-IDF (Term Frequency-Inverse Document Frequency)**

The intuition: **A word is important if it appears frequently in a document BUT rarely across all documents.**

**Formula:**

$$\text{TF-IDF}(t, d) = \text{TF}(t, d) \times \text{IDF}(t)$$

Where:
- $\text{TF}(t, d) = \frac{\text{count of term } t \text{ in document } d}{\text{total terms in document } d}$
- $\text{IDF}(t) = \log\left(\frac{\text{total documents}}{\text{documents containing } t}\right)$

**Example:**

Say "immigration" appears:

- 5 times in Document A (100 words total) → TF = 5/100 = 0.05
- In 1,000 out of 50,000 total documents → IDF = log(50,000/1,000) = log(50) ≈ 3.91
- **TF-IDF** = 0.05 × 3.91 = **0.196**

Compare to "trump":

- 5 times in Document A → TF = 0.05  
- In 45,000 out of 50,000 documents → IDF = log(50,000/45,000) = log(1.11) ≈ 0.10
- **TF-IDF** = 0.05 × 0.10 = **0.005**

**Result:** "immigration" gets higher weight because it's more *distinctive*.

```{r}
# Apply TF-IDF weighting
dfm_tfidf <- dfm_tfidf(dfm_truth, scheme_tf = "prop")

# Now look at top features
topfeatures(dfm_tfidf, 20)
```

You'll now see more distinctive terms—words that characterize *specific* posts rather than generic vocabulary.

With TF–IDF, the most prominent features are no longer simply the most frequent words, but the terms that appear **often in some posts and rarely in others**.  
This is why emojis, expressive markers (“joy,” “pray,” “thank”), and politically charged terms (“trump,” “biden,” “#truth”) rise to the top: they are not used uniformly across the corpus, so they carry more discriminative weight.

TF–IDF down-weights very common words and up-weights words that help differentiate documents, making the resulting features more useful for tasks such as clustering, topic modeling, and similarity analysis.


**When to use TF–IDF**

- ✓ Information retrieval (e.g., search ranking)
- ✓ Document similarity
- ✓ Clustering or grouping documents
- ✓ Finding distinctive terms or keywords
- ✗ When an algorithm requires raw word counts
- ✗ When rare words are not meaningful for the task

**Note:** 

TF–IDF is great for highlighting distinctive vocabulary, but some text analysis methods are based on **count data** and need the original document–feature matrix.  
We will see examples of those later in the course.


---

## Part 4: Dimensionality Reduction—Finding Structure

Our DFM has ~25,000 features (columns). Challenges:

1. **Computational**: Can't visualize 25,000 dimensions
2. **Statistical**: Curse of dimensionality (data is sparse)
3. **Interpretational**: Hard to understand 25,000 patterns

**Solution:** Reduce to 2-10 dimensions while preserving *most* of the variance.

### 4.1 Latent Semantic Analysis (LSA)—Understanding the Math

You've seen PCA (Principal Component Analysis). LSA is similar but uses **Singular Value Decomposition (SVD)** on text data.

**Concept: What is LSA actually doing?**

**Step 1:** Start with our DFM (documents × terms):

$$\mathbf{X}_{n \times m}$$

where n = documents, m = terms

**Step 2:** Apply SVD to decompose:

$$\mathbf{X} = \mathbf{U} \mathbf{\Sigma} \mathbf{V}^T$$

Where:

- $\mathbf{U}_{n \times k}$: **Document-topic matrix** (how much each document relates to each topic)
- $\mathbf{\Sigma}_{k \times k}$: **Singular values** (importance of each topic)
- $\mathbf{V}_{m \times k}$: **Term-topic matrix** (how much each term relates to each topic)
- k: Number of latent dimensions (we choose this, e.g., k=10)

**Intuition behind LSA:**

Imagine we have posts about:

- "Trump election fraud stolen votes"
- "Biden gas prices inflation economy"

Even if posts don't share exact words, LSA finds **latent concepts**:

- **Latent Topic 1**: Election integrity (high weights for: trump, election, fraud, stolen, vote)
- **Latent Topic 2**: Economic issues (high weights for: biden, gas, price, inflation, economi)

**Why it works:**

- Words that appear in similar contexts get grouped (distributional semantics)
- Reduces noise (keeping only k strongest patterns)
- Captures **synonymy** (car/automobile → same topic) and **polysemy** (bank: river/finance → different topics)

**Mathematical detail:**

The singular values in $\mathbf{\Sigma}$ represent the "strength" of each dimension:

$$\mathbf{\Sigma} = \begin{bmatrix} \sigma_1 & 0 & \cdots & 0 \\ 0 & \sigma_2 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \sigma_k \end{bmatrix}$$

Where $\sigma_1 > \sigma_2 > \cdots > \sigma_k$

Proportion of variance explained by dimension i:
$$\frac{\sigma_i^2}{\sum_{j=1}^{k} \sigma_j^2}$$

```{r}
# 1. Sample a manageable number of documents for LSA
# (2,000–5,000 is usually fine; adjust based on your machine)
dfm_small <- dfm_sample(dfm_tfidf, size = 3000)

# 2. Fit an LSA model with 10 latent dimensions
lsa_model <- textmodel_lsa(dfm_small, nd = 10)

# 3. Document embeddings (low-dimensional representation)
doc_embeddings <- lsa_model$docs

# 4. Term loadings (which words define each dimension)
term_loadings <- lsa_model$features

# 5. Variance explained by each dimension
variance_explained <- (lsa_model$sk^2) / sum(lsa_model$sk^2)
variance_explained

```

#### Why We Use a Sample for LSA

LSA (Latent Semantic Analysis) is based on a mathematical operation called the
singular value decomposition (SVD).  
Even when applied to a sparse document–feature matrix, SVD requires creating
very large intermediate matrices.

Our full DFM contains over **800,000 documents** and **138,000 features**, far
beyond what a typical computer can process for SVD. Attempting LSA on the full
matrix would require hundreds of gigabytes of memory.

To demonstrate the idea without overwhelming the computer, we apply LSA to a
**small random sample** of documents (e.g., 2,000–5,000 posts).  
This preserves the structure of the method while keeping the computation fast
and manageable.

Sampling in this way is standard practice when introducing dimensionality
reduction on large text corpora.


**Interpreting LSA dimensions:**

```{r}
# Top terms for each dimension
for (i in 1:5) {
  cat("\n--- Dimension", i, "---\n")
  top_terms <- sort(term_loadings[,i], decreasing = TRUE)[1:10]
  print(top_terms)
}
```

#### Interpreting LSA Dimensions

Each LSA dimension captures a pattern of terms that tend to appear together across the sampled posts.  
Unlike topic modeling, these dimensions are not “topics” in the semantic sense, but **directions of maximum variation** in the document–term matrix.

Because we used a small random subset of posts and tf–idf weights, the dimensions primarily reflect:

- distinctive hashtags,
- highly personalized vocabulary,
- emotionally expressive markers (emoji, “joy,” “pray”),
rather than broad political themes.

**What the sample dimensions suggest:**

- **Dimension 1**: Dominated by specialized hashtags and user handles  
  → reflects idiosyncratic or community-specific phrasing rather than a general theme.

- **Dimension 2**: Strong emoji/emotion axis (“emoji,” “joy,” “pray,” “fire,” “heart”)  
  → captures emotionally expressive or reaction-style posts.

- **Dimension 3**: Driven by specific user mentions and playful slang (“humpty-dumpti,” “lol”)  
  → likely reflects informal, humorous, or interpersonal content.

- **Dimension 4**: Mix of religious hashtags and expressive emoji  
  → indicates posts with religious framing combined with emotional markers.

- **Dimension 5**: A mixture of distinctive words (#fjbiden, “panicmod,” “exposematthew”)  
  → appears to capture targeted or niche discourse clusters.

**Critical insight:**  

The dimensions you see are *emergent mathematical structures*—they reflect whatever patterns dominate the sampled text.  
With a different sample or the full corpus, the dimensions would likely shift toward clearer thematic axes.

#### What “Variance Explained” Tells Us

Each LSA dimension captures a portion of the overall variation in the document–term matrix.  
The variance explained values show **how much of the structure in the data each dimension accounts for**.

Higher values mean:
- that dimension represents a strong underlying pattern,
- and the corresponding term combinations are influential in differentiating documents.

In our sample, the first few dimensions explain slightly more variance (around 10–12%), but the values are relatively close.  
This indicates that **the text does not cluster around a small set of dominant themes**; instead, many weak patterns coexist, which is typical for short, noisy social media posts.


**Visualizing documents in LSA space:**

```{r}
# Convert the first three LSA dimensions into a data frame
lsa_df <- as.data.frame(lsa_model$docs[, 1:3])
colnames(lsa_df) <- c("Dim1", "Dim2", "Dim3")

# Extract docvars from the sampled dfm (dfm_small)
lsa_df$high_follower <- docvars(dfm_small, "high_follower")
lsa_df$engagement <- docvars(dfm_small, "engagement")

# 3D visualization
plot_ly(lsa_df,
        x = ~Dim1, y = ~Dim2, z = ~Dim3,
        color = ~high_follower,
        size = ~engagement,
        type = "scatter3d",
        mode = "markers") %>%
  layout(title = "Truth Social Posts in Latent Semantic Space")

```

Short social media posts often produce extremely sparse matrices.  
When we apply tf–idf and then sample documents, many rows may contain only a
few nonzero features. In such cases, LSA has very little structure to learn, 
and the resulting document embeddings can collapse into a small cluster of
points. 

This illustrates an important practical lesson: dimensionality reduction 
methods require enough informative variation in the data to produce meaningful 
latent dimensions.

:::callout-note

### A potential fix: trimming (i.e removing low frequency/weight terms) before sampling

```r
# Trim vocabulary so tiny features don't dominate
dfm_trimmed <- dfm_trim(dfm_truth, min_docfreq = 5)

# Sample documents
dfm_small <- dfm_sample(dfm_trimmed, size = 3000)

# Apply LSA to counts (recommended)
lsa_model <- textmodel_lsa(dfm_small, nd = 10)
```

Then visualise.

:::

#### Limitations of LSA

1. **Requires enough signal in the data**  
   With very short or sparse texts (common in social media), LSA may struggle to find meaningful structure, producing dimensions that collapse or are hard to interpret.

2. **Linear assumptions**  
   LSA assumes that underlying themes combine linearly in a high-dimensional space, which may not reflect how language actually works.

3. **Interpretability challenges**  
   Each dimension mixes positive and negative loadings, and may emphasize highly distinctive words (hashtags, usernames) rather than coherent themes.

4. **No probabilistic interpretation**  
   LSA cannot express results as "Document X is 60% about theme A," unlike probabilistic topic models.

5. **Ignores word order and context**  
   “good” and “not good” contribute similarly, and semantic nuances are lost.

These limitations, especially the difficulty of interpreting dimensions in short social-media texts, motivate the use of **probabilistic topic models** like LDA, which are designed to infer coherent latent themes from sparse text corpora.


---

## Part 5: Topic Modeling—Discovering Themes

### 5.1 Latent Dirichlet Allocation (LDA)—A Probabilistic Approach

Instead of k-means, we will use another unsupervised technique called topic modelling. As the name implies, this clustering technique is more suitable for text data. The most popular of such families of models is called **Latent Dirichlet Allocation (LDA)**. We will not go into too much details about how it works, but you can read more about it [here](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation).

Find out more about LDA in [here](https://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-allocation/).

#### Concept: The LDA Generative Story

LDA imagines that each post is produced by mixing several latent topics.  
A “topic” is simply a probability distribution over words.

For each document, LDA assumes the following process:

1. **Choose a mixture of topics for the document**  
   
   Example: 60% “election,” 30% “economy,” 10% “media.”  
   This mixture varies across documents.

2. **For each word position in the document:**  
   
   - Pick a topic according to the document’s topic mixture  
   - Pick a word from that topic’s word distribution  

In other words, topics generate words, and documents generate topic proportions.

---

#### Mathematical Formulation (K topics)

- Each **document $d$** has a topic distribution  
  $$
  \theta_d \sim \text{Dirichlet}(\alpha)
  $$
  where $\theta_d$ is a vector of topic proportions (sums to 1).

- Each **topic $k$** has a word distribution  
  
  $$
  \phi_k \sim \text{Dirichlet}(\beta)
  $$

  where $\phi_k$ gives the probability of each word under that topic.

To generate each word:

1. Choose a **topic** for this word:

$$
z_{d,i} \sim \text{Categorical}(\theta_d)
$$

This means: pick one topic according to the document’s topic proportions  
(e.g., if the document is 60% “Election”, that topic is more likely).

2. Choose a **word** from that topic:

$$
w_{d,i} \sim \text{Categorical}(\phi_{z_{d,i}})
$$

This means: pick one word according to the topic’s word distribution  
(e.g., the “Election” topic might generate “vote”, “fraud”, “ballot” with high probability).


---

##### What LDA Actually Does (Inference)

We observe only the words \( w_{d,i} \).  
LDA “inverts” the generative process to infer:

- the **topic mixture for each document** (\( \theta_d \))  
- the **word distribution for each topic** (\( \phi_k \))  
- the **topic assignment of each word** (\( z_{d,i} \))

This is why LDA is called a **latent variable model**: the structure we care about is hidden and must be inferred.

---

#### Key Difference From LSA

- **LSA**

  - Uses linear algebra (SVD)
  - Produces continuous dimensions
  - Dimensions can mix positive/negative weights
  - Harder to interpret semantically

- **LDA**

  - Fully probabilistic model
  - Topics are discrete word distributions (easy to interpret)
  - Documents are mixtures of topics (e.g., 40% topic A, 25% topic B)
  - Designed to model the way text might realistically be generated

This makes LDA much more interpretable for thematic analysis, especially on large text corpora.


:::callout-important
### LDA cannot be fit on a TF–IDF matrix — it requires **raw word counts**.

LDA is a **generative probability model**.  
It assumes documents are created by repeatedly *counting* word draws from topic distributions.

If you replace counts with TF–IDF weights, the generative assumptions are violated and  
LDA cannot interpret the input as valid token frequencies.

**Always use a count-based DFM for LDA.**
:::


```{r}

# This is essential — LDA needs stopwords for co-occurrence structure.
tokens_lda <- tokens(
  corp_truth,
  remove_punct   = TRUE,
  remove_symbols = TRUE,
  remove_numbers = FALSE,
  remove_url     = TRUE
)

dfm_lda_raw <- dfm(tokens_lda)
dim(dfm_lda_raw)

set.seed(42)

set.seed(42)

# 1. Sample documents
sample_ids <- sample(seq_len(ndoc(dfm_lda_raw)), 15000)

dfm_lda <- dfm_lda_raw[sample_ids, ]

# 2. Remove empty documents (probably none)
dfm_lda <- dfm_lda[rowSums(dfm_lda) > 0, ]

# 3. Fit LDA using Gibbs sampling
lda_model <- LDA(
  dfm_lda, k = 8,
  method = "Gibbs",
  control = list(
    seed   = 42,
    burnin = 50,
    iter   = 500,
    thin   = 20
  )
)

# 4. Inspect topics
terms(lda_model, 15)


```

:::callout-tip

After trimming rare terms, some social media posts become empty (no remaining
tokens) as posts tend to be short. Since LDA requires each document to contain at least one word, we must
remove empty rows before fitting the model.

:::

**Interpreting LDA output:**


LDA identifies clusters of words that tend to appear together across documents. Because Truth Social posts are extremely short (often fewer than 10 words), many topics are shaped not only by content words but also by high-frequency function words such as *you*, *in*, *that*, and *we*. This is normal for social-media corpora: stopwords provide the connective structure that allows the model to detect co-occurrence patterns, and removing them would cause topics to collapse.

Despite the short and noisy nature of the posts, the model uncovers several meaningful themes. Some topics represent **conversational or personal language**, others reflect **political discourse**, and one topic is dominated by **emotional expression using emojis**—a characteristic feature of posts on this platform. Several topics are clearly associated with discussions of **Biden**, **Trump**, **the election**, or broader national issues, while others capture **collective mobilization language** (“we”, “they”, “people”) or **personal commentary**.

Overall, the model illustrates a crucial point for students:

> *Even with short texts, LDA can reveal broad thematic patterns—provided stopwords are retained and a sufficiently large sample is used.*


#### LDA Topics Identified in Truth Social Posts

| Topic | Interpretation / Theme | Example Top Words |
|-------|-------------------------|-------------------|
| **1. Conversational / Direct Address** | Informal replies, personal engagement, rhetorical questions | you, your, what, just, like, know, right |
| **2. Temporal & Locational Markers** | Describing events, timing, context, movement | in, on, at, from, after, today, years |
| **3. Political Commentary (Biden / National Issues)** | Discussion of Biden, America, governance, national identity | biden, america, country, joe, world, all |
| **4. Affective / Emoji-Driven Expression** | Emotional reactions, prayers, memes, humor, platform actions | emoji, joy, pray, fire, #truth, clap, boom |
| **5. Narrative / Evaluation of Actors** | Commentary, criticism, speculation, descriptions of events or people | he, was, not, would, about, no, been |
| **6. Personal Voice & Sentiment** | First-person opinions, feelings, and personal reflections | i, my, me, don’t, truth, i’m, more |
| **7. Collective / Group Identity Language** | “We” vs. “they” framing, mobilization, appeals to community | we, they, people, their, will, have, get |
| **8. Trump & Electoral Politics** | Trump-centered discourse, election commentary, political news | trump, president, election, news, now |

> These topics provide an interpretable map of public discourse on Truth Social, illustrating how LDA uncovers structure even in short, informal, and emoji-rich posts.

**Examining document–topic distributions**

LDA gives each document a **topic mixture**—a probability for each topic.
These values tell us *how strongly* a post belongs to each learned theme.

```{r}
# Get topic proportions for each document in the LDA sample
doc_topics <- posterior(lda_model)$topics

# Example: Topic distribution for document #100
doc_topics[100, ]
```

This post is primarily about topic 5 i.e the narrative / evaluation of actors topic.

Next, we identify the **dominant topic** (the topic with the highest probability).

```{r}
# Add dominant topic back to the sampled documents
sampled_docs <- data.frame(
  doc_id = rownames(doc_topics),
  dominant_topic = topics(lda_model, 1)
)
```

Now we merge this back into the metadata from our sample:

```{r}
# dfm_lda_raw already contains all original metadata
# sample_ids was used to create the LDA sample
dfm_lda <- dfm_lda_raw[sample_ids, ]

# Fit the LDA model (already done)
# lda_model <- LDA(dfm_lda, ...)

# Build topic-labeled metadata table
topic_df <- cbind(
  as.data.frame(docvars(dfm_lda)),
  dominant_topic = topics(lda_model, 1)
)

topic_df <- tibble::as_tibble(topic_df)

head(topic_df)

```

---

#### **Analytical questions using LDA**

These examples show how topic models enable deeper analysis.

---

**1. Temporal evolution: Do certain themes surge over time?**

```{r}
topic_df <- topic_df %>%
  mutate(post_date = lubridate::as_datetime(post_date))

topic_df %>%
  mutate(week = lubridate::floor_date(post_date, "week")) %>%
  count(week, dominant_topic) %>%
  ggplot(aes(x = week, y = n, color = factor(dominant_topic))) +
  geom_line() +
  labs(title = "Topic Prevalence Over Time (Weekly Aggregated)",
       x = "Week", y = "Number of Posts",
       color = "Topic")
```

---

**2. User specialization: Do high-follower accounts behave differently?**

```{r}
topic_df %>%
  group_by(high_follower, dominant_topic) %>%
  summarize(n = n()) %>%
  ggplot(aes(x = factor(dominant_topic), y = n, fill = high_follower)) +
  geom_col(position = "fill") +
  labs(title = "Topic Preferences by User Influence",
       x = "Topic", y = "Proportion of Posts")
```

---

**3. Engagement by topic: Which topics attract more interaction?**

```{r}
topic_df %>%
  group_by(dominant_topic) %>%
  summarize(mean_engagement = mean(like_count + reply_count + retruth_count)) %>%
  arrange(desc(mean_engagement))
```



Topic 8 (Trump/election) tends to generate more reactions, followed by Topic 3 which also deals with polictical topics.


#### Choosing K: How many topics?

**Perplexity and Coherence: What Do They Mean in Topic Modeling?**

When choosing the number of topics **K** for LDA, we need metrics that tell us whether the model is capturing real structure in the data.

Below are the two most widely used metrics:

1. Perplexity

*Definition (intuitive):*

Perplexity measures **how surprised the model is** when predicting unseen documents.
It is essentially the inverse probability of the test data, normalized by length.

👉 **Lower perplexity → better generalization → less “surprise.”**

**Why it matters:**

A model with too few topics cannot represent documents well → **high perplexity**.
A model with too many topics overfits tiny patterns → perplexity may drop only minimally.

**Math (optional, simple form):**

For held-out text:

$$
\text{Perplexity} = \exp\left(-\frac{1}{N} \sum_{i=1}^N \log p(w_i)\right)
$$

* $N$: number of words
* $p(w_i)$: probability assigned by the model

**Interpretation:**

* A perplexity of 200 means:
  “On average, the model is as uncertain as choosing among 200 equally likely words.”
* Lower values mean the model predicts real text better.

**Limitation:**

Perplexity correlates poorly with *interpretability*.
A model with excellent perplexity may produce topics humans cannot label.

---

2. Topic Coherence

**Definition (intuitive):**

Coherence measures **how strongly the top words in a topic tend to occur together** in actual documents.

👉 **Higher coherence → more semantically meaningful topics.**

**Why it matters:**

A topic containing words like:

```
election, vote, ballot, fraud, recount
```

has **high coherence** → words co-occur frequently.

A topic containing:

```
joy, border, thank, mail, bigly
```

has **low coherence** → words rarely co-occur → topic makes no sense.

**How it's calculated (conceptually):**

For the top words in a topic:

* count how often each pair appears in the same documents
* combine these into a score (PMI, UMass, or UCI coherence)

Example formula for a pair (simplified UMass coherence):

$$
C = \sum_{i<j} \log \frac{D(w_i, w_j) + 1}{D(w_j)}
$$

* $D(w_i, w_j)$: number of documents containing both words
* $D(w_j)$: number of docs containing word (w_j)

**Interpretation**

* Coherence evaluates **semantic quality**, not predictive accuracy.
* It aligns better with **human judgment** than perplexity does.

---

**Why we need both**

| Metric               | Measures                      | Good for               | Bad for             |
| -------------------- | ----------------------------- | ---------------------- | ------------------- |
| **Perplexity**       | Predictive fit to unseen data | Statistical accuracy   | Meaningfulness      |
| **Coherence**        | Word co-occurrence strength   | Topic interpretability | Predictive modeling |
| **Interpretability** | Human ability to label topics | Teaching, reporting    | Subjective          |

### Combined guideline:

* Use **perplexity** to avoid under/overfitting.
* Use **coherence** to ensure topics represent real themes.
* Use **interpretability** as the final decision:
  *“Can you name the topic?”*

This is the standard model selection approach taught in text mining, NLP, and computational social science.


Here is how coherence comparison works:

```{r}
library(quanteda.textstats)

# Compute coherence for different K
k_values <- 2:15
coherence_values <- numeric(length(k_values))

for (i in seq_along(k_values)) {
  k <- k_values[i]
  m <- textmodel_lda(dfm_lda, k = k)
  coh <- textstat_coherence(m, dfm_lda)
  coherence_values[i] <- mean(coh$coherence)
}

# Plot coherence
ggplot(data.frame(k = k_values, coherence = coherence_values),
       aes(x = k, y = coherence)) +
  geom_line() +
  geom_point() +
  labs(
    x = "Number of Topics (K)",
    y = "Mean Coherence",
    title = "Topic Coherence Across Different Numbers of Topics"
  ) +
  theme_minimal()

```

**Guideline:**
You're trying to find the maximum of the curve here, but **interpretability always wins**. And in this particular case, coherence doesn't seem to be informative!

For our dataset, **8 topics** is reasonable and yields interpretable themes.

---

#### LDA Limitations

1. **Bag-of-words assumption:** Word order is ignored (“not good” ≈ “good”).
2. **Fixed K:** You must choose the number of topics ahead of time.
3. **No hierarchy:** Topics exist side-by-side, not in a tree.
4. **Exchangeability:** LDA ignores the order of documents (time, threads).
5. **Struggles with very short texts:** Needs careful preprocessing.

These motivate more modern topic models such as **BERTopic**.

### 5.2 BERTopic—Modern Deep Learning Approach

**Concept: Why BERTopic is Different**

Traditional methods (LSA, LDA) use **bag-of-words**: "not good" and "good" are treated similarly.

**BERTopic** uses **transformer embeddings** (from BERT) that understand context:

- "not good" → negative semantic vector
- "good" → positive semantic vector
- These vectors are far apart in embedding space!

**The BERTopic Pipeline:**

1. **Embed documents** using pre-trained transformers (BERT, RoBERTa, etc.)

   - Each document → 768-dimensional dense vector
   - Vectors capture semantic meaning

2. **Reduce dimensionality** using UMAP (Uniform Manifold Approximation and Projection)

   - 768 dimensions → 5-10 dimensions
   - Preserves local structure (similar posts stay close)

3. **Cluster** using HDBSCAN (Hierarchical Density-Based Spatial Clustering)

   - Finds dense regions in embedding space
   - Automatically determines number of topics
   - Identifies outliers (posts that don't fit any topic)

4. **Generate topic representations** using c-TF-IDF
   - Like TF-IDF but at cluster level
   - Most representative words for each cluster/topic

**Key advantages over LDA:**

| Feature | LDA | BERTopic |
|---------|-----|----------|
| **Embedding** | Bag-of-words counts | Contextualized BERT vectors |
| **Understands context** | No ("not good" = "good") | Yes (semantically different) |
| **Number of topics** | Must specify K | Automatically determined |
| **Outlier handling** | Forces all docs into topics | Separate outlier cluster (-1) |
| **Hierarchical structure** | No | Yes (topics can merge) |
| **Topic coherence** | Moderate | Generally higher |
| **Computation** | Fast | Slower (needs GPU for large corpora) |

🏡 You'll see how to implement BERTopic in the lab this week. Try it out on the Truth Social dataset after that

**Analytical insights you might be interested in:**

- Which topics surge after specific events (Jan 6, midterm elections)?
- Are new topics emerging or old ones fading?
- Can we predict engagement based on topic?

**When to use BERTopic vs LDA:**

Choose **BERTopic** when:
- Context matters (sentiment, negation, nuance)
- You don't know how many topics to expect
- You have computational resources (GPU helpful)
- You want state-of-the-art topic coherence

Choose **LDA** when:
- You need speed (millions of documents)
- Interpretability via probabilities is crucial
- You have theoretical K in mind
- Working with limited computational resources

**BERTopic Limitations:**
1. **Computational cost**: Slow on very large datasets without GPU
2. **Black box embeddings**: Can't easily see why posts are similar
3. **Non-deterministic**: Different runs may yield different topics (use `seed` parameter)
4. **Requires sufficient data**: Needs enough posts per topic (min_topic_size parameter)

---

## Part 6: Keyness Analysis—What Makes Groups Different?

### 6.1 Comparing Vocabularies Across Groups

**Analytical question:** What language distinguishes high-follower users from low-follower users on Truth Social? Or posts with high engagement vs low engagement?

**Concept: Keyness**

**Keyness** measures how much more (or less) frequently a word appears in a *target* group compared to a *reference* group.

It answers: "What words are *characteristic* or *distinctive* of this group?"

**Why this matters:**
- Understand group identities and messaging strategies
- Detect coordinated campaigns (if verified users overuse specific terms)
- Identify polarization markers (us vs. them language)

### 6.2 Keyness Statistics

Several measures of keyness exist:

**1. Log-likelihood ratio (G²):**

- Statistical test of independence
- Asks: "Is the word frequency significantly different between groups?"
- Higher values = more "key" to the target group

**2. Chi-squared (χ²):**

- Another statistical independence test
- Similar interpretation to log-likelihood

**3. Log ratio (simple ratio):**

- Ratio of proportions: $\log\left(\frac{p_{\text{target}}}{p_{\text{reference}}}\right)$
- Positive = more frequent in target
- Negative = more frequent in reference
- Intuitive but doesn't account for statistical significance

```{r}
# Ensure grouping variable exists
docvars(dfm_truth, "high_follower") <- docvars(corp_truth, "high_follower")

# Group DFM by high vs low follower users
dfm_groups <- dfm_truth %>%
  dfm_group(groups = high_follower)

# Compute keyness (TRUE = high followers)
keyness_hf <- textstat_keyness(dfm_groups,
                               target = "TRUE",
                               measure = "lr")   # Log ratio

# Top distinctive words for high-follower users
head(keyness_hf, 20)

# Top distinctive words for low-follower users (negative end)
tail(keyness_hf, 20)

# Visualization
textplot_keyness(
  keyness_hf,
  n = 20,
  color = c("#17c3b2", "#C63C4A")
)

```



#### Interpreting Keyness: High-follower vs. Low-follower Users

The keyness plot compares the vocabulary of high-follower and low-follower users by showing which words are *statistically distinctive* for each group. Positive bars (turquoise) represent terms disproportionately used by verified accounts; negative bars (red) represent terms distinctive to unverified accounts.

##### **What stands out**

**1. High-follower users**

The most distinctive features include:

* **Tracking parameters and metadata** (e.g., `utm_source`, `ftag`, `cad-01-1`)
* **Branded or campaign-related emoji** (`purple_heart`, `red_circle`, `first_place_med`)
* **User mentions typical of organized groups** (`@dory`, `@godwin`, etc.)

**Interpretation:**

High-follower accounts appear to use **more structured, coordinated, or professionally managed language**. The presence of tracking codes and branded emoji suggests automated posting tools, cross-platform promotion, or campaign/organizational messaging.

---

**2. Lower-follower users**

Distinctive terms include:

* **Emotion-heavy vocabulary** (`rage`, `pray`, `hankey`, `face_vomit`)
* **Everyday material references** (`amazon`, `shirt`, `tangerine`)
* **General expressive slang and emoji**

**Interpretation:**
Low-follower users tend to produce **more personal, emotional, and varied posts**. Their vocabulary reflects spontaneous expression rather than coordinated communication.

---

**Why this matters analytically**

Keyness reveals **stylistic differences** between user groups:

| Verified Users                                    | Unverified Users                       |
| ------------------------------------------------- | -------------------------------------- |
| Structured, branded, sometimes automated language | Personal, informal, emotional language |
| Frequent platform metadata                        | Topic- and mood-driven vocabulary      |
| Possible organizational messaging                 | Individual-level expression            |

These patterns can indicate *how different groups use the platform differently*—whether for outreach, mobilization, identity expression, or personal communication.

---

**Important caveats**

You should understand that:

* **Keyness ≠ importance.**
  A word is “key” only because it appears more often in one group, not because it is meaningful.

* **Sample size matters.**
  Verified users are relatively few; a handful of repetitive posts can strongly shape their keyness profile.

* **Artifacts can dominate.**
  Terms like `utm_source` reveal posting workflows, not ideological content.

* **Always contextualize results.**
  Keyness shows contrasts but requires domain knowledge to interpret responsibly.

---
 **Takeaway**

Keyness is a powerful exploratory tool: it highlights **how language differs between groups**, helping us identify stylistic patterns, organizational behaviors, and communicative norms. But it must always be interpreted with caution and supported by qualitative examination of real posts.


### 6.3 Critical Considerations with Keyness

**Statistical vs. Substantive Significance:**
- A word can be statistically "key" but appear rarely (low frequency)
- Always check both keyness score AND raw frequency
- High keyness + high frequency = truly distinctive

**Sample size matters:**
- Large groups will show more "significant" keyness even for small differences
- Consider effect size (log ratio) alongside p-values

**Stopwords redux:**
- If you kept stopwords, keyness might highlight "the", "is", "very"
- These can be stylistic markers (informal vs. formal) but less interesting substantively

---

## Part 7: Practical Considerations and Ethics

### 7.1 Choosing Your Preprocessing Pipeline

There's no universal "correct" preprocessing. Your choices depend on your **research question**:

**Example research questions and appropriate pipelines:**

| Research Question | Remove Stopwords? | Stem/Lemmatize? | N-grams? | Other |
|-------------------|-------------------|-----------------|----------|-------|
| What topics dominate Truth Social? | Yes | Stem | Unigrams + bigrams | Remove URLs, @mentions |
| Can we predict post engagement? | Maybe | Lemmatize | Unigrams + bigrams | Keep punctuation (! might indicate emotion) |
| Who is a bot vs. human? | No | No | Character n-grams | Keep everything (style matters) |
| Sentiment toward Biden? | No (keep "not") | Lemmatize | Trigrams (capture negation) | Keep punctuation |
| Network of user mentions? | N/A | N/A | N/A | Extract @mentions only |

**The preprocessing checklist:**

1. **Lowercase?** 
   
   - Usually yes (unless case matters: "US" vs "us")
   
2. **Punctuation?**
   
   - Remove for topic modeling
   - Keep for sentiment (!!!) or bot detection

3. **Numbers?**
   
   - Remove if not meaningful
   - Keep for temporal references ("2020 election", "Jan 6")

4. **Stopwords?**
   
   - Remove for topic/similarity (reduce noise)
   - Keep for authorship/style (they're discriminative)

5. **Stemming vs. Lemmatization vs. Neither?**
   
   - Stem: Speed, crude but acceptable topic modeling
   - Lemmatize: Accuracy, sentiment analysis, linguistic analysis
   - Neither: Exact matching matters (legal, compliance)

6. **N-grams?**
   
   - Bigrams almost always useful (named entities, phrases)
   - Trigrams+ if you have enough data and specific multi-word phrases matter

**Document your choices:**

Different preprocessing = different results. Always report your pipeline for reproducibility.

### 7.2 Evaluating Your Results

**How do you know if your analysis is "good"?**

**For topic models:**

1. **Quantitative metrics:**
   
   - Perplexity (LDA): Lower is better
   - Coherence score: Higher is better
   - Silhouette score (clustering): Higher is better

2. **Qualitative evaluation:**
   
   - Can you name each topic coherently?
   - Are topics substantively distinct?
   - Do they align with domain knowledge?

3. **Validation:**
   
   - Do topics correlate with metadata (time, user type)?
   - Can you predict something useful from topics (engagement)?
   - Do human annotators agree with topic assignments?

**For similarity:**

- Spot-check: Read high-similarity pairs—do they seem similar?
- Validation set: Hand-label some pairs, calculate precision/recall
- Downstream task: Does similarity improve recommendation accuracy?

**Red flags:**

- Topics that are just lists of random words (noisy corpus)
- All documents assigned to 1-2 topics (K too small or documents very similar)
- Topics that are mostly stopwords (preprocessing failed)
- Results that contradict domain knowledge (possible data quality issues)

### 7.3 Ethical Considerations with Truth Social Data

**Privacy:**

- Truth Social posts are public, but:
  - Users may not expect academic scrutiny
  - Identifying individuals in publications can cause harm
  - Even "public" data requires ethical consideration

**Best practices:**

- Aggregate analyses (group-level patterns, not individuals)
- Anonymize examples (don't quote with usernames)
- Consider: Would publication harm vulnerable users?

**Bias in data:**

- Truth Social ≠ representative of US population
- Self-selected conservative-leaning user base
- Inferences about "Americans" or "conservatives" overgeneralize

**Bias in methods:**

- Stopword lists reflect linguistic biases (English, formal register)
- Pre-trained embeddings (BERT) trained on different corpora
- Topic models find patterns—don't assume they reflect "truth"

**Responsible interpretation:**

- Topics ≠ what people "believe" (they might share skeptically)
- Frequency ≠ importance (loud minorities vs. silent majorities)
- Correlation ≠ causation (does Topic X *cause* engagement or vice versa?)

**Researcher positionality:**

- Your background shapes which patterns you notice/interpret
- Involve diverse perspectives in interpretation
- Acknowledge limitations and alternative interpretations

### 7.4 Reproducibility

Make your analysis reproducible:

```{r}
# Document software versions
sessionInfo()

# Set seeds for randomness
set.seed(42)

# Save preprocessing artifacts
saveRDS(dfm_truth, "dfm_truth_2024-12-03.rds")
saveRDS(lda_model, "lda_k8_2024-12-03.rds")

# Document decisions
# "We removed posts shorter than 10 characters as likely spam"
# "We chose K=8 based on coherence scores and interpretability"
```
---

## Conclusion: The Art and Science of Text Mining

Text mining is both **technical** and **interpretive**:

**Technical skills you've learned:**
- Representing text as structured data (corpus, tokens, DFM)
- Preprocessing pipelines (stopwords, stemming, n-grams)
- Dimensionality reduction (LSA, PCA-like thinking)
- Probabilistic topic models (LDA)
- Modern deep learning approaches (BERTopic)
- Comparative analysis (keyness)
- Similarity measures (cosine, Jaccard)

**Analytical skills you've developed:**
- Asking: What is this method actually doing mathematically?
- Understanding trade-offs (speed vs. accuracy, interpretability vs. performance)
- Choosing appropriate methods for research questions
- Validating results (quantitative + qualitative)
- Interpreting patterns critically (what do topics *mean*?)

**Key takeaways:**

1. **No single "right" way:** Preprocessing and methods depend on your goals
2. **Transparency matters:** Document choices and limitations
3. **Validate, validate, validate:** Quantitative metrics + human judgment
4. **Context is crucial:** Methods find patterns; you provide interpretation
5. **Ethics aren't optional:** Consider privacy, bias, and responsible use

**Moving forward:**

- **Practice:** Apply these methods to different corpora (news, reviews, transcripts)
- **Compare:** Run LDA and BERTopic on same data—how do results differ?
- **Combine:** Use topic models + supervised learning for applied problems
- **Critique:** Read papers and ask: What preprocessing did they do? How might that affect results?

Text mining opens up a world of unstructured data. The methods are powerful, but **your critical thinking** is what transforms patterns into insights.

---

## Further Resources

**Textbooks:**

- *Text Mining with R* by Silge & Robinson (tidy text approach)
- *Speech and Language Processing* by Jurafsky & Martin (comprehensive NLP)
- *Quantitative Text Analysis* by Grimmer, Roberts, Stewart (political science focus)

**Tutorials:**

- Quanteda tutorials: https://tutorials.quanteda.io/
- TidyText documentation: https://www.tidytextmining.com/
- BERTopic documentation: https://maartengr.github.io/BERTopic/

**Papers:**

- Blei et al. (2003): "Latent Dirichlet Allocation" (original LDA paper)
- Grootendorst (2022): "BERTopic: Neural topic modeling with a class-based TF-IDF procedure"
- Grimmer & Stewart (2013): "Text as Data: The Promise and Pitfalls of Automatic Content Analysis"
- - Vaswani et al. (2027): "Attention is all you need"

**Applied examples:**

- Political science: Ideology detection, policy analysis
- Marketing: Customer feedback analysis, brand perception
- Digital humanities: Literary analysis, historical text mining
- Public health: Social media health discourse, misinformation tracking

