---
title: "Week 11 Part 1: Text Mining Recap: Discovering Structure in Text"
subtitle: "Theme: Applications"
author: "Dr Ghita Berrada"
format:
  html:
    from: markdown+emoji
    code-fold: false
    toc: true
    theme: yeti
execute:
  warning: false
  message: false
---


```{r}
# Load required libraries

library(quanteda)              # text processing + DFM operations
library(quanteda.textstats)    # LSA + text statistics
library(quanteda.textmodels)   # models (e.g LSA)
library(quanteda.textplots)    # keyness plot
library(topicmodels)           # LDA
library(bertopicr)             # BERTopic
library(qs)                    # fast serialization for loading dfm
library(tidyverse)             # useful for data manipulation (optional)

```

Last week, we saw the bag-of-words approach to transform text into a numerical matrix (a Document-Feature Matrix, or DFM).

But this immediately raises a question:

> What problems arise when we try to analyze such a massive matrix directly?

# Introduction: The Problem of High-Dimensional Text (3 min)

Let's load the DFM from last week:

```{r}
# Load pre-saved DFM

dfm_truth <- qread("data/dfm_truth.qs")

dim(dfm_truth) # check DFM dimensions
sparsity(dfm_truth) #percentage of non-zeros values in DFM
```

## *From High-Dimensional Text to Structure: Why We Need LSA, LDA/BERTopic, and Keyness*

After preprocessing our Truth Social corpus, our DFM contains:

* **823,927 documents**
* **138,175 features (unique terms)**

This is an **extremely high-dimensional and sparse matrix**.

### **Problems with high dimensionality:**

1. **Computational**

   * Hard to visualize, cluster, or model tens of thousands of dimensions.

2. **Statistical (“curse of dimensionality”)**

   * Most entries are zero → sparsity prevents clear signal detection.

3. **Interpretational**

   * No human can make sense of 138,000 axes of variation.

### **Solution:**

We need techniques that extract **lower-dimensional structure**, identify **themes**, or highlight **discriminative words**.

Today we examine **four complementary strategies**:

1. **LSA** — dimensionality reduction (mathematical structure)
2. **LDA** — probabilistic topic modeling (generative structure)
3. **BERTopic** — contextual semantic topic modeling (neural structure)
4. **Keyness** — comparative lexical analysis (contrastive structure)

Each reflects a different philosophy of how structure in text should be defined and extracted.

---

Absolutely — here is a **tight, intuitive, example-driven recap** of the four methods.
Each one now includes a **1–2 sentence real example** (especially for LDA, as requested).

This fits naturally into a 2–3 minute verbal recap before deeper explanation.

---

# **Four Philosophical Approaches to Making Sense of Text** *(Recap)*

These are the four main strategies we’ve learned for discovering structure in high-dimensional text.
Each method rests on a **different idea of what “structure” means**.

---

## **1. LSA: “Find the mathematical directions of maximum variance”**

*(Dimensionality Reduction)*

**Assumption:**
Structure = **statistical variation** in the document–term matrix.

**Goal:**
Identify the axes (dimensions) along which documents differ the most.

**Quick example:**
If posts use the words *Trump, election, fraud* together and *Biden, inflation, gas* together, LSA might find a **left–right political axis** and a **policy vs. election axis**, even if the words don’t co-occur perfectly.

**Think:**
PCA for text — a way to *compress* the space, not interpret topics.

---

## **2. LDA: “Documents are mixtures of hidden topics”**

*(Probabilistic Topic Modeling)*

**Assumption:**
Each document is created by mixing a small number of **latent topics**, and each topic is a distribution of words.

**Goal:**
Infer the topic–word distributions and document–topic proportions.

**Quick example:**
For the post:

> *“Trump won the election. Biden’s economy is a disaster.”*
> LDA might infer something like:

* 60% “Election” topic
* 30% “Economy” topic
* 10% “Media/Commentary” topic

Even though the post mixes themes, LDA **untangles** them.

**Think:**
Reverse-engineering a hidden generative process.

---

## **3. BERTopic: “Cluster documents based on semantic embeddings”**

*(Contextual Semantic Topic Modeling)*

**Assumption:**
Meaning is captured by **contextual embeddings**, not raw word counts.

**Goal:**
Turn each document into a dense semantic vector and cluster them.

**Quick example:**
The posts

* “Gas prices are insane right now”
* “Fuel costs keep skyrocketing”
  use totally different words, but BERT embeddings place them **close together**, so BERTopic groups them in the same “Economy/Gas Prices” cluster.

**Think:**
Semantic similarity → dimensionality reduction → density clustering.

---

## **4. Keyness: “Find what *distinguishes* one group from another”**

*(Contrastive Lexical Analysis)*

**Assumption:**
Structure appears in **differences** between groups.

**Goal:**
Identify which words are disproportionately used by one category over another.

**Quick example:**
Comparing posts before vs. after an election, keyness might show:

* **Before:** “vote”, “polls”, “ballots”
* **After:** “fraud”, “results”, “counting”

One group’s most distinctive words tell us what defines it.

**Think:**
What words “pop out” statistically when you compare categories?


---

Great — here is the **fully rebuilt LSA recap section**, integrating:

* A clear conceptual foundation
* An intuitive SVD explanation (with Σ, U, V interpreted plainly)
* A small numerical example
* Explanation of why TF-IDF is needed
* Sampling logic for large corpora
* Working LSA code for your DFM
* Code for interpreting latent dimensions
* Guidance on choosing (k)
* What LSA is actually useful for

This version is intentionally **detailed yet accessible**, suitable for students reviewing on their own *and* for use during lecture.

---

# Method 1: LSA — The Linear Algebra Approach (Recap)

## 1. Why LSA Exists

Our Truth Social DFM is extremely large:

* **823,927 documents**
* **138,175 features**

We cannot visualize or reason about this space directly.

But real text often varies along only a few important **hidden patterns** (latent dimensions):

* degree of political content
* election vs. economy themes
* Trump-oriented vs. Biden-oriented language

**LSA is a technique designed to uncover these hidden dimensions.**

---

## 2. Intuition: What Is a Latent Dimension?

A **latent dimension** is a pattern in how words tend to co-occur across documents.

Example:

Posts about:

* “Trump election fraud stolen votes”
* “Biden gas prices inflation economy”

share no vocabulary but still reflect:

* **Dimension 1: political intensity**
* **Dimension 2: election vs. economy**

LSA discovers these patterns directly from the document–term matrix.

You, the human, assign meaning *after* seeing which words and documents load on each dimension.

---

## 3. A Simple Numerical Example

Consider this tiny document-term matrix:

```
        trump  biden  election  economy
Doc1      3      0       2         0
Doc2      0      2       0         3
Doc3      1      1       1         1
```

What structure do we see?

* **Doc1** → election-related
* **Doc2** → economy-related
* **Doc3** → mixture of both

We would like to automatically discover two dimensions:

1. A **general political dimension**
2. A **contrast dimension**: election vs economy

This is exactly what LSA does on a massive scale.

---

## 4. The SVD Decomposition 

LSA applies **Singular Value Decomposition (SVD)** to the matrix:

$$
X = U \Sigma V^T
$$

These letters look abstract, but each one has a **very concrete meaning for text**.

---

### **(1) Σ — importance of each latent dimension**

Σ contains the **singular values**, one per latent dimension.

* **Large singular value → important, real pattern in the corpus**
* **Small singular value → mostly noise**

Think of singular values as the “strength” or “clarity” of each discovered theme.

We keep only the top (k) strongest dimensions.

---

### **(2) V — which *words* define each dimension**

The columns of (V) tell us:

> Which words are most strongly associated with each latent dimension.

Example:

If dimension 2 separates election from economy:

* High-positive loadings: `trump, election, ballots, fraud`
* High-negative loadings: `biden, inflation, gas, prices`

This is how we interpret what a dimension *means*.

---

### **(3) U — how *documents* relate to each dimension**

The columns of (U) tell us:

> How strongly each document expresses each dimension.

For example:

* Doc1 scores high on the **election** dimension
* Doc2 scores high on the **economy** dimension
* Doc3 sits in the middle

This helps with **clustering, visualization, and similarity measurement**.

---

### Summary table of what SVD provides

| Matrix | Meaning                  | Intuition                             |
| ------ | ------------------------ | ------------------------------------- |
| **Σ**  | strength of each pattern | “How big is this theme?”              |
| **V**  | word loadings            | “What vocabulary defines this theme?” |
| **U**  | document loadings        | “Which docs express this theme?”      |

This is the core of how LSA makes sense of high-dimensional text.

---

## 5. Why We Apply TF–IDF First

Raw counts behave poorly:

* common words dominate variance
* long posts artificially dominate structure
* rare informative words are ignored

**TF–IDF downweights common words and upweights meaningful ones.**

This produces the right matrix for discovering latent patterns.

---

## 6. Practical Note: Use a Sample, Not the Full Dataset

Running SVD on an 823k × 138k matrix is computationally prohibitive.

For demonstration we sample:

```{r}
set.seed(123)
dfm_lsa_demo <- dfm_truth[sample(ndoc(dfm_truth), 5000), ]
```

This keeps everything conceptually correct but computationally feasible.

---

## 7. LSA Demo Code

```{r}
# Sample for demonstration
set.seed(123)

# Apply TF–IDF weighting
dfm_lsa_tfidf <- dfm_tfidf(dfm_lsa_demo)

# Fit LSA with 10 latent dimensions
lsa_model <- textmodel_lsa(dfm_lsa_tfidf, nd = 10)

# Variance explained by each dimension (singular values)
variance <- (lsa_model$sk^2) / sum(lsa_model$sk^2)
variance
```

```{r}
variance |>
  as_tibble(.name_repair = "unique") |>
  rename(variance_explained = value) |>
  mutate(
    dimension = row_number(),
    label = scales::number(variance_explained, accuracy = 0.001)
  ) |>
  ggplot(aes(dimension, variance_explained)) +
  geom_col(fill = "#4C72B0") +  # nicer blue color
  geom_text(
    aes(label = label),
    vjust = -0.4,
    size = 3.5
  ) +
  labs(
    title = "Variance Explained by LSA Dimensions",
    x = "Dimension",
    y = "Proportion of Variance"
  ) +
  scale_x_continuous(breaks = 1:length(variance)) +
  ylim(0, max(variance) * 1.15) +  # add padding so labels fit above bars
  theme_minimal()
```

This distribution suggests:

- Dim 1 is very strong (30% variance alone).
- Dims 2 & 3 still capture meaningful structure.
- Dims 4–10 are “medium strength” and not noise — but they’re less dominant.

How many dimensions would we choose in practice?

For dimensionality reduction / visualization:

👉 k = 2–3 (the dominant structural patterns)

For document embeddings / similarity / clustering:

👉 k = 10–50
(we have meaningful variance beyond 3 dimensions; social media corpora are multi-thematic)

For LSA demo in class:

👉 k = 10 is perfect
—it shows the tapering structure and gives interpretable dimensions.

---

## 8. How to Interpret the Dimensions (Code + Explanation)

### **Interpret the vocabulary defining each dimension**

Selecting the first dimension:

```{r}
dim1_df <- lsa_model$features |>
  as_tibble(rownames = "feature") |>
  select(feature, V1) |>
  filter(!is.na(V1)) # <- safety for malformed tokens
```

✔ Top positive-loading words for dimension 1

```{r}
dim1_df |> arrange(desc(V1)) |> slice_head(n = 20)
```

✔ Top negative-loading words for dimension 1
```{r}
dim1_df |> arrange(V1) |> slice_head(n = 20)

```

Visualising loadings
```{r}
plot_df <- bind_rows(
  dim1_df |> arrange(desc(V1)) |> slice_head(n = 10) |> mutate(direction = "Positive"),
  dim1_df |> arrange(V1) |> slice_head(n = 10) |> mutate(direction = "Negative")
)

ggplot(plot_df, aes(x = reorder(feature, V1), y = V1, fill = direction)) +
  geom_col() +
  coord_flip() +
  labs(
    title = "Top Positive and Negative Word Loadings for LSA Dimension 1",
    x = "Feature", y = "Loading"
  ) +
  theme_minimal()

```

#### ⭐ Interpreting LSA Dimension 1

The words with the **largest absolute loadings** (ignoring the sign) tell us what a latent dimension is capturing.

For Dimension 1, the high-magnitude loadings are almost all:

* emotional words
* emoji-like tokens
* reaction indicators
* expressive social media vocabulary

So we can interpret **Dimension 1** as:

> A latent dimension of *expressiveness / emoji usage* across posts.

The **sign** (positive vs negative) does *not* have semantic meaning in SVD/LSA.
We could multiply the whole dimension by -1 and nothing would change.
What matters is the **magnitude** of the loading, not its sign.

On the other side of the dimension we mostly see **tiny coefficients** on rare tokens such as usernames or odd Unicode strings. These are not meaningful patterns. They are there to satisfy the mathematical constraints of the decomposition, not because they define a real linguistic contrast.

---

#### ⭐ How we could improve this if we wanted more meaningful LSA dimensions

Right now, Dimension 1 is dominated by emoji and reaction tokens because they are:

* relatively frequent in the sample
* high-variance (some posts use many, some use none)
* strongly co-occurring within the same kinds of posts

If we wanted LSA to pick up **more substantive patterns** (e.g. issues, topics, frames), we would need to clean the input before running TF–IDF + LSA. A more suitable workflow would be:

1. **Remove emojis, usernames and other non-word tokens as features**

   ```r
   dfm_clean <- dfm_select(
     dfm_truth,
     pattern = "^[a-z]+$",   # keep only alphabetic tokens
     selection = "keep"
   )
   ```

2. **Trim very rare words**

   ```r
   dfm_clean <- dfm_trim(dfm_clean, min_termfreq = 20)
   ```

   This removes one-off junk: typos, ultra-rare hashtags, random strings.

3. **Recompute TF–IDF and re-fit LSA**

   ```r
   dfm_lsa_tfidf <- dfm_tfidf(dfm_clean)
   lsa_model <- textmodel_lsa(dfm_lsa_tfidf, nd = 10)
   ```

4. **Re-extract the loadings for each dimension and interpret again**

   With a cleaned DFM, the leading dimensions are more likely to reflect:

   * political vs non-political vocabulary
   * issue-based contrasts (election vs economy, etc.)
   * sentiment or polarity
   * stylistic differences that are not just emoji usage

---

#### ⭐ Bottom line

The LSA result is **not wrong** – it is telling us something real about the data:

* In this 5,000-post TF–IDF-weighted sample, the **largest source of variation** is how much emoji/reactive vocabulary posts use.
* That’s not very helpful for **topic exploration**, but it *is* exactly the kind of high-variance pattern SVD is designed to find.

This is a useful lesson:

* how LSA works in practice,
* why it struggles with **short, sparse, noisy social media text**,
* why **preprocessing and trimming** matter a lot, and
* why methods like **LDA** and **BERTopic** are often better suited when the goal is to recover interpretable *themes* rather than stylistic dimensions.


---

## 9. How to Choose (k) (number of dimensions)

For LSA, choose (k) using:

### ✔ **Variance explained (singular values)**

Look for the **elbow** where additional dimensions add little new information.

### ✔ **Goal of analysis**

* For demos: (k = 5) to (10)
* For visualization: (k = 2) or (3)
* For similarity/clustering: (k = 50) to (300)

There is no single “correct” value — it depends on use-case.

---

## 10. What LSA Is Good For (and Not Good For)

### **LSA is great for:**

* reducing dimensionality
* creating document embeddings
* clustering documents
* computing similarity
* removing noise
* visualization and preprocessing for downstream ML

### **LSA is not great for:**

* producing interpretable topics
* handling extremely sparse social media data
* capturing semantics (use BERTopic)

---

## ⭐ Summary

LSA takes a massive document-term matrix and:

1. **Finds a few strong latent dimensions** (patterns)
2. **Shows which words define those dimensions** (V)
3. **Shows which documents express those dimensions** (U)
4. **Ranks dimensions by importance** (Σ)
5. **Transforms documents into a lower-dimensional space**
   where clustering and similarity become easier

It is a powerful tool for reducing complexity, even if the dimensions require human interpretation.

---

# **Method 2: Keyness — Identifying Distinctive Words Between Groups (5 min)**

Before we explore latent topics, it helps to understand **explicit differences** in word usage between groups of posts.
Keyness provides a simple, intuitive way to compare two groups:

> **Which words occur disproportionately in one group of documents compared to another?**

This method uses **raw counts**, so we can apply it directly to our
**5,000-document DFM**: `dfm_lsa_demo`.

---

## **Why Start With Keyness?**

* Fast, interpretable, and visual.
* Helps understand distributional differences before abstract models.
* Shows that even without topic modeling, **vocabulary structure already exists**.
* A motivating step toward LDA and BERTopic.

---

## **How Keyness Works**

`quanteda` compares two sets of documents:

* **Target group** → documents we want to analyze
* **Reference group** → all other documents

The goal: test whether each word is **equally frequent** in the two groups.

If not, the word is **key** to differentiating them.

---

## ⭐ The Log-Likelihood Ratio (G²): The Default and Correct Metric Choice Here

For each word, we build a 2×2 frequency table:

|                  | Target group  | Reference group | Total                           |
| ---------------- | ------------- | --------------- | ------------------------------- |
| **Word present** | $O_{1}$       | $O_{2}$         | $O_{1}+O_{2}$                   |
| **Word absent**  | $N_{1}-O_{1}$ | $N_{2}-O_{2}$  | $(N_{1}+N_{2}) - (O_{1}+O_{2})$ |
| **Total docs**   | $N_{1}$       | $N_{2}$         | $N_{1}+N_{2}$                   |

Where:

* $O_{1}$ = number of documents in the **target** group containing the word
* $O_{2}$ = number of documents in the **reference** group containing the word
* $N_{1}$, $N_{2}$ = total documents in target/reference groups

Expected counts if the word were equally distributed:

$$
E_{1} = \frac{(O_{1}+O_{2}), N_{1}}{N_{1}+N_{2}},
\qquad
E_{2} = \frac{(O_{1}+O_{2}), N_{2}}{N_{1}+N_{2}}
$$

The **log-likelihood ratio** is:

$$
G^{2} = 2 \left[
O_{1}\ln\left(\frac{O_{1}}{E_{1}}\right) +
O_{2}\ln\left(\frac{O_{2}}{E_{2}}\right)
\right]
$$

Interpretation:

* **Large positive G²** → strongly associated with the **target** group
* **Large negative G²** → strongly associated with the **reference** group
* **Near zero** → not distinctive

### Why G² is ideal for Truth Social text

* extremely sparse text
* many rare words
* highly skewed word frequency distributions

G² is robust in exactly these settings.

---

## ⭐ Other Keyness Metrics (and why we do *not* use them)

### **1. Chi-square ($\chi^2$)**

Formula:

$$
\chi^{2} =
\frac{(O_{1}-E_{1})^2}{E_{1}} +
\frac{(O_{2}-E_{2})^2}{E_{2}}
$$

Use when:

* expected frequencies ($E_{1}, E_{2} > 5$)
* corpora are large and dense
* word counts are not rare

Not appropriate here because **almost all expected counts < 1**.

---

### **2. Difference in proportions ($\Delta p$)**

$$
\Delta p =
\frac{O_{1}}{N_{1}} -
\frac{O_{2}}{N_{2}}
$$

Use for:

* simple effect size
* easy interpretation

Not appropriate alone because:

* no statistical significance
* very unstable for rare tokens
* misleading in sparse corpora

We may use it as a *supplement* in some contexts, but not as the main metric.

---

## **3. Log ratio (effect size; not significance)**

$$
\log\text{Ratio} =
\log_2 \left(
\frac{O_{1}/N_{1}}{O_{2}/N_{2}}
\right)
$$

Use when:

* comparing corpora of very different sizes
* wanting an intuitive “times more frequent” measure

Not used here because:

* unstable for rare words
* often used *after* G², not instead of it

---

### ✔ Therefore: **G² is the correct, robust choice for our dataset.**

---

## **Implementation (using `dfm_lsa_demo`)**

We create a grouping variable inside the DFM.
Example: documents mentioning “trump” vs. not.

### Load a sample of Truth Social posts (5000) and build a DFM for keyness

```{r}
# Load the sample of 5,000 posts
posts <- read_csv("data/truth_social_sample.csv")

# Build a corpus
corp <- corpus(posts, text_field = "text")

# Light preprocessing: remove URLs + punctuation, keep stopwords
toks <- tokens(
  corp,
  remove_punct = TRUE,
  remove_url = TRUE
)

dfm_key <- dfm(toks)

dfm_key
```

Keyness works best on:

- raw counts
- light preprocessing (remove punctuation, URLs, lowercase)
- keep stopwords (they help distinguish stylistic patterns)

### Compute Keyness Using Log-Likelihood Ratio (G²)

```{r}
keyness_lr <- textstat_keyness(
  dfm_key,
  target = docvars(dfm_key)$mention_trump == TRUE,
  measure = "lr"
)
```

### Visualize keyness

```{r}
textplot_keyness(keyness_lr, n = 20)
```

---

# **Interpreting a Keyness Plot**

A typical output shows:

* **Right side:** words characteristic of posts mentioning "trump"
* **Left side:** words characteristic of posts not mentioning "trump"
* **Bar length:** magnitude of the difference (G²)

This tells us:

* Vocabulary clusters around meaningful themes.
* Distinctive language emerges even without topic modeling.
* Social media discourse contains strong lexical signals.

---

## **Optional: Extracting Most Key Words Programmatically**

```{r}
keyness_lr %>%
  slice_max(G2, n = 20)

```

Words characteristic of the *reference* group:

```{r}
keyness_lr %>% slice_min(G2, n = 20)
```

---

## ⭐ "Which Keyness Statistic Should I Use?" — Cheat Sheet

| Statistic                                  | Measures                          | Best used when…                         | Not good for…               | Recommended here?          |
| ------------------------------------------ | --------------------------------- | --------------------------------------- | --------------------------- | -------------------------- |
| **Log-likelihood ratio (G²)**              | Significance                      | sparse corpora, rare words, short texts | N/A                         | ⭐ **Yes**                  |
| **Chi-square ((\chi^2))**                  | Significance                      | dense corpora, high expected counts     | sparse text; expected < 5   | ❌ No                       |
| **Difference in proportions ((\Delta p))** | Effect size                       | interpretability                        | rare words; no significance | ❌ No (supplement only)     |
| **Log ratio**                              | Effect size (“times more common”) | comparing corpora sizes                 | instability with rare words | ❌ No (optional supplement) |

**Bottom line:**
For short, sparse, social media text, **log-likelihood ratio (G²)** is the only reliable keyness statistic.


---

Perfect — here is a **merged, polished, fully student-facing LDA section** combining:

### **From Response 1**

✔ clearer section intro
✔ better conceptual foundation
✔ nicer “generative story”
✔ cleaner topic-number section structure
✔ clearer summary
✔ cleaner implementation section

### **From Response 2**

✔ stronger explanations of

* why stopwords
* why raw counts
* why Dirichlet
  ✔ fuller details in each rationale
  ✔ clearer intuition throughout

### **Plus improvements**

✔ symbols explained the first time they appear
✔ no meta-comments
✔ consistent voice (“we”)
✔ ready to paste into your recap file

---

# **Method 3: LDA — The Probabilistic Generative Model**

Latent Dirichlet Allocation (LDA) is built on a simple but powerful idea:

> Documents express **multiple underlying themes**, and each theme uses certain words more often than others.

LDA attempts to **reverse-engineer** those themes (topics) and how strongly each appears in each document.

---

## 🔶 **Conceptual Foundation**

LDA assumes each document is created through this process:

### **1. Each document draws a topic mixture**

A document chooses proportions for its topics:

$$
\theta_d = (\theta_{d1}, \theta_{d2}, \ldots, \theta_{dK})
$$

(where $\theta_d$ describes how much Topic 1, Topic 2, … contribute to that document)

### **2. Each topic has a word distribution**

Topic (k) has probabilities over words:

$$
\phi_k = (p(w_1 \mid k),, p(w_2 \mid k),, \ldots)
$$

### **3. To generate each word position in the document:**

1. pick a topic according to $\theta_d$
2. pick a word according to $\phi_k$

This explains why documents can contain multiple themes.

### **LDA works backwards**

We observe only the words. LDA tries to infer:

* **what topics exist**
* **which words define each topic**
* **how much each topic appears in each document**

---

## 🔶 **Why the Dirichlet Distribution Appears**

The Dirichlet distribution is a convenient way to generate **mixtures of probabilities**.

It controls how “focused” or “spread out” a document’s topic mixture is:

* **α < 1 → sparse mixture**
  documents focus on a *few* topics (ideal for short social-media posts)

* **α = 1 → neutral mixture**
  topics equally likely

* **α > 1 → diffuse mixture**
  documents spread weight across *many* topics

Short posts (like those on Truth Social) usually behave like **α < 1**:
they concentrate heavily on one theme, with smaller contributions from others.

---

## 🔶 **Why Stopwords Should Be **Kept** for LDA**

LDA relies on **patterns of word co-occurrence**.
Stopwords (in, on, to, you, what, do, is…) provide structure:

* they distinguish conversational from declarative writing
* they help separate rhetorical styles
* they form reliable co-occurrence scaffolding
* they prevent topics from collapsing into broad bags of content words

Removing stopwords often results in:

* generic "politics" topics
* harder-to-interpret clusters
* fewer stylistic distinctions

Keeping stopwords preserves contextual structure that LDA uses to separate topics.

However, a small set of extremely frequent function words (“the”, “a”, “to”, “of”, “in”, etc.) appear in almost every document and do not carry meaningful structure.

Removing only these hyper-common stopwords reduces the number of “pure stopword topics” while preserving linguistic information.

We also optionally include bigrams to help LDA capture short, meaningful phrases (e.g., “fake_news”, “deep_state”, “trump_won”).

---

## 🔶 **Why Raw Counts (Not TF–IDF)**

LDA’s generative story assumes that:

> Every word is drawn from a topic like a discrete count.

TF–IDF produces **weights** (e.g., 0.024) rather than counts (e.g., 3 occurrences).
Weights break the probabilistic assumptions of the model.

Therefore:

> For LDA: **use raw counts, not TF–IDF.**

This keeps the model consistent with its underlying mathematics.

---

## ⭐ **Choosing the Number of Topics (K)**

Choosing K is as much an **interpretive** decision as a **quantitative** one.

We combine several tools.

---

### 🔷 1. **Interpretability — the most important criterion**

A good topic:

* has top words forming a meaningful theme
* is distinct from other topics
* matches documents assigned heavily to it

If a topic cannot be named, it is not useful — even if it scores well on metrics.

---

### 🔷 2. **Topic Coherence — do the top words belong together?**

Coherence measures how often the top words in a topic appear **together** in documents.

High coherence → clean, thematic topics
Low coherence → vague, generic topics

Where coherence can be explored:

* `topicdoc::topic_coherence` documentation:
  [https://www.rdocumentation.org/packages/topicdoc/versions/0.1.1/topics/topic_coherence](https://www.rdocumentation.org/packages/topicdoc/versions/0.1.1/topics/topic_coherence)
* Last year's DS202A Week 10 lab (coherence, exclusivity, perplexity):
  [https://lse-dsi.github.io/DS202/2024-2025/autumn-term/weeks/week10/lab-updated-answers.html](https://lse-dsi.github.io/DS202/2024-2025/autumn-term/weeks/week10/lab-updated-answers.html)

---

### 🔷 3. **Exclusivity — do topics have their own distinctive words?**

High exclusivity:

* “ballot,” “absentee,” “precinct” belong to a voting topic
* and **not** to other topics

Low exclusivity:

* multiple topics share the same common vocabulary
* harder to distinguish them

Exclusivity complements coherence:
both are essential for clean, interpretable modeling.

Where exclusivity can be explored:

* Last year's DS202A Week 10 lab (coherence, exclusivity, perplexity):
  [https://lse-dsi.github.io/DS202/2024-2025/autumn-term/weeks/week10/lab-updated-answers.html](https://lse-dsi.github.io/DS202/2024-2025/autumn-term/weeks/week10/lab-updated-answers.html)

---

### 🔷 4. **Perplexity — predictive performance**

Perplexity measures:

> How well does the model predict unseen text?

Lower perplexity → better prediction
But:

* it often prefers **too many topics**,
* which can produce uninterpretable results

Perplexity is useful diagnostically,
but interpretability remains the core criterion.

Overview:
[https://info5940.infosci.cornell.edu/notes/text-analysis/topic-modeling/](https://info5940.infosci.cornell.edu/notes/text-analysis/topic-modeling/)

---

### 🔷 5. **Heuristic: K should be below √N**

A widely used rule of thumb:

$$
K \lesssim \sqrt{N}
$$

With **N = 5,000 sampled posts**:

$$
\sqrt{5000} \approx 70
$$

Thus:

* K = 8–12 → stable, interpretable
* K = 20–40 → possible, but more fragmented
* K > 70 → typically overfitting noise

This prevents extremely granular, unstable topics.

---

### 🔷 6. **Automated exploration with `ldatuning`**

This package helps compare multiple values of K:

```r
library(ldatuning)

results <- FindTopicsNumber(
  dfm_lda,
  topics = seq(5, 40, by = 5),
  metrics = c("Griffiths2004", "CaoJuan2009", "Arun2010", "Deveaud2014"),
  method = "Gibbs",
  control = list(seed = 42),
  verbose = TRUE
)

FindTopicsNumber_plot(results)
```

These diagnostics identify promising topic counts,
but interpretability determines the final choice.

---

## ⭐ **Implementation (on the 5,000-post sample)**

Below is the exact workflow used when running LDA.

---

### 🔶 **Step 1 — Build corpus → tokens → DFM**

```{r}

sample <- read_csv("data/truth_social_sample.csv")

corp_lda <- corpus(sample, text_field = "text")

# Minimal cleaning: remove URLs & punctuation, keep stopwords except hyper-frequent ones
tokens_lda <- tokens(
  posts$text,
  remove_punct = TRUE,
  remove_url = TRUE
)

# Remove ONLY extremely frequent stopwords (hyper-common function words)
ultra_stop <- c("the","a","an","to","of","in","and","for","on","is","it")

tokens_lda <- tokens_remove(tokens_lda, ultra_stop)

# OPTIONAL: add bigrams
tokens_lda <- tokens_ngrams(tokens_lda, n = 1:2)

# Build DFM
dfm_lda <- dfm(tokens_lda)

# Remove empty docs
dfm_lda <- dfm_lda[rowSums(dfm_lda) > 0, ]
```

---

### 🔶 **Step 2 — Fit the LDA model (K = 10)**

```{r}
set.seed(42)

lda_model <- LDA(
  dfm_lda,
  k = 10,
  method = "Gibbs",
  control = list(
    iter = 500,
    burnin = 50,
    thin = 20,
    seed = 42
  )
)
```

---

### 🔶 **Step 3 — Inspect results**

#### Top words per topic

```{r}
terms(lda_model, 10)
```

#### Topic mixture for each document

```{r}
doc_topics <- posterior(lda_model)$topics
head(doc_topics)
```

---

## ⭐ **What Topics Look Like in This Corpus**

The topics produced by LDA are not always purely “themes.”
With short social-media posts, topics often reflect:

* **style** (emoji use, punctuation, questioning)
* **interactional language** (“I”, “you”, “we”)
* **narrative reporting** (“he”, “his”, “was”)
* **true thematic clusters** (Biden, election, Trump)

This is because LDA groups words based on **co-occurrence**, not meaning.

Below we interpret each topic using the actual top terms from your LDA model:

---

### 🔶 Topic 1 — Emoji / Expressive Reactions

**Key words:** `<`, `emoji`, `<_emoji`, `>_<`, `us`, `emoji_us`, `joy`, `emoji_joy`

This topic is dominated by emoji tokens and expressive symbols.

**Interpretation:**

> Posts that are highly emotional, reaction-oriented, or meme-like.
> The strongest stylistic signal in the corpus.

---

### 🔶 Topic 2 — Biden / Presidential Politics + News Links

**Key words:** `biden`, `trump`, `president`, `joe`, `please`, `today`, `utm_source`

Mixes two elements:

* vocabulary about Biden/presidency
* URL tracking identifiers (`utm_source`) from shared news articles

**Interpretation:**

> Posts discussing or sharing political news, mostly about Biden and national governance.

---

### 🔶 Topic 3 — Masculine / Third-Person Narrative

**Key words:** `he`, `has`, `his`, `him`, `with`, `was`, `been`, `years`

**Interpretation:**

> Posts describing what someone (often a male political figure) did, said, or represents.
> A narrative, reporting-style topic.

---

### 🔶 Topic 4 — Evaluations & Reactions (Non-Emoji)

**Key words:** `up`, `good`, `not`, `most`, `too`, `everyone`, `#truthtruths`

**Interpretation:**

> Posts expressing evaluations (“good/not”), reactions to events, or general commentary.

---

### 🔶 Topic 5 — General Semantic / Background Language

**Key words:** `they`, `that`, `what`, `about`, `truth`, `would`, `she`

A general-purpose topic that appears because posts are short.

**Interpretation:**

> Background linguistic scaffolding shared across many posts.

---

### 🔶 Topic 6 — Collective Identity & Social Messaging

**Key words:** `are`, `we`, `our`, `people`, `will`, `all`, `as`, `no`

**Interpretation:**

> Posts expressing a collective viewpoint, group identity, or calls to action.
> Common in political communities.

---

### 🔶 Topic 7 — Election-Related Discourse

**Key words:** `election`, `was`, `who`, `again`, `never`, `many`, `big`

**Interpretation:**

> Posts discussing election narratives — often emotionally or rhetorically framed.

---

### 🔶 Topic 8 — Movement / Direction / Structural Words

**Key words:** `from`, `out`, `over`, `stop`, `first`, `left`, `put`

**Interpretation:**

> Posts describing changes, transitions, or directional actions.
> Another narrative-style topic.

---

### 🔶 Topic 9 — Commentary & Questioning

**Key words:** `just`, `should`, `news`, `new`, `why`, `make`, `now`, `other`

**Interpretation:**

> Opinionated or questioning posts reacting to news or events.

---

### 🔶 Topic 10 — Personal Address / Informal Interaction

**Key words:** `i`, `you`, `my`, `your`, `like`, `know`, `can`, `do`

**Interpretation:**

> Conversational posts — replies, personal reactions, arguments, and informal engagement.

---

### ⭐ **Summary Table**

| Topic | Interpretation                  | Representative Words         |
| ----- | ------------------------------- | ---------------------------- |
| 1     | Emoji / expressive style        | emoji, <, joy                |
| 2     | Biden + political news sharing  | biden, president, utm_source |
| 3     | Third-person narrative          | he, his, was                 |
| 4     | Evaluations & reactions         | good, not, everyone          |
| 5     | Background/semantic scaffolding | that, what, about            |
| 6     | Collective identity             | we, our, people              |
| 7     | Election discussion             | election, again, many        |
| 8     | Movement / directional          | from, out, over              |
| 9     | Commentary / questioning        | why, should, news            |
| 10    | Informal personal interaction   | i, you, my                   |


> “These results are exactly what we expect from LDA applied to short social-media posts:
> not just themes, but **styles, narrative structures, conversational modes**, and **political clusters**.
> LDA reveals the dominant **co-occurrence structures** in the data, even when they are stylistic rather than topical.”

---

## ⭐ **Summary**

LDA is effective when:

* documents contain multiple overlapping themes
* co-occurrence provides strong signals
* interpretability is important
* topic mixtures are useful for downstream tasks

It is less suited for extremely short, highly contextual posts—
which is where **BERTopic** excels as a complementary method.

---


# ⭐ **Method 4: BERTopic — Finding Themes Using Semantic Meaning**

LDA and LSA both rely on **bag-of-words** representations. This means they treat:

* “good” and “not good” as almost identical
* “bank” (river) and “bank” (money) as the same
* “Biden won the election” and “Biden lost the election” as similar

Why?
Because they ignore *context*.

BERTopic solves this problem by using **semantic embeddings** — numerical vectors that capture *meaning*, not just word counts.

---

## 🔶 **Why BERTopic exists**

Bag-of-words methods struggle with:

* nuances (sarcasm, negation, sentiment shifts)
* synonyms (“car”, “vehicle”, “automobile”)
* polysemy (“bank”, “charge”, “shot”)
* short texts (like Truth Social posts)

BERTopic uses deep learning embeddings so that:

> Posts with similar *meaning* end up close together — even if they use different words.

This makes BERTopic particularly powerful for social media, where posts are short and language is inconsistent.

---

## 🔶 **Intuition Behind Semantic Embeddings (No Transformer Knowledge Needed)**

Think of an embedding model as a machine that:

1. Reads a sentence
2. Tries to understand its meaning from context
3. Converts it into a list of numbers (a vector), e.g. length 384
4. Places similar meanings near each other in this vector space

Example:

| Text                         | Embedding Meaning                        |
| ---------------------------- | ---------------------------------------- |
| “Trump won the election”     | close to other posts about Trump victory |
| “Trump lost the election”    | far away — opposite meaning              |
| “Biden’s economy is failing” | close to other economy-criticism posts   |

The key idea:

> Texts with similar *meaning* → close together in the embedding space.

These embeddings are the foundation of BERTopic.

---

## 🔶 **The BERTopic Pipeline (Explained Step-by-Step)**

### **Step 1 — Embed each document (convert text → meaning-vector)**

Each post becomes a dense vector representing its meaning.
This replaces bag-of-words with something context-aware.

We use a small, fast model:

> **all-MiniLM-L6-v2**
> (high quality, very efficient — perfect for 5,000 post sample)

---

### **Step 2 — Reduce dimensionality with UMAP**

Embedding vectors are high-dimensional (384 or 768).
Clustering them directly is slow and unstable.

UMAP reduces them to ~5–10 dimensions **while preserving neighborhood structure**:

* similar posts stay close
* dissimilar posts stay far

Why UMAP instead of PCA?
Because meanings form **non-linear shapes** in vector space; PCA cannot preserve these relationships well.

UMAP is ideal for visualizing and clustering embeddings.

---

### **Step 3 — Cluster documents with HDBSCAN**

You already know DBSCAN — a density-based clustering algorithm.

HDBSCAN is DBSCAN’s more stable, flexible older sibling:

| DBSCAN                              | HDBSCAN                        |
| ----------------------------------- | ------------------------------ |
| Requires a fixed eps                | Learns density automatically   |
| Fails when clusters vary in density | Handles variable densities     |
| No natural outlier class            | Automatically assigns outliers |

This matters for social media because:

* some themes are dense (repeated memes)
* some are sparse (one-off political claims)
* many posts are random noise → marked as outliers

BERTopic relies on HDBSCAN to *discover the number of topics automatically*.

No need to choose K.
HDBSCAN finds natural clusters in meaning-space.

---

### **Step 4 — c-TF-IDF: creating topic labels**

After clustering documents, BERTopic must extract **which words describe each topic**.

This is done with **c-TF-IDF**, short for “class-based TF-IDF.”

Regular TF-IDF asks:

> Which words distinguish *documents*?

c-TF-IDF asks:

> Which words distinguish *clusters/topics*?

The idea is identical to keyness:

* words that appear often in one cluster
* but rarely elsewhere
  → become the defining topic words.

This is why BERTopic topics tend to be coherent and descriptive.

---

## ⭐ **Choosing Parameters in BERTopic**

These matter much less than LDA’s parameters, but they shape the granularity of topics.

### **1. embedding_model**

* `"all-MiniLM-L6-v2"` → fast, excellent quality
* `"all-mpnet-base-v2"` → slower, highest quality

We use MiniLM for teaching.

---

### **2. min_topic_size**

Controls how large a cluster must be before being considered a “topic.”

* 20 → many fine-grained topics
* 80 → few broad topics

For **5,000 sampled posts**, a value around **40** gives a good balance.

---

### **3. n_neighbors (UMAP)**

Controls how local the embedding structure is:

* small → more clusters
* large → fewer clusters

Default is fine for teaching.

---

### **4. Outliers**

HDBSCAN labels some posts as **-1 = outlier**.
This is normal and often desirable — it prevents noisy posts from contaminating topics.

---

# ⭐ **Implementation (on the 5,000-post sample)**

```{r}
# Load the sample
posts <- read_csv("data/truth_social_sample.csv")
# Use your sampled texts (5,000 posts)
texts <- posts$text

# 1. Create a BERTopic model
model <- BERTopic$new(
  embedding_model = "all-MiniLM-L6-v2",
  min_topic_size = 40
)

# 2. Fit the model and get document-topic assignments
result <- model$fit_transform(texts)

# result is a list with:
#   - result$topics: topic ID per document
#   - result$probabilities: topic probability matrix

topics <- result$topics

# 3. Inspect top words per topic
topic_words <- model$get_topics()

# For example, inspect Topic 1:
topic_words[[1]][1:10]
```

This implementation mirrors the workflow for LDA so students can compare them.

---

## ⭐ **When BERTopic Excels**

BERTopic is usually the best choice when:

* posts are short
* nuance matters (sarcasm, negation, framing)
* vocabulary varies wildly
* number of topics is unknown
* you want context-aware similarities
* you want more stable, interpretable themes than LSA/LDA provide

It typically outperforms LDA on social-media datasets.

---

# ⭐ **When LDA Might Still Be Better**

LDA is preferable when:

* you need **probability distributions** over topics per document
  (BERTopic gives cluster assignments, not mixtures)
* reproducibility matters (BERTopic varies due to embeddings + UMAP + HDBSCAN)
* computational resources are limited
* corpus is extremely large (>100k documents) without GPU

---

# ⭐ **Summary — How BERTopic Complements LDA**

| Challenge                             | LDA    | BERTopic      |
| ------------------------------------- | ------ | ------------- |
| Captures context (negation, synonyms) | ❌      | ✅             |
| Works well on short posts             | ❌      | ✅             |
| Needs K specified                     | ✅      | ❌ (automatic) |
| Produces topic mixtures               | ✅      | ❌             |
| Fast on huge corpora                  | ⚠️     | ❌             |
| Topic coherence                       | medium | high          |

LSA → linear structure
LDA → probabilistic themes
BERTopic → semantic meaning

Together they illustrate three fundamentally different philosophies of “structure in text.”



---

# Questions for Reflection

Before moving to causal ML, consider:

1. Why does LSA struggle with very sparse data (short social media posts)?
2. Why must LDA use raw counts instead of TF-IDF?
3. How does BERTopic handle "not good" differently than LDA?
4. When would you choose LDA over BERTopic despite BERTopic being "better"?
5. What does it mean that LSA dimensions aren't "topics"?

---

# Next: Causal Inference & Fairness

After the break, we'll shift from **description** to **causation**:

- How do we move from "what patterns exist?" to "what causes what?"
- Same methods (LSA, LDA, BERTopic) but now as tools for causal inference
- Plus: Algorithmic fairness - "is it just?"