---
title: "🗓️ Week07 -Introduction to unsupervised learning and dimensionality reduction"
subtitle: "DS202: Data Science for Social Scientists"
author: "Theme: Unsupervised Learning"
date: "13 November 2025"
format: 
  html:
    page-layout: full
    toc: true
    toc-depth: 2
self-contained: true
editor: 
  markdown: 
    wrap: 72
---

## PART 1: FROM SUPERVISED TO UNSUPERVISED LEARNING

### What We've Done So Far: Supervised Learning

In previous weeks, we focused on **supervised learning**: predicting a
known outcome from features.

**Example**: Predicting heart disease (outcome) from blood pressure,
smoking, cholesterol (features)

**Key characteristic**: We have **labeled data** (we know the outcome
for each person)

**Methods**: Logistic regression, decision trees, random forests,
XGBoost, neural networks

### What If We Have No Outcome? Unsupervised Learning

Often, we collect rich data but **don't have a specific outcome to
predict**. Instead, we want to:

-   **Discover hidden patterns** in the data
-   **Understand structure** we didn't know existed
-   **Simplify complexity** for visualization and interpretation

This is **unsupervised learning**: learning from data without labels.

### The Four Families of Unsupervised Learning

**1. Clustering** (Next week!)

-   **Goal**: Group similar observations together
-   **Example**: Identify patient subtypes based on risk profiles
-   **Methods**: K-means, hierarchical clustering, DBSCAN

**2. Association Rules** (not tackled in this course)

-   **Goal**: Find patterns of co-occurrence
-   **Example**: If someone has high BP and diabetes, they likely have
    high cholesterol
-   **Methods**: Apriori, FP-Growth

**3. Anomaly Detection** (Week after next i.e W09!)

-   **Goal**: Identify unusual, outlier observations
-   **Example**: Detect patients with atypical risk profiles
-   **Methods**: Isolation Forest, One-Class SVM, autoencoders

**4. Dimensionality Reduction** (**TODAY'S FOCUS**)

-   **Goal**: Reduce many variables → few meaningful dimensions
-   **Example**: 10 correlated risk factors → 2-3 "meta-risk" dimensions
-   **Methods**: PCA, MCA, FAMD, UMAP, autoencoders

### Why Dimensionality Reduction Matters

**The Curse of Dimensionality**: As features increase, several problems
emerge:

1.  **Visualization impossible**: We can't really plot 4+ dimensions!
    And if we were to create plots for each pair of variables, then,
    with $p$ variables, we'd need $\frac{p(p-1)}{2}$ pairwise plots:

-   p=5 → 10 plots (manageable)
-   p=10 → 45 plots (tedious)
-   p=20 → 190 plots (impossible!)

2.  **Computation become expensive with higher dimensions**:

> Suppose you need all pairwise distances between 4,000 people—for
> example, for $k$-NN, clustering, or similarity search.
>
> The number of distances you must compute is $$
> \frac{n(n-1)}{2}
> $$ which for (n = 4000) is $$
> \frac{4000 \cdot 3999}{2} \approx 8{,}000{,}000 \text{ distances.}
> $$
>
> The *algorithmic complexity* of computing all distances is $$
> O(n^2 d)
> $$ because each distance requires ($d$) operations (subtract → square
> → sum).
>
> This means that as dimensionality increases, the computation grows
> **linearly in (**$d$):
>
> -   **2D:** ($8\text{M} \times 2$) → baseline
> -   **10D:** ($8\text{M} \times 10$) → **5× slower**
> -   **50D:** ($8\text{M} \times 50$) → **25× slower**
>
> Higher-dimensional data doesn’t change how *many* distances you
> compute—it changes how *expensive* each one is. This is one of the
> core reasons dimensionality reduction dramatically speeds up k-NN,
> clustering, and similarity search.

3.  **Overfitting risk**: Model memorizes noise with too many features
4.  **Interpretation overload**: Can't understand 10+ numbers per person
5.  **Multicollinearity**: Correlated features create unstable models

**Solution**: Transform $p$ variables into $k$ dimensions ($k << p$)
that preserve essential information.

------------------------------------------------------------------------

## PART 2: THE FRAMINGHAM HEART STUDY & RESEARCH QUESTIONS

### Background: The Study That Defined Risk Factors

The **Framingham Heart Study** (begun 1948) was the first prospective
study of cardiovascular disease. It:

-   Followed \~5,000 adults in Framingham, Massachusetts
-   Collected extensive health data every \~2 years
-   **Identified** the concept of "risk factors" (smoking, high BP,
    cholesterol)
-   **Revolutionized** how we understand heart disease

### Our Dataset

```{r setup, message=FALSE, warning=FALSE}
library(tidyverse)
library(tidymodels)
library(janitor)
library(recipes)
library(rsample)
library(parsnip)
library(workflows)
library(tune)
library(yardstick)
library(embed)
library(FactoMineR)
library(factoextra)
library(corrplot)
library(patchwork)
library(umap)
library(skimr)

theme_set(theme_minimal())
```

```{r load-data}
# Load from GitHub (public dataset)
fram_url <- "https://raw.githubusercontent.com/GauravPadawe/Framingham-Heart-Study/master/framingham.csv"
fram_raw <- read_csv(fram_url, show_col_types = FALSE)

# Data preparation
fram <- fram_raw %>%
  mutate(
    # Create factors
    sex = factor(male, levels = c(0,1), labels = c("Female", "Male")),
    education_level = factor(education, levels = 1:4, 
                            labels = c("Some HS", "HS Grad", "Some College", "College")),
    smoking_status = factor(currentSmoker, labels = c("Non-smoker", "Smoker")),
    bp_medication = factor(BPMeds, labels = c("No", "Yes")),
    diabetes = factor(diabetes, labels = c("No", "Yes")),
    # Outcome: 10-year CHD risk
    chd_risk = factor(TenYearCHD, labels = c("Low Risk", "High Risk"))
  ) %>%
  # Select relevant variables  
  select(age, sex, education_level, smoking_status, cigsPerDay, bp_medication,
         sysBP, diaBP, BMI, heartRate, glucose, totChol, diabetes, chd_risk) 

cat("Dataset:", nrow(fram), "participants with", ncol(fram)-1, "variables\n")
glimpse(fram)
```

This code produces a cross-sectional dataset because:

-   The downloaded `framingham.csv` contains only one row per
    participant, with baseline measurements and a 10-year CHD outcome.
-   No repeated exams, dates, or identifiers are present in the dataset.
-   The transformations applied simply relabel or recode variables; they
    do not introduce any longitudinal structure.

Therefore, the resulting object is inherently cross-sectional: one
observation per individual.

The following code block shows us that our dataset is rather clean (the
few variables with missing values have less than 9% missing values) and
has a mix of numeric and categorical variables:

```{r}
skim(fram)
```

Let's explore the missingness further:

```{r}
naniar::vis_miss(fram) +
  theme(axis.text.x = element_text(angle = 45, hjust = 1))

```

***Interpreting Missing Data Mechanisms from a Missingness Plot***

**Visual summary of the dataset**

-   Missingness present in: *education_level (\~2%), cigsPerDay (\~1%),
    bp_medication (\~1%), BMI (\~0.5%), heartRate (\<0.1%), glucose
    (\~9%), totChol (\~1%)*
-   Missing values appear in **distinct row bands** rather than being
    uniformly scattered.
-   No strong co-occurrence of missingness across variables.

**1. MCAR (Missing Completely At Random)**

*Definition:* Missingness unrelated to any variable (observed or
unobserved). *Visual signature:* Uniform, “salt-and-pepper” pattern;
evenly scattered missing cells.

*Interpretation:*

-   The plot does **not** show uniform random scatter.
-   Missingness appears clustered within certain row ranges. → **MCAR is
    unlikely.**

**2. MAR (Missing At Random)**

*Definition:* Missingness depends on *observed* variables (e.g., test
performed only in certain subgroups). *Visual signature:* Structured
missingness—clusters or bands across rows, often corresponding to exam
waves, demographic groups, or study modules.

*Interpretation:*

-   Glucose missingness forms a clear band (\~rows 500–3500).
-   Other variables have their own distinct bands.
-   Patterns differ by variable and do not overlap uniformly. → **The
    plot is most consistent with MAR.**

**3. MNAR (Missing Not At Random)**

*Definition:* Missingness depends on the *unobserved* value itself
(e.g., high glucose results not recorded). **Visual signature:** Hard to
detect visually; often missingness tied to expected extreme values.

*Interpretation:*

-   No pattern suggests missingness is driven by a variable’s own
    (unseen) value. → **No evidence of MNAR from the plot**, though it
    cannot be fully ruled out.

**Conclusion**

Based on the observed patterns:

> The missing data mechanism is most consistent with MAR (Missing At
> Random).
>
> MCAR can be ruled out, and there is no strong reason to suspect MNAR
> from the visual pattern alone.

**Optional statistical checks (recommended in practice)**

-   **Little’s MCAR test** to formally reject MCAR.
-   **Logistic regression on missingness indicators** (e.g.,
    `glucose_NA ~ age + sex + BMI + …`) Significant predictors →
    supports MAR.

This means that we'll have to impute missing values before performing
our analysis.

### Research Question (Cross-Sectional)

**Primary Question**: What are the fundamental dimensions that
characterize cardiovascular risk profiles **at baseline**?

**Why This Matters**:

1.  **Screening**: Can we create a simple "risk score" from many
    variables?
2.  **Patient stratification**: Do natural risk groups emerge?
3.  **Interpretability**: "High general CV risk" vs. listing 10+ numbers
4.  **Feature engineering**: Reduce correlated features for better
    prediction

**Data Structure**: Each person measured once (\~4,000 participants)

### Initial Exploration

```{r eda}
# Numeric variables summary
fram %>%
  select(where(is.numeric), -cigsPerDay) %>%
  summary()

# Class balance (important for later!)
table(fram$chd_risk)
prop.table(table(fram$chd_risk))
```

**Key Findings**:

-   **Age range**: 32-70 years (middle to older adults)
-   **Class imbalance**: 85% low risk, 15% high risk (highly imbalanced
    dataset! As expected in a medical setting)
-   **Mix ofdata types**: Numeric (BP, BMI) + Categorical (smoking,
    education)

### The Multicollinearity Problem

Let's check how correlated the numeric features are:

```{r correlation}
# Select numeric variables
fram_numeric <- fram %>% 
  select(age, sysBP, diaBP, BMI, heartRate, glucose, totChol, cigsPerDay)

# Correlation matrix
cor_matrix <- cor(fram_numeric)

# Visualize
corrplot(cor_matrix, method = "color", type = "upper",
         addCoef.col = "black", number.cex = 0.7,
         title = "Multicollinearity: Are Variables Redundant?",
         mar = c(0,0,2,0), tl.col = "black")
```

**Multicollinearity is low in this dataset.** The only strongly
correlated pair is `sysBP` and `diaBP` (`r paste("r ≈", 0.79)`
), which reflect the
same underlying construct (blood pressure). All other predictors show
weak to moderate correlations (`|r| < 0.4`), indicating no major
redundancy.

**Implication**:

Although there are 8 numeric predictors, **most correlations are weak.**
The only clearly redundant pair is:

-   `sysBP ↔ diaBP` (`r paste("r ≈", 0.79)`
) → both measure blood pressure

This suggests one underlying “blood pressure” dimension.

-   Dimensionality reduction can:

    -   Combine sysBP + diaBP into a single, more stable BP factor
    -   Capture subtle joint patterns across weakly correlated variables
        (glucose, BMI, cholesterol, age…)
    -   Reduce from 8 inputs to 2–3 interpretable latent dimensions,
        improving model simplicity, noise reduction, visualization and
        clustering performance

------------------------------------------------------------------------

## PART 3 — Cross-Sectional Analysis

Up to now, we have explored individual variables (their distributions,
missingness patterns) and pairwise associations (correlations,
scatterplots).

Cross-sectional dimensionality reduction goes further:

> **How do all variables relate simultaneously?** **Are there underlying
> latent dimensions that summarize risk profiles?** **Can we reduce
> complexity while preserving essential information?**

To answer these questions, this part introduces **five complementary
methods**:

-   **PCA** (numeric, linear structure)
-   **MCA** (categorical structure)
-   **FAMD** (mixed numeric + categorical)
-   **UMAP** (non-linear geometry)
-   **Autoencoders** (non-linear neural representation learning)

Before applying any of them, we must create a **complete, coherent
dataset** with consistent preprocessing. That is the focus of Part 3.1.

------------------------------------------------------------------------

### 3.1 Preprocessing & MAR Imputation

Dimensionality reduction requires:

1.  a dataset with **no missing values**,
2.  appropriately **standardized numeric variables**,
3.  properly handled **categorical variables**,
4.  a **single preprocessing pipeline** that all methods can share.

The goal is to produce one high-quality, fully imputed dataset that PCA,
MCA, FAMD, UMAP, and Autoencoders can all use.

------------------------------------------------------------------------

#### 3.1.1 Reminder: Why We Don’t Drop Missing Values

The Framingham dataset is mostly complete, but several variables contain
non-negligible missingness (glucose \~9%, education \~2.5%, cigsPerDay
\~0.7%, etc.). Dropping missing rows:

-   removes hundreds of participants,
-   distorts the joint structure of variables,
-   implicitly assumes **MCAR**, which is unrealistic in health and
    social datasets.

Even small amounts of systematic missingness can bias dimensionality
reduction methods.

------------------------------------------------------------------------

#### 3.1.2 Reminder: The MAR Assumption

The missingness patterns in this dataset are most consistent with
**MAR**:

-   Missing values depend on other *observed* characteristics (e.g.,
    glucose missingness varies by demographic profiles or lifestyle
    factors).

Under MAR, predictive imputation is appropriate and does not distort
multivariate analyses.

------------------------------------------------------------------------

#### 3.1.3 Reminder: Why Use Bagged-Tree Imputation

Bagged-tree imputation is well suited because it:

-   handles **mixed numeric + categorical** variables,
-   captures **non-linear** relationships,
-   performs robustly under **MAR**,
-   avoids distortions introduced by mean/median or k-NN imputation.

It is an excellent default choice when preparing data for multivariate
analysis.

------------------------------------------------------------------------

#### 3.1.4 The Unified Preprocessing Pipeline (Used for All DR Methods)

We now construct the **single preprocessing pipeline** for all methods
in Part 3.

##### This pipeline must:

-   **Impute all missing values** with bagged trees
-   **Standardize numeric variables**
-   **Leave categorical variables unchanged**
-   Produce a **complete, consistent dataset**
-   Be applied **once** (no repeated imputations that could create
    inconsistencies)

This dataset will feed:

-   PCA
-   MCA
-   FAMD
-   UMAP
-   Autoencoders

so consistency is essential.

##### ❗ Why standardize *after* imputation?

Because standardization uses:

$$
(x - \text{mean}) / \text{sd}
$$

which requires *actual values*. If we standardized first, missing values
would break the transformation and impair imputation.

The correct order is:

1.  **Impute raw values**
2.  **Then standardize**

This preserves real variance structure and ensures PCA/UMAP/Autoencoders
behave correctly.

------------------------------------------------------------------------

#### 3.1.5 Code: MAR-Compatible Bagged-Tree Imputation + Standardization

```{r}
# Start with the cleaned dataset from earlier
fram_clean <- fram %>%
  clean_names()

# Unified preprocessing recipe
impute_recipe <- recipe(~ ., data = fram_clean) %>%
  # Impute ALL predictors using bagged trees (works for numeric + categorical)
  step_impute_bag(all_predictors()) %>%
  
  # Standardize numeric predictors (required for PCA/UMAP/Autoencoders)
  step_normalize(all_numeric_predictors())

# Prepare the recipe
impute_prep <- prep(impute_recipe)

# Impute and standardize the dataset
fram_dr <- bake(impute_prep, new_data = NULL)

glimpse(fram_dr)
```

------------------------------------------------------------------------

#### 3.1.6 Why This Pipeline Works for PCA, MCA, FAMD, UMAP, and Autoencoders

| Method | Needs Standardization? | Needs Categorical Variables? | Compatible with This Pipeline? |
|----|----|----|----|
| **PCA** | Yes | No | ✔ numeric subset |
| **MCA** | No | Yes | ✔ categorical subset |
| **FAMD** | Yes (numeric only) | Yes | ✔ mixed data |
| **UMAP** | Yes | No | ✔ numeric subset |
| **Autoencoders** | Yes | No | ✔ numeric subset |

This pipeline is intentionally minimal and unified:

-   everything is imputed exactly once,
-   no rows are dropped,
-   numeric and categorical variables remain available for their
    respective DR methods,
-   all methods operate on the same underlying participant-level
    information.

This avoids inconsistencies (e.g., PCA using a differently imputed
dataset than FAMD).

------------------------------------------------------------------------

#### 3.1.7 Important Note About Leakage (for Part 4)

Here, we are in an **unsupervised** setting. It is acceptable to:

-   impute on the full dataset,
-   standardize on the full dataset.

However:

> **For supervised learning (Part 4), imputation must be done *inside*
> the training split to avoid information leakage.**

We will revisit this when building predictive models.

------------------------------------------------------------------------

#### 3.1.8 Summary of Preprocessing

-   Missing data are imputed under a **MAR** assumption.
-   Bagged-tree models provide robust multivariate imputations.
-   Numeric variables are standardized (after imputation).
-   Categorical variables remain factors for MCA/FAMD.
-   The resulting dataset is the basis for **all cross-sectional
    dimensionality reduction methods**.

------------------------------------------------------------------------

### 3.2 Principal Component Analysis (PCA)\*\*

*A method to reduce numeric variables into a smaller set of
interpretable dimensions*

Principal Component Analysis (PCA) is one of the core tools of
unsupervised learning. It takes a dataset with many numeric variables
and finds a smaller number of **new** variables—called *principal
components*—that capture the most important patterns in the data.

In our dataset, we have **8 numeric predictors**:

-   age
-   systolic blood pressure (sysBP)
-   diastolic blood pressure (diaBP)
-   BMI
-   glucose
-   total cholesterol
-   heart rate
-   cigarettes per day

PCA will help us:

-   understand relationships among these variables
-   summarize them into a few interpretable “risk dimensions”
-   visualize the data in 2D or 3D
-   prepare for clustering or anomaly detection

------------------------------------------------------------------------

#### 3.2.1 Why Use PCA Here?

There are several reasons PCA is extremely useful for the Framingham
dataset:

##### **1. Too many variables to visualize directly**

With 8 numeric variables, the number of pairwise scatterplots is:

$$
\frac{8 \times 7}{2} = 28
$$

Too many to inspect. PCA compresses everything into 2–3 dimensions we
*can* visualize.

------------------------------------------------------------------------

##### **2. Several variables measure the same underlying construct**

Examples:

-   sysBP and diaBP both reflect **blood pressure**
-   BMI, glucose, cholesterol reflect **metabolic health**

PCA automatically discovers and summarizes this structure.

------------------------------------------------------------------------

##### **3. It creates interpretable “risk axes”**

Examples:

-   “overall cardiovascular burden”
-   “metabolic vs lifestyle dimension”
-   “blood pressure axis”

------------------------------------------------------------------------

##### **4. PCA improves downstream methods**

PCA often improves:

-   clustering
-   k-NN
-   anomaly detection
-   visualization
-   computation speed

because it removes redundancy and noise.

------------------------------------------------------------------------

#### 3.2.2 PCA Intuition: Rotating the Axes to Follow the Data\*\*

Suppose we plot **Age** and **Blood Pressure**:

```         
     BP
     ↑
  140|     ●
     |    ● ●
  130|   ● ● ●         ← Diagonal cloud (Age and BP correlate)
     |  ● ● ●
  120| ● ● ●
     |● ●
  110|●
     └─────────────→ Age
      30  40  50  60
```

We see a **diagonal cloud**.

##### PCA's key idea: rotate the axes

```         
     PC2 (perpendicular to cloud)
      ↑
      |
      |   
      |      Cloud of points
      |     /
      |    /
      |   /
      |  /    ← PC1 (main axis of variation)
      | /
      |/
      └────────────→
```

##### Interpretation

-   **PC1** (main diagonal): captures the strongest trend
-   **PC2** (perpendicular): captures the weaker remaining variation

If Age and BP correlate strongly:

-   **PC1 might explain \~80–90% of variation**
-   effectively reducing 2D → 1D

The same idea extends to 8 variables.

------------------------------------------------------------------------

#### 3.2.3 What PCA Computes (Clear, Intuitive Mathematics)\*\*

PCA constructs new variables (PCs) with three properties:

------------------------------------------------------------------------

##### **1. PCs are linear combinations of the original variables**

```         
PC1 = w₁×Age + w₂×SysBP + w₃×DiaBP + ... + w₈×TotChol
PC2 = v₁×Age + v₂×SysBP + v₃×DiaBP + ... + v₈×TotChol
```

The weights (*loadings*):

-   show which variables matter most
-   determine the meaning of each PC

------------------------------------------------------------------------

##### **2. PCs are uncorrelated**

-   PC1 explains the most variation
-   PC2 explains the second-most, in a **different direction**
-   PC3 the next-most, and so on

They form a new orthogonal coordinate system.

------------------------------------------------------------------------

##### **3. PCs are ordered by importance**

$$
\text{Var(PC1)} \ge \text{Var(PC2)} \ge \ldots
$$

This ranking makes dimensionality reduction possible.

------------------------------------------------------------------------

##### **4. Intuition**

Imagine the dataset as an 8D point cloud.

PCA asks:

-   “In which direction is the cloud *widest*?” → PC1
-   “What is the next-widest direction *perpendicular* to PC1?” → PC2
-   etc.

It reorients the coordinate system to align with the true structure.

------------------------------------------------------------------------

#### 3.2.4 Why Standardization Is Absolutely Required\*\*

PCA is based on **variance**, and variance depends on scale.

Example:

```         
Variable    Range      Variance
Glucose     40–400     ~1000
Age         30–70      ~100  
BMI         15–50      ~25
```

If unscaled:

-   PCA becomes dominated by glucose simply due to its larger variance
-   PC1 becomes almost entirely about glucose
-   interpretation collapses

Solution: **z-score standardization**

$$
z = \frac{x - \text{mean}}{\text{sd}}
$$

After standardization:

-   all variables have mean = 0, SD = 1
-   all variables contribute fairly

------------------------------------------------------------------------

#### 3.2.5 Running PCA on the Preprocessed Data\*\*

We already created `fram_dr`, our fully imputed + standardized dataset.

##### **Extract the numeric variables for PCA**

```{r}
fram_numeric <- fram_dr %>% 
  select(where(is.numeric))
```

##### **Run PCA using recipes**

```{r}
pca_recipe <- recipe(~ ., data = fram_numeric) %>%
  step_pca(all_predictors(), num_comp = 8)

pca_prep <- prep(pca_recipe)
pca_results <- bake(pca_prep, new_data = NULL)

# Extract PCA object from the step
pca_obj <- pca_prep$steps[[1]]$res
```

**`pca_results` now contains PC1–PC8 for all participants.**

------------------------------------------------------------------------

#### 3.2.6 How Much Variation Each PC Explains

```{r}
eigen_df <- broom::tidy(pca_obj, matrix = "eigenvalues")

eigen_df %>%
  transmute(
    PC,
    variance = percent,
    cumulative = cumulative
  )
```

-   PC1 explains \~30% of the variance
-   PC2 explains \~14% of the variance
-   PC3 explains \~13% of the variance

##### Scree plot

```{r}
eigen_df %>%
  ggplot(aes(PC, percent)) +
  geom_col() +
  geom_line(group = 1) +
  labs(
    title = "PCA Scree Plot",
    y = "Variance Explained (%)"
  )
```

------------------------------------------------------------------------

#### 3.2.7 Loadings: What Each PC Represents\*\*

Loadings quantify each variable’s contribution to each PC.

```{r}
loadings <- broom::tidy(pca_obj, matrix = "loadings")
```

##### Loadings plot

```{r}
loadings %>%
  filter(PC %in% c(1, 2)) %>%
  ggplot(aes(value, reorder(column, value), fill = factor(PC))) +
  geom_col() +
  facet_wrap(~PC, scales = "free_x") +
  geom_vline(xintercept = 0, linetype = "dashed") +
  labs(
    title = "PCA Loadings (PC1 & PC2)",
    x = "Loading Weight",
    y = "Variable",
    fill = "Component"
  ) +
  theme(legend.position = "none")
```

##### Overall idea

Principal Component Analysis (PCA) finds new axes (PC1, PC2, …) that
represent dominant patterns of variation in your dataset. **Loadings**
tell you how strongly each original variable contributes to each
principal component and in what direction.

This plot shows loadings for **PC1** (left, red) and **PC2** (right,
teal) for variables such as blood pressure, cholesterol, BMI, glucose,
etc.

##### Interpretation of PC1 (Component 1 — red bars)

PC1 has **positive loadings for nearly every variable**, especially:

-   **sys_bp (systolic BP)** – large positive
-   **dia_bp (diastolic BP)**
-   **bmi**
-   **age**
-   **tot_chol**
-   **heart_rate/glucose (moderate)**

Only **cigs_per_day** is slightly negative.

##### Meaning of PC1

Because almost all health metrics increase together along this axis,
**PC1 appears to capture overall “cardiometabolic risk” or general
health severity**:

-   Higher blood pressure
-   Higher BMI
-   Higher cholesterol
-   Older age

Individuals with a high PC1 score generally have **worse
cardiovascular/metabolic profiles**.

------------------------------------------------------------------------

##### Interpretation of PC2 (Component 2 — teal bars)

PC2 has **mixed signs**:

**Positive loadings**

-   **age (strong positive)**
-   **tot_chol (moderate positive)**
-   **glucose (small positive)**

**Negative loadings**

-   **cigs_per_day (strong negative)**
-   **heart_rate (moderate negative)**
-   **dia_bp (moderate negative)**

**Meaning of PC2**

This axis contrasts **older, higher-cholesterol individuals** (positive
side) with **smokers and people with higher heart rates / higher
diastolic BP** (negative side).

So PC2 seems to represent a **lifestyle vs. aging** contrast:

-   **Positive PC2:** older, higher cholesterol, higher glucose
-   **Negative PC2:** younger but more smoking and elevated heart rate
    (“stress” profile)

------------------------------------------------------------------------

##### Putting It Together

**PC1**

A **global severity axis**: people with high BP, BMI, cholesterol, and
age score high.

**PC2**

A **contrast axis** separating:

-   **Older / metabolic-heavy profiles** (high cholesterol/glucose) vs.
-   **Younger / lifestyle-risk profiles** (smoking, elevated heart rate)

------------------------------------------------------------------------

**Summary**

-   **PC1**: captures general cardiometabolic burden.
-   **PC2**: splits variability between *aging/metabolic factors* and
    *smoking/heart-rate–related factors*.
-   Variables with large absolute values contribute most strongly to
    each component.

------------------------------------------------------------------------

#### 3.2.8 Visualizing Individuals in PCA Space

```{r}
pca_results %>%
  bind_cols(fram_dr %>% select(chd_risk)) %>%
  ggplot(aes(PC1, PC2, color = chd_risk)) +
  geom_point(alpha = 0.5) +
  labs(
    title = "Individuals in PCA Space",
    subtitle = "PC1: General Risk  |  PC2: Lifestyle vs Metabolic",
    x = "PC1",
    y = "PC2",
    color = "CHD Risk"
  )
```

**Interpretation**

The PCA provides a clearer view of the underlying structure of the
cardiovascular risk data than looking at each of the eight variables
separately. The first principal component (PC1) captures a broad
**general cardiometabolic risk gradient**, driven by factors such as
blood pressure, BMI, cholesterol, and age. Individuals with higher
values on these variables tend to score higher along PC1, and while
people labeled as “High CHD Risk” appear slightly more frequently on the
right side of the PC1 axis, the separation is far from sharp. Instead,
PC1 reflects a *continuous spectrum* of risk rather than a distinct
divide between groups.

The second principal component (PC2) represents a **lifestyle–metabolic
contrast**, with smoking and heart rate contributing in the opposite
direction from age, cholesterol, and glucose. This axis captures
meaningful physiological differences in the population, but it does
**not** correspond to large differences in CHD risk classification.
Individuals from both risk groups are distributed broadly on both the
positive and negative sides of PC2.

Overall, the PCA score plot shows a **dense, overlapping cloud** of
points, indicating that the population does not form clear-cut clusters.
Instead, the data reflect a continuum of risk profiles, with gradual
transitions rather than discrete subgroups. A few points appear as
**outliers**, corresponding to individuals with extreme values on one or
more risk factors, but these do not form separate clusters.

Despite the limited separation between CHD risk groups, PCA is still
valuable here. By reducing eight correlated variables into two
interpretable dimensions, it highlights the major axes along which
individuals differ, makes the general risk gradient visually accessible,
clarifies distinctions between metabolic and lifestyle patterns, and
identifies extreme cases. This two-dimensional view provides a far more
intuitive understanding of the data’s structure than examining the
original variables in isolation.

------------------------------------------------------------------------

#### 3.2.9 Why PCA Fails on Categorical Variables

This is a **mathematical incompatibility**, not a technical limitation.

##### **1. Euclidean distances are meaningless for one-hot encodings**

Examples:

```         
SomeHS = [1,0,0,0]
HSGrad = [0,1,0,0]
College = [0,0,1,0]
Grad    = [0,0,0,1]
```

Distances:

```         
d(SomeHS, HSGrad)  = √2
d(SomeHS, College) = √2
d(HSGrad, Grad)    = √2
```

PCA believes all transitions are equally distant — nonsense.

------------------------------------------------------------------------

##### **2. PCA averages categories**

```         
PC1 = 0.4×SomeHS + 0.3×HSGrad + 0.3×College
```

These combinations have **no interpretation**.

------------------------------------------------------------------------

##### **3. One-hot encoding inflates dimensionality**

Categories become:

-   2 smoking levels → 2 dimensions
-   4 education levels → 4 dimensions

PCA treats them as independent numeric dimensions — incorrect.

------------------------------------------------------------------------

##### **4. Dummy variables are sparse**

Sparse 0/1 vectors create *artificial* variance that PCA misinterprets.

------------------------------------------------------------------------

##### ✔ Correct method: **MCA** (Multiple Correspondence Analysis)

This is PCA’s categorical counterpart, using **chi-square distances**.

Covered in Section 3.3.

------------------------------------------------------------------------

#### 3.2.10 Summary

-   PCA identifies new, uncorrelated dimensions explaining maximal
    variation.
-   PCs are linear combinations of the original numeric variables.
-   Standardization is required.
-   PCA reveals interpretable cardiovascular/metabolic structure.
-   PCA is **not** appropriate for categorical variables → MCA is.
-   PCA prepares the ground for clustering, anomaly detection, and
    visualization.

------------------------------------------------------------------------

### 3.3 Multiple Correspondence Analysis (MCA)

*A dimensionality reduction method for categorical variables*

MCA is the categorical analogue of PCA. Where PCA handles **numeric**
variables using **Euclidean geometry**, MCA handles **categorical**
variables using **chi-square geometry**, which is appropriate for
variables like:

-   sex
-   education level
-   smoking status
-   blood-pressure medication
-   diabetes
-   CHD risk

Our dataset contains 6 such categorical variables. MCA reduces them into
a small number of **latent categorical dimensions** that summarize their
relationships.

------------------------------------------------------------------------

#### 3.3.1 Why Use MCA Here?

Categorical variables contain structure:

-   some categories often appear together
-   others contrast strongly
-   some combinations describe distinct groups
-   some categories are rare and therefore **informative**
-   others are common and therefore **less discriminative**

However:

-   categories have **no inherent order**
-   “distances” between categories are not numeric
-   dummy variables distort similarity
-   PCA on one-hot encoded variables is **mathematically invalid**

MCA solves these problems.

------------------------------------------------------------------------

#### 3.3.2 What MCA Does (Conceptual Overview)

MCA finds a set of latent dimensions that:

-   summarize co-occurrence patterns among categories
-   group similar categories together
-   place contrasting categories opposite one another
-   position individuals near the categories they possess
-   are ordered by decreasing “inertia” (variance)

These latent dimensions often reflect interpretable contrasts, such as:

-   **healthy vs at-risk profile**
-   **demographic gradient**
-   **lifestyle pattern**

MCA is PCA’s categorical sibling.

------------------------------------------------------------------------

#### 3.3.3 MCA Intuition: The Geometry of Categories

In PCA, numeric variables form continuous axes and individuals are
placed in a Euclidean space.

In MCA, each **category** becomes a point in space, and similarity is
measured using **chi-square distance**.

##### **Why Chi-Square Distance? (Intuition)**

Chi-square distance gives:

-   **more weight to rare categories** (because they distinguish people)

-   **less weight to very common categories** (because they contain
    little information)

This is essential because categorical data has:

-   unbalanced distributions
-   rare but important categories
-   co-occurrence patterns that Euclidean distance cannot capture

MCA uses chi-square geometry to ensure that:

-   two people are close if they share **informative** category
    combinations
-   categories are close if they tend to appear together
-   categories are far apart if they contrast sharply

This creates a meaningful categorical map.

------------------------------------------------------------------------

##### **ASCII Diagram: Interpreting Individuals and Categories**

```         
                     ↑  Dim2 (secondary contrast)
                     |
          (Smoker)   |             ● Individuals who smoke
               ●     |            /
                     |        (High CHD Risk)
                     |        ●
    (Male)      ●    |      /
                 |    |     /
                 |    |    /
(Female)     ●---+----+----------------→ Dim1 (main contrast)
                 |    |
                 |    |   ● (Non-smoker) 
                 |    |
                 |    |
                 |    ● (Low CHD Risk)
```

**How to read this**

-   \*\* Axes = latent categorical contrasts\*\* Dim1 and Dim2 summarize
    *major* patterns in how categories relate.

-   \*\* Category points = category levels\*\* "Smoker", "Female", "High
    CHD Risk" etc.

-   \*\* Individuals = dots\*\* Positioned near the categories
    describing their profile.

-   \*\* Proximity\*\*

    -   categories close together → frequently co-occur
    -   categories opposite sides of an axis → strongly contrast
    -   individuals near “Smoker” tend to be smokers
    -   individuals near “Low CHD Risk” tend to have healthy profiles

**Example interpretation**

If “Smoker”, “Male”, and “High CHD Risk” all lie on the right side of
Dim1:

-   they tend to occur together
-   Dim1 represents a **health risk gradient**
-   individuals on the right tend to have riskier profiles

This is the essence of MCA interpretation.

------------------------------------------------------------------------

#### 3.3.4 What MCA Computes

MCA operates on the **indicator matrix** created from categorical
variables:

```         
Sex_Female   Sex_Male   Smoker_No   Smoker_Yes   ...
     1           0           1           0
     0           1           0           1
     1           0           1           0
```

From this matrix, MCA finds components that:

-   are **linear combinations of category dummies**
-   are mutually **orthogonal** (uncorrelated)
-   maximize **chi-square variance** (called “inertia”)
-   summarize associations between categories

Mathematically, MCA components look like:

```         
Dim1 = a₁·(Male) + a₂·(Female) + a₃·(Smoker) + a₄·(Non-Smoker) + ...
```

The weights (a_i):

-   determine how strongly each category contributes
-   define the meaning of each dimension

This is the categorical analogue of PCA loadings.

------------------------------------------------------------------------

#### 3.3.5 Running MCA on the Categorical Variables

Prepare the data:

```{r}
fram_cat <- fram_dr %>%
  select(sex, education_level, smoking_status, bp_medication, diabetes, chd_risk)
```

Run MCA:

```{r}
mca_res <- MCA(fram_cat, graph = FALSE, ncp = 5)
```

Extract coordinates:

```{r}
mca_ind <- as_tibble(mca_res$ind$coord)
mca_var <- as_tibble(mca_res$var$coord)
```

------------------------------------------------------------------------

#### 3.3.6 Variance Explained in MCA

```{r}
eigenvalues <- mca_res$eig

eigenvalues %>%
  as_tibble() %>%
  rename(
    eigenvalue = `eigenvalue`,
    percent = `percentage of variance`,
    cumulative = `cumulative percentage of variance`
  )
```

**Interpretation**

In Multiple Correspondence Analysis (MCA), the variation in the
categorical data is quantified using **inertia**, which plays the same
role that **variance** plays in PCA. Because MCA works on a large matrix
of 0/1 indicators (one column per category level), inertia measures how
much “spread” or structure exists in those indicators. Each MCA
dimension captures a portion of this total inertia, showing how much of
the underlying pattern it explains.

Our MCA results show that:

-   **Dimension 1 explains 16.1% of the total inertia**,
-   **Dimension 2 explains 15.2%**,
-   **Dimension 3 explains 12.7%**,
-   and together the first three dimensions account for **about 44%** of
    the overall variation in the categorical data.

This is completely typical for MCA: unlike PCA, where a single component
may dominate, MCA spreads the information more evenly because each
category contributes its own piece of structure. As a result, it’s
normal for the first few dimensions to explain relatively modest amounts
of inertia. In your case, the first four dimensions reach **56%
cumulative inertia**, showing that meaningful patterns are spread across
multiple axes.

Overall, our MCA results show a standard and healthy structure: no
single dimension dominates, but the first two or three dimensions
provide a useful low-dimensional view of the main associations among
categories.

------------------------------------------------------------------------

#### 3.3.7 MCA Category Map (Loadings)

```{r}
fviz_mca_var(mca_res, repel = TRUE) +
  labs(
    title = "MCA Category Map",
    subtitle = "Which categories co-occur or oppose each other?",
    x = "Dimension 1",
    y = "Dimension 2"
  )
```

**Interpretation of the MCA Category Map**

In Multiple Correspondence Analysis, categories that **frequently
co-occur in the same individuals are positioned close together**, while
categories that rarely appear together are placed far apart or even on
opposite sides of a dimension. Interpreting the map therefore amounts to
identifying which categories cluster and which ones oppose each other.

In your MCA map, **Dimension 1 (horizontal)** captures a clear lifestyle
and demographic gradient. On the right side, *Male*, *Smoker*, *Some
HS*, and *High CHD Risk* cluster together, showing that these categories
tend to appear in the same individuals. On the left, *Female*,
*Non-smoker*, *Low Risk*, *Some College*, and *diabetes_No* form a
tightly grouped cluster, indicating a consistently healthier profile.
This axis therefore reflects a contrast between **higher-risk
lifestyle/demographic combinations** and **lower-risk, health-protective
ones**.

**Dimension 2 (vertical)** reflects a different pattern—one dominated by
medical conditions. *diabetes_Yes* and *bp_medication_Yes* sit high on
this axis and close to each other, showing a strong co-occurrence
between diabetes and the use of blood-pressure medication. Their
distance from *diabetes_No* and *bp_medication_No* at the lower end of
the plot makes this a straightforward **chronic disease and
medical-treatment axis**.

Taken together, the two dimensions create a meaningful structure:

-   **Bottom-left:** low-risk, non-smoking, female, healthier profiles
-   **Bottom-right:** lifestyle-oriented risk (male, smoker, high CHD
    risk)
-   **Top-right:** medically treated or clinically diagnosed risk
    (diabetes, BP medication)
-   **Top-left:** combinations that are very rare, which is why this
    quadrant is relatively empty

Overall, the MCA map provides an intuitive visualization of how
demographic characteristics, lifestyle behaviors, chronic conditions,
and risk labels **align and co-occur** in your dataset, with the spatial
arrangement directly reflecting the strength of those associations.

------------------------------------------------------------------------

#### 3.3.8 MCA Individual Map

```{r}
fviz_mca_ind(mca_res, 
             habillage = fram_dr$chd_risk,
             addEllipses = TRUE,
             alpha = 0.3) +
  labs(
    title = "Individuals in MCA Space",
    subtitle = "People with similar categorical profiles appear close to one another"
  )
```

**Summary of the MCA Individual Map (with comparison to PCA)**

The MCA individual map positions people based on how similar their
**categorical profiles** are. Individuals who share the same patterns of
smoking status, sex, education, diabetes, medication use, and CHD risk
appear close together, while those with different profiles are farther
apart.

**Dimension 1 (16.1%)** captures a lifestyle and demographic gradient.
High-risk individuals tend to fall slightly to the **right**, aligning
with categories such as *Male* and *Smoker*, while low-risk individuals
appear more to the **left**, where *Female*, *Non-smoker*, and *Low
Risk* categories cluster. Although the groups overlap substantially,
their centers of gravity differ, indicating consistent patterns in how
these categorical traits co-occur.

**Dimension 2 (15.2%)** reflects a chronic-condition or
medical-treatment axis. Individuals higher on this dimension often
correspond to profiles involving *diabetes_Yes* and *bp_medication_Yes*,
while those lower on the axis tend to have none of these conditions.
High-risk individuals show more spread upward, suggesting greater
variability in medical treatment or diagnosed conditions within that
group.

The ellipses show that **low-risk individuals form a more compact
cluster**, whereas **high-risk individuals are more dispersed**,
consistent with more heterogeneous risk-factor combinations.

Compared with the earlier PCA plot, the MCA map shows similar broad
overlap between high- and low-risk groups, but the structure it reveals
is different. PCA emphasized **physiological** variation (blood
pressure, cholesterol, BMI), whereas MCA highlights **behavioral,
demographic, and medical-status** patterns. The result is a categorical
“profile space” where risk-related traits cluster in predictable ways,
even though the groups are not cleanly separable.

------------------------------------------------------------------------

#### 3.3.9 When MCA Works Well (and When It Doesn’t)

##### Works well when:

-   all variables are categorical
-   categories form interpretable structure
-   you want a categorical “risk axis”
-   demographic/lifestyle patterns matter
-   you want to use these components for clustering later

##### Struggles when:

-   categories have many rare levels
-   relationship between numeric and categorical variables matters →
    **FAMD**
-   categorical data is extremely imbalanced

For our dataset, MCA is exactly the right tool to understand categorical
structure.

------------------------------------------------------------------------

#### 3.3.10 Summary

MCA:

-   handles categorical data correctly
-   uses chi-square geometry
-   groups co-occurring categories
-   contrasts opposing categories
-   positions individuals near their categorical profile
-   reveals interpretable latent dimensions

Typical interpretation for this dataset:

-   **Dim1** ≈ overall cardiovascular risk profile
-   **Dim2** ≈ demographic / education pattern

MCA will later be combined with PCA through **FAMD** to analyze the full
mixed dataset.

------------------------------------------------------------------------

### 3.4 Factor Analysis of Mixed Data (FAMD)

*A method to jointly reduce numeric **and** categorical variables into
shared dimensions*

So far:

-   **PCA** handled **numeric** risk factors.
-   **MCA** handled **categorical** lifestyle/demographic variables.

But the real Framingham dataset is **mixed**:

-   numeric: age, BP, BMI, glucose, cholesterol, heart rate, cigarettes
    per day
-   categorical: sex, education_level, smoking_status, bp_medication,
    diabetes

You usually care about **both together**.

FAMD (Factor Analysis of Mixed Data) is designed precisely for this
situation: it finds low-dimensional **latent risk dimensions** that
simultaneously summarize:

-   numeric profiles (like PCA), **and**
-   categorical profiles (like MCA),

in a **single, unified space**.

------------------------------------------------------------------------

#### 3.4.1 Why We Need FAMD (Not “PCA + MCA”)

It is tempting to say:

> “Just run PCA on the numeric variables and MCA on the categorical
> ones, then somehow combine them.”

This does **not** work well, for three reasons.

##### 1. You lose cross-type relationships

-   PCA only “sees” numeric variables (BP, BMI, glucose…).
-   MCA only “sees” categorical variables (sex, education, smoking…).

If you do them separately, you throw away:

-   how low education is associated with higher BMI and glucose,
-   how smoking interacts with blood pressure,
-   how sex differences appear in numeric profiles.

FAMD **keeps** all of these cross-type links.

------------------------------------------------------------------------

##### 2. PCA and MCA live in different geometric spaces

-   PCA uses **Euclidean distance on standardized numeric variables**.
-   MCA uses **chi-square distance on category profiles**.

Their scales and geometries are fundamentally different.

You cannot simply concatenate PCA and MCA components and pretend they
live in the same space.

FAMD builds **one** space in which:

-   numeric and categorical variables contribute on **comparable
    scales**,
-   distances between individuals reflect **both** types of information.

------------------------------------------------------------------------

##### 3. Separate analyses can double-count or under-weight parts of the data

If you:

-   take 3 PCA components **plus** 3 MCA components, you may:

    -   overweight variables with many categories,
    -   underweight key numeric variables,
    -   create artificial “extra dimensions”.

FAMD instead optimizes a **single criterion**:

> find axes that jointly explain as much variation as possible in
> numeric **and** categorical variables, with each original variable
> contributing fairly, regardless of how many levels it has.

------------------------------------------------------------------------

#### 3.4.2 Intuition: One Common Space for Clinical + Lifestyle Risk

Think of two separate pictures:

##### Numeric “risk cloud” (PCA style)

Imagine plotting age, BP, BMI, glucose, cholesterol, etc. PCA finds a
**diagonal “risk axis”**:

-   low on that axis = younger, normal BP, normal BMI, normal glucose
-   high on that axis = older, higher BP, heavier, higher glucose

Very loosely:

```         
Numeric space (simplified):

   high BP, high BMI, high glucose   ●
                                    ●
                                 ●
                              ●
                           ●
                        ●
                     ●
    low BP, low BMI ●────────────────────────→  numeric risk axis
```

------------------------------------------------------------------------

##### Categorical “profile cloud” (MCA style)

Now imagine only categories:

-   sex ∈ {Female, Male}
-   education_level ∈ {Some HS, HS Grad, Some College, College}
-   smoking_status ∈ {Non-smoker, Smoker}
-   bp_medication ∈ {No, Yes}
-   diabetes ∈ {No, Yes}

MCA reveals that some combinations co-occur:

-   **Male + Smoker + Some HS + No BP meds + Diabetes** cluster
    together,
-   **Female + Non-smoker + College + No Diabetes** cluster somewhere
    else.

Conceptually:

```         
Categorical space (simplified):

   “Male + Smoker + low education”   ●
                                   ●
                                ●
                             ●
   “Mixed profiles”        ●
                         ●
                      ●
   “Female + Non-smoker + College”  ●
```

------------------------------------------------------------------------

##### FAMD’s job

In reality, **both clouds describe the same people**. FAMD asks:

> *“Is there a single set of axes that organizes both numeric risk and
> categorical profiles at the same time?”*

You can picture this as overlaying the two clouds:

```         
Integrated space (very schematic):

   high numeric risk
   (high BP, BMI, glucose)
          ●   Male + Smoker + low education
         ● ●
       ●   \
     ●       \
   ●          \
 ●             \      Female + Non-smoker + College
●_______________\_______________________________→  FAMD Dim 1
 low numeric risk
```

-   On the **right**: people with high BP/BMI/glucose, more often Male,
    Smoker, lower education, Diabetes = **high-risk profile**.
-   On the **left**: lower numeric risk, more often Female, Non-smoker,
    higher education, No Diabetes = **low-risk profile**.

FAMD finds that **diagonal axis** (here “FAMD Dimension 1”) that:

-   **lines up** with numeric deterioration,
-   **also lines up** with riskier category profiles.

That is the main intuition: **one axis summarising both kinds of risk at
once.**

------------------------------------------------------------------------

#### 3.4.3 How FAMD Treats Numeric and Categorical Variables

FAMD is built to mimic:

-   **PCA** on numeric variables, and
-   **MCA** on categorical variables,

but in a **single model**.

##### Numeric variables (as in PCA)

-   Each numeric variable (age, sysBP, diaBP, BMI, heart_rate, glucose,
    tot_chol, cigs_per_day) is **standardized** (already done by our
    recipe: mean 0, SD 1).
-   FAMD then uses these standardized variables exactly as PCA would:
    they contribute via **correlations/variance**.

So for the numeric part alone, FAMD behaves like PCA.

------------------------------------------------------------------------

##### Categorical variables (as in MCA)

-   Each categorical variable (sex, education_level, smoking_status,
    bp_medication, diabetes) is internally expanded into indicator (0/1)
    columns.
-   FAMD uses **chi-square distances**, just like MCA, to measure
    similarity between category profiles.
-   Categories that co-occur more than expected by chance lie closer in
    the FAMD space.

So for the categorical part alone, FAMD behaves like MCA.

------------------------------------------------------------------------

##### Balancing numeric and categorical influence

To avoid categorical variables with many levels dominating the analysis,
FAMD:

-   **weights** each categorical variable so that it contributes
    **approximately the same** as one numeric variable,
-   within a categorical variable, its levels share that weight.

Rough idea:

-   “Education” with 4 levels should not automatically count 4× more
    than “Sex” with 2 levels.
-   All original variables (numeric or categorical) get a **comparable
    voice** in defining the dimensions.

The end result:

> FAMD finds components (dimensions) that **maximize total explained
> variation** across numeric **and** categorical variables, under a
> unified distance metric.

------------------------------------------------------------------------

#### 3.4.4 Setting Up FAMD on the Framingham Data

We use the **already imputed + standardized dataset** `fram_dr` from
Section 3.1.

From that data, we construct a **mixed** table with:

-   8 numeric variables
-   5 categorical variables

and **exclude** the outcome `chd_risk` from the FAMD model (we will use
it only for colouring plots).

```{r}
fram_mixed <- fram_dr %>%
  select(
    age, sys_bp, dia_bp, bmi, heart_rate, glucose, tot_chol, cigs_per_day,
    sex, education_level, smoking_status, bp_medication, diabetes
  )

# ---- FIX: convert to clean data.frame with unique rownames ----
fram_mixed <- as.data.frame(fram_mixed)
rownames(fram_mixed) <- paste0("row_", seq_len(nrow(fram_mixed)))
# --------------------------------------------------------------


# Check structure: numeric + factors
skimr::skim(fram_mixed)
```

Now we run FAMD:

```{r}
set.seed(123)

famd_result <- FAMD(
  fram_mixed,
  ncp   = 5,     # keep first 5 dimensions
  graph = FALSE  # no base R plots
)
```

`famd_result` now contains:

-   eigenvalues (variance explained),
-   coordinates for individuals,
-   coordinates for variables and category levels,
-   contributions of each variable/level to each dimension.

------------------------------------------------------------------------

#### 3.4.5 How Much Variation the FAMD Dimensions Explain

Just like PCA, we first look at the **eigenvalues** (percentage of total
variation explained by each dimension).

```{r}
famd_eig <- get_eigenvalue(famd_result)

famd_eig
```

To make this more readable:

```{r}
famd_eig %>%
  as.data.frame() %>%
  rownames_to_column("Dimension") %>%
  mutate(
    `Variance %`      = round(variance.percent, 2),
    `Cumulative %`    = round(cumulative.variance.percent, 2)
  ) %>%
  select(Dimension, `Variance %`, `Cumulative %`)
```

This shows us that:

-   **Dim 1**: \~18% of total mixed variation
-   **Dim 2**: \~12% of total mixed variation
-   **Dim 3**: \~10% of total mixed variation

Total after 3 dims: **40%** of the overall signal.

This is **normal**: with more variables (numeric + many category
indicators), some of the variation is noise or very fine-grained detail,
so each dimension explains a bit less than in a pure numeric PCA.

##### Scree plot

```{r}
fviz_screeplot(famd_result, addlabels = TRUE) +
  labs(
    title    = "FAMD: Variance Explained by Dimensions",
    subtitle = "Combines numeric variance and categorical inertia",
    x        = "Dimension",
    y        = "Percent of total variation"
  ) +
  theme_minimal()
```

You interpret the scree plot like PCA:

-   look for the “elbow”;
-   keep enough dimensions to explain a reasonable proportion (often
    30–50%) depending on how much detail you need.

------------------------------------------------------------------------

#### 3.4.6 Variable Plot: What Do the FAMD Dimensions Mean?

Now we want to understand **what each dimension represents**.

FAMD provides:

-   numeric variables as **arrows** (like PCA),
-   categorical *levels* as **points** (like MCA), all in the **same
    plot**.

```{r}
fviz_famd_var(
  famd_result,
  repel       = TRUE,
  col.var     = "contrib",                      # colour by contribution
  gradient.cols = c("#00AFBB", "#E7B800", "#FC4E07")
) +
  labs(
    title    = "FAMD Variable Map",
    subtitle = "Numeric variables (arrows) + categorical levels (points)"
  ) +
  theme_minimal()
```

**Summary of the FAMD Variable Map (with interpretation + comparison to
PCA/MCA)**

This FAMD variable map shows how **numeric variables (arrows)** and
**categorical levels (points)** jointly structure the dataset. In FAMD,
arrows point in the direction of *increasing values* for numeric
variables, while categorical levels appear near the regions of the space
where individuals with those characteristics tend to fall. Variables
that point in the same direction are positively associated; variables on
opposite sides represent contrasting profiles.

*Dimension 1 (18.3%): Physiological / cardiometabolic load*

The strongest contributors to Dim1 are the numeric variables **systolic
BP, diastolic BP, age, BMI,** and **total cholesterol**—all pointing to
the right. This means individuals with higher values on these measures
lie further to the right of the map. Dim1 therefore represents a
**physiological risk axis**, almost identical to PC1 in your PCA (where
the same variables dominated). Categorical levels related to medical
diagnoses (e.g., diabetes, medication use) lie close to the origin,
indicating that these variables do not strongly shape Dim1.

*Dimension 2 (12.3%): Smoking intensity and lifestyle*

Dim2 is dominated by **cigs_per_day** and **smoking_status**, which sit
high on the vertical axis. This identifies Dim2 as a **smoking/lifestyle
axis**, separating smokers (and heavier smokers) from non-smokers. This
matches the structure seen in the MCA: smoking categories formed one of
the strongest contrasts in the categorical space. FAMD reinforces this
pattern and also quantifies it through the numeric “cigs_per_day” arrow.

*Variables near the center contribute less*

Because FAMD re-balances numeric and categorical variables, variables
near the center—such as **glucose, heart rate, diabetes,** and
**bp_medication**—do not drive Dim1 or Dim2. They may become more
important in later dimensions or simply vary more weakly.

**How this compares to PCA and MCA**

*Compared to PCA*

-   **Dim1 in FAMD is extremely similar to PC1 in PCA:** both capture a
    cardiometabolic burden direction (BP, BMI, cholesterol, age).
-   **Dim2 differs:** PCA’s second component was a mixed
    lifestyle–metabolic axis, but in FAMD, smoking dominates Dim2 more
    cleanly because categorical smoking levels and cigarettes-per-day
    reinforce one another.
-   FAMD reveals *the same physiological gradient* as PCA but also
    incorporates *categorical information* that PCA cannot include.

*Compared to MCA*

-   MCA showed that smoking categories were one of the clearest
    categorical contrasts; FAMD confirms this by showing smoking as the
    main driver of Dim2.
-   MCA also revealed a chronic-condition axis (diabetes, medication
    use), but in FAMD these variables contribute less—likely because
    numeric variables like BP and cholesterol explain more of the shared
    variance.
-   FAMD integrates numeric and categorical structures simultaneously,
    so its dimensions represent **joint patterns**, whereas MCA
    represents **categorical-only structure**.

**Overall Interpretation**

The FAMD variable map reveals two coherent axes:

1.  **Dim1 = Cardiometabolic physiological load** Driven by BP, BMI,
    cholesterol, age (consistent with PCA).

2.  **Dim2 = Smoking intensity and lifestyle pattern** Driven by both
    numeric and categorical smoking variables (consistent with MCA).

Together, the map shows how mixed data types interact: physiological
risk patterns occupy one dimension, lifestyle behaviors another. This
gives a more complete picture than PCA or MCA alone, combining the
strengths of both approaches while preserving interpretability.

#### 3.4.7 Which Variables and Levels Drive Each Dimension?

To quantify which variables matter most for each dimension, we can
inspect contributions.

##### Variable contributions

```{r}
fviz_contrib(
  famd_result,
  choice = "var",
  axes   = 1,
  top    = 10
) +
  labs(
    title = "Top Contributors to FAMD Dimension 1"
  ) +
  theme_minimal()

fviz_contrib(
  famd_result,
  choice = "var",
  axes   = 2,
  top    = 10
) +
  labs(
    title = "Top Contributors to FAMD Dimension 2"
  ) +
  theme_minimal()
```

These plots show:

-   for Dim 1: `sys_bp` and `dia_bp` contribute strongly.
-   for Dim 2: `cigs_per_day` and `smoking_status` contribute strongly.

------------------------------------------------------------------------

#### 3.4.8 When to Use FAMD vs PCA or MCA

You now have three main linear tools:

-   **PCA** → numeric only
-   **MCA** → categorical only
-   **FAMD** → both together

A practical rule:

-   If your analysis focuses purely on laboratory / numeric variables →
    **PCA**.
-   If your analysis focuses on survey-like, all-categorical data →
    **MCA**.
-   If your data are mixed (which is most real-world datasets) and you
    care about how numeric and categorical parts interact → **FAMD**.

In this Framingham context:

-   PCA gave you **“numeric risk axes”**.

-   MCA showed **how lifestyle/demographic categories relate**.

-   FAMD provides the **integrated picture**:

    -   how clinical measurements and lifestyle/demographics jointly
        structure cardiovascular risk.

This FAMD space will be especially useful later for:

-   clustering individuals into **risk phenotypes**,
-   understanding **which combinations** of education, smoking, sex,
    etc., go with particular numeric profiles,
-   and reducing dimensionality before supervised learning (if you want
    a small set of mixed-type risk scores).

------------------------------------------------------------------------

------------------------------------------------------------------------

### 3.5 Uniform Manifold Approximation and Projection (UMAP)

*A non-linear method that uncovers curved, clustered, and complex
structure in numeric data*

UMAP is a modern dimensionality-reduction method designed to capture
**non-linear geometry**: curved shapes, branching patterns, complex
clusters, and local neighborhoods.

In contrast to PCA/MCA/FAMD — which uncover only **linear** structures —
UMAP can reveal more intricate cardiometabolic relationships such as:

-   subgroups (phenotypes) with non-linear separation,
-   metabolic trajectories that follow curved manifolds,
-   interactions and thresholds invisible to linear methods.

UMAP is therefore a crucial complement to PCA and FAMD in the Framingham
dataset.

------------------------------------------------------------------------

#### 3.5.1 Why UMAP? When Linear Methods Aren’t Enough

All linear DR methods — **PCA**, **MCA**, **FAMD** — assume that
patterns in the data can be captured by **straight lines** (linear
combinations of variables).

This works extremely well for:

-   global trends
-   linear relations between predictors
-   variance maximization

But many real-world biological and behavioral processes are **not
linear**.

##### **Common non-linear structures in health data**

**A. Threshold effects**

BP \< 120 (healthy) vs BP \> 140 (hypertension) The risk change is **not
gradual** — it jumps at defined points.

Linear PCA cannot detect these sharp transitions.

------------------------------------------------------------------------

**B. Synergistic interactions**

Risk factors often multiply rather than add:

-   BMI × smoking
-   glucose × blood pressure
-   age × cholesterol

A purely linear combination misses these curved interaction surfaces.

------------------------------------------------------------------------

**C. Curved or branched clusters**

Patient populations may form:

-   metabolic subtypes,
-   behaviorally defined groups,
-   risk trajectories following curved paths.

Linear methods flatten all of these into straight axes.

------------------------------------------------------------------------

#### 3.5.2 How Do We Detect Non-Linearity?

Since we cannot visually check all pairwise scatterplots (8 variables →
**28** pairs), we need *systematic diagnostics*.

Below are **three practical, scalable strategies**.

------------------------------------------------------------------------

##### **Strategy 1: Check PCA Variance Explained**

If PCA or FAMD captures most of the structure, their first dimensions
will explain a high fraction of the total variance. When the early
components explain only a modest amount, it suggests that linear methods
may be missing nonlinear structure—so methods like UMAP can reveal
additional patterns.

```{r}
percent <- eigenvalues[,  "percentage of variance"] / 100
cum_var_3pc <- sum(percent[1:3])

if (cum_var_3pc < 0.6) {
  cat("→ LOW! (<60%) suggests linear methods miss structure → Try UMAP\n")
} else {
  cat("→ MODERATE. PCA captures main trends, but UMAP might reveal clusters\n")
}
```

**Rule of thumb:**

-   **\< 60%** → linear methods are likely missing non-linear patterns
-   **60–75%** → linear structure somewhat captured, but UMAP may reveal
    clusters
-   **\> 75%** → PCA captures most structure (still, UMAP may refine
    clusters)

------------------------------------------------------------------------

##### **Strategy 2: Compare Linear vs Non-Linear Smooths**

Example: is the relationship between SysBP and CHD risk linear?

```{r}
fram %>%
  ggplot(aes(sysBP, as.numeric(chd_risk)-1)) +
  geom_point(alpha = 0.1, position = position_jitter(height = 0.02)) +
  geom_smooth(method = "lm", color = "blue", se = FALSE, size = 1.2) +
  geom_smooth(method = "loess", color = "red", se = FALSE, size = 1.2) +
  labs(
    title = "Is SysBP → CHD Risk Linear (blue) or Non-Linear (red)?",
    subtitle = "Large red–blue gaps indicate non-linearity",
    x = "Systolic Blood Pressure",
    y = "CHD Risk (0 = Low, 1 = High)"
  ) +
  theme_minimal()
```

-   If **red and blue curves match** → linear relationship
-   If **they diverge substantially** → non-linear relationship → UMAP
    helpful

This diagnostic can be used for any predictor.

------------------------------------------------------------------------

##### **Strategy 3: Domain Knowledge**

Medical processes frequently exhibit:

-   **Thresholds** (e.g., hypertension/diabetes cutoffs)
-   **U-shaped curves** (e.g., exercise dose)
-   **Synergistic interactions**
-   **Distinct physiological phenotypes**

If you *expect* non-linear behavior, UMAP is appropriate.

------------------------------------------------------------------------

##### **Decision Rule**

Use **UMAP** when:

-   PCA’s first 3 PCs explain **\<60% variance**
-   predictor–outcome relationships look **curved**
-   domain expertise suggests **thresholds** or **interactions**
-   you suspect **multiple non-linear risk subtypes**

And as you noted:

> **No need for t-SNE — UMAP is strictly better and you already have
> enough methods.**

------------------------------------------------------------------------

#### 3.5.3 UMAP: The Core Idea (High-Level Intuition)

UMAP models the high-dimensional cardiometabolic space using:

##### **1. Local Neighborhoods**

UMAP finds each individual’s *k* nearest neighbors using Euclidean
distance (on standardized numeric data).

This forms a **neighborhood graph** capturing local similarity:

```         
Person A → similar to → B, C, D
Person B → similar to → A, C, E
...
```

This graph encodes non-linear structure the same way a road map encodes
geodesic distances.

------------------------------------------------------------------------

##### **2. Manifold Geometry**

The idea: the data lie on a *lower-dimensional curved surface* (a
manifold) inside 8D space.

UMAP tries to recover this manifold.

------------------------------------------------------------------------

##### **3. Optimizing a 2D Embedding**

UMAP then places points in 2D such that:

-   close neighbors stay close,
-   far points do not collapse together,
-   curved shapes remain curved,
-   clusters remain intact.

Unlike PCA, the axes no longer correspond to “variance directions” —
they represent the geometry of the manifold.

------------------------------------------------------------------------

#### 3.5.4 Preparing Data for UMAP

UMAP requires:

-   **numeric variables**,
-   **comparable scale** (standardization),
-   **no missing values**.

Our unified preprocessing pipeline (`fram_dr`) satisfies this.

Select numeric predictors:

```{r}
fram_umap <- fram_dr %>%
  select(age, sys_bp, dia_bp, bmi, heart_rate, glucose, tot_chol, cigs_per_day)
```

We’ll add `chd_risk` back only for plotting.

------------------------------------------------------------------------

#### 3.5.5 Running UMAP (Recommended Settings)

```{r}

set.seed(123)

umap_config <- umap.defaults
umap_config$n_neighbors <- 15
umap_config$min_dist <- 0.1
umap_config$metric    <- "euclidean"

umap_fit <- umap(fram_umap, config = umap_config)

umap_df <- as.data.frame(umap_fit$layout) %>%
  rename(UMAP1 = V1, UMAP2 = V2) %>%
  bind_cols(fram_dr %>% select(chd_risk))
```

Parameters explained:

-   **n_neighbors = 15** balances local vs global structure
-   **min_dist = 0.1** controls tightness of clusters
-   **metric = "euclidean"** correct for standardized numeric predictors

------------------------------------------------------------------------

#### 3.5.6 Visualizing Individuals in UMAP Space

```{r}
ggplot(umap_df, aes(UMAP1, UMAP2, color = chd_risk)) +
  geom_point(alpha = 0.5) +
  labs(
    title = "UMAP Embedding of Numeric Cardiometabolic Variables",
    subtitle = "Reveals non-linear phenotypes and curved risk patterns",
    x = "UMAP 1", y = "UMAP 2"
  ) +
  theme_minimal()
```

UMAP reveals that cardiometabolic profiles form a continuous nonlinear
spectrum rather than discrete clusters. Individuals with similar numeric
risk factors lie close together, but low- and high-risk individuals
overlap extensively — confirming that CHD risk emerges gradually across
the manifold rather than forming distinct subgroups. Compared to PCA’s
linear gradients and FAMD/MCA’s structured categorical patterns, UMAP
exposes the underlying curved geometry of numeric health variation but
still shows no separable phenotypes associated with risk.

UMAP often shows **branching** or **looping** patterns that reflect real
non-linear dynamics.

------------------------------------------------------------------------

#### 3.5.7 Understanding What Drives the UMAP Embedding

Because UMAP is non-linear, it does not produce loadings. But variable
influence can be examined with:

##### **A. Correlations with UMAP axes**

```{r}
cor(fram_umap, umap_df[,c("UMAP1","UMAP2")])
```

##### **B. Coloring UMAP by a variable**

Example: glucose

```{r}
ggplot(umap_df %>% bind_cols(fram_umap),
       aes(UMAP1, UMAP2, color = glucose)) +
  geom_point(alpha = 0.5) +
  scale_color_viridis_c() +
  labs(title = "UMAP Colored by Glucose Level") +
  theme_minimal()
```

This reveals **high-glucose arcs**, **BP clusters**, **BMI lobes**, etc.

------------------------------------------------------------------------

#### 3.5.8 Choosing UMAP Parameters

| Parameter | Meaning | Effect |
|----|----|----|
| **n_neighbors** | neighborhood size | small → fine clusters; big → smoother global path |
| **min_dist** | cluster tightness | small → tight blobs; large → spread-out |
| **metric** | distance function | Euclidean is appropriate here |

Robust structure is that which remains stable across a range of
parameter choices.

------------------------------------------------------------------------

#### 3.5.9 When Should You Prefer UMAP?

UMAP is preferred when:

-   structure is expected to be **non-linear**,
-   PCA explains **\<60%** of the variance in first 3 PCs,
-   medical reasoning suggests **thresholds**, **interactions**, or
    **curved processes**,
-   you suspect **distinct subtypes** or phenotypes,
-   you want the most expressive 2D visualization.

UMAP complements PCA/FAMD rather than replacing them.

------------------------------------------------------------------------

#### 3.5.10 Summary

-   UMAP reveals **non-linear structure** that PCA/MCA/FAMD cannot
    capture.
-   It is well suited for cardiometabolic data where risk evolves along
    **curved**, **branched**, or **clustered** trajectories.
-   It preserves **local neighborhoods** and **global manifold shape**.
-   When linear methods fall short — due to thresholds, interactions, or
    low PCA variance — UMAP provides a far richer map of the data.
-   It is the most expressive 2D tool for visualizing numeric
    health-risk structure.

------------------------------------------------------------------------

### 3.6 Autoencoders (AEs)

*A neural-network method for non-linear dimensionality reduction*

Autoencoders (AEs) are neural networks designed to learn **compressed
representations** of data. They are particularly powerful when:

-   relationships are **highly non-linear**,
-   patterns lie on **complex manifolds**,
-   interactions are **multiplicative or threshold-based**,
-   structure cannot be captured by PCA/FAMD (linear) or even UMAP
    (local non-linear).

In cardiometabolic datasets like Framingham, AEs can detect subtle
combinations of risk factors missed by other methods.

However — unlike PCA/FAMD — autoencoders are **less interpretable**.
Their strength lies in representation learning, not feature explanation.

------------------------------------------------------------------------

#### 3.6.1 What Is an Autoencoder? (Intuition)

An autoencoder is composed of two parts:

```         
INPUT → [ENCODER] → LATENT SPACE (bottleneck) → [DECODER] → OUTPUT
```

It tries to **reconstruct the input** after compressing it:

```         
x  →  f(x)  →  z  →  g(z)  →  x̂
```

Where:

-   **x** = original 8D numeric vector
-   **z** = low-dimensional representation (2D or 3D)
-   **x̂** = reconstructed version of x
-   **f(x)** = encoder (non-linear transformations)
-   **g(z)** = decoder (non-linear transformations)

The network is trained to minimize:

$$
\text{Loss} = ||x - \hat{x}||^2
$$

Thus, the model learns:

-   which variables matter most,
-   how to combine them non-linearly,
-   what low-dimensional structure preserves information.

This is “representation learning.”

------------------------------------------------------------------------

#### 3.6.2 Why Use Autoencoders? (Advantages Over PCA/UMAP)

Autoencoders excel when relationships between variables involve:

##### **1. Multi-factor interactions**

For example:

-   BMI × age × glucose
-   smoking × cholesterol
-   BP × BMI × glucose × heart rate

These interactions create **curved, folded spaces**.

AEs learn these implicitly.

------------------------------------------------------------------------

##### **2. Complex manifolds**

Some patterns look like this in high-D:

```         
  (curved surface)
   /\_
  /   \__
 /        \
```

PCA flattens it into a line. UMAP preserves local curvature but cannot
model global structure. Autoencoders can model the full manifold.

------------------------------------------------------------------------

##### **3. Reconstruction-based anomaly detection**

AEs can identify individuals whose reconstruction error is large:

-   unusual metabolic profiles
-   rare risk combinations
-   outliers not captured by PCA

------------------------------------------------------------------------

##### **4. Neural networks scale better with dimension**

As $p$ increases, PCA becomes unstable. AEs naturally handle dozens or
hundreds of predictors.

------------------------------------------------------------------------

#### 3.6.3 When NOT to Use Autoencoders

Autoencoders are **NOT appropriate** when:

-   you need **interpretability**
-   sample size is **small** (\< 2,000 rows)
-   data are mostly categorical (AEs prefer numeric)
-   the goal is *understanding* rather than *compression*

They are powerful — but not transparent.

------------------------------------------------------------------------

#### 3.6.4 Data Preparation for Autoencoders

Autoencoders require:

✔ **Numeric variables only** ✔ **Complete data** ✔ **Standardized data**
✔ **Matrix input** (not tibble)

Our preprocessing pipeline already satisfies these.

```{r}
data_ae <- fram_dr %>%
  select(age, sys_bp, dia_bp, bmi, heart_rate, glucose, tot_chol, cigs_per_day) %>%
  as.matrix()
```

------------------------------------------------------------------------

#### 3.6.5 Architecture: The 8 → 3 → 8 Autoencoder

We want to learn a **3D latent representation**, analogous to 3 PCs.

### Encoder:

-   8 → 16 neurons (non-linear activation)
-   16 → 8 neurons
-   8 → 3 latent neurons

### Decoder (mirror):

-   3 → 8
-   8 → 16
-   16 → 8 (output layer)

This architecture is expressive enough without overfitting.

------------------------------------------------------------------------

#### 3.6.6 Comparison: Autoencoders vs PCA vs UMAP

| Method | Linear? | Non-linear? | Interpretable? | Good for clusters? | Good for manifolds? |
|----|----|----|----|----|----|
| **PCA** | ✔ linear | ✖ | ⭐⭐⭐ | ⭐ | ⭐ |
| **UMAP** | ✖ | ✔ local | ⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐ |
| **Autoencoders** | ✖ | ✔ global + local | ⭐ (low) | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |

AEs capture the **most complex structure**, but with **least
interpretability**.

------------------------------------------------------------------------

#### 3.6.8 When to Use Autoencoders

Use AEs when:

-   you have **non-linear interactions**,
-   you want **deep representations** for clustering, risk modeling, or
    anomaly detection,
-   PCA/UMAP seem insufficient,
-   sample size is ≥ 2,000,
-   data are numeric and standardized.

Do **NOT** use AEs for purely interpretive purposes.

------------------------------------------------------------------------

#### 3.6.9 Summary

-   Autoencoders are neural networks that learn non-linear
    low-dimensional representations.
-   They outperform PCA and UMAP when complex interactions or manifold
    structure exist.
-   They require careful preprocessing (imputation + standardization).
-   They provide little interpretability but excellent expressive power.
-   Latent dimensions can reveal deep structure in cardiometabolic risk
    patterns.
-   Reconstruction error allows detection of unusual or atypical
    individuals.

------------------------------------------------------------------------

## PART 4 — Predictive Modeling With Supervised Learning

Up to now (Parts 1–3), you have:

-   understood the Framingham dataset,
-   identified its cross-sectional structure,
-   explored missingness and MAR imputation,
-   applied dimensionality reduction (PCA, MCA, FAMD, UMAP,
    Autoencoders),
-   learned how each method reveals different aspects of the underlying
    risk structure.

Now the natural question is:

> **Can we use these representations to *predict* 10-year CHD risk?**

This is where **supervised learning** enters.

Supervised learning uses:

-   **Predictors** (age, BP, glucose, PCA/FAMD dimensions, etc.)
-   **Outcome** (CHD risk: Low Risk vs High Risk)

to train a model that forecasts the outcome for new patients.

------------------------------------------------------------------------

### 4.1 The Goal of Predictive Modeling in This Context\*\*

Our outcome variable is:

**CHD_risk** = Low Risk (0) vs High Risk (1)

Supervised learning will help us answer:

##### **1. Which variables (or dimensions) are most predictive?**

Age? glucose? BP? metabolic dimensions? lifestyle dimensions?

##### **2. Does dimensionality reduction improve prediction?**

Especially helpful when:

-   predictors are correlated,
-   relationships are non-linear,
-   noise obscures patterns,
-   we want a compact model (2–5 latent dimensions).

##### **3. Can we build accurate risk scores / classifiers?**

Using models such as:

-   logistic regression,
-   penalized logistic regression (lasso / elastic net),
-   random forest,
-   gradient boosting (XGBoost),
-   neural networks.

##### **4. How do low-dimensional representations compare to raw variables?**

For example:

-   PCA components as predictors
-   vs raw numeric predictors
-   vs FAMD components
-   vs UMAP embeddings
-   vs Autoencoder latent vectors

------------------------------------------------------------------------

### 4.2 Critical Principle: NO DATA LEAKAGE

This is the single most important rule of supervised learning:

> **Any preprocessing step that looks at the data must be done *inside*
> the training split only.**

Otherwise, information from the test set leaks into the model.

This includes:

-   standardization (`step_normalize`)
-   imputation (`step_impute_bag`)
-   dimensionality reduction (`step_pca`, `step_umap`, autoencoder
    training)
-   feature selection
-   neighborhood graph construction (UMAP!)
-   outcome smoothing
-   hyperparameter tuning

**In Part 3**, leakage was fine because everything was unsupervised.

**In Part 4**, it is NOT acceptable.

Therefore, we **must rebuild** a new preprocessing pipeline **inside the
training split**.

------------------------------------------------------------------------

### 4.3 Train/Test Split

```{r}
set.seed(123)
splits <- initial_split(fram, prop = 0.8, strata = chd_risk)
train_data <- training(splits)
test_data  <- testing(splits)
```

We **must not** reuse the earlier cross-sectional imputed dataset
(`fram_dr`) because that imputation used all rows.

------------------------------------------------------------------------

### 4.4 Workflow 1: Logistic Regression With Raw Predictors

This provides a **baseline**.

#### 4.4.1 Preprocessing: Imputation + Standardization

Inside the recipe:

```{r}
logreg_recipe <- recipe(chd_risk ~ ., data = train_data) %>%
  step_impute_bag(all_predictors()) %>%    # impute using ONLY training data
  step_dummy(all_nominal_predictors()) %>% # convert factors to dummies
  step_normalize(all_numeric_predictors())
```

------------------------------------------------------------------------

#### 4.4.2 Model Specification

```{r}
logreg_model <- logistic_reg() %>%
  set_engine("glm")
```

------------------------------------------------------------------------

#### 4.4.3 Workflow & Training

```{r}
logreg_wf <- workflow() %>%
  add_recipe(logreg_recipe) %>%
  add_model(logreg_model)

logreg_fit <- fit(logreg_wf, data = train_data)
```

------------------------------------------------------------------------

#### 4.4.4 Evaluate on Test Data

```{r}
logreg_class <- predict(logreg_fit, test_data, type = "class") %>%
  bind_cols(test_data %>% select(chd_risk))

f_meas(
  logreg_class,
  truth = chd_risk,
  estimate = .pred_class,
  event_level = "second"
)
```

This is our **baseline to beat**.

------------------------------------------------------------------------

### 4.5 Workflow 2: Logistic Regression + PCA Components\*\*

Dimensionality reduction is applied *within the training split*, so PCA
is “legal.”

#### 4.5.1 PCA Recipe

```{r}
pca_recipe <- recipe(chd_risk ~ ., data = train_data) %>%
  step_impute_bag(all_predictors()) %>%
  step_normalize(all_numeric_predictors()) %>%
  
  # Apply PCA ONLY to numeric variables
  step_pca(all_numeric_predictors(), num_comp = 5)
```

We select the first 5 PCs (they capture \~60% variance).

------------------------------------------------------------------------

#### 4.5.2 Model

Same logistic regression model:

```{r}
pca_logreg_wf <- workflow() %>%
  add_recipe(pca_recipe) %>%
  add_model(logreg_model)
```

------------------------------------------------------------------------

#### 4.5.3 Train + Evaluate

```{r}
pca_logreg_fit <- fit(pca_logreg_wf, data = train_data)

pca_class <- predict(pca_logreg_fit, test_data, type = "class") %>%
  bind_cols(test_data %>% select(chd_risk))

f_meas(
  pca_class,
  truth = chd_risk,
  estimate = .pred_class,
  event_level = "second"
)
```

Interpretation:

-   sometimes PCA helps by reducing noise
-   sometimes PCA loses subtle predictive signals
-   PCA is not optimized for prediction, only variance compression

You could also try FAMD+logistic regression, UMAP/auto-encoders+Random
Forest and compare.

### 4.6 Key Takeaways

##### **1. Linear methods (LogReg, PCA, FAMD) are interpretable but limited**

Good for understanding risk factors, but not always best predictors.

##### **2. Non-linear methods (UMAP, Autoencoders, XGBoost) detect deeper structure**

They capture interactions, thresholds, and curved boundaries.

##### **3. No method is “best” universally**

The best choice depends on:

-   interpretability needs
-   model complexity
-   data size
-   expected non-linearity

##### **4. Dimensionality reduction helps prediction**

Especially:

-   FAMD for mixed data
-   UMAP + XGBoost
-   Autoencoders for deep signal extraction

##### **5. Leakage must be avoided**

All preprocessing must be done inside the training split.

------------------------------------------------------------------------

## PART 5: PREPROCESSING CHECKLIST & DECISION FRAMEWORK

### Preprocessing Requirements by Method

| Method | Standardize? | Missing Data? | Categorical Handling | Sample Size | Key Notes |
|----|----|----|----|----|----|
| **PCA** | ✅ Required | ❌ Remove or impute | ❌ Numeric only | Any | **NEVER** apply to one-hot encoded categoricals |
| **MCA** | ❌ Not needed | ❌ Remove or treat as category | ✅ Native | Any | Chi-square internally handles scale |
| **FAMD** | ✅ Numeric only | ❌ Remove or impute | ✅ Native | Any | Best for real-world mixed data |
| **UMAP** | ✅ Recommended | ❌ Remove | ⚠️ One-hot OK but inflates dim | \>500 | Sensitive to scale, needs neighbors |
| **Autoencoders** | ✅ Required | ❌ Remove | ⚠️ One-hot increases input size | \>2000 | Needs normalized inputs (0-1 or standardized) |

### Decision Framework Flowchart

```         
START: What type of data do you have?
│
├─ All NUMERIC variables
│  │
│  └─ Are relationships LINEAR?
│     │
│     ├─ YES / MOSTLY
│     │  └─ → PCA
│     │
│     └─ NO / NON-LINEAR
│        │
│        └─ Sample size?
│           │
│           ├─ <500: PCA (safer)
│           ├─ 500-2000: UMAP
│           └─ >2000: UMAP or Autoencoders
│
├─ All CATEGORICAL variables
│  └─ → MCA
│
└─ MIXED (numeric + categorical)
   │
   └─ → FAMD (almost always!)
```

### Method Comparison Summary Table

| Method | Data Type | Relationships | Sample Size | Interpretability | Speed | Longitudinal |
|----|----|----|----|----|----|----|
| **PCA** | Numeric | Linear | Any | ⭐⭐⭐ High | ⚡⚡⚡ Fast | ✅ Yes (change/wide) |
| **MCA** | Categorical | Associations | Any | ⭐⭐ Medium | ⚡⚡⚡ Fast | ✅ Yes (transitions) |
| **FAMD** | Mixed | Linear | Any | ⭐⭐ Medium | ⚡⚡ Medium | ✅ Yes (best for mixed) |
| **UMAP** | Numeric | Non-linear | \>500 | ⭐⭐ Medium | ⚡⚡ Medium | ✅ Yes (wide better) |
| **Autoencoders** | Numeric | Very complex | \>2000 | ⭐ Low | ⚡ Slow | ⚠️ Yes (LSTM better) |

------------------------------------------------------------------------

## PART 6: KEY TAKEAWAYS

### Core Concepts

1.  **Dimensionality reduction solves the curse of dimensionality** by
    transforming many correlated variables into few meaningful
    dimensions while preserving essential information.

2.  **Linear methods (PCA/MCA/FAMD)**:

    -   Work when relationships are roughly linear or associations are
        categorical
    -   Provide interpretable loadings/contributions
    -   Fast and stable with any sample size
    -   **PCA = SVD + standardization** for numeric data only
    -   **MCA** uses chi-square distance for categorical data
    -   **FAMD ≠ PCA+MCA** - it's a unified optimization integrating
        both types simultaneously

3.  **Non-linear methods (UMAP/Autoencoders)**:

    -   Capture complex, curved relationships linear methods miss
    -   UMAP preserves local neighborhood structure via manifold
        learning
    -   Autoencoders learn hierarchical representations through neural
        networks
    -   Require larger samples and less interpretable

4.  **In machine learning pipelines**, dimensionality reduction:

    -   Reduces multicollinearity (helps linear models most)
    -   Prevents overfitting (especially with small n or high p)
    -   Improves computational efficiency
    -   Enables visualization and interpretation
    -   Trade-off: Information loss vs. noise reduction

### Practical Guidance

**When to Use Dimensionality Reduction**: ✅ High correlation between
predictors (multicollinearity) ✅p ≈ n or p \> n (more features than
samples) ✅ Need visualization (reduce to 2-3 dimensions) ✅ Want
interpretable "meta-features" ✅ Computational constraints ✅ As
preprocessing for clustering or anomaly detection

**When NOT to Use**:

❌ Few features already (\<5) that are all meaningful ❌ Tree-based
models with sufficient data (they handle correlations) ❌ Need
feature-specific coefficients for interpretation ❌ Very low variance
explained (\<50%) - losing too much information

### Next Steps

**Next Week - Clustering**:

-   Use PCA/FAMD/UMAP coordinates as inputs
-   K-means, DBSCAN
-   Identify patient subtypes, risk groups

**Week After - Anomaly Detection**:

-   DBSCAN
-   Autoencoder reconstruction error
-   Isolation forests
-   LOF
-   Identify unusual risk profiles
