---
title: "🗓️ Week 4 - Classification & Metrics & Resampling Methods"
subtitle: "Theme: Supervised Learning"
author: "Dr. Ghita Berrada"
date: "25 October 2024"
format: 
  html:
    from: markdown+emoji
    page-layout: full
    toc: true
    toc-depth: 2
self-contained: true
---

Welcome to Week 4! This week, we go back to classification. But, before we start, let's do a bit of set up

⚙️ Set up

**Loading R libraries**

```{r import, message=FALSE, warning=FALSE}
# Tidyverse packages, spelled out individually for clarity
library(dplyr)
library(ggplot2)
library(archive)
library(readr)
library(stringr)
library(forcats)
library(tidyr)

# Tidymodels packages, spelled out individually for clarity
library(parsnip)
library(yardstick)
library(recipes)
library(workflows)
library(rsample) # to split data in training and test sets
library(kknn) 
library(tune)#library needed for K-NN
```

**(Re)downloading the dataset we'll use for most of the lecture**

```{r}
df<-read_csv(archive_read("https://info.stackoverflowsolutions.com/rs/719-EMH-566/images/stack-overflow-developer-survey-2022.zip", file = 3), col_types = cols())
```

**Cleaning up the dataset**

```{r}
filtered_gender <- c("Man", "Woman", "Non-binary")
survey_results <- df %>% 
  dplyr::filter(
    Country == "United Kingdom of Great Britain and Northern Ireland",
    Employment == "Employed, full-time",
    ConvertedCompYearly > 0,
    ConvertedCompYearly < 2e6
  )

## identify non-ICs, to remove
managers_ctos <- survey_results %>%
  dplyr::filter(str_detect(DevType, "Engineering manager|Product manager|Senior executive/VP"))

## identify academics, to remove
academics <- survey_results %>%
  dplyr::filter(str_detect(DevType, "Academic researcher|Scientist|Educator"))

filtered_devtype <- c(
  "Other", "Student",
  "Marketing or sales professional"
)

results <- survey_results %>%
  anti_join(managers_ctos) %>%
  anti_join(academics) %>%
  transmute(ResponseId,
            EdLevel = fct_collapse(EdLevel,
                                   `Less than bachelor's` = c(
                                     "Primary/elementary school",
                                     "Secondary school (e.g. American high school, German Realschule or Gymnasium, etc.)",
                                     "Some college/university study without earning a degree",
                                     "Associate degree (A.A., A.S., etc.)"
                                   ),
                                   `Other` = "Something else",
                                   `Bachelor's degree` = "Bachelor’s degree (B.A., B.S., B.Eng., etc.)",
                                   `Graduate degree` = c(
                                     "Other doctoral degree (Ph.D., Ed.D., etc.)",
                                     "Master’s degree (M.A., M.S., M.Eng., MBA, etc.)",
                                     "Professional degree (JD, MD, etc.)"
                                    )
            ),
            DevType,
            RemoteWork,
            YearsCodePro = parse_number(YearsCodePro),
            Gender = case_when(
              str_detect(Gender, "Non-binary") ~ "Non-binary",
              TRUE ~ Gender
            ),
            ConvertedCompYearly
  ) %>%
  dplyr::filter(Gender %in% filtered_gender)

results_parsed <- results %>%
  mutate(DevType = str_split(DevType, pattern = ";")) %>%
  unnest(DevType) %>%
  mutate(
    DevType = case_when(
      str_detect(str_to_lower(DevType), "data scientist") ~ "Data scientist",
      str_detect(str_to_lower(DevType), "data or business") ~ "Data analyst",
      str_detect(str_to_lower(DevType), "desktop") ~ "Desktop",
      str_detect(str_to_lower(DevType), "embedded") ~ "Embedded",
      str_detect(str_to_lower(DevType), "devops") ~ "DevOps",
      str_detect(DevType, "Engineer, data") ~ "Data engineer",
      str_detect(str_to_lower(DevType), "site reliability") ~ "DevOps",
      TRUE ~ DevType
    ),
    DevType = str_remove_all(DevType, "Developer, "),
    DevType = str_to_sentence(DevType),
    DevType = str_replace_all(DevType, "Qa", "QA"),
    DevType = str_replace_all(DevType, "Sre", "SRE"),
    DevType = str_replace_all(DevType, "Devops", "DevOps")
  ) %>%
  dplyr::filter(
    !DevType %in% filtered_devtype,
    !is.na(DevType)
  )

cleaned_df <- results_parsed %>%
  drop_na(YearsCodePro)%>%
  dplyr::filter(
    ConvertedCompYearly < 1e7,
    YearsCodePro < 60
  ) %>%
  dplyr::filter(Gender %in% c("Man", "Woman")) %>%
  select(-ResponseId)
```

# Part 1: Let's go back to logistic regression once again

## 1.1 The principles

Consider a **binary** response:

$$
Y = \begin{cases}
0 \\
1
\end{cases}
$$

We model the probability that $Y = 1$ using the **logistic function** (aka. sigmoid curve):

$$
Pr(Y = 1|X) = p(X) = \frac{e^{\beta_0 + \beta_1X}}{1 + e^{\beta_0 + \beta_1 X}}
$$

This function is mathematically guaranteed to be bounded by the interval $[0, 1]$.

### 1.1.1 Insights into the logistic regression function

Let's see how this mathematical function looks like:

```{r}
plot_logistic_function <- function(X_range, beta0, beta1, color){
  
  df <- expand.grid(X=X_range, beta0=beta0, beta1=beta1)
  
  df <- 
    df %>% group_by(beta0, beta1) %>% 
    transform(X=X, Y=exp(beta0+beta1*X)/(1+exp(beta0+beta1*X)))
  
  df <- df %>% 
    transform(beta0=factor(beta0, ordered = TRUE),
              beta1=factor(beta1, ordered = TRUE))
  
  g <- 
    if(missing(color)){
      ggplot(df, aes(x=X, y=Y))
    }else if(color=="beta0"){
      ggplot(df, aes(x=X, y=Y, color=beta0, linetype=beta0))
    }else{
      ggplot(df, aes(x=X, y=Y, color=beta1, linetype=beta1))
    }
  
  g <- 
    g + 
    geom_line(linewidth=1.4) +
  
    xlab("X") + ylab("Y") +
  
    # THEMING
    theme_bw() +
  	theme(
          axis.title.x = element_text(size=24,margin = margin(t = 10, r = 0, b = 0, l = 0)),
          axis.title.y = element_text(size=24, margin = margin(t = 0,  r = 10, b = 0, l = 0)),
          legend.text = element_text(size=18),
          legend.position="right",
          plot.title=element_text(size=26),
          axis.text.x = element_text(size=18),
          axis.text.y = element_text(size=18),
          panel.grid.minor = element_blank(),
          aspect.ratio = 1/2)
  return(g)
}
```

```{r}
plot_logistic_function(X_range=(-400:400)/100, beta0=c(-2, 0, 1), beta1=c(1), color="beta0") + 
labs(title=expression(paste("If I keep ", ~beta[1]==1," fixed and vary ", ~beta[0] )))
```

```{r}
plot_logistic_function(X_range=seq(-4, 4, 0.1), beta0=c(0), beta1=c(-1, 0, 1, 2), color="beta1") + 
labs(title=expression(paste("If I keep ", ~beta[0]==0," fixed and vary ", ~beta[1] )))
```

## 1.2 Going back to our problem of predicting whether developers are experienced or not

Just as last week, we want to try and predict whether developers have more then 10 years experience or not.

For that, just as last week, we create a variable `not_rookie` (of type factor) in our dataset to indicate whether a developer has more than 10 years experience (based on a lookup of the `YearsCodePro` variable). We then split the data into training and test set in preparation for fitting a logistic regression model.

```{r message=FALSE, error=FALSE}

dataset <- 
    dataset <- 
    cleaned_df %>% 
    mutate(
      experienced = as.factor(if_else(YearsCodePro > 10, "Yes", "No")),
    ) %>%
    tidyr::drop_na(experienced)
  

# Split the data with 75% being used to train the model
survey_split <- initial_split(dataset, prop = 0.75,strata=Gender) #the option strata=Gender allows me to keep the original proportions of genders from the dataset 
# Create tibbles of the training and test set

survey_train <- training(survey_split)
survey_test <- testing(survey_split)
```

We fit a logistic regression model on the training data:

```{r}
set.seed(123)
logistic_model1 <-
  logistic_reg() %>%
  set_engine("glm") %>% 
  set_mode("classification") %>%
  fit(experienced ~ ConvertedCompYearly, data=survey_train)

# Look at the coefficients
logistic_model1$fit
```

What does this tell me?

My coefficient $\beta_0$ is equal to $-0.6033368$ and my coefficient $\beta_1$ is equal to $4.379922e-07$.

This means that:

1.  In the case where the annual salary was zero, the probability of being experienced is:

$$
\frac{e^{-0.6033368}}{1 + e^{-0.6033368}} \approx 0.353$$

2.  For every 1 USD increase in the annual salary, the probability of being experienced increases by a factor of

    $$
    \frac{e^{-0.6033368 + 4.379922e-07}}{1 + e^{-0.6033368 + 4.379922e-07}}\approx 0.353
    $$

How should I go about evaluating the performance of my model?

```{r}
logistic_model1 %>% 
  augment(new_data=survey_train,type.predict = "response") %>%
  conf_mat(truth = experienced, estimate = .pred_class) 
```

Let's visualise the same thing a bit better:

```{r}
logistic_model1 %>% 
    augment(new_data=survey_train,type.predict = "response") %>% 
    conf_mat(truth = experienced, estimate = .pred_class) %>%
    autoplot(type="heatmap")
```

What I have here is called a **confusion matrix**: it represents counts from predicted and actual values.

Perhaps, a bit of vocabulary to know in the context of binary classification:

-   the two classes in your classification are called positive class (i.e the class you're interested in, here developers with more than ten years' experience), and negative class

-   knowing that, you define true positives (TP) as the instances in your dataset that are of the positive class and are correctly classified by the model as belonging to the positive class, true negatives (TN) are the instances that are of the negative class and are correctly predicted by the model as belonging to the negative class, false positives (FP) are instances of the negative class that are misclassified by the model as instances of the positive class and false negatives (FN) are instances of the positive class that are falsely predicted by the model as belonging to the negative class

-   whether the FN or FP count is more important depends on your application

Now, let's have a look at a few more metrics:

```{r}
logistic_model1 %>% 
    augment(new_data=survey_train,type.predict = "response") %>% 
    conf_mat(truth = experienced, estimate = .pred_class) %>%
    summary(event_level="second")
```

-   Accuracy

    It's the proportion of correct predictions i.e

    $$
    \begin{equation}
    accuracy = \frac{TP+TN}{P+N}\
    \end{equation}
    $$

    where $P$ is the total number of positive instances in the dataset and $N$ the total number of negative instances in the dataset.

    It's only a good metric when the classes within your dataset are balanced. Can you guess what the problem might be in case of imbalance?

    How do you check if classes are balanced?

    ```{r}
    table(survey_train$experienced)
    ```

    Unfortunately, our dataset is heavily imbalanced: about $37$ % of the developers in our training set are experienced while $63$ % are not!

    So accuracy is **not** an appropriate metric in this case.

    What metrics can we use in case of imbalance?

-   Precision, recall/sensitivity and F1-score

    Precision is the proportion of true positives among all the predicted positive cases.

    $$
    \begin{align}
    precision &= \frac{TP}{TP+FP}\\
              &= \frac{TP}{\textrm{all predicted positives}}
    \end{align}
    $$

    It is a good metric when the cost of false positives is high, e.g spam detection (the consequences of misclassifying a legitimate email as spam can be significant as it may result in important communications being filtered out and so the cost of a false positive is higher than that of a false negative in this case).

    Recall (or sensitivity) is the proportion of true positives among all actual positive cases.

    $$
    \begin{align}
    recall &= \frac{TP}{TP+FN}\\
              &= \frac{TP}{\textrm{all actual positives}}
    \end{align}
    $$

    High recall is critical in applications where the cost of false negatives is high, e.g medical applications. If you think back to this week's lab, a false negative meant a patient that had diabetes being misdiagnosed as being healthy resulting in delayed diagnosis and delayed appropriate treatment (and likely deterioration of the patient's condition in the meanwhile).

    Most applications, though, will need a balance between precision and recall. That's where the F-score (especially in the form of the balanced F1-score) comes in: it's the harmonic mean of precision and recall.

    $$
    \textrm{F1-score}=2\frac{precision * recall}{precision + recall}
    $$

    A more general score $F_\beta$ exists if you want to emphasize precision over recall or vice versa:

    $$
    \textrm{F}_{\beta}\textrm{-score} = (1+\beta^2)\frac{precision * recall}{(\beta^2.precision)+ recall}
    $$

    The recall is terrible on my training set ($0.00826$) - this means that our model barely finds any developers with more than 10 years experience- and the precision rather low ($0.692$), resulting in a very low F1-score ($0.0163$). If your goal was to recruit developers, you might want to emphasize recall (having experienced developers is key) or balance precision and recall and focus on the F1-score (developer experience is crucial but maybe salary costs, which likely increase with experience, are an important factor too): in this case, our model is very bad from the outset (it performs very badly on training data).

-   Specificity

    The specificity is the proportion of negatives that are correctly identified as negatives.

    $$
    specificity = \frac{TN}{TN+FP}
    $$

    It is important in contexts where the classification of negatives is of high priority.

    It's particularly useful in the medical context. Consider the example of a medical test for diagnosing a disease; e.g the HbA1C blood test used to diagnose diabetes. Specificity refers to the test's ability to correctly reject healthy patients without the condition (i.e without diabetes). Supposing HbA1C had high specificity, a positive result in such test with high specificity would be useful for "ruling in" disease, i.e determining that the patient does in fact have diabetes, since the test rarely gives positive results in healthy patients.

    Our model's specificity is high ($0.998$): if a developer is classified as not being experienced, its probability of being so for this model is 99.8%.

-   Balanced accuracy

    But specificity is also used in another context. It is one of the components used to define balanced accuracy, a metric designed to overcome the problem that accuracy has when it comes to imbalanced data.

    $$
    \begin{equation}
    \textrm{balanced}\_\textrm{accuracy}= \frac{sensitivity+specificity}{2}
    \end{equation}
    $$

The closer the balanced accuracy is to 1, the better the model is able to correctly classify observations (for both classes).

For our model, the performance based on balanced accuracy is no better than random: the balanced accuracy is $0.503$.

-   ROC Curve/ Area Under the Curve (AUC)

The ROC curve is a plot of the true positive rate - $TPR$- (recall) against the false positive rate ($FPR$).

The false positive rate is defined as follows:

$$
\begin{align}
FPR &= \frac{FP}{FP+TN}\\
    &= \frac{FP}{\textrm{all actual negatives}}\\
    &= 1-specificity
\end{align}
$$

It is a good metric when you want to visualise the trade-off between precision and recall. It's **a visual representation of model performance across all thresholds.**

Here's what it looks like for our model:

```{r}
logistic_model1 %>% 
    augment(new_data=survey_train,type.predict = "response") %>% 
    roc_curve(truth = experienced, .pred_Yes, event_level="second") %>%
    autoplot()
```

As a baseline, a random classifier is expected to give points lying along the diagonal ($FPR=TPR$).

Classifiers that give curves closer to the top-left corner indicate a better performance. The closer the curve comes to the 45-degree diagonal of the ROC space, the less accurate the test.

A metric that goes along the ROC curve is the AUC (Area Under the Curve) metric. The AUC is a metric that measures the performance of a model across all classification thresholds. The closer to 1, the better the model: a random classifier is one whose ROC lies on the diagonal $FPR=TPR$ and its AUC would be 0.5. It is a good metric when you want to compare models across different thresholds.

```{r}
logistic_model1 %>% 
    augment(new_data=survey_train,type.predict = "response") %>% 
    roc_auc(truth = experienced, .pred_Yes, event_level="second") 
```

Our model performs slightly better than random ($AUC=0.639$): not a particularly good model.

**A recap of the metrics**

| **Metric**            | **Description**                                                                                       | **`yardstick` function**                                                         |
|-------------------|----------------------|-------------------------------|
| Accuracy              | Proportion of correct predictions                                                                     | [`accuracy()`](https://yardstick.tidymodels.org/reference/accuracy.html)         |
| Sensitivity (=recall) | The proportion of positive results out of the number of samples that were actually positive.          | [`sens()`](https://yardstick.tidymodels.org/reference/sens.html)                 |
| Specificity           | The proportion of negatives that are correctly identified as negatives.                               | [`spec()`](https://yardstick.tidymodels.org/reference/spec.html)                 |
| Precision             | The proportion of true positive results out of the number of samples that were predicted as positive. | [`precision()`](https://yardstick.tidymodels.org/reference/precision.html)       |
| Recall (=sensitivity) | The proportion of true positive results out of the number of samples that were actually positive.     | [`recall()`](https://yardstick.tidymodels.org/reference/recall.html)             |
| F-measure             | The harmonic mean of precision and recall.                                                            | [`f_meas()`](https://yardstick.tidymodels.org/reference/f_meas.html)             |
| Balanced accuracy     | Mean of sensitivity and specificity                                                                   | [`bal_accuracy()`](https://yardstick.tidymodels.org/reference/bal_accuracy.html) |
| ROC Curve             | Plot of recall/true positive rate ($TPR$) against false positive rate ($FPR$)                         | [`roc_curve()`](https://yardstick.tidymodels.org/reference/roc_curve.html)       |
| AUC                   | Area under the ROC curve                                                                              | [`roc_auc()`](https://yardstick.tidymodels.org/reference/roc_auc.html)           |

Obviously, we can't stop the evaluation of our model at the training set. What about the performance on the test set?

```{r}
logistic_model1 %>% 
    augment(new_data=survey_test,type.predict = "response") %>% 
    conf_mat(truth = experienced, estimate = .pred_class) %>%
    autoplot(type="heatmap")
```

```{r}
logistic_model1 %>% 
    augment(new_data=survey_test,type.predict = "response") %>% 
    conf_mat(truth = experienced, estimate = .pred_class) %>%
    summary(event_level="second")
```

What do we learn from this?

The performance degrades significantly on the test set: the model doesn't quite generalize. And what was a bad model on the training is even more so on the test set!

What could we do to improve matters?

-   choose the right prediction threshold (just as you did in the lab this week). Classifiers predict probabilities, not classes directly. By adjusting the prediction threshold, you can influence the confusion matrix and precision-recall trade-off. Except for ROC/AUC, our metrics implicitly assumed that our prediction threshold was 0.5.

-   our dataset is imbalanced (i.e one class is overrepresented -the class of rookies- compared to the other -the class of experienced developers). We could mitigate the problem by either undersampling the overrepresented class (i.e selecting fewer samples from the overrepresented class) or oversampling the underrepresented class (i.e creating extra samples of the underrepresented class) to balance out the set.

## 1.3 A slightly different problem: predicting remote work

Now I want to look at a slightly different problem. What if instead of predicting whether developers have more then 10 years experience, I tried predicting whether they worked in the office, in hybrid mode or remotely? As you can see, this is no longer a binary classification problem.

🎯 ACTION POINT: Does this really change anything to how I model my problem and evaluate it?

# Part 2: Classifying differently: K-NN

## 2.1 The principles

**Basic idea (images and example from [here](https://www.analyticsvidhya.com/blog/2018/03/introduction-k-neighbours-algorithm-clustering/)):**

Look at this example. Say you want to predict the class of the blue star (BS) and see whether it's a red circle (RC) or a green square (GS).

![](images/paste-1.png)

$K$-NN will take "votes" from the nearest neighbours (nearest defined based on pre-specified distance usually Euclidian distance) and will only look at $K$ neighbours (hence the $K$ in $K$-NN).

In this case, if we take $K=3$, we see that the blue star is surrounded by 3 red circles and no green squares, so the star would be classified as a red circle.

![](images/paste-2.png)

**K-NN: the steps**

1.  Load the data

2.  Initialise the value of \$K\$

3.  To get the predicted class, iterate over the test data points

    -   Calculate the distance between test data point and each row of the training set. By default, use the Euclidian distance. Other distance metrics that could be used are the Manhattan distance, Minkowski distance, Chebyshev, cosine, etc. If there are categorical variables, the Hamming distance can be used.

    -   Sort the calculated distances in ascending order based on distance values

    -   Get top $K$ rows from the sorted array of distances

    -   Get the most frequent class from these rows

    -   Return the most frequent class as the predicted class for the current test data point

## 2.2 Applying it to our problem of predicting whether developers are experienced or not

How about we try $K$-NN to the same prediction problem as before: predicting whether developers are experienced or not

```{r}
knn_fit <-
  nearest_neighbor(neighbors = 5) %>% 
  set_mode("classification") %>% 
  set_engine("kknn") %>% 
  fit(experienced ~ ConvertedCompYearly, data = survey_train)
```

Let's evaluate the model now, first on the training set then the test set:

```{r}
knn_fit %>% 
    augment(new_data=survey_train) %>% 
    conf_mat(truth = experienced, estimate = .pred_class) %>%
    autoplot(type="heatmap")
```

```{r}
knn_fit %>% 
    augment(new_data=survey_train) %>% 
    conf_mat(truth = experienced, estimate = .pred_class) %>%
    summary(event_level="second")
```

```{r}
knn_fit %>% 
    augment(new_data=survey_train) %>% 
    roc_curve(truth = experienced, .pred_Yes, event_level="second") %>%
    autoplot()
```

```{r}
knn_fit %>% 
    augment(new_data=survey_train) %>% 
    roc_auc(truth = experienced, .pred_Yes, event_level="second") 
```

$K$-NN is performing better on the training set than the logistic regression model.

Does it bear out on the test set?

```{r}
knn_fit %>% 
    augment(new_data=survey_test) %>% 
    conf_mat(truth = experienced, estimate = .pred_class) %>%
    autoplot(type="heatmap")
```

```{r}
knn_fit %>% 
    augment(new_data=survey_test) %>% 
    conf_mat(truth = experienced, estimate = .pred_class) %>%
    summary(event_level="second")
```

```{r}
knn_fit %>% 
    augment(new_data=survey_test) %>% 
    roc_curve(truth = experienced, .pred_Yes, event_level="second") %>%
    autoplot()
```

```{r}
knn_fit %>% 
    augment(new_data=survey_test) %>% 
    roc_auc(truth = experienced, .pred_Yes, event_level="second")
```

The performance degrades slightly on the test set but still remains better than that of logistic regression.

📚 **Notes**: When independent variables in training data are measured in different units, it is important to standardize variables before calculating distance. So, it is usually good practice to normalize the dataset before applying $K$-NN. You can do that with `recipes` and `step_normalize`

Because it relies on distance calculations, $K$-NN is sensitive to outliers

## 2.3 What about predicting remote work?

Same as before: I want to look at predicting predicting whether they worked in the office, in hybrid mode or remotely instead.

🎯 ACTION POINT: How should I go about modifying my $K$-NN model and its evaluation?

# Part 3: Cross-validation

## 3.1 What is overfitting/underfitting?

What if I chose $K$ to be 1 in my previous model?

```{r}
knn_fit2 <-
  nearest_neighbor(neighbors = 1) %>% 
  set_mode("classification") %>% 
  set_engine("kknn") %>% 
  fit(experienced ~ ConvertedCompYearly, data = survey_train)
```

```{r}
knn_fit2 %>% 
    augment(new_data=survey_train) %>% 
    conf_mat(truth = experienced, estimate = .pred_class) %>%
    autoplot(type="heatmap")
```

```{r}
knn_fit2 %>% 
    augment(new_data=survey_train) %>% 
    conf_mat(truth = experienced, estimate = .pred_class) %>%
    summary(event_level="second")
```

My training set metrics seems rather unchanged. But, what happens for the test set?

```{r}
knn_fit2 %>% 
    augment(new_data=survey_test) %>% 
    conf_mat(truth = experienced, estimate = .pred_class) %>%
    autoplot(type="heatmap")
```

```{r}
knn_fit2 %>% 
    augment(new_data=survey_test) %>% 
    conf_mat(truth = experienced, estimate = .pred_class) %>%
    summary(event_level="second")
```

My performance degrades more significantly between training and test set than it did when $K$ was 5. This is a sign of **overfitting**: the model fits to the training data too closely and doesn't generalize well to the test data.

**Underfitting** occurs when you have a model that is too simple and can't capture the relationship between input and output variables in the dataset.

Once we're done with training our model (e.g linear/logistic regression), we need to evaluate its out-of-sample performance

We could do that on a test set but:

-   what if we have limited data?
-   what if we're not sure the test set has a similar distribution than the training set?

Simply, using a single training/test set split is not enough to produce a generalizable model.

Enters cross-validation.

## 3.2 What is cross-validation?

-   Enters K-Fold cross-validation, which is a resampling procedure used to evaluate models on limited data:
    -   the data is randomly split into $K$ equal-sized subsamples on which the model is then evaluated ($K$ is usually set as 5 or 10)
    -   the performance of the model is averaged over those subsamples
    -   this makes for a less biased performance since all data would have been seen

![Source:https://upload.wikimedia.org/wikipedia/commons/4/4b/KfoldCV.gif](KfoldCV.gif)

```{r}
knn_spec <-  nearest_neighbor(neighbors = 5) %>% 
             set_mode("classification") %>% 
             set_engine("kknn")
cv <- vfold_cv(survey_train,strata=Gender)
rec <- recipe(experienced ~ ConvertedCompYearly, data = dataset)
w_flow <- 
  workflow() %>% 
  add_recipe(rec) %>% 
  add_model(knn_spec)
res <- fit_resamples(w_flow, cv) %>% collect_metrics()
```