---
title: "🗓️ Week 11 - Modeling with fairness in mind"
subtitle: "Theme: Applications"
author: "Dr. Ghita Berrada"
format: 
  html:
    from: markdown+emoji
    page-layout: full
    toc: true
    toc-depth: 2
self-contained: true
---

**⚙️ Setup**

You'll probably need to install a few libraries before loading them:

``` r

install.packages('baguette')
install.packages('desirability2')
install.packages('GGally')
install.packages('readmission')
```

Let's load the libraries needed today:

```{r}
library(tidymodels)
library(baguette)
library(desirability2)
library(GGally)
library(readmission)
library(naniar)
library(skimr)  
library(rsample)
library(xgboost)
library(kernlab)
library(dplyr)
library(themis)
```

# A machine learning fairness case study: fair prediction of hospital readmission

Until now, we've only been considering model performance under the lens of classic performance metrics such as accuracy, F1-score and AUC to only name a few.

But what if that's not enough?

What if an algorithm can perform well in the classical sense but still be a bad model? In what cases, do you see this happening?

The COMPAS algorithm is the archetype of such an algorithm. For more of technical discussion on COMPAS, see [this description of COMPAS](https://afraenkel.github.io/fairness-book/content/04-compas.html) and [this notebook](https://github.com/propublica/compas-analysis/blob/master/Compas%20Analysis.ipynb) as well as [this analysis](https://rpubs.com/omesner/survival).

## 1.1 Exploring the data

A hospital can train a machine learning model to predict reasonably well whether a diabetes patient will be readmitted within 30 days based on information on the patient's stay at the hospital such as demographics, diagnostic results, payment, and medications. What harms to patients, however, could result from using such a model?

That's what we'll find out in this part.

([Obermayer et al., 2019](https://doi.org/10.1126/science.aax2342)) published a study in 2019 that showed that if the wrong variable was used to predict healthcare need (in their paper, health costs as a proxy for health needs), the outcome was an "algorithm \[that\] falsely concludes that Black patients are healthier than equally sick White patients".

In this part, we follow the same analysis as in [here](https://www.tidymodels.org/learn/work/fairness-readmission/).

For the purposes of our analysis, we use a publicly available dataset containing information on 71,515 hospital stays from diabetes patients. It comes from a study by (Strack et al.,[2014](https://www.tidymodels.org/learn/work/fairness-readmission/#ref-strack2014)), where the authors model the effectiveness of a particular lab test in predicting readmission. A version of that data is available in the [readmission R package](https://simonpcouch.github.io/readmission/).

```{r}
glimpse(readmission)
```

```{r}
skim(readmission)
```

```{r}
vis_miss(readmission)
```

There are plenty of missing values to deal with! But before we do, let's have a closer look at the variable `readmitted`.

The first variable in the `readmission` data, `readmitted`, tells whether the patient was readmitted within 30 days of discharge.

We’ll be using this variable as a proxy for “unmet need for additional care”: readmission within one month indicates that the patient would have benefited from additional attention during their hospital stay.

If a machine learning model consistently identifies lesser need (via prediction of non-readmission) in one subgroup than another, the subgroup allocated lesser need will go without care they’d benefit from. We’d like to train a model that is both fair with regard to how it treats `race` groups and is as efficient as possible (with regards to the usual metrics).

Let's have a closer look at our outcome variable `readmitted` :

```{r}
readmission %>%
  count(readmitted)
```

We are in the presence of a heavily imbalanced dataset: a minority of patients (8.8% of patients) were readmitted within 30 days after being discharged from the hospital.

Now, taking a look at counts of protected classes:

```{r}
readmission %>%
  count(race)
```

A vast majority of patients are labeled `"Caucasian"` (74.8%) or `"African American"` (18%). The counts for the remaining racial categorizations are quite a bit smaller, and when we split the data up into resamples, those counts will reduce even further. As a result, the variability associated with the estimates `"Asian"`, `"Hispanic"`, `"Other"`, and `"Unknown"` will be larger than those for `"African American"` and `"Causasian"` .

For example:

```{r}
readmission %>%
  # randomly split into groups
  mutate(., group = sample(1:10, nrow(.), replace = TRUE)) %>%
  group_by(race, group) %>%
  # compute proportion readmitted by race + random `group`ing
  summarize(prop = mean(readmitted == "Yes"), n = n()) %>%
  # compute variation in the proportion by race.
  # note that, by default, the output from above is grouped by race only.
  summarize(mean = mean(prop), sd = sd(prop), n = sum(n))
```

The standard deviations `sd` are much larger for groups with smaller numbers of observations `n`, even if the average proportions `mean` are similar. This variation affects our analysis downstream as fairness metrics measure variation across groups: the noise associated with proportions calculated for `race` groups with fewer values may overwhelm the signal associated with actual disparities in care for those groups.

We have various options to address this:

-   **Remove rows corresponding infrequent classes.** In this example, this would mean removing all rows with `race` values other than `"Caucasian"` or `"African American"`. This is the approach taken by other public analyses of this data, including the original study that this data came from. An analysis resulting from this approach would ignore any disparities in care for `race` groups other than `"Caucasian"` or `"African American"`.

-   **Construct custom fairness metrics that address the increased variability associated with smaller counts.** We could scale the variability of the estimates for each `race` group before calculating fairness metrics.

-   **Bin the values for infrequent classes.** This would entail collapsing values of `race` other than `"Caucasian"` and `"African American"` into one factor level. This approach is somewhat of a hybrid of the two approaches above: we lose some granularity in information regarding care for `race` groups other than `"Caucasian"` and `"African American"`, but reduce the variability associated with the estimates for those groups in the process.

In this analysis, we go with the last option i.e binning of values of infrequent classes

::: callout-note
We won’t construct custom fairness metrics in this analysis but you could do so using the `new_groupwise_metric()` function in `yardstick`.
:::

Let's recode the `race` column in our dataset:

```{r}
readmission_collapsed <-
  readmission %>%
  mutate(
    race = case_when(
      !(race %in% c("Caucasian", "African American")) ~ "Other",
      .default = race
    ),
    race = factor(race)
  )

readmission_collapsed %>%
  count(race)
```

Now, let's visualise the remaining predictors in the dataset:

```{r}

readmission_collapsed %>%
  ggplot(aes(x = age)) +
  geom_bar() +
  facet_grid(rows = vars(admission_source))
  
```

As can be expected:

-   most patients in this data are in their 60s and 70s

-   emergencies account for most admissions in this data, though many others are from referrals or other sources.

```{r}
readmission_collapsed %>%
  ggplot(aes(x = insurer)) +
  geom_bar()
```

The insurance information for most patients is missing (not covered by insurance at all? or other data entry problem?). Most patients for which the data is available are covered under Medicare (as could be expected since most patients represented in the data are in their 60s or 70s).

::: callout-tip
Insurance/payment method is one way in which societal unfairness may manifest in the source data aside from the variables on protected groups themselves, as also highlighted in the (Obermeyer et al, 2019) study:

-   Medicaid coverage is only available to people below a certain income

-   Insurance cover (and quality) is tied to a job

-   many self-pay patients do not have medical insurance because they cannot afford it

    Relatedly, poverty rates and medical insurance rates differ drastically among racial groups in the U.S.
:::

```{r}
readmission_collapsed %>%
  pivot_longer(starts_with("n_")) %>%
  ggplot(aes(x = value)) +
  geom_histogram() +
  facet_wrap(vars(name), scales = "free_x")
```

For many patients, this was the first inpatient visit to this hospital system. During their stay, many patients receive 10-20 medications and experience several procedures and multiple diagnoses.

## 1.2 Preparing the data for modeling

Now that we have a clearer view of the data, let's first start by splitting it into training and test sets and setting up the parameters for cross-validation.

Because the data is so large (and computation times will take ages!), I'll slice the data to a more manageable size. Because of the analysis I want to do later I need to ensure my slicing preserves the same proportions of `race` and `sex` as in the original data.

```{r}
set.seed(42)
readmission_collapsed <- readmission_collapsed %>%
  mutate(strata = interaction(race, sex, drop = TRUE))
# Step 2: Slice the data (e.g., 10% of the original dataset)
slice_split <- initial_split(readmission_collapsed, prop = 0.05, strata = strata)

# Extract the stratified slice
sliced_data <- training(slice_split)
sliced_data <- sliced_data %>% select(-strata)
```

```{r}
set.seed(42)
readmission_splits <- initial_split(sliced_data, strata = readmitted)
readmission_train  <- training(readmission_splits)
readmission_test   <- testing(readmission_splits)
```

```{r}
readmission_folds <- vfold_cv(readmission_train, strata = readmitted, v=5)
```

Note as usual the use of the `strata` option in both data splitting and creation of cross-validation folds to deal with class imbalance.

It's now time to write a couple of recipes to pre-process our data.

```{r}
recipe_basic <-
  recipe(readmitted ~ ., data = sliced_data) %>%
  step_unknown(all_nominal_predictors()) %>%
  step_dummy(all_nominal_predictors()) %>%
  step_nzv(all_predictors(), -all_outcomes()) %>%
  step_YeoJohnson(all_numeric_predictors()) %>%
  step_normalize(all_numeric_predictors()) %>%
  step_rose(readmitted) 
```

This first recipe uses all the features in the dataset to predict `readmitted` , assigns missing values in a factor level to `"unknown"` , applies a transformation (the Yeo-Johnson transformation) to numeric features so that the distribution of these features becomes more symmetric and closer to a normal distribution, normalizes the numeric features and finally applies a `step_dummy` to convert categorical features into numeric features. We also include a `step_rose` in the recipe, which is the implementation of the **Random Oversampling Examples (ROSE) algorithm:** the rough idea here is that the algorithm tries to estimate the probability distribution of each class contained in the dataset and then draw samples from that estimated distribution to balance out the class samples in the data. We also use `step_nzv` : it's a step that that "diagnoses predictors that have one unique value (i.e. are zero variance predictors) or predictors that have both of the following characteristics:

1.  they have very few unique values relative to the number of samples and

2.  the ratio of the frequency of the most common value to the frequency of the second most common value is large." and then removes them (see [documentation](https://recipes.tidymodels.org/reference/step_nzv.html))

In our data, age is encoded as bins instead of numeric values so the next recipe is a variation on the first one that encodes age as a numeric feature:

```{r}
# e.g. "[10-20]" -> 15
age_bin_to_midpoint <- function(age_bin) {
  # ensure factors are treated as their label
  age <- as.character(age_bin) 
  # take the second character, e.g. "[10-20]" -> "1"
  age <- substr(age, 2, 2)
  # convert to numeric, e.g. "1" -> 1
  age <- as.numeric(age)
  # scale to bin's midpoint, e.g. 1 -> 10 + 5 -> 15
  age * 10 + 5
}

recipe_age <-
  recipe(readmitted ~ ., data = sliced_data) %>%
  step_mutate(age_num = age_bin_to_midpoint(age)) %>%
  step_rm(age) %>%
  step_unknown(all_nominal_predictors()) %>%
  step_dummy(all_nominal_predictors()) %>%
  step_nzv(all_predictors(), -all_outcomes()) %>%
  step_YeoJohnson(all_numeric_predictors()) %>%
  step_normalize(all_numeric_predictors()) %>%
  step_rose(readmitted)
```

Instead of replacing missing values in the two categorical features where they are missing (i.e `blood_glucose` and `insurer` ), we'll try variants of the above recipes that impute the missing values instead using bagged tree imputation:

```{r}
recipe_basic_bag <-
  recipe(readmitted ~ ., data = sliced_data) %>%
  step_impute_bag(all_nominal_predictors()) %>%
  step_dummy(all_nominal_predictors())%>%
  step_nzv(all_predictors(), -all_outcomes()) %>%
  step_YeoJohnson(all_numeric_predictors()) %>%
  step_normalize(all_numeric_predictors()) %>%
  step_rose(readmitted) 

recipe_age_bag <-
  recipe(readmitted ~ ., data = sliced_data) %>%
  step_mutate(age_num = age_bin_to_midpoint(age)) %>%
  step_rm(age) %>%
  step_impute_bag(all_nominal_predictors()) %>%
  step_dummy(all_nominal_predictors())%>%
  step_nzv(all_predictors(), -all_outcomes()) %>%
  step_YeoJohnson(all_numeric_predictors()) %>%
  step_normalize(all_numeric_predictors()) %>%
  step_rose(readmitted) 
```

Now, we're ready to do our modeling.

## 1.3 Training and evaluating models

We'll train a logistic regression model, an XGBoost model and an SVM model (with RBF kernel).

```{r}
spec_lr <- 
  logistic_reg("classification")

spec_bt <-
  boost_tree("classification", mtry = tune(), learn_rate = tune(), trees = 500)

svm_mod <-
  svm_rbf(cost = tune(), rbf_sigma = tune()) %>%
  set_mode("classification") %>%
  set_engine("kernlab")
```

We combine the models within a workflowset (using the recipes we defined previously).

```{r}
wflow_set <- 
  workflow_set(
    preproc = list(basic = recipe_basic, age = recipe_age, basic_bagged = recipe_basic_bag, age_bagged = recipe_age_bag),
    models = list(lr = spec_lr, bt = spec_bt, svm = svm_mod)
  )
```

Up until now, this is fairly standard fare. The interesting part of the modeling comes next in the selection of metrics. This is where on top of the usual metrics, we'll select additional metrics that take into account fairness considerations.

In addition to usual classification metrics, `bal_accuracy()` and `roc_auc()` (selected since since both are suited in the class imbalance setting), we’ll compute a set of fairness metrics: `equal_opportunity()`, `equalized_odds()`, and `demographic_parity()`.

-   `equal_opportunity()`: Equal opportunity is satisfied when a model’s predictions have the same true positive and false negative rates across protected groups. In our case study, a model satisfies equal opportunity if it correctly predicts readmission and incorrectly predicts non-readmission at the same rate across `race` groups. This metric represents the interests of the patient: a patient would want to be just as likely to receive additional care resources as another patient of equal need, and no more likely to go without unneeded care than another. This metric does not consider false positives so it does not penalize disparately providing additional care resources to a patient who may not need them.

-   `equalized_odds()`: Equalized odds is satisfied when a model’s predictions have the same false positive, true positive, false negative, and true negative rates across protected groups. This is a special case of `equal_opportunity`, where additional constraints are placed on the false positive and true negative rates. In our example, a model would satisfy equalized odds if it correctly predicts both readmission and non-readmission *and* incorrectly predicts readmission and non-readmission at the same rate across `race` groups. Just as with equal opportunity, the stakeholders for the metric in this case can be generally understood to be those who are subject to the model’s predictions, except that this metric also aims to prevent disparately 1) providing additional care resources to those who may not need them and 2) identifying patients who do *not* need additional care resources correctly.

-   `demographic_parity()`: Demographic parity is satisfied when a model’s predictions have the same predicted positive rate across groups. In this example, a model satisfies demographic parity if it predicts readmission at the same rate across `race` groups. Note that this metric does not depend on the true outcome value, `readmitted`. The interests of a stakeholder who would like to see additional care resources provisioned at the same rate across `race` groups, even if the actual need for those resources differs among groups, are represented by this metric. Since demographic parity is broadly accepted as part of a legal definition of machine learning fairness, hospital systems might consider this metric to protect themselves legally ([1607.4, n.d.](https://www.tidymodels.org/learn/work/fairness-readmission/#ref-ecfr)).

For each of the above fairness metrics, values closer to zero indicate that a model is fairer.

The above three metrics are defined specifically with fairness in mind. By another view of fairness, though, `bal_accuracy()` and `roc_auc()` are *also* fairness metrics. Some stakeholders may believe that the model that performs best regardless of group membership—i.e. the model that predicts readmission most accurately across groups—is the fairest model.

To evaluate each of these metrics against the specified workflows, we create a metric set as follows:

```{r}
m_set <-
  metric_set(
    bal_accuracy,
    roc_auc,
    equal_opportunity(race),
    equalized_odds(race),
    demographic_parity(race)
  )
```

::: callout-note
The first two inputs, `bal_accuracy()` and `roc_auc()`, are standard [yardstick](https://yardstick.tidymodels.org/) metrics. The latter three are also yardstick metrics like any other, though are created using the [*metric factories*](https://yardstick.tidymodels.org/reference/new_groupwise_metric.html) `equal_opportunity()`, `equalized_odds()`, and `demographic_parity()`. When passed a data-column, metric factories output yardstick metrics.
:::

We can finally proceed to training and evaluating our models.

### Evaluation

We can now evaluate the workflows we’ve defined against resamples using our metric set. The `workflow_map()` function will call `tune_grid()` on each workflow. We use `doParallel::registerDoParallel()` to run operations in parallel and speed up proceedings

```{r}
doParallel::registerDoParallel()

set.seed(42)
wflow_set_fit <- 
  workflow_map(
    wflow_set, 
    verbose = TRUE, 
    seed = 1, 
    metrics = m_set,
    resamples = readmission_folds
  )
```

Let's explore the results

```{r}
wflow_set_fit %>%
  collect_metrics() %>%
  pivot_wider(
    id_cols = c(wflow_id, .config), 
    names_from = .metric, 
    values_from = mean
  ) %>%
  select(-c(wflow_id, .config)) %>%
  ggpairs() +
  theme(axis.text.x = element_text(angle = 45, vjust = .8, hjust = .9))
```

The above code plots a **pair plot** (or scatterplot matrix) that allows us to visualize the relationships and correlations between the different metrics (e.g., `bal_accuracy`, `demographic_parity`, `equal_opportunity`, `equalized_odds`, and `roc_auc`) we selected to measure for our models. Here's how to interpret the plot:

### Key Elements:

1.  **Diagonal Panels**:

    -   These display density plots or histograms of the individual metrics and give insights into the distribution of each metric.

    -   Example: `roc_auc` has a sharp peak around 0.575–0.6, suggesting a concentrated range of values.

2.  **Upper Triangular Panels**:

    -   These show the correlation coefficients between pairs of metrics.

    -   Example: `equalized_odds` and `equal_opportunity` have a near-perfect correlation (0.999\*\*\*), indicating they are essentially equivalent for this dataset.

3.  **Lower Triangular Panels**:

    -   These display scatterplots of pairwise relationships between metrics.

    -   Example: There is a strong positive linear trend between `bal_accuracy` and `demographic_parity`.

4.  **Correlation Coefficients (Numbers) and Significance (`*` or `***`)**:

    -   Asterisks indicate the statistical significance of the correlation.

        -   `***`: Very strong significance (p \< 0.001).

    -   Strong correlations (close to 1 or -1) suggest a robust relationship, while weak correlations (close to 0) suggest little to no relationship.

    -   Example: `bal_accuracy` and `roc_auc` have a very weak correlation (0.081), suggesting these metrics behave independently.

------------------------------------------------------------------------

### Observations:

1.  **Strong Correlations**:

    -   `demographic_parity`, `equal_opportunity`, and `equalized_odds` are strongly positively correlated (correlations \> 0.87). This suggests these fairness metrics behave similarly in this dataset.

2.  **Balanced Accuracy vs Fairness Metrics**:

    -   `bal_accuracy` has moderately strong correlations with fairness metrics (`demographic_parity`: 0.870, `equal_opportunity`: 0.872, etc.), implying that improving accuracy aligns with improving fairness in this context.

3.  **ROC AUC and Fairness Metrics**:

    -   `roc_auc` is weakly negatively correlated with fairness metrics (e.g., -0.134 with `demographic_parity`, -0.206 with `equal_opportunity`). This could indicate a potential trade-off between fairness and discriminative performance (as measured by ROC AUC).

------------------------------------------------------------------------

### Interpretation:

-   **Fairness Metrics Are Highly Interdependent**:

    -   The near-perfect correlations among `demographic_parity`, `equal_opportunity`, and `equalized_odds` suggest they measure overlapping aspects of fairness. You might focus on one or two of these instead of all three to reduce redundancy.

-   **Trade-offs Between Balanced Accuracy and Fairness**:

    -   The positive correlation between `bal_accuracy` and fairness metrics suggests some compatibility between improving balanced accuracy and fairness. However, the weak correlation with `roc_auc` suggests performance improvements measured by AUC might not align with fairness improvements.

-   **Recommendation**:

    -   Depending on the goal of your model (e.g., maximizing fairness or performance), you should prioritize metrics with compatible relationships (e.g., `bal_accuracy` and fairness metrics) and monitor potential trade-offs with `roc_auc`.

So, what this means is a model that performs very well in terms of AUC might not necessarily be the best in terms of fairness! However, best performing models in terms of balanced accuracy also perform best in terms of fairness.

Let's examine the best performing models in terms of balanced accuracy:

```{r}
rank_results(wflow_set_fit, rank_metric = "bal_accuracy") %>%
  filter(.metric == "bal_accuracy")
```

There isn't exactly much difference in terms of balanced accuracy between our models (and the overall balanced accuracy indicates a performance not much better than random) but SVM seems to have a bit of an edge over the other models.

Let's examine the results for the top performing model more closely:

```{r}
autoplot(wflow_set_fit, id = "basic_bagged_svm")
```

Neither `cost` nor `sigma` appear to have an effect on this model's AUC. The fairness metrics are stable and low at low `sigma` values but increase drastically at higher `sigma`'s. They are much more sensitive to `cost` fluctuations. The same can be said for balanced accuracy. The tricky thing here is setting `cost` and `sigma` such that balanced accuracy is as high as possible while keeping fairness metrics low!

::: callout-note
In machine learning fairness, “impossibility theorems” show that fairness definitions “are not mathematically or morally compatible in general” ([Mitchell et al. 2021](https://www.tidymodels.org/learn/work/fairness-readmission/#ref-mitchell2021)). More concretely, unless we live in a world with no inequality, there is no way to satisfy many definitions of fairness at once. However, recent research emphasizes that near-fairness among more limited sets of metrics, like the three we’ve used here, is both possible and relevant ([Bell et al. 2023](https://www.tidymodels.org/learn/work/fairness-readmission/#ref-bell2023)).
:::

To choose a model that performs well both with respect to a typical performance metric like `bal_accuracy()` and the fairness metrics we’ve chosen, we will make use of [desirability functions](https://www.tidyverse.org/blog/2023/05/desirability2/), which allow us to optimize based on multiple metrics at once.

```{r}
best_params <-
  # extract the tuning results for the boosted tree model
  extract_workflow_set_result(wflow_set_fit, "basic_bagged_svm") %>%
  # collect the metrics associated with it
  collect_metrics() %>%
  # pivot the metrics so that each is in a column
  pivot_wider(
    id_cols = c(cost, rbf_sigma), 
    names_from = .metric, 
    values_from = mean
  ) %>%
  mutate(
    # higher balanced accuracy are better; detect max and min from the data
    d_bal_accuracy = d_max(bal_accuracy, use_data = TRUE),
    # lower equalized odds are better; detect max and min from the data
    d_e_odds  = d_min(equalized_odds, use_data = TRUE),
    # compute overall desirability based on d_roc and d_e_odds
    d_overall = d_overall(across(starts_with("d_")))
  ) %>%
  # pick the model with the highest desirability value
  slice_max(d_overall)
```

```{r}
best_params %>% print(width=Inf)
```

We can use the parameters we've just calculated to finalize a workflow that we’ll use to generate our final model fit:

```{r}
final_model_config <-
  extract_workflow(wflow_set_fit, "basic_bagged_svm") %>%
  finalize_workflow(best_params)
```

Finally, generating our final model fit:

```{r}
final_model <- 
  last_fit(final_model_config, readmission_splits, metrics = m_set)
```

We can see the metrics associated with this final fit:

```{r}
collect_metrics(final_model)
```

The model we’ve selected has rather decent in terms of fairness with respect to the set of metrics we’ve chosen here. The balanced accuracy of the model is 52.1%, so not much better than random i.e predicting that someone would be readmitted or not is pretty much close to the toss of a coin! Not exactly good enough or usable in practice! The `roc_auc()` value of 0.558 indicates that the model indeed correctly predicts readmission in some cases, though there is still has lots of room for improvement. A further analysis of these models might measure performance using a metric that specifically evaluates predictions on observations from the minority class—as in, patients that did actually readmit—like [`sens()`](https://yardstick.tidymodels.org/reference/sens.html?q=sens#details).

We extract the model fit from the `last_fit` object:

```{r}
final_model_fit <- extract_workflow(final_model)
```

The `final_model_fit` object is now ready to predict on new data!

Throughout, we’ve demonstrated a fairness-oriented modeling analysis: modeling with fairness in mind is not simply a numerical optimization problem, but a holistic process of examining the moral meanings of fairness in our problem context and how they are represented—or not represented—by various mathematical notions of fairness.

While this analysis allowed us to train models that are rather fair with respect to a limited set of fairness metrics, it leaves many questions to be answered about the impacts of using such a model. Among other things:

-   We ended up with models that were rather fair but not performing well in the classical sense. There is always a tradeoff between fairness and other measures of performance. So how do we ensure fairness without sacrificing model "accuracy" and performance?

-   We’ve only evaluated the fairness of the model with `race` in mind. We also have information on the patient’s `sex`. Could this model behave unfairly with respect to the patient’s `sex`? A paper by ([Straw et al. 2022](https://doi.org/10.1136/bmjhci-2021-100457)) shows, in the context of liver disease detection, that models can be perfectly accurate on all demographics and yet be unfair to minority genders, in particular women, since the features taken into account are not suitable marker of disease for that particular demographic. How about with respect to the intersections of `race` and `sex`? A growing body of research shows us that black women experience pronounced discrimination as patients ([Johnson et al. 2019](https://www.tidymodels.org/learn/work/fairness-readmission/#ref-johnson2019); [Okoro, Hillman, and Cernasev 2020](https://www.tidymodels.org/learn/work/fairness-readmission/#ref-okoro2020); [Gopal et al. 2021](https://www.tidymodels.org/learn/work/fairness-readmission/#ref-gopal2021)).

-   The categorizations of `race` and `sex` that we have access to are coarse. Race/ethnicity and sex/gender are richly and diversely experienced, and the limited set of categories that patients are presented with (or unknowingly assigned to) in healthcare contexts are only a proxy for how a patient may experience them.

-   We’ve used *readmission* as a proxy for *need for additional care*. What factors might influence whether a patient willingly readmits? Or is admitted to the hospital in the first place? Note, especially, the disproportionate financial burden of healthcare for poorer patients.

-   We don’t know how the predictions resulting from this model will be used by practitioners or trusted by patients. The outputs of the model, of course, should be evaluated in conjunction with other evidence collected by a healthcare team throughout a patient’s stay. Knowing this, though, will the model’s outputs be differently interpreted for different groups in the patient population? For example, would a positive prediction of readmission be considered more seriously as evidence for needed additional care for a white man than for others? Further, given historical exploitation and unethical practice, would protected groups believe that algorithmic recommendation for additional care is likely discriminatory and mistrust the recommendations offered to them? (See [Rajkomar et al. 2018](https://www.tidymodels.org/learn/work/fairness-readmission/#ref-rajkomar2018).)

Machine learning models can both have significant positive impacts on our lives and at the same time cause significant harms. Given the tremendous reach of these models in our society, efforts to include fairness as a criteria for evaluating machine learning models are as necessary as ever.

But do you think including fairness metrics is enough to make the models ethical? Or do we need anything else?

**References**

Obermeyer, Ziad, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. “Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations.” Science 366 (6464): 447–53. <https://doi.org/10.1126/science.aax2342>.

607.4, 29 CFR §. n.d. “Information on Impact.”

Bell, Andrew, Lucius Bynum, Nazarii Drushchak, Tetiana Zakharchenko, Lucas Rosenblatt, and Julia Stoyanovich. 2023. “The Possibility of Fairness: Revisiting the Impossibility Theorem in Practice.” In *Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency*, 400–422. FAccT ’23. New York, NY, USA: Association for Computing Machinery. <https://doi.org/10.1145/3593013.3594007>.

Gopal, Dipesh P, Ula Chetty, Patrick O’Donnell, Camille Gajria, and Jodie Blackadder-Weinstein. 2021. “Implicit Bias in Healthcare: Clinical Practice, Research and Decision Making.” *Future Healthcare Journal* 8 (1): 40.

Johnson, Jasmine D, Ifeyinwa V Asiodu, Christine P McKenzie, Christine Tucker, Kristin P Tully, Katherine Bryant, Sarah Verbiest, and Alison M Stuebe. 2019. “Racial and Ethnic Inequities in Postpartum Pain Evaluation and Management.” *Obstetrics & Gynecology* 134 (6): 1155–62.

Kuhn, Max, and Julia Silge. 2022. *Tidy Modeling with r*. "O’Reilly Media, Inc.".

Mitchell, Shira, Eric Potash, Solon Barocas, Alexander D’Amour, and Kristian Lum. 2021. “Algorithmic Fairness: Choices, Assumptions, and Definitions.” *Annual Review of Statistics and Its Application* 8 (1): 141–63. <https://doi.org/10.1146/annurev-statistics-042720-125902>.

Okoro, Olihe N, Lisa A Hillman, and Alina Cernasev. 2020. “‘We Get Double Slammed!’: Healthcare Experiences of Perceived Discrimination Among Low-Income African-American Women.” *Women’s Health* 16: 1745506520953348.

Omi, Michael, and Howard Winant. 1994. *Racial Formation in the United States*. Second. Routledge.

Rajkomar, Alvin, Michaela Hardt, Michael D Howell, Greg Corrado, and Marshall H Chin. 2018. “Ensuring Fairness in Machine Learning to Advance Health Equity.” *Annals of Internal Medicine* 169 (12): 866–72.

Strack, Beata, Jonathan P DeShazo, Chris Gennings, Juan L Olmo, Sebastian Ventura, Krzysztof J Cios, John N Clore, et al. 2014. “Impact of HbA1c Measurement on Hospital Readmission Rates: Analysis of 70,000 Clinical Database Patient Records.” *BioMed Research International* 2014.

Straw, Isabel, Honghan Wu - "Investigating for bias in healthcare algorithms: a sex-stratified analysis of supervised machine learning models in liver disease prediction". BMJ Health & Care Informatics 2022;29:e100457. <https://doi.org/10.1136/bmjhci-2021-100457>