Interpreting and explaining machine learning models

# Interpreting and explaining machine learning models
### Computation Skills Workshop

---

# Interpretation

> Interpretability is the degree to which a human can understand the cause of a decision.

> Interpretability is the degree to which a human can consistently predict the model's result.

.footnote[
Miller, Tim. "Explanation in artificial intelligence: Insights from the social sciences." arXiv Preprint arXiv:1706.07269. (2017).

Kim, Been, Rajiv Khanna, and Oluwasanmi O. Koyejo. "Examples are not enough, learn to criticize! Criticism for interpretability." Advances in Neural Information Processing Systems (2016).
]

---

# Explanation

**Answer to the "why" question**

- Why did the government collapse?
- Why was my loan rejected?
- Why have we not been contacted by alien life yet?

.footnote[Miller, Tim. "Explanation in artificial intelligence: Insights from the social sciences." arXiv Preprint arXiv:1706.07269. (2017).]

Focus on specific observations

Good explanations are:

- Contrastive
- Selected
- Social
- Truthful
- Generalizable

---

# Global vs. local methods

- Interpretation `$\leadsto$` global methods
- Explanation `$\leadsto$` local methods

---

# White-box model

Models that lend themselves naturally to interpretation:

- Linear regression
- Logistic regression
- Generalized linear model
- Decision tree

---

# Black-box model

---

# Black-box model

- Random forests
- Boosted trees
- Neural networks
- Deep learning

---

---

# Predicting student debt

- [College Scorecard](https://collegescorecard.ed.gov/)
- [`rscorecard`](https://github.com/btskinner/rscorecard)

---

# Predicting student debt

```
## Rows: 1,732
## Columns: 14
## $ unitid    <dbl> 100654, 100663, 100706, 100724, 100751, 100830, 100858, 1009…
## $ name      <chr> "Alabama A & M University", "University of Alabama at Birmin…
## $ state     <chr> "AL", "AL", "AL", "AL", "AL", "AL", "AL", "AL", "AL", "AL", …
## $ type      <fct> "Public", "Public", "Public", "Public", "Public", "Public", …
## $ admrate   <dbl> 0.9175, 0.7366, 0.8257, 0.9690, 0.8268, 0.9044, 0.8067, 0.53…
## $ satavg    <dbl> 939, 1234, 1319, 946, 1261, 1082, 1300, 1230, 1066, NA, 1076…
## $ cost      <dbl> 23053, 24495, 23917, 21866, 29872, 19849, 31590, 32095, 3431…
## $ netcost   <dbl> 14990, 16953, 15860, 13650, 22597, 13987, 24104, 22107, 2071…
## $ avgfacsal <dbl> 69381, 99441, 87192, 64989, 92619, 71343, 96642, 56646, 5400…
## $ pctpell   <dbl> 0.7019, 0.3512, 0.2536, 0.7627, 0.1772, 0.4644, 0.1455, 0.23…
## $ comprate  <dbl> 0.2974, 0.6340, 0.5768, 0.3276, 0.7110, 0.3401, 0.7911, 0.69…
## $ firstgen  <dbl> 0.3658281, 0.3412237, 0.3101322, 0.3434343, 0.2257127, 0.381…
## $ debt      <dbl> 15250, 15085, 14000, 17500, 17671, 12000, 17500, 16000, 1425…
## $ locale    <fct> City, City, City, City, City, City, City, City, City, Suburb…
```

---

# Construct some models

---

# Global interpretation methods

---

# Permutation-based feature importance

---

# Permutation-based feature importance

* Calculate the increase in the model's prediction error after **permuting** the feature
    * Randomly shuffle the feature's values across observations
* Important feature
* Unimportant feature

```
For any given loss function do
1: compute loss function for original model
2: for variable i in {1,...,p} do
     | randomize values
     | apply given ML model
     | estimate loss function
     | compute feature importance (permuted loss / original loss)
   end
3. Sort variables by descending feature importance   
```

---

# Random forest feature importance

---

---

# Partial dependence plots

---

# Individual conditional expectation

- *Ceteris peribus* - "other things held constant"
- Marginal effect a feature has on the predictor
- Plot one line per observation that shows how the observation's prediction changes when a feature changes
- Partial dependence plot is average of all ICEs

```
For a selected predictor (x)
1. Determine grid space of j evenly spaced values across distribution of x
2: for value i in {1,...,j} of grid space do
     | set x to i for all observations
     | apply given ML model
     | estimate predicted value
     | if PDP: average predicted values across all observations
   end
```

---

# Net cost

---

# Type

---

# Interpreting with `DALEX`

---

# Local methods

---

# Shapley values

---

# University of Chicago

```
## # A tibble: 1 × 12
##   state type    admrate satavg  cost netcost avgfacsal pctpell comprate firstgen
##   <chr> <fct>     <dbl>  <dbl> <dbl>   <dbl>     <dbl>   <dbl>    <dbl>    <dbl>
## 1 IL    Privat…  0.0617   1528 78555   27315    166923   0.114    0.947    0.202
## # … with 2 more variables: debt <dbl>, locale <fct>
```

---

# Breakdown of random forest

---

# Breakdown of random forest

---

# Breakdown of random forest

---

# Shapley Additive Explanations (SHAP)

---

# Shapley Additive Explanations (SHAP)

- Average contributions of features are computed under different coalitions of feature orderings
- Randomly permute feature order using `$B$` combinations
- Average across individual breakdowns to calculate feature contribution to individual prediction

---

# Shapley Additive Explanations (SHAP)

---

# Shapley Additive Explanations (SHAP)

---

# LIME

---

# LIME

* Global `$\rightarrow$` local
* Interpretable model used to explain individual predictions of a black box model
* Assumes every complex model is linear on a local scale
* Simple model explains the predictions of the complex model **locally**
    * Local fidelity
    * Does not require global fidelity
* Works on tabular, text, and image data

---

# LIME

# LIME

1. For each prediction to explain, permute the observation `$n$` times
1. Let the complex model predict the outcome of all permuted observations
1. Calculate the distance from all permutations to the original observation
1. Convert the distance to a similarity score
1. Select `$m$` features best describing the complex model outcome from the permuted data
1. Fit a simple model to the permuted data, explaining the complex model outcome with the `$m$` features from the permuted data weighted by its similarity to the original observation
1. Extract the feature weights from the simple model and use these as explanations for the complex models local behavior

---

# `$10$` nearest neighbors

---

# Random forest

---

# Explaining with `DALEX` and `lime`