️Co-authors
Prof Dianne Cook
Department of Econometrics and Business Statistics, Melbourne, Monash University
Dr Emi Tanaka
Research School of Finance, Actuarial Studies and Statistics, Australian National University
A/Prof Susan VanderPlas
Department of Mathematics and Statistics, Utah State University
A/Prof Klaus Ackermann
Department of Econometrics and Business Statistics, Melbourne, Monash University, Australia
Challenges in interpreting residual plots reliably
Heteroskedasticity ?
Vertical spread of the points varies with the fitted values.
Visual inference (Buja, et al. 2009)
H_0: The linear regression model satisfies all model assumptions.
T(X): The residual v.s. fitted plot.
F_0(t): Simulate residuals from the fitted regression model (details on the next slide) to generate a collection of null plots.
Randomly place the observed plot, t_{\mathrm{obs}}, among 19 null plots to form a lineup. Ask N observers to identify the most different plot(s).
p = P(K \geq k) = \sum_{j=k}^{N} \binom{N}{j} \left(\frac{1}{20}\right)^j \left(1-\frac{1}{20}\right)^{N-j}.
Computer vision model
Modern computer vision models are well-suited for addressing this challenge.
Source: https://en.wikipedia.org/wiki/Convolutional_neural_network
To compare observed plot and null plots
How do we define a numerical measure of “difference” or “distance” between residual plots?
- pixel-wise sum of square differences?
- Structural Similarity Index Measure (SSIM)?
- scagnostics?
- …
-
=
?
Distance from “theoretically” good residual plots
Q
\overset{D}{\longrightarrow}
P
However, Q is unknown in practice \Rightarrow D can not be computed.
Simulation of training data
Non-linearity + Heteroskedasticity
Non-normality + Heteroskedasticity
Distribution of predictor
Estimation of D
We train a computer vision model to estimate D with 64,000 simulated residual plots:
\widehat{D} = f_{CV}(V_{h \times w}(\hat{\boldsymbol{e}}, \boldsymbol{\hat{y}})),
where V_{h \times w}(.) generates an h \times w image, and f_{CV}(.) predicts a non-negative distance.
Statistical testing
Example: Boston housing
library(autovi)
fitted_model <- lm(MEDV ~ RM + LSTAT + PTRATIO, data = housing)
checker <- residual_checker(fitted_model)
checker$plot_resid()
rotate_resid()
Null residuals are simulated from the fitted model assuming it is correctly specified.
# A tibble: 489 × 2
.fitted .resid
<dbl> <dbl>
1 632372. 82404.
2 525177. 24363.
3 646753. -16642.
4 624848. 7895.
5 611817. -25387.
6 551051. -128980.
7 504757. -37748.
8 445700. 33616.
9 281912. -17081.
10 453398. -103580.
# ℹ 479 more rows
check() and summary_plot()
── <AUTO_VI object>
Status:
- Fitted model: lm
- Keras model: (None, 32, 32, 3) + (None, 5) -> (None, 1)
- Output node index: 1
- Result:
- Observed visual signal strength: 6.484 (p-value = 0)
- Null visual signal strength: [100 draws]
- Mean: 1.169
- Quantiles:
╔══════════════════════════════════════════╗
║ 25% 50% 75% 80% 90% 95% 99% ║
║1.037 1.120 1.231 1.247 1.421 1.528 1.993 ║
╚══════════════════════════════════════════╝
- Bootstrapped visual signal strength: [100 draws]
- Mean: 6.28 (p-value = 0)
- Quantiles:
╔══════════════════════════════════════════╗
║ 25% 50% 75% 80% 90% 95% 99% ║
║5.960 6.267 6.614 6.693 6.891 7.112 7.217 ║
╚══════════════════════════════════════════╝
- Likelihood ratio: 0.7064 (boot) / 0 (null) = Extremely large 
Example: Left-triangle
Breusch–Pagan test p-value = 0.0457
💡Example: Dinosaur
Ramsey Regression Equation Specification Error test p-value = 0.742
Breusch–Pagan test p-value = 0.36
Shapiro-Wilk test p-value = 9.21e-05
Thanks! Any questions?
