Heteroskedasticity-consistent standard errors

History

Heteroskedasticity-consistent standard errors are introduced by Friedhelm Eicker, and popularized in econometrics by Halbert White.

Problem

Consider the linear regression model for the scalar y.

: y = \mathbf{x}^{\top} \boldsymbol{\beta} + \varepsilon, ,

where \mathbf{x} is a k × 1 column vector of explanatory variables (features), \boldsymbol{\beta} is a k × 1 column vector of parameters to be estimated, and \varepsilon is the residual error.

The ordinary least squares (OLS) estimator is

: \widehat \boldsymbol{\beta}_\mathrm{OLS} = (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^\top \mathbf{y}. ,

where \mathbf{y} is a vector of observations y_i, and \mathbf{X} denotes the matrix of stacked \mathbf{x}_i values observed in the data.

If the sample errors have equal variance \sigma^2 and are uncorrelated, then the least-squares estimate of \boldsymbol{\beta} is BLUE (best linear unbiased estimator), and its variance is estimated with

: \hat{\mathbb{V}}\left[\widehat\boldsymbol\beta_\mathrm{OLS}\right] = s^2 (\mathbf{X}^{\top}\mathbf{X})^{-1}, \quad s^2 = \frac{\sum_{i=0}^n \widehat \varepsilon_i^2}{n-k}

where \widehat \varepsilon_i = y_i - \mathbf{x}i^{\top} \widehat \boldsymbol{\beta}\mathrm{OLS} are the regression residuals.

When the error terms do not have constant variance (i.e., the assumption of \mathbb{E}[\varepsilon\varepsilon^{\top}] = \sigma^2 \mathbf{I}_n is untrue), the OLS estimator loses its desirable properties. The formula for variance now cannot be simplified:

: \mathbb{V}\left[\widehat\boldsymbol\beta_\mathrm{OLS}\right] = \mathbb{V}\big[ (\mathbf{X}^{\top}\mathbf{X})^{-1} \mathbf{X}^{\top}\mathbf{y} \big] = (\mathbf{X}^{\top}\mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{\Sigma} \mathbf{X} (\mathbf{X}^{\top}\mathbf{X})^{-1}

where \mathbf{\Sigma} = \mathbb{V}[\varepsilon].

While the OLS point estimator remains unbiased, it is not "best" in the sense of having minimum mean square error, and the OLS variance estimator \hat{\mathbb{V}} \left[ \widehat \boldsymbol{\beta}_\mathrm{OLS} \right] does not provide a consistent estimate of the variance of the OLS estimates.

For any non-linear model (for instance logit and probit models), however, heteroskedasticity has more severe consequences: the maximum likelihood estimates of the parameters will be biased (in an unknown direction), as well as inconsistent (unless the likelihood function is modified to correctly take into account the precise form of heteroskedasticity). As pointed out by Greene, “simply computing a robust covariance matrix for an otherwise inconsistent estimator does not give it redemption.”

Solution

If the regression errors \varepsilon_i are independent, but have distinct variances \sigma^2_i, then \mathbf{\Sigma} = \operatorname{diag}(\sigma_1^2, \ldots, \sigma_n^2) which can be estimated with \widehat\sigma_i^2 = \widehat \varepsilon_i^2. This provides White's (1980) estimator, often referred to as HCE (heteroskedasticity-consistent estimator):

: \begin{align} \hat{\mathbb{V}}\text{HCE} \big[ \widehat \boldsymbol{\beta}\text{OLS} \big] &= \frac{1}{n} \bigg(\frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^{\top} \bigg)^{-1} \bigg(\frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^\top \widehat{\varepsilon}_i^2 \bigg) \bigg(\frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^{\top} \bigg)^{-1} \ &= ( \mathbf{X}^{\top} \mathbf{X} )^{-1} ( \mathbf{X}^{\top} \operatorname{diag}(\widehat \varepsilon_1^2, \ldots, \widehat \varepsilon_n^2) \mathbf{X} ) ( \mathbf{X}^{\top} \mathbf{X})^{-1}, \end{align}

where as above \mathbf{X} denotes the matrix of stacked \mathbf{x}_i^{\top} values from the data. The estimator can be derived in terms of the generalized method of moments (GMM).

Also often discussed in the literature (including White's paper) is the covariance matrix \widehat\mathbf{\Omega}_n of the \sqrt{n}-consistent limiting distribution:

: \sqrt{n}(\widehat \boldsymbol{\beta}_n - \boldsymbol{\beta}) , \xrightarrow{d} , \mathcal{N}(\mathbf{0}, \mathbf{\Omega}),

where

: \mathbf{\Omega} = \mathbb{E}[\mathbf{X} \mathbf{X}^{\top}]^{-1} \mathbb{V}[\mathbf{X} \boldsymbol{\varepsilon}]\operatorname \mathbb{E}[\mathbf{X} \mathbf{X}^{\top}]^{-1},

and

: \begin{align} \widehat\mathbf{\Omega}_n &= \bigg(\frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^{\top} \bigg)^{-1} \bigg(\frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^{\top} \widehat \varepsilon_i^2 \bigg) \bigg(\frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^{\top} \bigg)^{-1} \ &= n ( \mathbf{X}^{\top} \mathbf{X} )^{-1} ( \mathbf{X}^{\top} \operatorname{diag}(\widehat \varepsilon_1^2, \ldots, \widehat \varepsilon_n^2) \mathbf{X} ) ( \mathbf{X}^{\top} \mathbf{X})^{-1} \end{align}

Thus,

: \widehat \mathbf{\Omega}n = n \cdot \hat{\mathbb{V}}\text{HCE}[\widehat \boldsymbol{\beta}_\text{OLS}]

and

: \widehat \mathbb{V}[\mathbf{X} \boldsymbol{\varepsilon}] = \frac{1}{n} \sum_i \mathbf{x}_i \mathbf{x}_i^{\top} \widehat \varepsilon_i^2 = \frac{1}{n} \mathbf{X}^{\top} \operatorname{diag}(\widehat \varepsilon_1^2, \ldots, \widehat \varepsilon_n^2) \mathbf{X}.

Precisely which covariance matrix is of concern is a matter of context.

Alternative estimators have been proposed in MacKinnon & White (1985) that correct for unequal variances of regression residuals due to different leverage. Unlike the asymptotic White's estimator, their estimators are unbiased when the data are homoscedastic.

Of the four widely available different options, often denoted as HC0-HC3, the HC3 specification appears to work best, with tests relying on the HC3 estimator featuring better power and closer proximity to the targeted size, especially in small samples. The larger the sample, the smaller the difference between the different estimators.

An alternative to explicitly modelling the heteroskedasticity is using a resampling method such as the wild bootstrap. Given that the studentized bootstrap, which standardizes the resampled statistic by its standard error, yields an asymptotic refinement, heteroskedasticity-robust standard errors remain nevertheless useful.

Instead of accounting for the heteroskedastic errors, most linear models can be transformed to feature homoskedastic error terms (unless the error term is heteroskedastic by construction, e.g. in a linear probability model). One way to do this is using weighted least squares, which also features improved efficiency properties.

Software

• EViews: EViews version 8 offers three different methods for robust least squares: M-estimation (Huber, 1973), S-estimation (Rousseeuw and Yohai, 1984), and MM-estimation (Yohai 1987).
• Julia: the CovarianceMatrices package offers several methods for heteroskedastic robust variance covariance matrices.
• MATLAB: See the hac function in the Econometrics toolbox.
• Python: The Statsmodel package offers various robust standard error estimates, see statsmodels.regression.linear_model.RegressionResults for further descriptions
• R: the vcovHC() command from the package or lm_robust() from the package.
• RATS: option is available in many of the regression and optimization commands (, , etc.).
• Stata: robust option applicable in many pseudo-likelihood based procedures.
• Gretl: the option --robust to several estimation commands (such as ols) in the context of a cross-sectional dataset produces robust standard errors.

References

References

1. (2006). "Applied Econometrics with R". UseR-2006 conference.
2. Eicker, Friedhelm. (1967). "Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability".
3. Huber, Peter J.. (1967). "Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability".
4. White, Halbert. (1980). "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity". [[Econometrica]].
5. (2015). "How Robust Standard Errors Expose Methodological Problems They Do Not Fix, and What to Do About It". Political Analysis.
6. (1963). "Asymptotic Normality and Consistency of the Least Squares Estimators for Families of Linear Regressions". The Annals of Mathematical Statistics.
7. (January 1967). "Limit theorems for regressions with unequal and dependent errors". Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics.
8. Giles, Dave. (May 8, 2013). "Robust Standard Errors for Nonlinear Models". Econometrics Beat.
9. Guggisberg, Michael. (2019). "Misspecified Discrete Choice Models and Huber-White Standard Errors". [[Journal of Econometric Methods]].
10. Greene, William H.. (2012). "Econometric Analysis". Pearson Education.
11. (1985). "Some Heteroskedastic-Consistent Covariance Matrix Estimators with Improved Finite Sample Properties". [[Journal of Econometrics]].
12. Long, J. Scott. (2000). "Using Heteroscedasticity Consistent Standard Errors in the Linear Regression Model". The American Statistician.
13. C., Davison, Anthony. (2010). "Bootstrap methods and their application". Cambridge Univ. Press.
14. "EViews 8 Robust Regression".
15. [https://github.com/gragusa/CovarianceMatrices.jl CovarianceMatrices: Robust Covariance Matrix Estimators]
16. "Heteroskedasticity and autocorrelation consistent covariance estimators". Econometrics Toolbox.
17. [https://cran.r-project.org/web/packages/sandwich/index.html sandwich: Robust Covariance Matrix Estimators]
18. (2008). "Applied Econometrics with R". Springer.
19. [https://cran.r-project.org/web/packages/estimatr/index.html estimatr: Fast Estimators for Design-Based Inference]
20. See online help for [https://www.stata.com/manuals15/p_robust.pdf _robust] option and [https://www.stata.com/manuals15/rregress.pdf regress] command.
21. "Robust covariance matrix estimation". Gretl User's Guide, chapter 22.