If you have two measures of the same confounder, you can just include both of them in your regression model

Corrigendum: There was an embarrassing mistake in the SEM part of the original version of this blog post which has now been fixed. Many thanks to David Kenny for catching it! Scroll to the very end if you want to know what I got wrong.

Sometimes, researchers worry about multicollinearity in situations where it’s actually a non-issue. Here’s one such scenario.

Imagine a situation where you are interested in the effect of X on Y (X → Y). You have measured some confounding construct (X ← construct → Y) with two different tests, covariate 1 and covariate 2. For example, you may be interested in how math self-concept (X) affects school grades (Y), and consider cognitive abilities a relevant confounder (math self-concept ← cognitive abilities → school grades), which you thus measure with two different cognitive tests (covariate 1, covariate 2). You may use two tests just to be on the safe side, or maybe one is an established measure whereas the other one is more novel but has better psychometric characteristics, or maybe whoever planned the study just threw in a bunch of measures, just for good measure.^[1]The two measures are imperfect proxies, so you can also think of this as a scenario with measurement error in the confounding construct, see Westfall & Yarkoni, 2016. But for now, I want to focus on what happens in a regression setting, rather than in an SEM setting, where people usually wouldn’t even start to think about multicollinearity (although it can sometimes lead to convergence issues, which is a slightly different topic). Figure 1 shows the situation I have simulated. I simulated a sample with 200 observations and repeated the whole thing 1,000 times, you can find the code here.

Covariate 1 and covariate 2 naturally end up highly correlated (here, r = .63), and this is often taken as a sign that they shouldn’t both be included at the same time because of multicollinearity. The main concern is usually that it makes the analysis unstable, or in some way unreliable, or maybe difficult to interpret. So, for our example, the fear is that including both measures of cognitive abilities messes up our conclusions regarding the effect of math self-concept on school grades.

To prevent messing things up, researchers may come up with workarounds, such as including only the more reliable one, or averaging the two tests. In case there are even more covariates involved, I have even seen people conduct a principal component analysis (PCA) to extract orthogonal components from their covariates. But what precisely happens when you just ignore the whole multicollinearity issue and include both?

Multicollinearity…

Recall that we are interested in X → Y. Thus, the coefficient that is of main interest to us is the coefficient of X (e.g., math self-concept), which tells us how the predicted Y (e.g., school grades) changes when we increase X while holding constant the covariates (e.g., cognitive abilities), which in the ideal scenario captures the effect of X on Y. However, the model also returns coefficients for our covariates 1 and 2 (e.g., two cognitive tests), and these are the two where multicollinearity kicks in; our model will be quite uncertain about their precise values.

Recall that in a multiple regression analysis, every coefficient makes a statement about what happens to the outcome when all other predictors are held constant. So, if covariate 1 increases by one point, but covariate 2 is held constant, what happens to the outcome? Both usually go up together, given that they are supposed to measure the same thing—a person who scores high on one test of cognitive abilities will also tend to score high on another one—and so this is a hard question for our model to answer. There just isn’t much information in the data about what happens when they vary independently. So, the model returns the numerical equivalent of a shruggie: wide standard errors. This wouldn’t have happened if you had listened to your parents included only one covariate, in which case you would have asked your model an easier question, resulting in narrower standard errors. In our simulated case, averaged across 1,000 repetitions, the standard errors go up by 10% to 21% (covariate 1: from 0.064 to 0.070, covariate 2: from 0.038 to 0.046; see table below for all values). This may not sound overly dramatic, but this standard error “inflation” increases when the covariates are even more correlated.

…but it’s a non-issue

But what happens beyond that? What does it mean for the thing that we are actually interested in, the coefficient of X? Nothing bad, actually. If you think of your model as a lean, mean prediction machine,^[2]A perspective spelled out in an earlier blog post and a recent preprint, but this idea goes way back. it’s still working perfectly fine. 

While the model is uncertain whether it’s covariate 1 or covariate 2 or a combination of them that does the heavy lifting, this uncertainty actually doesn’t matter for the rest of the model. That’s because when one of the covariates goes up, the other one does so, too. So, it doesn’t matter how precisely any predicted increase in the outcome is split between the two of them, in the end you come out at the same place.^[3]On Bluesky, Chris Adams pointed me toward chapter 23 in a textbook by Goldberger that discusses that while multicollinearity hinders precise inferences about individual coefficients, it can facilitate precise inferences about their sum (p.251). This is like a situation in which we observe two people pull up an heavy item with a rope. If one pulls for real while the other always just pretends to pull simultaneously, we have trouble distinguishing their contributions. Still, we can easily predict what will happen when they both move.

Considering the target quantity of interest—the effect of X on Y—the standard errors don’t suffer the least. If we include only one covariate, we get an average standard error of 0.076 to 0.081. If we include both, we get an average standard error of 0.075, which is actually a bit tighter. So, the standard errors certainly don’t speak against including both covariates. And when considering the point estimate, on average it’s actually less biased from its true value (here 0.3) when including both covariates (average estimate 0.380) instead of just one (average estimate 0.522 when including only Covariate 1, 0.395 when including only Covariate 2)!

That’s because any of the covariates is an imperfect measure of the actual confounder, with covariate 1 being worse than covariate 2 (because it has a lower loading from the latent factor, 1*true_confound versus 2*true_confound, see Figure 1 above). So, if we include only one covariate, that leaves some residual confounding, and then if we additionally include the second one, it picks up some of that residual confounding.^[4]This presumes the data-generating mechanism presented in Figure 1. As always, when the true data-generating mechanism isn’t known, there’s no guarantee that throwing in more covariates will reduce bias. 

In the end, we still end up with some bias, because even in combination, the covariates aren’t perfect. However, including both covariates is definitely preferable over including just one of them. So, this is a situation where there is multicollinearity, and it does affect the estimation of certain coefficients in the regression model, but that is no problem whatsoever for our research question.

Addendum: Florian Pargent has looked at a modified scenario in which one covariate is not measurement invariant across levels of X. You can read up on his results in his blog post titled “If you have two measures of the same confounder, it might be better to only control for the measurement-invariant one.”

TL;DR

Analysis: Y ~ X +…	Average estimate of effect of X	Average standard error	Bias (true value = 0.3)	RMSE of effect of X*	Note
Covariate 1	0.522	0.081	0.223	0.237	Large bias because covariate 1 is a mediocre measure of the confounder
Covariate 2	0.395	0.076	0.096	0.123	Less bias because covariate 2 is a better measure of the confounder
Covariate 1 + Covariate 2	0.380	0.075	0.080	0.112	Including both gets rid of some residual confounding without increasing the standard error of the effect of interest
*mean(Covariate 1, Covariate 2)	0.387	0.075	0.087	0.117	Including the average isn’t quite optimal because it assigns equal weights to covariates that vary in reliability
*latent, with latent =~ Covariate 1 + Covariate 2	0.300	0.084	0	0.087	No more bias! However, standard errors go up a bit. On average, we end up the closes to truth

*Supplementary analyses: What if we average the covariates? What about latent variable modeling?

Averaging the covariates

I mentioned before that sometimes people will average covariates and then only include the average. What happens if we do this in our simulated scenario? The answer is that we end up somewhat closer to the truth (average estimate of 0.387 when the true value is 0.300) than if we had included any of the covariates in isolation. However, in this particular scenario, we end up slightly worse than if we had included both covariates in isolation (which gave us, on average, 0.380). This is because the average gives us essentially an estimate of the latent variable, but it assigns each covariate the same weight, when in fact covariate 1 is a worse measure than covariate 2. So, our home-made aggregate measure does not provide the optimal weighting of the two covariates.^[5]If we had set the loadings for both covariates to the same value in the simulation, both covariates would be equally good measures, and the average would have been optimally weighted. There’s a nifty finding in econometrics that there is a way to get the optimal weighting—and that’s by including both covariates in a regression model. Check out this aptly named blog post, “Black Magic”,^[6]Thanks to Per Engzell, who pointed me to this. but note that there, the estimand of interest is different—it’s the coefficient of the latent construct. According to my limited understanding of the matter, I think this may imply that our analysis, in which we include both covariates separately, is the best we can do with a simple regression setup. 

(Pragmatically speaking, averaging the covariates may give you very similar results, as long as the covariates have about the same amount of reliable variance.)

SEMagic

Could we do better if we don’t just stick with simple regression and instead fit a latent variable model, with a latent confounder that is measured by the two covariates? It turns out that the answer is yes—here, if we fit a latent variable model, we get an average estimate that is actually unbiased (average estimate of 0.300). This does come at the cost of increased uncertainty; the standard error is now on average 0.084, somewhat higher than in our regression models (where it was 0.075). So, this is the first time in this post that we encounter a situation in which we trade a reduction in bias for an increase in variance. However, notice that the comparison might not be the most useful one: the standard errors tells us how much individual estimates vary around the mean estimate—and for the latent variable model, that mean estimate is on point; for the regression model, that mean estimate is already off. If our main concern is getting the right answer (rather than just a tight standard error in our analysis output), then we should calculate the root mean square error (RMSE in the table above) which quantifies how much individual estimates vary around the ground truth.^[7]Many thanks to Charles Driver who raised that point on Bluesky. And here, the latent variable variable clearly comes out on top with a RMSE of 0.087 compared to a RMSE of 0.112 in the regression model with both covariates. So, clearly, the latent variable model is a pretty sweet deal: We end up the least wrong, on average.

Further readings

Ben Bolker has put together a reading list titled “Why you shouldn’t worry about collinearity.”

Corrigendum

In the initial version of the manuscript, I wrote that the SEM model here was not identified because there are only two indicators. That’s actually wrong! While a latent factor model with two indicators is underidentified in isolation, this is no longer the case once the arrow to X and Y are included. So, SEM it is.