Reviewer notes: Odds ratios are really odd

When psychological researchers investigate binary outcomes,^[1]For example, hit vs. miss, correct vs. incorrect, diagnosis vs. no diagnosis, donated vs. didn’t donate, went on a date vs. sorry you’re just not her type. they routinely report odds ratios to quantify the strength of an effect or an association. This is quite understandable given that psychologists are taught to analyze binary outcomes with binary-logistic regression models, and those models in turn usually return odds ratios, which are usually considered more interpretable than the raw regression coefficients. And odds ratios do have certain nice statistical properties. The problem is that odds ratios, at the same time, are really odd ratios. Here’s a short explanation.^[2]The premise of the “reviewer notes” format is that I write something brief, for the benefit of my co-bloggers who have to proofread everything. For a more comprehensive read, check out Cameron Patrick’s excellent “You are what you ATE: Choosing an effect size measure for binary outcomes.” There are a lot of other things to be said about odds ratios, check out the responses to this Bluesky post for lots of references and the occasional hot take.

Problem 1: Odds ratios are frequently misinterpreted

Let’s say an intervention increases the probability of an outcome from 20% to 40% – that is, the outcome becomes twice as likely. What’s the corresponding odds ratio? If you think the answer is 2, congratulations – you have been interpreting odds ratios wrong all along. Don’t worry about getting ratioed, you are in very good company; the odds that researchers misinterpret odds ratios are fairly high.

If you ended up with the answer 2, you probably simply divided 40% by 20%. If you did so, you did not calculate the odds ratio but rather the risk ratio, sometimes also called relative risk. The risk is the probability of the outcome and the label is applied regardless of whether the outcome is a good thing or not, so we’re stuck with twisted things such as “the relative risk of recovery.”

To calculate the odds ratio, we instead have to start from the odds and divide these. A probability of 40% corresponds to 40% divided by (100% – 40% =) 60%, so the odds of the outcome with treatment are 2/3, which is approximately 0.66666667. A probability of 20% corresponds to 20% divided by (100% – 20% =) 80%, so the odds without treatment are 1/4. To get the odds ratio, we divide 2/3 by ¼. The resulting odds ratio is approximately 2.67.

How can we interpret the odds ratio? Well, quite literally, it’s the ratio of odds. If you routinely handle odds – maybe because you have a gambling habit – it should be self-explanatory. 

It also happens to be the exponentiated log-odds regression coefficient that is usually the default regression output in binary-logistic regression, so if you find that log-odds regression coefficient intuitive, just think of odds ratios as Euler’s number to the power of the log-odds. This is a kink shaming free space.

But if you’re more vanilla: You can think of the odds as “outcomes per non-outcomes.”^[3]Interpretation courtesy of Noah Greifer. For example, let’s say in the example above the outcome was recovery. In the treatment group, there are 2 recovered people for each 3 non-recovered people, or (approximately) 0.67 recoveries per non-recovery. In the control (i.e., no treatment) group, there is 1 recovered person per 4 non-recovered people, or 0.25 recoveries per non-recovery. So, in the treatment group, there are 2.67 times as many recoveries per non-recovery as in the control group, and that’s our odds ratio. Easy peasy lemon squeezy, albeit maybe not the first quantification one would think of.

Problem 2: When aggregating, everything collapses – except for odds ratios

This is the issue of non-collapsibility – another of those twisted labels because intuitively, one might think that it’s bad if things collapse. But here, collapsibility would be desirable,^[4]Fans of Einstürzende Neubauten unite. and odds ratios lack it.

Imagine that for everybody, you knew their probability of the outcome both with and without intervention. In that case, you could calculate an individual-level odds ratio for every person in your sample (e.g., Todd’s ratio).^[5]If that seems odd to you – for the present educational purposes, I endorse probabilistic potential outcomes à la Steyer. Don’t @ me. Now, what’s the odds ratio for the sample as a whole? One may think that it is the average across the individual-level odds ratios, or at least some form of weighted average. But this is not the case, and that’s the issue of non-collapsibility.

Obviously you don’t – and actually can’t – know what happens to everybody if they were in both groups, so this may seem like a fairly abstract problem. It gets a lot more tangible if we think in terms of subgroup effects. For example, imagine that we found out that for women, the effect of the treatment on recovery results in an odds ratio of 2.67. For men, we found an odds ratio of 2.25. What’s the odds ratio across women and men combined? One may naively think that it has to be somewhere between 2.67 and 2.25. Alas, that intuition would only be correct if odds ratios were collapsible, which they aren’t. In this example – see numbers below, across both women and men, the odds ratio turns out to be 1.86 – well below both subgroup specific estimates.

	Women				Men
	Recovery	No recovery			Recovery	No recovery
Treated	40	60		Treated	90	10
Untreated	20	80		Untreated	80	20

	Women + Men
	Recovery	No recovery
Treated	130	70
Untreated	100	100

OR_Women = (40/60)/(20/80) ≈ 2.67

OR_Men = (90/10)/(80/20) = 2.25

OR_Combined = (130/70)/(100/100) ≈ 1.86

To push things even further, it’s possible to generate constellations in which the subgroups have the same odds ratios – but when combined, a different odds ratio results.^[6]For example, if treatment increases the probability of recovery from 20% to 40% for women and from 60% to 80% for men. In that case, both subgroups have an odds ratio of 2.67. For both groups in combination, treatment increases the probability of recovery from 40% to 60%, which corresponds to an odds ratio of 2.25. This has some funny implications if you use odds ratios for clinical decision-making, as explored in this excellent short blog post by Noah Greifer.

Now, when an aggregate association looks different from subgroup associations, people are quick to think that a Simpson’s paradox situation is going on. However, Simpson’s paradox is usually reserved for situations in which some sort of confounding variable is part of the story. And for once, this is not what’s going on here. This one is on odds ratios, and it has all sorts of weird implications. For example, an odds ratio calculated from a coefficient for some predictor will change if another covariate is included that predicts the outcome even if that other covariate has nothing to do with the predictor of interest. You can imagine how that causes headaches when trying to make sense of results across studies with different adjustment sets.

To see that nothing sketchy is going on with the numbers I came up with, we can calculate the risk difference, which happens to be collapsible. For women, treatment increases the probability of recovery from 20% to 40%, or by 20 percentage points. For men, treatment increases the probability of recovery from 80% to 90%, or by 10 percentage points. In the combined sample, treatment increases the probability of recovery from 50% to 65%, or by 15 percentage points – which is right in the middle of the two gender-specific effects.

Recommendations

None of this means that odds ratios are inherently wrong.^[7]That being said, any odds ratio you encounter in the literature may of course still be wrong for other reasons than the choice of metric (such as confounding, p-hacking, and all the other usual suspects). They are just frequently misinterpreted and behave in ways that are counterintuitive for people who are used to collapsibility. Some people even argue that non-collapsibility is a non-issue because according to the argument, the subgroup-specific (i.e., conditional) odds ratios are the stable, transportable quantities that describe reality; the aggregate one is just a population-dependent artifact. And some features of odds ratios are actually desirable in the context of certain designs.^[8] Namely, case-control studies in which researchers start from cases (e.g., people with a certain disease) and then look for controls to figure out what is associated with the disease. Here, odds ratios should come out the same regardless of whether you calculate them “retrospectively” (start from cases, find controls, look back at “pre”dictors) or “prospectively” (start from sample, see who becomes a case).

Either way, the obvious solution here: if you have to report odds ratios for some reason, also report additional metrics. And make sure that all of them are interpreted properly. This requires some degree of precision in language. For example, I sometimes see authors write something along the lines of “the outcome was 20% more likely in the treatment group.” Now, what does that mean? For me, this would most obviously correspond to a risk ratio of 1.2. However, I’ve also seen it used to refer to the fact that the odds increase by 20%, which corresponds to an odds ratio of 1.2. And then some people may use this phrase to refer to a risk difference of 20 percentage points (e.g., from 10% to 30%), so there’s that.^[9]I don’t condone using % when referring to a risk difference, precisely because of the additional ambiguity it introduces. That being said, I don’t think everyone has learned the convention that differences in percentages are expressed in the somewhat convoluted unit “percentage points.” More research precise language is needed.

My usual recommendation would be to additionally report risk differences, which result in statements such as “treatment increases the probability of recovery by 20 percentage points.” Risk ratios are also fine, if one really wants to express something as a ratio. Both of these numbers should be supplemented with percentages to paint a more comprehensive picture; for example, a 10 percentage point increase from 5% to 15% may lead to different conclusions than a 10 percentage point increase from 45% to 55%.

To get these numbers, it’s not necessary to fit a different statistical model – the model just needs to be asked the right question during postestimation. There are various software options for this, my favorite one being the marginaleffects package. Admittedly, I may be biased here. See Rohrer & Arel-Bundock (2026) for an accessible introduction.

Last but not least, some people even prefer the log-odds coefficient that binary logistic models return by default. Konsta Happonen makes the case that these can be interpreted as changes in the mean of the latent process imagined behind the observed data. I imagine this is attractive to some latent variable aficionados, and I don’t want to yuck anybody’s yum. 

Reporting different metrics takes up very little space and so from the perspective of the author, it’s easy to accommodate different preferences. In contrast, it’s hard for a reader to translate between metrics, because often relevant information is missing (any meta-analyst will tell you that). So, be kind to your readers, report meaningful metrics and provide a clear and precise interpretation.

P.S. This was all just about calculating one metric to summarize a finding. Often, psychological researchers will go one step further and compare effects (or associations) – across subgroups, across different causes, across different outcomes, or across different studies. If we change the way we quantify the effect or the association, such comparison can suddenly yield qualitatively different results. For example, looking at an interaction, the risk ratio may indicate a larger effect in group A than in group B while the risk difference may indicate a larger effect in group B than in group A. This is one of those things that feel substantively counterintuitive but are just how numbers work. Luckily, we have written about this issue before – check out the section on the scale dependence of interactions in Rohrer & Arslan (2021).