The 100% CI

TL;DR: What’s an age-effect net of all time-varying covariates?
The sound of one hand clapping.

Recently, we submitted a paper with some age trajectories of measures of individuals’ (un-)well-being. We thought of these trajectories in the most descriptive way: How do these measures change across the life course, all things considered? And really while this might not be the most interesting research question because it doesn’t directly answer why stuff happens, I’m a fan of simple descriptive studies and think they should have a place in our domain; Paul Rozin wrote a great piece on the importance of descriptive studies.

Anyway, the editor asked us to justify why we did not include any time-varying covariates (e.g. income, education, number of children, health) in our analysis of age trajectories. I thought the editor had requested an actual justification; my co-author (an economist) thought the editor just wanted to tell us that we should throw in all sorts of covariates. I felt too lazy to re-run all analyses and create new figures and tables, plus I always get a weird twitch in my left eye when somebody asks for “statistical control” without additional justification, so instead I looked into the (scientific) literature on the midlife crisis and tried to figure out how people have justified the inclusion of control variables in the analyses of age effects on well-being.^[1]Ruben, on the other hand, would probably get a twitch in his right eye if he found out I did not automate making my figures and tables.

Cat ownership, a time-varying covariate. (Pic: pixabay.com)

Whether or not life satisfaction dips in middle adulthood (somewhere between age 45-64) before rising again in older age^[2]before dipping again in a terminal decline. If you read that footnote, you’re now in the mortality salience condition of my Terror Management Theory study. has been hotly debated by psychologists and economists. There are a lot of papers out there on that subject and personally, I’m totally agnostic regarding the existence of the midlife crisis – ask me again in 20 years, if I’m not too busy driving my Porsche. But there are a lot of interesting methodological questions that arise when trying to answer this question.

A brief list of stuff I don’t want to talk about in this post, which are important nonetheless:

• the Age-Period-Cohort conundrum: In short, this requires us to make certain assumptions when we want to identify age/period/cohort effects. That’s okay though, every researcher needs to make assumptions from time to time.
• longitudinal vs. cross-sectional data: Both can have their pros and cons.
• statistical control for covariates that don’t change over time, such as gender or race, as systematic differences in the composition in the age groups can cause issues
• what we can learn from lab studies in which researchers recruit older people and then compare their performance on an arbitrary task X to the performance of their convenient undergraduate sample. How do you reasonably match 60 year old people that decided to participate in lab studies onto a younger sample of psych majors that really just want to get their freakin’ course credit?
• lots of other interesting stuff you can do with longitudinal data that is more interesting than simple descriptive trajectories

But let’s get back to the topic our editor raised: Should we control for time-varying covariates such as income, marital status, health? The logic seems straightforward: Wouldn’t we want to “purge” the relationship between age and life satisfaction from other factors?

Quite obviously, a lot of stuff changes as we age. We get older, get our degrees, start a decent job and make some money,^[3]Or, alternatively, start a blog. maybe marry and settle down or travel to an Ashram to awaken our inner goddess and spite our conservative parents or maybe just get a lot of cats.

Mount Midlife Crisis, not to be confused with Mount Doom. (Art: Hakuin Ekaku)

To control for these variables might be wrong for two distinct reasons, and I will start with the somewhat more obscure one.

First, our time-varying covariate might actually be causally affected by life satisfaction. This argument has been raised regarding the statistical control of marital status by the late Norval Glenn (2009). He simulated a data situation in which (1) life satisfaction is stable across the life course and (2) starting from 21, only the 10 happiest people marry each year. He then demonstrated that controlling for marital status will result in a spurious trajectory, that is, a pronounced decline of life satisfaction over the life course even though we know that there’s no age effect in the underlying data. If you have read this blog before and the data situation sounds somewhat familiar to you: Marital status would be one of the infamous colliders that you should not control for because if you do, I will come after you.^[4]And you should be scared because I can deliver angry rants about inapproriate treatment of third variables. I might bring cookies and hope that you have decent coffee because this might take longer. If marital status is affected by age (the older you are, the more likely you are to be married), and if satisfied people are more likely to marry, marital status becomes a collider of its two causes and should not be controlled.

The second reason is somewhat more obvious: In many cases, the time-varying covariates will mediate the effects of age on your outcome. That is probably most obvious for health: Health declines with age. Decreases in health affect life satisfaction.^[5]They obviously do, though the fact that life satisfaction remains stable until a certain age despite decreases in health has been labeled the happiness paradox. So life satisfaction might decrease with age because of declining health. Now what does it mean if we control for this potential mediator?

Well, it means that we estimate the age effect net of the parts that are mediated through health. That is not inherently nonsensical, we just have to interpret the estimate properly. For example, Andrew Oswald was cited in Vol. 30 of the Observer: “[But] encouragingly, by the time you are 70, if you are still physically fit then on average you are as happy and mentally healthy as a 20 year old.” Now this might be indeed encouraging for people who think they are taking great care of their health and predict that they will be healthy by the time they are 70; but whether it’s encouraging on average strongly depends on the average health at age 70.
For example, if we assume that only the luckiest 1% of the population will be physically fit at that age, 99% will end up unhappier than 20 year olds (whether or not 20-year-olds are very happy is a different question). That doesn’t sound very optimistic any more, does it? The lucky one percent might also be very special with respect to other characteristics such as income, and a message such as “the wealthy will be still happy at age 70, whereas the poor are wasting away because of a lack of health care” again sounds not very encouraging. For the record, I’m not claiming that this is happening, but those are all scenarios that are aligned with the simple statement that those who are physically fit at age 70 are as mentally healthy as 20 year olds.

So the estimated association has its own justification but must be interpreted carefully.^[6]Actually, there is yet another problem that has been pointed out by Felix Thoemmes in the comments section of this post. A mediator is also almost always a collider, as it is per definition caused by the independent variable of interest and (most likely) by some other factors. So you would actually have to control the backdoor paths from the mediator in turn, or else your estimate of the direct effect, whatever it reflects, will actually be biased again. Additionally, it renders the “remaining” age effect hard to interpret, so it might not be very enlightening to look at age effects net of the effects of time-varying covariates. Let’s assume you “control” all sorts stuff that happens in life as people age – marital status, education, income, number of children, maybe also number of good friends, cat ownership, changes in health, and when we are already at it, why don’t we also control for the stuff that is underlying changes in health, such as functioning of organs and cell damage? – and still find a significant age effect.

What does that mean? Well, it means that you haven’t included all time-varying covariates that are relevant to life satisfaction because age effects must necessarily be mediated through something. The sheer passing of time only has effects because stuff happens in that time.
The “stuff” might be all sorts of things, and we might be inclined to consider that stuff more or less psychologically meaningful. For example, we might not consider changes in marital status or physical health to be “genuinely psychological”, so we might decide to control for these things to arrive at a purely psychological age effect. Such a “purely psychological” age effect might then be driven by e.g. people’s attitude towards the world. For example, people might get more optimistic and thus more satisfied controlling for other life circumstances. But I would again be careful with those interpretations, because of the collider problem outlined before and because of the somewhat arbitrary distinction between physical changes and social role changes as opposed to psychological changes.

In other words: what you should or shouldn’t control for always depends on your research question. If you study living things and control for life, don’t be surprised if your results seem a bit dull.

April Fool’s Post 2017

“The pursuit of knowledge is, I think, mainly actuated by love of power. And so are all advances in scientific technique.”
― Bertrand Russell

Today on the 100% CI we have to stop the jokes for a moment. Today, we will talk about being disappointed by your idols, about science criticism being taken too far. Here, we reveal how Dr. Andrew Gelman, the prominent statistician and statistics blogger, abused his power.

We do not make this accusation lightly, but our discoveries leave no other conclusion: In his skepticism of psychological science, Dr. Gelman lost sight of right and wrong. What follows is a summary of the evidence we obtained.

This March the four of us visited the Applied Statistics Center at Columbia University for a brief workshop on Stan, the probabilistic programming language. We were excited about this opportunity to learn from one of our personal heroes.

The course, however, did not live up to our expectations. Frequently, Dr. Gelman would interrupt the course with diatribes against psychological science. On the second-to-last afternoon, we were supposed to write our first Stan code in a silent study session. We were left alone in the Gelman lab and our minds wandered. Our attention was drawn to a particularly large file drawer that turned out to be unlocked. What we discovered can only be described as profoundly shocking:

Inside were literally dozens of study reports on Power Posing. At first glance, this did not seem particularly strange as Dr. Gelman is known to write frequently about this phenomenon on his blog.

But this particular file drawer problem was different: The lab log revealed that Dr Gelman was desperate to obtain evidence against the phenomenon – and failed repeatedly. Initially, he invested enormous resources to run experiments with extraordinary four digit sample sizes to “nail the coffin shut on Power Pose”, as a hand-written note on an early report reads. The data painted a very clear picture, and it was not to his liking. As it dawned on him that, contrary to his personal convictions, Power Posing might be a real phenomenon, he began to stack the deck.

Instead of simple self-reports, he tried manifest behavioral observations and even field studies where the effect was expected to vanish. Power Pose prevailed. He deliberately reduced study samples to the absurdly low numbers often criticized on his very own blog. But even in his last attempts with 1-β almost equal to ɑ: Power Pose prevailed. As more and more evidence in favor of Power Posing was gathered, the research became… sloppy. Conditions were dropped, outliers removed, moderators randomly added, and, yes, even p-values were rounded up. Much to Dr. Gelman’s frustration, Power Pose prevailed. He was *unable* to collect data in favor of the null hypothesis.

He thought he had one final Bayesian trick up his sleeve: By hiring a skilled hypnotist he manipulated his priors, his own beliefs (!) in Power Posing. But even with these inhumane levels of disbelief, the posterior always indicated beyond a doubt: Power Pose prevailed. It was almost like the data were trying to tell him something – but Dr. Gelman had forgotten how to listen to evidence a long time ago.

In a recent publication, Simmons and Simonsohn analyzed the evidential value of the published literature on Power Posing. The centerpiece of their research is a p-curve (figure below, left graph) on the basis of which they “conclusively reject the null hypothesis that the sample of existing studies examines a detectable effect.” Had Dr. Gelman not hidden his findings in a file drawer, Simmons and Simonsohn’s conclusions would have been dramatically different (right graph).

Initially, we couldn’t believe that he would go this far just to win an argument. We were sure there must have been some innocuous explanation – yet we also did not want to confront him with our suspicions right away. We wanted to catch him red-handed.

Thus, we decided to infiltrate one of his studies, which he was covertly advertising under the obvious pseudonym Mr. Dean Wangle. He administered the study wearing a fake moustache and a ridiculous French beret, but his voice is unmistakeable. Below is a video of an experimental session that we were able to record with a hidden camera. The footage is very tough to watch.

Combined, the evidence leaves only one conclusion: Andrew Gelman betrayed science in his war on power posing.

So… what is media psychology anyway?

Does playing violent video games increase aggression?^[1]No, but violent video game research kinda does. What makes advertisements persuasive?^[2]David Hasselhoff, obvs. Are 5%25%75% of the population addicted to social media?^[3]It’s almost like humans have a fundamental need for social interactions. Who are these people that watch porn?^[4]Literally everyone everywhere Why do we enjoy cat videos so much?^[5]WHY???

These are the typical research questions media psychologists are concerned with. Broadly, media psychology describes and explains human behavior, cognition, and affect with regards to the use and effects of media and technology. Thus, it’s a hybrid discipline that borrows heavily from social, cognitive, and educational psychology in both its theoretical approaches and empirical traditions. The difference between a social psychologist and a media psychologist that both study video game effects is that the former publishes their findings in JPSP while the latter designs “What Twilight character are you?” self-tests^[6]TEAM EDWARD FTW! for perezhilton.com to avoid starving. And so it goes.

New is always better

A number of media psychologists is interested in improving psychology’s research practices and quality of evidence. Under the editorial oversight of Nicole Krämer, the Journal of Media Psychology (JMP), the discipline’s flagship^[7]By “flagship” I mean one of two journals nominally dedicated to this research topic, the other being Media Psychology. It’s basically one of those People’s Front of Judea vs. Judean People’s Front situations. journal, not only signed the Transparency and Openness Promotion Guidelines, it has also become one of roughly fifty journals that offer the Registered Reports format.

To promote preregistration in general and the new submission format at JMP in particular, the journal launched a fully preregistered Special Issue on “Technology and Human Behavior” dedicated exclusively to empirical work that employs these practices. Andy Przybylski^[8]Who, for reasons I can’t fathom, prefers being referred to as a “motivational researcher” and I were fortunate enough to be the guest editors of this issue.

The papers in this issue are nothing short of amazing – do take a look at them even if it is outside of your usual area of interest. All materials, data, analysis scripts, reviews, and editorial letters are available here. I hope that these contributions will serve as an inspiration and model for other (media) researchers, and encourage scientists studying media to preregister designs and share their data and materials openly.

Media Psychology BC^[9]Before Chambers

If you already suspected that in all this interdisciplinary higgledy-piggledy, media psychology did not only inherit its parent disciplines’ merits, but also some of their flaws, you’re probably^[10]unerring-absolute-100%-pinpoint-unequivocally-no-ifs-and-buts-dead-on-the-money correct. Fortunately, Nicole was kind enough to allot our special issue editorial more space than usual in order to report a meta-scientific analysis of the journal’s past and to illustrate how some of the new practices can ameliorate the evidential value of research. For this reason, we surveyed a) availability of data, b) errors in the reporting of statistical analyses, and c) sample sizes and statistical power of all 147 studies in N = 146 original research articles published in JMP between volume 20/1, when it became an English-language publication, and volume 28/2 (the most recent issue at the time this analysis was planned). This blog post is a summary of the analyses in our editorial, which — including its underlying raw data, analysis code, and code book — is publicly available at https://osf.io/5cvkr/.

Availability of Data and Materials

Historically the availability of research data in psychology has been poor. Our sample of JMP publications suggests that media psychology is no exception to this, as we were not able to identify a single publication reporting a link to research data in a public repository or the journal’s supplementary materials.

Statistical Reporting Errors

A recent study by Nuijten et al. (2015) indicates a high rate of reporting errors in reported Null Hypothesis Significance Tests (NHSTs) in psychological research reports. To make sure such inconsistencies were avoided for our special issue, we validated all accepted research reports with statcheck 1.2.2, a package for the statistical programming language R that works like a spellchecker for NHSTs by automatically extracting reported statistics from documents and recomputing^[11]p-values are recomputed from the reported test statistics and degrees of freedom. Thus, for the purpose of recomputation, it is assumed that test statistics and degrees of freedom are correctly reported, and that any inconsistency is caused by errors in the reporting of p-values. The actual inconsistencies, however, can just as well be caused by errors in the reporting of test statistics and/or degrees of freedom. p-values.

For our own analyses, we scanned all n_emp = 131 JMP publications reporting data from at least one empirical study (147 studies in total) with statcheck to obtain an estimate for the reporting error rate in JMP. Statcheck extracted a total of 1036 NHSTs reported in n_nhst = 98 articles. Forty-one publications (41.8% of n_nhst) reported at least one inconsistent NHST (max = 21), i.e. reported test statistics and degrees of freedom did not match reported p-values. Sixteen publications (16.3% of n_nhst) reported at least one grossly inconsistent NHST (max = 4), i.e. the reported p-value is < .05 while the recomputed p-value is > .05, or vice-versa. Thus, a substantial proportion of publications in JMP seem to contain inaccurately reported statistical analyses, of which some might affect the conclusions drawn from them (see Figure 1).

Caution is advised when speculating about the causes of the inconsistencies. Many of them are probably clerical errors that do not alter the inferences or conclusions in any way.^[13]For example, in 20 cases the authors reported p = .000, which is mathematically impossible (for each of these p_recomputed < .001). Other inconsistencies might be explained by authors not declaring that their tests were one-tailed (which is relevant for their interpretation). However, with some concern, we observe it is unlikely to be the only cause, as in 19 out of 23 cases the reported p-values were equal to or smaller than .05 while the recomputed p-values were larger than .05, whereas the opposite pattern was observed in only four cases. Indeed, if incorrectly reported p-values resulted merely from clerical errors, we would expect inconsistencies in both directions to occur at approximately equal frequencies.

All of these inconsistencies can easily be detected using the freely available R package statcheck or, for those who do not use R, in your browser via www.statcheck.io.

Sample Sizes and Statistical Power

High statistical power is paramount in order to reliably detect true effects in a sample and, thus, to correctly reject the null hypothesis when it is false. Further, low power reduces the confidence that a statistically significant result actually reflects a true effect. A generally low-powered field is more likely to yield unreliable estimates of effect sizes and low reproducibility of results. We are not aware of any previous attempts to estimate average power in media psychology.

Strategy 1: Reported power analyses. One obvious strategy for estimating average statistical power is to examine the reported power analyses in empirical research articles. Searching all papers for the word “power” yielded 20 hits and just one of these was an article that reported an a priori determined sample size.^[14]In the 19 remaining articles power is mentioned, for example, to either demonstrate observed or post-hoc power (which is redundant with reported NHSTs), to suggest larger samples should be used in future research, or to explain why an observed nonsignificant “trend” would in fact be significant had the statistical power been higher.

Strategy 2: Analyze power given sample sizes. Another strategy is to examine the power for different effect sizes given the average sample size (S) found in the literature. The median sample size in JMP is 139 with a considerable range across all experiments and surveys (see Table 1). As in other fields, surveys tend to have healthy sample sizes apt to reliably detect medium to large relationships between variables.

For experiments (including quasi-experiments), the outlook is a bit different. With a median sample size per condition/cell of 30.67, the average power of experiments published in JMP to detect small differences between conditions (d = .20) is 12%, 49% for medium effects (d = .50), and 87% for large effects (d = .80). Even when assuming that the average effect examined in the media psychological literature could be as large as those in social psychology (d = .43), our results indicate that the chance that an experiment published in JMP will detect them is 38%, worse than flipping a coin.^[15]An operation that would be considerably less expensive.

Table 1. Sample sizes and power of studies published in JMP volumes 20/1 to 28/2.
n = Number of published studies; MD_S = Median sample size; MD_s/cell = Median sample size per condition; 1-ß_r=.1/d=.2 / 1-ß_r=.3/d=.5 / 1-ß_r=.5/d=.8 = Power to detect small/medium/large bivariate relationships/differences between conditions.
For between-subjects, mixed designs, and total we assumed independent t-tests. For within-subjects designs we assumed dependent t-tests. All tests two-tailed, α = .05. Power analyses were conducted with the R package pwr 1.20

Feeling the Future of Media Psychology

The above observations could lead readers to believe that we are concerned about the quality of publications in JMP in particular. If anything, the opposite is true, as this journal recently committed itself to a number of changes in its publishing practices to promote open, reproducible, high-quality research. These analyses are simply another step in a phase of sincere self-reflection. Thus, we would like these findings, troubling as they are, to be taken not as a verdict, but as an opportunity for researchers, journals, and organizations to reflect similarly on their own practices and hence improve the field as a whole.

One key area which could be improved in response to these challenges is how researchers create, test, and refine psychological theories used to study media. Like other psychology subfields, media psychology is characterized by frequent emergence of new theories which purport to explain phenomena of interest.^[16]As James Anderson recently put it in a very clever paper (as usual): “Someone entering the field in 2014 would have to learn 295 new theories the following year.” This generativity may, in part, be a consequence of the fuzzy boundaries between exploratory and confirmatory modes of social sciences research.

Both modes of research – confirming hypotheses and exploring uncharted territory – benefit from preregistration. Drawing this distinction helps the reader determine which hypotheses carefully test ideas derived from theory and previous empirical studies, and it liberates exploratory research from the pressure to present an artificial hypothesis-testing narrative.

As technology experts, media psychology researchers are well positioned to use and study new tools that shape our science. A range of new web-based platforms have been built by scientists and engineers at the Center for Open Science, including their flagship, the OSF, and preprint services like PsyArXiv. Designed to work with scientists’ existing research flows, these tools can help prevent data loss due to hardware malfunctions, misplacement,^[17]Including mindless grad students and hungry dogs or relocations of researchers, while enabling scientists to claim more credit by allowing others to use and cite their materials, protocols, and data. A public repository for media psychology research materials is already in place.

Like psychological science as a whole, media psychology faces a pressing credibility gap. Unlike some other areas of psychological inquiry,^[18]such as meta science however, media research — whether concerning the Internet, video games, or film — speaks directly to everyday life in the modern world. It affects how the public forms their perceptions of media effects, and how professional groups and governmental bodies make policies and recommendations. In part because it is key to professional policy, empirical findings disseminated to caregivers, practitioners, and educators should be built on an empirical foundation with sufficient rigor.

We are, on balance, optimistic that media psychologists can meet these challenges and lead the way for psychologists in other areas. This special issue and the registered reports submission track present an important step in this direction and we thank the JMP editorial board, our expert reviewers, and of course, the dedicated researchers who devoted their limited resources to this effort.

The promise of building an empirically-based understanding of how we use, shape, and are shaped by technology is an alluring one. We firmly believe that incremental steps taken towards scientific transparency and empirical rigor will help us realize this potential.

If you read this entire post, there’s a 97% chance you’re on Team Edward.

(this post was jointly written by Malte & Anne; in a perfect metaphor for academia, WordPress doesn’t know how to handle multiple authors)

We believe in scientific openness and transparency, and consider unrestricted access to data underlying publications indispensable. Therefore, we^[1]Not just the authors of this post, but all four of us. signed the Peer Reviewers’ Openness (PRO) Initiative, a commitment to only offer comprehensive review for or recommend the publication of a manuscript if the authors make their data and materials publicly available, unless they provide compelling reasons^[2]The data-hungry dog of a former grad student whose name you forgot is not a compelling reason. why they cannot do so (e.g. ethical or legal restrictions).

As reviewers, we enthusiastically support PRO and its values.
Also as reviewers, we think PRO can be a pain in the arse.

Ok, not really. But advocating good scientific practice (like data sharing) during peer review can result in a dilemma.

This is how it’s supposed to work: 1) You accept an invitation for review. 2) Before submitting your review, you ask the editor to relay a request to the authors to share their data and materials (unless they have already done so). 3) If authors agree – fantastic. If authors decline and refuse to provide a sensible rationale why their data cannot be shared – you reject the paper. Simple.
So far, so PRO. But here’s where it gets hairy: What happens when the action editor handling the paper refuses to relay the request for data, or even demands that such a request is removed from the written review?

Here is a reply Anne recently got from an editor after a repeated^[3]The editor apologised for overlooking the first email – most likely an honest mistake. Talk to Chris Chambers if you want to hear a few stories about the funny tendency of uncomfortable emails to get lost in the post. PRO request:

“We do not have these requirements in our instructions to authors, so we can not ask for this without discussing with our other editors and associate editors. Also, these would need to involve the publication team. For now, we can relieve you of the reviewing duties, since you seem to feel strongly about your position.
Let me know if this is how we should proceed so we do not delay the review process further for the authors.”

Much like judicial originalists insist on interpreting the US constitution literally as it was written by a bunch of old white dudes more than two centuries ago, editors will sometimes cite existing editorial guidelines by which authors obligate themselves to share data on request, but only after a paper has been published, which has got to be the “Don’t worry, I use protection” argument of academia.^[4]We picked a heterosexual male perspective here but we’re open to suggestions for other lewd examples. Also, we know that this system simply does. not. work.

As reviewers, it is our duty to evaluate submitted research reports, and data are not just an optional part of empirical research – they are the empirical research (the German Psychological Society agrees!). You wouldn’t accept a research report based on the promise that “theory and hypotheses are available on request”, right?^[5]Except when reviewing for Cyberpsychology.

PRO sets “data or it didn’t happen” as a new minimum standard for scientific publications. As a consequence, comprehensive review should only be offered for papers that meet this minimum standard. The technically correct^[6]The best kind of being correct. application of the PRO philosophy for the particular case of the Unimpressed Editor is straightforward: When they decide – on behalf of the authors! – that data should or will not be shared, the principled consequence is to withdraw the offer to review the submission. As they say, PRO before the status quo.

Withdrawing from the process, however, decreases the chance that the data will be made publicly accessible, and thus runs counter to PRO’s ideals. As we say in German, “Operation gelungen, Patient tot” – surgery successful, patient deceased.

Adhering strictly to PRO would work great if everybody participated: The pressure on non-compliant journals would become too heavy. Then again, if everybody already participated, PRO wouldn’t be a thing. In the world of February 2017, editors can just appoint the next best reviewer^[7]Withdrawing from review might still have an impact in the absence of a major boycott by causing the editors additional hassle and delaying the review process – then again, this latter part would unfairly harm the authors, too. who might simply not care about open data – and couldn’t you push for a better outcome if you kept your foot in the door? Then again, if all PRO signatories eroded the initiative’s values that way, the day of reaching the critical mass for a significant boycott would never come.

A major concern here is that the authors are never given the chance to consider the request although they might be receptive to the arguments presented. If increased rates of data sharing is the ultimate goal, what is more effective: boycotting journals that actively suppress such demands by invited reviewers, or loosening up the demands and merely suggest that data should be shared so at least the gist of it gets through?

There are two very different ways to respond to such editorial decisions, and we feel torn because each seems to betray the values of open, valuable, proper scientific research. You ask: What is the best strategy in the long run? Door in the face! Foot in the door! Help, I’m trapped in a revolving door! We would really like to hear your thoughts on this!

Academia is a strange place. There are a lot of implicit norms and unspoken rules which, to make it worse, can vary by field, subfield, across countries, and over time. For example: How do you write an email in an academic setting? Should your mails be polite or is it already impolite to waste the reader’s time with polite fluff? How do you address a professor who you (1) have never met in person, (2) met in person once but they likely don’t remember, (3) had a beer with at a conference but they likely don’t remember? Do you shake hands? How do you start a collaboration and are you sure you want to wear this pair of jeans/cat shirt/three piece suit to the next conference?

It takes time to figure these things out and to finally feel comfortable in academic interactions – even more so for students from working class families who can’t draw on experiences from their parents, or for students from parts of the world with considerably different academic norms.

So how can we help people to feel welcome in our strange insider’s club?

I have three suggestions, and as it happens, none of them is about tone. Actually, I believe that changing the tone is a quite ineffective way to make people feel welcome and valued because it is just cosmetics: Communication can be extremely hostile while maintaining a picture-perfect all friendly nonviolent surface. I suspect that people who champion tone monitoring hope that talking nicely for long enough will transform attitudes. However, I faintly remember learning in the first year of my undergraduate that Sapir-Whorf is not well substantiated.^[1]This memory also spoiled Arrival for me. Still a great movie though.^[2]After publishing this post, it has been pointed out three times that I shouldn’t bash linguistic relativity. To add more nuance to my argument, let me add that, as far as I know, there is substantial evidence for a weaker form of the Sapir-Whorf account (which also seems to be misnamed), which I consider plausible. Of course, I’m only bashing the form of linguistic relativity displayed in Arrival.^[3]After publishing this post, my boyfriend read it and I additionally have to add the disclaimer that yes, Arrival was a great movie and, yes, maybe, assuming that those aliens are so different from humans and way more advanced etc., probably in a parallel universe that does not adhere to our physics, maybe it could work like this.

Furthermore, setting well-intended rules about the tone of interactions might just add another layer of conventions that poses yet another obstacle for outsiders. What I suggest is that we do not tackle tone, but instead try to change the underlying climate.

Start admitting that you are sometimes wrong

Many students start with the assumption that people with the fancy “Dr.” attached to their name or (gasp) professors have privileged access to the secrets of the world and are thus close to infallibility. Anne pointed me towards Perry’s Scheme, a model of how college students come to understand knowledge, that succinctly summarizes this first level of understanding: The authorities know.

However, social interactions get pretty one-sided if one side assumes that the other side is never wrong, and it unnecessarily reinforces power differentials (that exist anyway, and that are probably not always conducive to scientific progress, but we will keep this for another blog post). It also greatly obscures how science – as opposed to esotericism – is supposed to work.
Anecdotal data ahead: I have never felt particularly unwelcome in academia, and I blame this on the fact that both my parents have a PhD. Now before we all get excited about social transmission of educational attainment, I will quickly add that I was not raised by the doctors but by my down-to-earth mother-of-eight catholic grandma. However, I still got the strong impression that academic rank does not predict how often a person is right about things that are outside of their specific narrow subfield. Of course there is a German word for this idea: Fachidiot, a narrowly specialized person who is an idiot when it comes to anything else. In fact, I might have had a phase in which I firmly believed that a PhD indicates that a person is always wrong.^[4]I’m sorry, Mum. It wasn’t you, it was puberty.

Coincidentally, this also relates to the one piece of career advice I got from my dad: It’s important to hang out at conferences because there you can actually see with your own eyes that everybody cooks with water, which is the German way to say that everybody puts their pants on one leg at a time.
There is a quick fix to the misconception that academics are always right: Just communicate that you are fallible and be honest about the things that you are uncertain about. If you need a role model for this type of behavior, I recommend Stefan Schmukle, who has been my academic advisor since the second year of my undergraduate and is probably the main reason why I did not leave psychology for a more lucrative and less frustrating career path. Stefan openly admits his knowledge gaps when he teaches and stresses that he keeps learning a lot. Funnily enough, it does not undermine his authority^[5]You know what does undermine your authority in front of the students? Pretending to know something that you don’t know while not even being aware that the smarter students can easily tell you are just pretending. There is a German word for the student’s feeling in such a situation, it’s called fremdschämen. in front of his students according to the data available to me, which includes both quantitative (student evaluations and teaching awards) and qualitative (intensive student interviews over a beer or two) evidence. 

Positive side effects of admitting that you are sometimes wrong might include (1) students feeling more respected because of your honesty, (2) students learning that psychology is not an arcane art accessible only to privileged old white men, and (3) sending a strong signal that you are, in fact and despite all your glorious achievements, a human being. Which already leads to my second suggestion.

Show others it’s okay to have a life. Have a life.

This is important not only because you probably enjoy having a life, but it also avoids any sort of the mystification of what it means to be an academic. If we establish the norm that being an academic implies working from early morning until late in the night, seven days a week and especially between Christmas and New Year’s Eve, a lot of people might actually decide that they don’t want to feel welcome in academia. If your subfield actually requires this type of commitment, then please be frank about it so that junior researchers can decide early on whether they want to sacrifice literally everything else that makes life fun.
However, if your job does not require to sacrifice your life completely, it’s great to signal to others that you are, in fact, a human being with a family, hobbies, and other stuff that you do in your free time, like binge-watching Gilmore Girls or blindfolded speed runs of your favorite childhood video games.^[6]The author of this article only indulges in one of these two activities but knows for a fact that at least one tenured individual in her proximity indulges in the other one. 

I don’t have any data to back up this claim, but I’m pretty sure most humans enjoy the company of other humans above the company of restless and efficient publication machines. Overworking is not a sustainable lifestyle for most people, and it does not create a particularly welcoming climate. It also leads to a race to the bottom which makes life worse for everyone, so maybe work less (and unionize). As a senior scientist, don’t make overworking the norm.

As it happens, this point also maps onto the one piece of solid career-related advice my mother passed on to me. Her professor told her that she was spending too much time in the library instead of getting to know her peers in the evenings. In my personal interpretation, I’m not reading that as advice to go “networking”, but to do things that are actually fun because we know what all work and no play did to Jack.

Don’t act as if willpower/grit/self-control/discipline/ambition/perseverance will lead to success

In the current predominant culture, especially in academia – and a bit more in the US than in my control group, Germany – success is often equated with the result of some sort of internal strength. If only you tried a bit harder, if only you got a bit more organized, if only you started getting up earlier, if only you gave a bit more, if only you networked more efficiently, your efforts would finally pay off. It’s all fine and dandy to try your best and to try to actively regulate your behavior, but I fear we have brought this to a point at which the attitude is getting toxic.
First, it opens the door to self-exploitation. Second, it makes people more willing to comply with exploitative structures, which is great for the maintenance of the status quo, but not so much for early career researchers who end up working endless hours. Third, if internal strength inevitably leads to success, having no success implies that you lack some sort of internal strength, or worse, that you are a failure.

But, most importantly, it’s just not true that trying as hard as possible will lead to success, and that success will lead to some sort of bliss that compensates for all the hard work.
Success depends on multiple factors, and even if we assume that effort contributes quite a bit, there is still plenty of factors outside of our control: innate abilities, external factors such as being surrounded by people that support you (vs. having an advisor who is still fully absorbed in the rat race and exploits you for their own purposes) and a lot of randomness. Anyone who has ever submitted a paper to a journal will probably agree that there is a lot of randomness in the current academic system in psychology – if you’ve never encountered some level of arbitrary decision making, you’ve been pretty lucky (q.e.d.).
Then, the story goes, you should of course accept the things outside of your control but work hard with those that you can control – such as your ambitions and your perseverance. But can we even control these things? Frankly, I don’t know. But Ruben pointed out that we know few interventions that improve conscientiousness reliably, and that grit (which is basically conscientiousness) is partially heritable. Based on my experience, trying hard is much harder for some people than for others. I can indeed be as disciplined as I want, but I cannot will what I want.^[7]I’m pretty sure there is a reason why Schopenhauer is not particularly popular with motivational coaches. 

Last but not least, I don’t think that bliss necessarily awaits those who work hard and end up being successful. We have yet to hear of the lucky person who got tenured and immediately reached a state of inner peace as a result. In fact, when I look into the office next door, I get the impression that the daily grind is not that different with a nice title in front of your name. (It certainly is more comfortable with respects to financial security, but not everybody can end up being a professor, so maybe we need structural change instead of individual struggle to tackle the precarious employment situation in academia.) However, this outlook does not seem too dull to me: In our lab, we are being nice to each other and we agree that our job is (to some extent) about doing science – not that much about gaming the system to get somewhere where you can finally, if you are lucky, do science.

TL;DR: It’s fine if you are sometimes wrong, don’t sweat it. Don’t make overworking the norm. Don’t give students the impression they just have to try hard enough to make it because deep down, we all know that this is not how it works.

A research parasite, a destructo-critic, a second-stringer, and a methodological terrorist walk into a bar. Their collective skepticism creates a singularity, so they morph into a flairless superbug and start a blog just to make things worse for everyone.
This is, roughly, our origin story. Who are we? We are The 100% CI, bound by a shared passion for horrible puns and improving our inferences through scientific openness and meta science.

You know what I need in my life right now? Another blog on meta science!, said no one ever. Ok, sure, that’s fair, BUT:

• We are 4 Germans, which approximates 1 Gelman according to our calculations^[1]Analysis scripts are available upon request.^[2]If you request them, we will not respond to your emails for several months. Also a grad student ate lost them.
• We will be blogging about other stuff. This week alone we will have posts on
  • Strategies to make people feel welcome in academia (Julia)
  • Reproducible websites (Ruben)
  • Requesting data during peer review (Malte & Anne)
  • EEG infant research on biological motion perception (Anne)
• All our kitten pics are organically sourced and come with appropriate author citation.
• We have the best meta science. It’s great. Everybody loves our meta science.
• You literally have nothing better to do if you kept reading until this point.

So maybe give us a chance at least? We already paid the domain rent for the next 12 months. That probably wasn’t smart, but it’s who we are.

We are The 100% CI. Always confident. Sometimes credible.

Mit freundlichen Grüßen

Anne Scheel
Ruben Arslan
Malte Elson
Julia Rohrer