Requesting data during peer review: Foot in the door, or door in the face?

(this post was jointly written by Malte & Anne; in a perfect metaphor for academia, WordPress doesn’t know how to handle multiple authors)

 

We believe in scientific openness and transparency, and consider unrestricted access to data underlying publications indispensable. Therefore, we[1]Not just the authors of this post, but all four of us. signed the Peer Reviewers’ Openness (PRO) Initiative, a commitment to only offer comprehensive review for or recommend the publication of a manuscript if the authors make their data and materials publicly available, unless they provide compelling reasons[2]The data-hungry dog of a former grad student whose name you forgot is not a compelling reason. why they cannot do so (e.g. ethical or legal restrictions).

As reviewers, we enthusiastically support PRO and its values.
Also as reviewers, we think PRO can be a pain in the arse.

Ok, not really. But advocating good scientific practice (like data sharing) during peer review can result in a dilemma.

This is how it’s supposed to work: 1) You accept an invitation for review. 2) Before submitting your review, you ask the editor to relay a request to the authors to share their data and materials (unless they have already done so). 3) If authors agree – fantastic. If authors decline and refuse to provide a sensible rationale why their data cannot be shared – you reject the paper. Simple.
So far, so PRO. But here’s where it gets hairy: What happens when the action editor handling the paper refuses to relay the request for data, or even demands that such a request is removed from the written review?

Here is a reply Anne recently got from an editor after a repeated[3]The editor apologised for overlooking the first email – most likely an honest mistake. Talk to Chris Chambers if you want to hear a few stories about the funny tendency of uncomfortable emails to get lost in the post. PRO request:

“We do not have these requirements in our instructions to authors, so we can not ask for this without discussing with our other editors and associate editors. Also, these would need to involve the publication team. For now, we can relieve you of the reviewing duties, since you seem to feel strongly about your position.
Let me know if this is how we should proceed so we do not delay the review process further for the authors.”

Not seeing data before recommending a paper for publication is like die Katze im Sack kaufen (buying the cat in a bag, photo: angela n.)

Much like judicial originalists insist on interpreting the US constitution literally as it was written by a bunch of old white dudes more than two centuries ago, editors will sometimes cite existing editorial guidelines by which authors obligate themselves to share data on request, but only after a paper has been published, which has got to be the “Don’t worry, I use protection” argument of academia.[4]We picked a heterosexual male perspective here but we’re open to suggestions for other lewd examples. Also, we know that this system simply does. not. work.

As reviewers, it is our duty to evaluate submitted research reports, and data are not just an optional part of empirical research – they are the empirical research (the German Psychological Society agrees!). You wouldn’t accept a research report based on the promise that “theory and hypotheses are available on request”, right?[5]Except when reviewing for Cyberpsychology.

PRO sets “data or it didn’t happen” as a new minimum standard for scientific publications. As a consequence, comprehensive review should only be offered for papers that meet this minimum standard. The technically correct[6]The best kind of being correct. application of the PRO philosophy for the particular case of the Unimpressed Editor is straightforward: When they decide – on behalf of the authors! – that data should or will not be shared, the principled consequence is to withdraw the offer to review the submission. As they say, PRO before the status quo.

Withdrawing from the process, however, decreases the chance that the data will be made publicly accessible, and thus runs counter to PRO’s ideals. As we say in German, “Operation gelungen, Patient tot” – surgery successful, patient deceased.


We at The 100% CI stopped using doors after repeatedly banging various body parts against them (Photo: Max Pixel)

Adhering strictly to PRO would work great if everybody participated: The pressure on non-compliant journals would become too heavy. Then again, if everybody already participated, PRO wouldn’t be a thing. In the world of February 2017, editors can just appoint the next best reviewer[7]Withdrawing from review might still have an impact in the absence of a major boycott by causing the editors additional hassle and delaying the review process – then again, this latter part would unfairly harm the authors, too. who might simply not care about open data – and couldn’t you push for a better outcome if you kept your foot in the door? Then again, if all PRO signatories eroded the initiative’s values that way, the day of reaching the critical mass for a significant boycott would never come.

A major concern here is that the authors are never given the chance to consider the request although they might be receptive to the arguments presented. If increased rates of data sharing is the ultimate goal, what is more effective: boycotting journals that actively suppress such demands by invited reviewers, or loosening up the demands and merely suggest that data should be shared so at least the gist of it gets through?

There are two very different ways to respond to such editorial decisions, and we feel torn because each seems to betray the values of open, valuable, proper scientific research. You ask: What is the best strategy in the long run? Door in the face! Foot in the door! Help, I’m trapped in a revolving door! We would really like to hear your thoughts on this!

RE: Door in the face
Dear Editor,

Thank you very much for the quick response.

Of course I would have preferred a different outcome, but I respect your decision not to request something from the authors that wasn’t part of the editorial guidelines they implicitly agreed to when they submitted their manuscript.

What I do not agree with are the journal’s editorial guidelines themselves for the reasons I provided in my previous email. It seems counterproductive to invite peers as “gatekeepers” while withholding relevant information that are necessary for them to fulfill their duty until the gatekeeping process has been completed.

Your decision not even to relay my request for data sharing to the authors (although they might gladly do so!), unfortunately, bars me from providing a comprehensive review of the submission. It is literally impossible for me to conclude a recommendation about the research as a whole when I’m only able to consider parts of it.

Therefore, I ask that you unassign me as a reviewer, and not invite me again for review except for individual manuscripts that meet these standards, or until the editorial policy has changed.

Sincerely,
Reviewer #2

RE: Foot in the door
Dear Editor,

Thank you very much for the quick response.

Of course I would have preferred a different outcome, but I respect your decision not to request something from the authors that wasn’t part of the editorial guidelines they implicitly agreed to when they submitted their manuscript.

In fact, those same principles should apply to me as a reviewer, as I, too, agreed to review the submission under those rules. Therefore, in spite of the differences in my own personal standards versus those presented in your editorial guidelines, I have decided to complete my review of the manuscript as originally agreed upon.

You will see that I have included a brief paragraph on the benefits of data sharing in my review. I neither demand the authors share their data nor will I hold it against them if they refuse to do so at this point. I simply hope they are persuaded by the scientific arguments presented in my review and elsewhere — In fact, I hope that you are too.

I appreciate this open and friendly exchange, and I hope that you will consider changing the editorial guidelines to increase the openness, robustness, and quality of the research published in your journal.

Sincerely,
Reviewer #2

Footnotes

Footnotes
1 Not just the authors of this post, but all four of us.
2 The data-hungry dog of a former grad student whose name you forgot is not a compelling reason.
3 The editor apologised for overlooking the first email – most likely an honest mistake. Talk to Chris Chambers if you want to hear a few stories about the funny tendency of uncomfortable emails to get lost in the post.
4 We picked a heterosexual male perspective here but we’re open to suggestions for other lewd examples.
5 Except when reviewing for Cyberpsychology.
6 The best kind of being correct.
7 Withdrawing from review might still have an impact in the absence of a major boycott by causing the editors additional hassle and delaying the review process – then again, this latter part would unfairly harm the authors, too.

14 thoughts on “Requesting data during peer review: Foot in the door, or door in the face?”

  1. This is very timely for me. I just experienced my first case of an editor refusing to pass along the request for open data and materials this week. Because open data and materials are not required by the journal the editor was “not in a position to request such information from one set of authors and not another prior to review.” Instead, she suggested that I simply ask the authors to provide the data and materials as part of my full review.
    In the end, I declined to provide a full review and submitted a slightly-wordier version of PRO’s recommended minimal review (cannot recommend for publication, but would review a revision that provides open data and materials). I feel like withholding the full review is perhaps the only real power I have in these situations, but maybe the gentler approach, as described in your foot in the door e-mail, would ultimately yield better results. I’m very interested in people’s thoughts on how to approach editorial refusals of the request.

  2. Hilarious, but also (sadly) very true. The open science/open data movement has won the easy battle (convincing those with little to lose). The next stage is going to be a lot harder, because it is going to involve a lot of people admitting to themselves that they have been wrong for the past 10, 20, or 30 years.

  3. Thoughtful blog post. As an Editor, I can affirm that PRO requests can indeed be a pain in the arse. I think that’s part of the point. Editors don’t want to have to negotiate with prospective reviewers and then with authors (sometimes through several rounds of email, and sometimes ultimately failing to satisfy the reviewer) on a case-by-case basis. One hopes that they will instead be motivated to move toward standardizing processes that encourage authors to make data available to reviewers and that automatically inform prospective reviewers as to how they can access the data or, if they cannot, why not. Psych Science has such a system in place (with an upgrade that will soon be implemented). I think PRO helped me make the case for these systems with my Senior and Associate Editors (although some of them are worried that it is going to cost us in terms of submissions).

  4. Asking editors to relay requests doesn’t work.

    I have zero sympathy for authors who didn’t make data available when they submitted. Either they don’t want to share or they have been living under a rock for past five years.

    So I write my minimal review and requests revise and resubmit. If it’s inconvient for authors to have another round of review, they should have submitted the full paper. I would do the same if every other page was missing or their discussion got cut out.

    Oh and editors get pissed.
    Sincerely,
    Reviewer 2.

  5. I would go with the Door in the Face approach. As Stephen says above, withholding your review and boycotting a journal that doesn’t adhere to open data is the only real power you have. Journal editors find it generally quite hard to find reviewers. I think a boycott can become quite effective even if only a few reviewers are part of PRO.

    I am a bit puzzled by the description of PRO though. My understanding was that you refuse to review if the authors don’t do anything about data availability (i.e. they don’t make data available or add a statement why they can’t). I didn’t read the PRO guidelines as “rejecting” the manuscript. The latter implies remaining part of the review process and I think following the principles of PRO you should simply refuse to partake in the review process altogether in this situation (which is essentially what Door in the Face is).

    1. I understand the PRO like you Samsung: I don’t reject manuscript or recommend their rejection, I review papers and say that for a successful revision, authors must (1) share the data and code or (2) provide convincing reasons (not to me, but in the paper) why they cannot.

  6. I’ve just been in the same situation this week with a journal refusing to pass along the request for open data and materials. It seemed that it was not the action editor’s decision, but more that of the journal board (“We have thought about this request carefully and we are currently considering the journal’s position on the openness initiative. While we are broadly in favour of it, we feel it is vital to take a consistent position in the journal and we also feel we need to flag up our position before authors submit their work. I therefore won’t ask one author to do this since we are not asking all authors to do this (though we may do in the future”).
    I therefore submitted my review (major revision) asking for the minimum quality requirements for an open scientific manuscript, considering that if the journal cannot ask the authors, I –as a reviewer– can. I envisage 3 possibilities now: 1) the action editor does not consider my review (=> I’ll boycott the journal), 2) I’m invited to review the second round and the authors did not pay attention to my review (=> I recommend rejection), 3) I’m invited to review and the authors considered my review (=> I’ll offer a comprehensive review).
    Seems to me that this is in line with the PRO initiative, and it takes into consideration the fact that some authors may not be familiar with the need for open science. Whatever the authors’ or the journal’s decision, I made the message clear (hopefully).

  7. It seems that the foot in the door approach, as stated above, is not really an option for anyone who has signed the PRO agreement. As I understand that agreement, we are committing ourselves to only provide reviews when the materials and data are open. If editors, for example, do not allow us to even request that the data etc are open, I think we obliged by the PRO agreement to not provide a review.
    On a more general note, in the description of the PRO agreement, the issue of lack of cooperation by editors was not considered. However, as we see above, and also in the case JEP:LMC, this is turning out to be an important issue. It would be good if there was a PRO discussion forum or mailing list to discuss this and the many other new issues that are likely to arise in the future.

    1. Mark, maybe I should have mentioned that when I said “I submitted my review (major revision) asking for the minimum quality requirements for an open scientific manuscript”, it was actually the sole content of my review (no other comment on the manuscript, I didn’t read it!).

      1. I see. And when I said that we should not provide a review if certain standards are not met, I really meant to say we should not provide a typical review of the content of the article but instead we should do something like what you did, e.g. send in a review recommending a major revision asking for open data.

    1. This indeed works, but only when the review process isn’t double-blind (which in my field, for example, would be very uncommon)

Comments are closed.