Someone recently told me I was wrong about something. You can read the exchange here (Woodley et al., our reply).[1]minus some points exchanged non-publicly in peer review Despite my best efforts, I’m not sure how graciously I reacted. Mostly, I thought my critic was wrong, but I was forced to deal with his arguments, sought out the literature, expert advice, and my co-authors. It may not have led me to substantially change my mind, but I went through the motions.
Which made me think: these motions are even more unfamiliar to me than any form of physical exercise. The criticism stings, worse than I imagine side stitches might feel. Clearly, I’m rarely wrong! Am I?
I know that I’ve held back my own criticisms of research sometimes because I was not sure whether they would be received as intended – good faith criticism trying to improve science. Given the competitive nature of science, people might perceive me as trying to rain on their parade, make a name for myself as a destructo-critic, or just bring down my competitors. And they might be right about this at times. So, if they use this as an excuse to dismiss the criticism, the social costs of criticism for me might outweigh the benefits to science.
The thing is, I don’t know how others will react. Maybe I’m too shy about sending out criticism, or maybe when I’m criticising people now I’m really only venting, not truly arguing in good faith. That also won’t lead to any improvements in studies. It’s a coordination problem. Without knowing how the criticised person will react, my incentives favour not criticising them publicly.
And how can you solve a coordination problem?[2]Well, I’m really not the one to ask, because this dog is my spirit animal. Well, this post serves as a signal[3]Now, you might also say that this could just be an empty virtue signal. After all, what prevents me from ignoring all of these comments? I don’t have a good solution to this criticism. I don’t want all anonymous criticism to be public and non-deletable by default, because, you know, the internet. I commit to reacting to criticisms of published first-authored work publicly within two months, but may in some cases put up a placeholder until I’m able to fully respond. I guess I could appoint a trusted third party to judge my responses, but I’m currently not sure what nature of criticism to expect and who would be trusted by my (anonymous) critics more than me. So, since some measure of trust is needed, it might as well be in me. to you, that I intend not to dismiss criticism. And if you worry about the social costs of offending a lowly postdoc who last visited a gym exclusively to go the sauna and whose wit is as acerbic as a warm glass of milk, I won’t blame you. Instead, I put up an anonymous submission form and humbly request: Tell me I’m wrong.
https://formr.org/tellmeimwrong
Use it to let me know that I’m wrong about something, if you are unsure if I would react well when criticised face-to-face. I want to hear it, but I cannot pretend that I always find criticism easy to stomach and cannot guarantee “no hard feelings”. This is why this form exists.
I’m mainly interested in finding out I have been wrong in preprints and papers that I have first authored, because this is where I consider it 1) likely not to be told and 2) costly to be wrong. So, open season on those. I intend to react to all good-faith submissions regarding these publicly, in the form of entries on https://rubenarslan.github.io[4]This website doesn’t exist yet, because as far as I know, I’ve never been wrong. Really, it’s high time for someone to tell or corrections, or retractions where appropriate.
Still, the form is open for anything. Fair game includes anything I’ve written, said, or implied, including my assessments of other research, my politics (which may affect the research questions I focus on), my professional behaviour (which affects the people I work with), and so on. I reserve the right to react less publicly by default when I’m only a co-author, because I don’t want to scare off future collaborators. If you tell me I wasn’t funny at a conference or horrible on Twitter, I might just cry to myself a little rather than react publicly.[5]Although I admit this also has humor potential when you can cope: http://slatestarcodex.com/2018/02/13/more-testimonials-for-ssc/
I’m mainly looking for criticism of things I can change. So “drop out of science, you’re too stupid and lazy to make it” is actionable insight, “you’re ugly and will never amount to anything” is out of scope. I’m not trying to set this as a new social norm. I’ll happily report back on my experiences when I have gathered some. I know anonymous comment boxes have a bad reputation for good reason. [6]If you want to set up such a form for yourself without waiting for my experiences, I put up the JSON-file that allows you to recreate such a form for yourself on formr.org.[7]This post, in turn, is mainly inspired by gwern’s mistake page.
In the near future, I also plan to set bug bounties, i.e. sums of money you can earn if you find errors in my published work. I’ll limit these to work published after this post, though, so please don’t let that keep you. Although academics love working for free in peer review, I thought this might be necessary to get people to read and criticise published work, where they don’t yet know what good the invested work will do.
Footnotes
↑1 | minus some points exchanged non-publicly in peer review |
---|---|
↑2 | Well, I’m really not the one to ask, because this dog is my spirit animal. |
↑3 | Now, you might also say that this could just be an empty virtue signal. After all, what prevents me from ignoring all of these comments? I don’t have a good solution to this criticism. I don’t want all anonymous criticism to be public and non-deletable by default, because, you know, the internet. I commit to reacting to criticisms of published first-authored work publicly within two months, but may in some cases put up a placeholder until I’m able to fully respond. I guess I could appoint a trusted third party to judge my responses, but I’m currently not sure what nature of criticism to expect and who would be trusted by my (anonymous) critics more than me. So, since some measure of trust is needed, it might as well be in me. |
↑4 | This website doesn’t exist yet, because as far as I know, I’ve never been wrong. Really, it’s high time for someone to tell |
↑5 | Although I admit this also has humor potential when you can cope: http://slatestarcodex.com/2018/02/13/more-testimonials-for-ssc/ |
↑6 | If you want to set up such a form for yourself without waiting for my experiences, I put up the JSON-file that allows you to recreate such a form for yourself on formr.org. |
↑7 | This post, in turn, is mainly inspired by gwern’s mistake page. |
A good scientist should be humble, because there’s always the possibility of being wrong. In fact, it’s so common for prominent scientists to be wrong that it’s even got a name: Nobel disease!
Good for you for seeking criticism and always acknowledging the possibility that you’re in error!
Speaking of Nobel disease, I meant to post this link: https://rationalwiki.org/wiki/Nobel_disease
In response to this and the other naysayers. You criticise but not in any constructive way. Who is to say that what you believe to be good would be successful? Reading some of these criticisms one would think you were not watching the same show. Criticism void of justification is mere opinion. If your going to criticise please just give us some valid facts to support your collective cases.