Thursday, May 20, 2010

Is professional commenting worth the risk?

Journal articles are the basis for the advancement of science and the discussion of novel data and analyses as they are published is one feature that unites all fields. Between journal clubs, informal discussions, conferences, emails, phone calls, Morse code, etc., we all engage in debate over our results and those of others. But other than in the memories of those involved, these discussions are largely "off the record" and invisible to anyone new entering a particular field.

To foster this type of discussion in a more open and accessible format, some journals have added the option to comment on published papers online. Notably, the BMC and PLoS journals were at the forefront of this movement and Nature has recently enabled this as well. There are others, but you get the point - some high profile papers are available for individuals to comment on and for everyone else to see what those people said.

In an ideal world, this is a great option. It allows for the compilation of people's reaction to certain studies in a publicly accessible way. It brings the water cooler to the source and can provide insight to those who are new to the field. Which papers have which potential flaws? Who are the major players in the debate on this work? It could all be right there.

In practice, how does this work? Well, if you've looked through these journals you will probably note that there are not many comments. An occasional paper will spark some debate, but the mechanism is widely underutilized and the community has not really embraced it. Why?

Well, one reason might be the concern about the repercussions of publicly attacking (even nicely) someone's work. Maybe one of the authors doesn't appreciate the "debate" and consciously or unconsciously holds that against you as a reviewer of your paper/proposal/tenure review package down the road. Maybe it's not even an author on the paper, but someone who reads your comment but will never be known to you. Is the benefit of public debate to the community more important than possible repercussions to your personal work?

I bring this up because I came across a paper just the other day that has what I believe to be fundamental flaws in its approach. The paper's conclusions are interesting, but inherently problematic based on the author's assumptions stemming from their apparent lack of familiarity of the pertinent literature. Knowing that the results might get picked up upon, but the assumptions underlying those results might not be obvious to those interested in the final outcome, I considered commenting on the paper. Thus, I asked some colleagues whether it was important to point this out and the reaction ranged from "This is important for science" to "You would be stupid to risk it".

I know there are plenty of studies on people's interest in public literature commenting (I'm just too lazy to look them up right now), but doesn't it just come down to risk? If there is any perception that one's comments could have negative consequences for one's research, what is the motivation to participate? Do science martyrs get tenure?

This is basically the same debate that goes on in the blogosphere between those who prefer to blog under a pseudonym and those who think that any comment or blog without a "real" name associated with it isn't worth a damn. PLoS, for instance, makes it very clear that all comments must be obviously attributable to an identifiable individual (although they with keep your email address secret, because no one can find that out).

I wonder how commenting would be affected by allowing pseudonyms to comment on professional journal articles. Would early career scientists feel emboldened to engage in public "on the record" debate? Would it empower those who refuse to open themselves to any additional risk to their livelihood? Would the debate improve or devolve into a hotbed of uncivil behavior?


  1. I wonder whether the degree of fear of adverse outcome has now outstripped the real risk. Are we giving advice that is too reactionary?

  2. I do believe that-—and it may sound like I am cynical or worse, disgruntled-—the whole system of academic reviews is stacked against early-career academics. Any type of review--whether it is an article for peer reviewed journal, review for grant proposals, and of course the holy grail of tenure and promotion reviews--gets unconsciously or consciously biased if a senior established professor is involved.

    For peer-reviewed journals, anonymous peer reviews is an honest and fair mechanism to establish the quality of the work, but that is true only in principle. Haven't we all experienced the situations where senior professor is added to the author list only for his name, so that the paper goes in an high-impact journal? Or that if unfamiliar names--young researchers, or researchers from developing world--are on the paper, the reviewers are somewhat more skeptical than they would be if they see a bigshot name? I wonder if this can be avoided if the review process is made double-blind, where reviewers will not get to see who is the author.

    So I am not surprised why you get the comments asking you not to risk it, as you prepare to stand naked in front of all the established ones pointing out the truth.

  3. Another question: Should we be training graduate students in how to strike the right tone for public critique of a paper?

  4. Your colleagues are both right. Discussion IS important for science, but unfortunately scientists' egos are way too fragile and their dispositions far too vindictive for the benefits of public, non-anonymous commenting to outweigh the possible negative consequences. What's that they say when you're arrested? "Anything you say can and will be used against you." I think that pretty much sums it up!

  5. Dr Becca showing her experience as a hooligan youth?

    The truth of the matter is that it only takes one example of someone enduring consequences of their public dispute with a paper to keep everyone else from putting themselves out there. As one colleague pointed out, there are ways to "argue" in the literature that are more generally accepted (and recognizable as something productive on your CV) than leaving comments on people's papers on the publisher's website. Is that ideal for science? I don't know.

  6. I agree with Prof-like Substance. It is "dangerous" to comment publicly on a paper. Comments on the Internet are forever (hence most bloggers use pseudonyms). It is hard to strike the right tone in text with a stranger. In a formal referee report, there is a context for remarks that allow criticisms to be taken constructively (by most sane scientists) that is just not there for something put out into the world for anyone to read. I am far more likely to hold a scientific discussion over the phone or email than over blog or public comment. I've mailed authors with questions before, and that has lead to some really nice discussion. The relative privacy of email vs the relative public space of commenting/blogs/journal clubs makes all the difference.

    FWIW, I also think some people are inhibited from commenting in case they are not sure of themselves, and don't want to be publicly wrong either.

  7. I also agree with the dangerous comments. For better or worse, context is important. The stuff i would say about a paper over beers with a friend, in journal club, at a conference or as a reviewer would vary substantially in tone. Id like to think, though, not in conclusion. This could be especially problematic for those in politically charged fields (climate change, stem cells, etc)

  8. I was once an over enthusiastic student, keen to point out every flaw I thought I saw. I believed I was was practicing being a good scientist, but I was naive to the technical difficulties, costs, and time associated with many methods. This was before my first paper - before I realized scientists cannot help but become emotionally attached to a project that has consumed several years of their life.

    For this reason I am thankful the critiques of papers I analyzed as a student are not all over the web!

    I have high standards for anything I let out under my real identity. However, anonymous postings allow me to push the boundaries and test out ideas that I can abandon. The peer review process is anonymous and and editors offer some protective from unfair or malicious comments. It could work online to, but only if policed to keep out crazies and haters.

  9. I'm not sure I agree that it would need to be policed. I think one of the strengths of the idea is that everything is out there for others to evaluate. Even if anonymous or pseudonymous comments were allowed, there would still be the opportunity for the reader to take in all the comments and decide for themselves.

  10. A very nice post!

    I had the same urge regarding a paper not so long ago, while still on tenure track. The paper attempted to refute the results of our (mine and my collaborators') earlier high-profile paper, but the refutal was based on a set of assumptions completely invalid for the experimental situation at hand.

    I actually drafted a response, and was encouraged to submit it by a senior collaborator on the project. However, another, more conservative senior colleague said not to do it, that comments and refutals are not what you want yourself associated with *WHILE ON TENURE TRACK*.

    I never submitted it and I think it was the right decision. Since you are on tenure track, I strongly recommend you don't either. You don't want that kind of exposure -- I think subconsciously most people will consider you are overstepping your early-career authority, and the author will likely not be your friend for openly challenging him/her.

    A thing that you may consider doing is sending a polite comment to the author directly, and starting a discussion where no one is publically challenged/humilated.

    Or as I did, take a deep breath and let go. The first author was a postdoc looking for a prof position, and he didn't need me to cramp his style. There's plenty of time to straighten it later directly.

  11. Dr Becca showing her experience as a hooligan youth?

    It's what they say on TV!

  12. I called into question some work done once by a past collaborator during a videoconference broadcast to various sites. The collaborator got pissed and immediately asked who I was before they would even answer the question. She was pretty snide to me in her response until my boss leaned over the microphone and told them I was write. The look on their face as they shit an intellectual brick was priceless. So it may be important to science but egos do exist and it might be worth a letter to the editor if you can drum up some more senior colleagues to back your concerns and sign beneath the comments line.