BY EYAL SHAHAR
View original article at Brownstone.org.
I think the answer is “yes,” but here is the story for you to judge.
Each year the CDC publishes estimates of the effectiveness of the flu vaccine in the previous season. Reported by several networks, these estimates are based on a research design that is called a test-negative case-control study.
Over the years, the various authors have shared a similar analytical strategy: Early respiratory events were considered differently from later respiratory events (on the assumption that no effect is expected until immunity is built up). “Early” has typically been within two weeks post-vaccination.
Several months ago, I realized that this special handling of early events is a source of bias called “immortal time.” To expose the bias, I used causal diagrams, formally called “directed acyclic graphs (DAGs).” DAGs were introduced in landmark publications from the 1990s and are widely recognized in epidemiology as a methodological tool.
I wrote a short paper with a scary title: “Immortal time bias in test-negative studies of the flu vaccine.” I illustrated the bias by two simple DAGs that encoded the analytical approaches as described in those CDC-associated studies. The paper may be summarized in three bullet points:
Immortal time is an overlooked bias in test-negative, case-control studies of the flu vaccine.
The causal structure corresponds to misclassification bias or selection bias, depending on how early events were handled.
The bias can be avoided by considering all events and estimating built-up vaccine effectiveness by consecutive post-vaccination days.
I submitted the paper sequentially to three respected epidemiology journals. Surprisingly, the editor-in-chief of each journal rejected the paper within one week using boilerplate text. It was not sent for peer review. Why?
There are three possible reasons:
The message was not sufficiently important.
The paper was poorly written.
Soliciting peer review was not needed. The editor decided that the bias did not exist.
I can quickly eliminate the first reason. Exposing entrenched bias in studies of the annual flu vaccine is of utmost importance. I didn’t need to compete with any “more important” papers.
Was it poorly written? I have published many scientific papers and two books. I am not a newcomer to epidemiology and even served as an associate editor for one of the three journals. So, that was not the reason.
We are left with the third reason. Three editors (possibly more in-house) decided that my analysis was obviously wrong. They did not need any external peer review. Was that the case indeed? How likely is it that three editors have reached a unanimous, independent verdict on my analysis without soliciting peer reviews?
I could have written a whole page on my expertise in the domains of causal diagrams and biases, but here is a short version of my record. I suggested a new taxonomy of biases based on DAGs and identified new types of bias. I authored about 20 DAG-related publications, including a book chapter. (Some of my articles were cited in the paper.) For sure, my methodological analysis was not “obviously wrong.” More likely, it was obviously correct.
So, why was my paper rejected without peer review? May I speculate?
This paper has a startling message that discredits the CDC. All those studies about the effectiveness of the annual flu vaccine contained a mistaken analytical decision. The reported estimates are biased. Had my paper been published, it might have been picked up by some reporters. You can imagine the headlines.
The editors were dealing with a hot potato. They did not want to take the risk that peer-reviewers would endorse the paper, which would force them to publish it. An upfront rejection was a safe way out.
I tried once more. In my fourth submission to an epidemiology journal, I included the following cover letter:
“Dear Editor,
This short paper addresses immortal time in test-negative studies of the flu vaccine. Here, two common approaches to handling early post-vaccination events are encoded by causal diagrams to identify and classify a source of bias. At the end, I explained how the bias may be avoided.
To my knowledge, the message is novel and relevant to future studies that employ the test-negative design to estimate vaccine effectiveness. Therefore, I believe that the paper deserves peer review.”
This time it took a little longer to get a response. Perhaps the editors were busy, or they contemplated how to handle the “hot potato.” No surprise was in store, however. About three weeks later, a boilerplate message landed in my inbox. Two paragraphs are pasted below:
“Your manuscript has been reviewed by two members of the editorial board. I am sorry to inform you that its priority rating was not sufficiently high to warrant our accepting it for publication…The judgment about priority is made in view of the many other papers we receive and our space limitations. Thus, even informative papers have to be rejected on occasion. This preliminary review is done on all papers so that authors can receive a quick decision if it is going to be negative. Thus we have no reviewers’ comments to forward to you.”
Then, they graciously added:
“Other reasons why we are rejecting your manuscript include: After an initial editorial assessment, we have decided not to send the paper for external peer review. Although the topic is of interest, the manuscript does not align closely enough with the journal format.“
So, which do you think it was: a priority rating, alignment with the journal format, or a hot potato?
I gave up and posted the paper as a preprint. It is now in the public domain, but not very many will read it. It was not certified “peer-reviewed.” And the CDC will continue to report annual estimates of vaccine effectiveness from studies that contain that bias (not the only one, by the way.)
However, every editor who rejected my paper and every author of the CDC papers on the flu vaccine will get the preprint in their inbox. I will also try to reach editorial board members and others. If someone thinks that my analysis is faulty, perhaps they will respond. Otherwise, I expect my inbox to remain free of replies. Good enough for me. Less so for public health.
We still have freedom of scientific dissent, albeit with some exceptions.
Thank you for posting a report on these incidents.