1 in 5 Brain Injury Trials Show Errors, Signs of Spin

This is for TBI but ask your doctor how badly spin affects stroke research. Your doctor, if competent, should be cognizant of this problem and be able to discuss it.

1 in 5 Brain Injury Trials Show Errors, Signs of Spin

LOS ANGELES — A new report shows that spin, including signs of exaggeration and mathematical errors, was seen in 21% of 150 randomized traumatic brain injury clinical trials published in leading medical journals.

"This is concerning result," said general physician Lucas Piason F. Martins, MD, of Brazil's Bahiana School of Medicine and Public Health. "Many of these trials have been included in clinical guidelines and cited extensively in systematic reviews and meta-analyses, especially those related to hypothermia therapy."

Martins presented the findings at the American Association of Neurological Surgeons 2023 annual meeting.

Defining Spin

In recent years, medical researchers have sought to define and identify spin in medical literature. According to a 2017 report in PLOS Biology, "spin refers to reporting practices that distort the interpretation of results and mislead readers so that results are viewed in a more favorable light."

Any spin can be dangerous, Martins said, because it "can potentially mislead readers and affect the interpretation of study results, which in turn can impact clinical decision-making."

For the new report, a systematic review, Martins and colleagues examined 150 studies published in 18 top-ranked journals including the Journal of Neurotrauma (26%), the Journal of Neurosurgery (15%), Critical Care Medicine (9%), and The New England Journal of Medicine (8%).

Studies were published between 1960 and 2020. The review protocol was previously published in BMJ Open.

According to the report, most of the 32 studies with spin (75%) had a "focus on statistically significant results not based on primary outcome."

For example, Martins told Medscape Medical News that the abstract for a study about drug treatment of brain contusions highlighted a secondary result instead of the main finding that the medication had no effect. Another study of treatment for severe closed head injuries, he said, focused on a subgroup outcome.

As Martins noted, it's potentially problematic for studies to have several outcomes, measure outcomes in different ways, and have multiple time points without a predefined primary outcome. "A positive finding based on such strategies could potentially be explained by chance alone," he said.

The researchers also reported that 65% of the studies with spin highlighted "the beneficial effect of the treatment despite statistically non-significant results" and that 9% had incorrect statistical analysis.

The findings are especially noteworthy because "the trials we analyzed were deemed to have the highest quality of methodology," Martins said.

The researchers didn't identify specific studies that they deemed to have spin, and they won't do so, Martins said. The authors do plan to reveal which journals were most spin-heavy but only when these findings are published.

Were the study authors trying to mislead readers? Not necessarily, he said. Researchers "may search for positive results to confirm their beliefs, although with good intentions," Martins said, adding that the researchers found that "positive research tends to be more cited."

They also reported that studies with smaller sample sizes were more likely to have spin (P = .04).

At 21%, the percentage of studies with spin was lower than that found in some previous reports that analyzed medical literature in other specialties.

A 2019 study of 93 randomized clinical studies in cardiology, for example, found spin in 57% of abstracts and 67% of full texts. The lower number in the new study may be due to its especially conservative definition of spin, Martins said.

Appropriate Methodology

Duke University Medical Center cardiologist Richard Krasuski, MD, who co-authored the 2019 study into spin in cardiology studies, told Medscape Medical News that the new analysis follows appropriate methodology and appears to be valid.

It makes sense, he said, that smaller studies had more spin: "It is much harder to show statistical significance in small studies and softer endpoints can be harder to predict. Small neutral trials are also much harder to publish in high-level journals. This all increases the tendency to spin the results so the reviewer and eventually the reader is more captivated."

Why is there so much spin in medical research? "As an investigator, you always hope to positively impact patient health and outcomes, so there is a tendency to look at secondary analyses to have something good to emphasize," he said. "This is an inherent trait in most of us, to find something good we can focus on. I do believe that much of this is subconscious and perhaps with noble intent."

Krasuski said that he advises trainees to look at the methodology of studies, not just the abstract or discussion sections. "You don't have to be a trained statistician to identify how well the findings match the author's interpretation," he said.

"Always try to identify what the primary outcome of the study was at the time of the design and whether the investigators achieved their objective. As a reviewer, my own personal experience in research into spin makes me more cognizant of its existence, and I generally require authors to reword and tone down their message if it is not supported by the data."

What's next? The investigators want to look for spin in the wider neurosurgery literature, Martins said, with an eye toward developing "practical strategies to assess spin and give pragmatic recommendations for good practice in clinical research."

No study funding is reported. Martins has no disclosures, and several study authors report funding from the UK National Institute for Health Research. Krasuski has no disclosures.

American Association of Neurological Surgeons 2023. Presented April 21, 2023.