I would expect our fucking failures of stroke associations to go thru all stroke research and identify the wasted ones. Then boycott those senior researchers and mentors from further grants.
Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers
BMJ 2018; 363 doi: https://doi.org/10.1136/bmj.k4645 (Published 12 November 2018) Cite this as: BMJ 2018;363:k4645- Correspondence to: P Glasziou Paul_Glasziou@bond.edu.au
In
their history of the evolution of guidelines for reporting medical
research, Doug Altman and Iveta Simera showed that poor design, conduct,
and reporting of medical research have been concerns for over a
century: “The quality of published papers is a fair reflection of the
deficiencies of what is still the common type of clinical evidence. A
little thought suffices to show that the greater part cannot be taken as
serious evidence at all.”1
Indeed,
more than 250 years ago, the Scottish doctor James Lind declared in the
introduction to his review of reports on treating scurvy: “Before this
subject could be set in a clear and proper light it was necessary to
remove a great deal of rubbish.”2
Quantifying
the extent of poor reporting of medical research seems not to have
begun until 1966 (box). After assessing 295 publications in 10 “most
frequently read” medical journals, Schor (a statistician) and Karten (a
medical student) concluded: “In almost 73% of the reports . . .
conclusions were drawn when the justification for these conclusions was
invalid.”3
However,
the title of their article, “Statistical evaluation of medical journal
manuscripts,” was unlikely to ignite action among clinicians to deal
with a situation that threatened their patients’ wellbeing.
The wake-up call came 30 years later, in 1994, with an editorial in The BMJ by the journal’s chief statistical adviser, Doug Altman. He described as a scandal4
that “huge sums of money are spent annually on research that is
seriously flawed through the use of inappropriate designs,
unrepresentative samples, small samples, incorrect methods of analysis,
and faulty interpretation.” In 2015, the journal’s readers rated the
article the most important in The BMJ in the previous 20 years.5 It has been cited over 700 times.
Milestones in the recognition and reduction of avoidable research waste
- 1966: Schor and Karten find 73% of research conclusions not justified
- 1986: Simes shows impact of non-publication; call for universal registration
- 1994: Altman declares “scandal of poor medical research” in The BMJ
- 1996: CONSORT statement published
- 1997, 2000: ClinicalTrials.gov founded, made internet accessible
- 2006: Altman founds EQUATOR Centre in Oxford
- 2009: Chalmers and Glasziou publish “Avoidable waste in research” in Lancet
- 2010: National Institute for Health Research initiates adding value in research programme
- 2012: Begg and Ellis fail to replicate key preclinical studies
- 2013: AllTrials campaign launched
- 2014: Lancet series of five papers on avoidable waste
- 2015: Conference in Edinburgh and the REWARD statement
- 2015: Ensuring Value in Research (EVIR) Funders’ Forum founded
- 2017: Cochrane-REWARD prizes to NIHR, SYRCLE, and COMET
- 2018: Cochrane-REWARD Prizes to EQUATOR, TrialsTracker, and James Lind Alliance
Altman
was blunt: “What should we think about researchers who use the wrong
techniques, use the right techniques wrongly, misinterpret their
results, report their results selectively, cite the literature
selectively, and draw unjustified conclusions? We should be appalled.
Yet numerous studies have shown that all of the above phenomena are
common. This is surely a scandal”4
Altman’s conclusion was that we needed “less research, better research, and research done for the right reasons.”
Responses to Altman’s challenge
Altman’s challenge contributed to two important developments. Drummond Rennie, a senior editor of JAMA,
was influential in both. Firstly, in 1989, Rennie initiated a series of
congresses on peer review in biomedical publication, an international
forum in which these problems could be identified and confronted.
Secondly, he suggested that clinical trial reporting was so poor that a
reporting guideline was needed. The resulting CONSORT (Consolidated
Standards of Reporting Trials) statement6
prompted the development of other reporting guidelines. In 2006,
recognising the need to compile, improve, and disseminate these
guidelines, Altman established the first EQUATOR centre for enhancing
the quality and transparency of health research.7
Unfortunately,
medical journals’ efforts to improve reporting do not tackle
fundamental deficiencies in the design and conduct of research. Another
Altman article8
suggested that journals could help to prevent poor research by
promoting electronic publication of research protocols; this could occur
within the emerging framework of registering trials at their inception.
Subsequent reports of the registered research could be linked
electronically, through to deposition of whole datasets.
Better
trial registration has made it possible to assess the extent of biased
under-reporting of whole studies, first shown in medicine in 1986 with
comparisons of analyses based on all trials registered with all those
that had been reported in public showing that the reported trials
overestimated benefit by between 10% and 15%.9 Other studies showed biased reporting of outcomes within studies.1011
Commenting
on the practical consequences of these reporting biases, Alessandro
Liberati, a health services researcher and patient, complained about the
waste and suffering caused by the biased under-reporting of research.12
To tackle that concern, the AllTrials campaign was launched in 2013,
calling for all past and present clinical trials to be registered, and
their results reported.
How much money is wasted?
To
try to quantify Altman’s “huge sums of money” wasted, we estimated in
2009 that 85% of the more than $100bn a year spent on medical research
globally was being wasted avoidably.13
This waste arises from the multiplicative effects at different stages
of research: over 50% of research is not published; over 50% has
avoidable design flaws; and over 50% is unusable or incompletely
reported, or both.14
This
staggering sum brought attention to Altman’s scandal. One of England’s
principal research funders, the National Institute for Health Research,
soon established the “Adding value in research” programme to monitor and
tackle the problems. The Lancet commissioned five articles
with over 40 authors, including Altman and his former students, giving
more detail about sources of, solutions to, and recommendations for
reducing avoidable research waste (www.thelancet.com/series/research).
Altman’s
observations and critique were largely in applied clinical research,
but similar problems have become apparent in preclinical research. Most
notably, a 2012 study failed to replicate the findings of 47 of 53 “key”
preclinical studies, even though the attempted replications had mostly
been done by the original authors.15
This
finding challenged researchers to reconsider whether the research
enterprise was working. The causes can largely be found in the issues
Altman had raised in 1994: “inappropriate designs, unrepresentative
samples, small samples, incorrect methods of analysis, and faulty
interpretation,” together with pervasive biased under-reporting of
research.
What have been the drivers that led us to this
waste? The lack of motivation to tackle research waste has been driven
by the lack of awareness: funders and researchers often express
surprise, or even disbelief, at the scale of the problem. Instead,
academic and commercial incentives in the research system reward poor
research practices such as reanalysing data until statistical
significance is found (P hacking), spin, incomplete reporting, and
withholding protocols and data. Fortunately, both awareness and
incentives seem to be changing.
Where are we now?
At
Edinburgh’s 2015 conference on research waste, Altman’s keynote address
began, “It is my great pleasure to be here and make you feel quite sad
about how bad things are.”16
He proceeded to enthral and sadden us with a catalogue of editorials
and research which had pointed out the extensive and ongoing problems in
medical research. However, in 2016, a report by the journalology
researcher David Moher and his colleagues promoted some optimism that
practical steps could be taken to reduce waste.17 They documented the global response to the 2014 Lancet series using examples of individual initiatives to reduce research waste in different stakeholder groups.
Among the responses has been the establishment of the Evidence-Based Research Network (www.EBRNetwork.com)
in 2016 to lobby for all proposals for new research to be supported by
references to systematic reviews of relevant existing research.18
This is supported by the call from nine influentially placed coauthors,
including former and current chief scientific advisers to the British
government, for information syntheses that are inclusive, rigorous,
transparent, and accessible for policy makers,19 a call we consider long overdue.20
But
perhaps most notable and potentially influential development is the
Ensuring Value In Research (EVIR) Funders’ Forum, established by the
National Institute for Health Research in England, ZonMw in the
Netherlands, and the Patient-Centred Outcomes Research Institute in the
United States.21
The forum now includes over 30 research funders internationally,
working together to reduce research waste and improve value. Given their
influence through funding decisions and regulation, research funders
have a pivotal role in fostering improvements. However, their potential
influence remains underexploited: an audit of the adoption by funders of
relevant recommendations in the Lancet series showed low levels of uptake, with a few influential outliers.22
Cochrane-REWARD prize
At
the 2015 Edinburgh conference, the REWARD (Reduce Research Waste and
Reward Diligence) Alliance was founded to facilitate work on the
recommendations from the Lancet series. It has established the
Cochrane-REWARD prize to recognise and disseminate scalable initiatives
aimed at reducing avoidable waste in research. At the 2017 World
Congress on Research Integrity in Amsterdam, the first prize was awarded
to NIHR for its initial response and ongoing work in monitoring and
reducing research waste. The second prize was shared by the Systematic
Review Centre for Laboratory Animal Experimentation (SYRCLE), based in
Nijmegen, the Netherlands, for reducing waste in animal research through
systematic reviews and better regulation; and the Core Outcome Measures
for Effectiveness Trials (COMET) Initiative, which promotes the
agreement and adoption of core outcome measures for assessing the
effects of interventions, as pioneered in the 1990s by the
rheumatologists who created the Outcome Measures in Rheumatoid Arthritis
Clinical Trials (OMERACT) Group.
The 2018
Cochrane-REWARD prize was awarded to the EQUATOR Network for Good
Reports, and in recognition of the considerable work of Altman’s Oxford
Centre for Medical Statistics in improving research reporting and
reducing research waste more generally.23
Second prize went jointly to the James Lind Alliance for its extensive
and innovative efforts to improve research prioritisation and to the
Trials Tracker established by Ben Goldacre and his colleagues at the
Oxford Centre for Evidence Based Medicine. The tracker exposes the
institutions and people who are acquiescing in under-reporting of
research (it has shown that academia is now more culpable than
industry).
What needs to happen now?
Initiatives
such as those recognised by the REWARD prize are heartening, but their
work is poorly supported by funders and institutions. So, although the
pace of change in reporting guidance has been rapid, change in improving
practice has remained slow, with overall publication rates remaining at
about 50%, and vast other areas of medical research unobserved and
unimproved.
The recent evidence based medicine
manifesto, citing Altman’s 1994 “scandal” paper, proposes nine steps
towards “better evidence for better healthcare.”24
It links the problems of poor research to poor information for clinical
decision making. For Altman, these decision making consequences were
central. As a statistician, he was appalled by the poor quality of the
design, conduct, and reporting of research, not just for its deleterious
effects on the medical knowledge base but, more importantly, for its
“serious consequences for clinical practice, research, policy making,
and ultimately for patients.”
Given Doug Altman’s
pivotal role in raising awareness of the scandal of poor medical
research and in establishing the EQUATOR centres, it is unsurprising
that he received a Lifetime Achievement Award from The BMJ in
2016. To realise the full value of his legacy, research funders,
research regulators, research organisations, journals, and the many
people Doug taught and inspired must act together to design, conduct,
and report better research done for the right reasons. The continuing
ethical, scientific, and economic deficiencies of medical research
remain scandalous.
Biography
Paul
Glasziou is a GP researcher and the founding director of the Centre for
Research in Evidence-Based Practice at Bond University. He is also
chair of the REWARD Alliance.
Iain Chalmers has a longstanding interest in biased under-reporting of research. He is coordinator of the James Lind Initiative.
No comments:
Post a Comment