A randomized trial alerting authors, with or without coauthors or editors, that research they cited in systematic reviews and guidelines has been retracted

ABSTRACT Retracted clinical trials may be influential in citing systematic reviews and clinical guidelines. We assessed the influence of 27 retracted trials on systematic reviews and clinical guidelines (citing publications), then alerted authors to these retractions. Citing publications were randomized to up to three e-mails to contact author with/without up to two coauthors, with/without the editor. After one year we assessed corrective action. We included 88 citing publications; 51% (45/88) had findings likely to change if retracted trials were removed, 87% (39/45) likely substantially. 51% (44/86) of contacted citing publications replied. Including three authors rather than the contact author alone was more likely to elicit a reply (P = 0.03). Including the editor did not increase replies (P = 0.66). Whether findings were judged likely to change, and size of the likely change, had no effect on response rate or action taken. One year after e-mails were sent only nine publications had published notifications. E-Mail alerts to authors and editors are inadequate to correct the impact of retracted publications in citing systematic reviews and guidelines. Changes to bibliographic and referencing systems, and submission processes are needed. Citing publications with retracted citations should be marked until authors resolve concerns.


Introduction
Once concerns are raised with journals, delays in the publication of expressions of concern or retraction notices may be lengthy (Grey, Avenell, and Bolland 2021;Loadsman 2019).Even after the retraction of publications, citations usually continue (Candal-Pedreira et al. 2020;Mott, Fairhurst, and Torgerson 2019), although these later citations may constitute a minority of all citations for the retracted work (Bolland, Grey, and Avenell 2022).The continued citation of retracted clinical trials is particularly likely to put patients at risk (Steen 2011(Steen , 2012)).Previously, Avenell et al. (2019) concluded that the findings of 13/32 (41%) systematic reviews, clinical guidelines and other reviews would likely change if the retracted trials they cited were removed.
Presently, there are no integrated systems to alert researchers that they have cited work with expressions of concern or retractions.The Zotero, EndNote, LibKey and Papers bibliographic systems can automatically alert researchers to entries in Retraction Watch's database of expressions of concern or retractions (Retraction Watch's website 2022).Initiatives like this should start to reduce the citation of research in academic literature written after work has been retracted, but may not aid correction of literature which has already been published that cites retracted work.
If a researcher finds that their publication cites retracted work, what action should they take?Detailed guidance is lacking.Committee on Publication Ethics (COPE) guidance only states that "Articles that relied on subsequently retracted articles in reaching their own conclusions, such as systematic reviews or meta-analyses, may themselves need to be corrected or retracted" (Committee on Publication Ethics 2019).Of all publications, systematic reviews and clinical guidelines are likely to have the greatest influence on clinical practice and patient care and policy, thus it is particularly important to develop effective systems to alert citing authors to expressions of concern and retractions to ensure their publications are rapidly updated.
Following suspicions aroused during the conduct of a Cochrane review (Avenell et al. 2009), 33 randomized trials by Yoshihiro Sato and Jun Iwamoto were investigated, and findings published (Bolland et al. 2016).Of these publications, twenty-eight were retracted between 2016-2019 in the field of osteoporosis; research particularly relevant for older people with dementia, stroke, and Parkinson's disease.In this paper, we examined the influence of 27 of the retracted trials on citing systematic reviews and guidelines, where authors appeared unaware that they had cited retracted work.We assessed whether corrective action was needed, and alerted authors to these retractions.We did not know who to alert by e-mail for these systematic reviews or guidelines, or how many reminders might be needed.Therefore, citing publications were randomized to notification of the contact author or the contact author and coauthors, with or without notification of the journal editor.We assessed the responsiveness of recipients, and the incidence and nature of any corrective action undertaken.

Materials and methods
The study protocol is available upon request to the corresponding author.Ethical approval was not required.

Literature searching
By 1 November 2019, 28/33 affected trial reports from the original investigation (Bolland et al. 2016) had been retracted.One affected trial report was retracted on 24 October 2019 and was not included in this study, as the authors would likely have had insufficient time to receive notification before our study commenced.Web of Science (WoS) was searched on 22 November 2019 for publications citing the 27 included trial reports (see list in Appendix 1), and Scopus on 16 January 2020 for any additional citing publications.Citing publications were only included if they were systematic reviews, including meta-analyses, or clinical guidelines.We also hand searched personal files for citing clinical guidelines relating to osteoporosis.Self-citing systematic reviews with Yoshihiro Sato or Jun Iwamoto as first authors were excluded.Systematic reviews or clinical guidelines which only cited retracted trial reports in their introduction, discussion, or as excluded studies (e.g., Cochrane reviews' category of excluded studies) were not included, thereby limiting citing publications to those whose findings were most likely to be affected.
From July to September 2020, we searched Web of Science, and the websites of journals and guideline organizations, to identify any published evidence (e.g., correction notice, correspondence in the journal) to indicate that authors of citing publications had identified that they had cited one or more of the 27 retracted trials.Citing publications which were unaware of the retractions were included in our trial.

Assessment of impact
For each citing publication, two authors independently categorized the likely impact of the retracted trial reports: whether the findings were likely to change when the retracted trial(s) was removed (Yes/No/Uncertain), the size of any likely change (substantial, moderate, minor), and whether reanalysis of data by the authors was required.Substantial effects were judged if there was a change in the direction of effect, e.g., from negative to neutral, or neutral to positive.Moderate effects were changes within the same effect size.Minor changes were small and did not change the interpretation of the result.
When there was disagreement, a third author made an independent assessment and the majority view was taken.If the citing publication provided a quality assessment rating for the retracted trial report(s), this was also extracted.

Randomization
In a factorial design, with 1:1 randomization, all citing publications were randomized to either an e-mail to the contact author and the journal editor, or only the contact author, using a random number generator (MS Excel 2010).If no editor contact was available, a publisher contact was used, or an organizational contact for guideline organizations.Citing publications with two or more authors were randomized by the same process to an e-mail to the contact author only, or up to 3 authors (out of contact, first, second and final authors).In all cases, coauthors and editors were copied into e-mails to contact authors.

E-mail notifications
Between 15 September 2020 and 14 October 2020, e-mails were sent to contact authors of each citing publication, and depending on randomization, copying in the editor, or other authors.Each e-mail provided the full reference of the citing publication, a list of the retracted trial reports cited, and a pdf of the citing publication highlighting the retracted trial reports.Contacts were advised that a full list of all retractions of publications by the Sato/Iwamoto group was available from the Retraction Watch's database (http://retractiondatabase.org/RetractionSearch.aspx?).
Recipients were asked to confirm receipt of the e-mail and asked whether they believed any action needed to be taken for the citing publication.E-Mails did not contain our assessments of the impact on the citing publication of removing the retracted trial report(s).
If e-mails "bounced," alternative e-mail addresses for the same authors were sought, e.g., from institutional websites or newer publications.If no e-mail address could be located, another author was sought as a replacement.E-Mails were sent monthly for a maximum of three months.Once any author or editor replied for a citing publication, no further e-mails were sent for that publication.If authors or editors replied asking what they should do, our reply stated that there was no official guidance, but they might consider publishing a notice with the publication.

Assessment of outcomes
We followed-up for outcomes for one year after initial e-mails were sent.The outcomes were: any reply yes/no, time for a reply from first author, time for any reply, action taken (notice published, including date, whether and to what extent the notice provided the impact of the retracted trial(s) and whether re-analysis was undertaken); notice planned, as notified to us; unclear action planned; no action planned.We searched PubMed, Web of Science or Scopus, journal and organizational websites for any notice published relating to the citing publication.
Two authors independently categorized and analyzed themes in the text of replies to our e-mails, which were checked by a third author.
Comparisons between groups or by assessed impact for categorical variables were done using chi-square tests and for continuous variables using the Wilcoxon test for non-parametric data.All analyses were performed using R (R 3.5.1,2019, R Foundation for Statistical Computing, Vienna, Austria) and P < 0.05 was considered statistically significant.

Results of the search
Figure 1 details the flow chart for the identification of the 88 citing systematic reviews and clinical guidelines included in the trial, published between 2003 and 2020, citing at least one of the 27 retracted trial reports (see Appendix 2), Of these citing publications, 10/88 (11%) cited a trial report that already had been retracted or had an expression of concern prior to submission.
The 27 retracted trial reports had multiple integrity issues in common, including failure of randomization, improbable recruitment and productivity, remarkably positive outcomes inconsistent with existing literature, inconsistencies between and within trials, failure of ethical oversight and admitted data fabrication (Bolland et al. 2016;Gross 2016).

Description of citing publications
Of the 88 citing publications, 18 (20%) were clinical guidelines or position statements from professional organizations relating to a wide range of

Quality assessment of retracted trial reports by citing publications
42/88 (48%) citing publications with 69 retracted trial report citations performed a quality assessment of those reports.Of these 69, 41 were assessed using qualitative scoring [18 Jadad scale (Jadad et al. 1996); 9 PEDro scale (PEDro scale 2022); 25 by a risk of bias approach including 20 based on Cochrane risk of bias (Higgins et al. 2011)], and three with a descriptive approach.Of the 41 reports with a quality score, seven were given the highest possible score, 28 above the middle score, two below the middle score, and four were not reported.Of the 25 articles with a risk of bias assessment, 14 were scored as low risk across the majority of categories, six as not low risk across the majority of the categories, one a description only, and four were not reported.Thus, the majority of the retracted trials were assessed as low or moderate risk of bias or moderate to high quality.None of the citing publications documented any concerns regarding the publication integrity of the trial reports.

Analysis of impact of retracted trial reports
Table 1 shows that, overall, 45 (51%) of citing publications had findings that we assessed would likely change if the retracted trial reports were removed; in 39/45 (87%) there was likely to be a substantial change, i.e., a change in the direction of that finding; and 20/45 (44%) were assessed as needing reanalysis of data.Agreement between the two main assessors was good: whether findings were likely to change -agreement 85%, kappa 0.75; the size of effect if findings changed -agreement 96%, kappa 0.84; need for reanalysis -agreement 100%.The assessed impacts on citing publications were similar across the randomized groups (Table 1).
Outcome assessment E-Mails regarding two citing publications were not sent because of events between searching and e-mail dispatch.One publication was retracted because meta-analysis findings relied entirely on retracted trial reports (Zhang et al. 2014), and one because the guideline had been updated removing the retracted trial report (Healthcare Improvement Scotland 2015), with guideline staff aware of the retracted trial report through other correspondence with us.Thus, e-mails were sent for 86 publications from a total of 88 randomized.

E-mail replies
We received a reply for around half (44/86, 51%) of citing publications that were emailed (Table 2).No author or editor indicated that a correction notice had been planned before our e-mail arrived.26/41 (63%) of e-mails to up to three authors produced a reply compared to 18/45 (40%) e-mails to the contact/only author (P = 0.03).Copying the editor had no effect on the proportion of replies received (P = 0.66).Only 9/18 (50%) of guideline authors or organizations provided any e-mail reply.For all replies, the mean time to the first reply was 15 days and the median (range) 2 days (0-63).There was no difference in time to first reply between e-mails to the contact/only author and up to three authors (P = 0.41), or if the journal editor or guideline organization contact was included in the e-mails (P = 0.29).Whether we received any reply, the time for any reply, and  4) 3 ( 16) 1 ( 50) 2 ( 8) 3 ( 14) 5 ( 11) Minor 1 ( 4) 0 (0) 0 (0) 1 ( 4) 0 ( 0 the time for a reply by the first author did not differ by year of the citing publication, (<median of 7 years old versus ≥ 7y), whether the publication cited a retracted trial report at the time of submission or not, or whether the findings were judged likely to change by removal of the retracted trial, the size of the likely change or the need for data reanalysis.Among the 44 citing publications for which we received a reply, 35 (80%) either planned no action or it was unclear what they intended to do (Table 2).Three said that they planned a notice in their initial e-mails, but no notice was published within one year of being contacted.
In the four cases we judged findings likely to be changed with substantial effects, one paper was retracted (Zhong and Chen 2009), and another author indicated that previous findings were now "called into question" (Peterson 2014).One Cochrane review wrote that the retracted trial report had "minimal impact" (Gillespie et al. 2012), whilst another indicated that the effect was "uncertain" (Qaseem et al. 2008).In the one case where we had considered that reanalysis was required, this was undertaken and new results presented (Murad et al. 2012).The same author re-analyzed data in a further study (Murad et al. 2011).
In summary, one year after our notification e-mails, for 39 of the 44 publications whose findings we considered would change, no action was taken.Four of the remaining 42 publications emailed had also publicly alerted their readers (Herrmann et al. 2007;McCarus 2006;Murad et al. 2012;Qaseem et al. 2017).
We pooled outcomes into two groups (no action planned or unclear if action was planned, and action planned or taken).There were no differences between outcomes whether we contacted up to three authors or only the contact author, whether we contacted the journal or guideline organization or not, or whether the findings were judged likely to change by the removal of the retracted trial, the size of the likely change or the need for data reanalysis.
Two citing publications (Tricco et al. 2017 in JAMA Internal Medicine;Invernizzi et al. 2009 in Parkinsonism and Related Disorders) were published by journals which had themselves retracted one of the source trial reports, in one case after the citing publication came out, and one before.Both citing publications had been cited multiple times in 2021, and neither has been corrected.

Themes identified in e-mail replies
Contact authors from four citing publications asked us for advice as to what they should do.In another four cases the contact author already knew of retractions relating to their publication, but had not provided any update in public.
For 12 citing publications correspondents thought that there was no or minimal impact, so did not plan to alert readers.For three of these 12, we had judged likely substantial impact from removal of the retracted trial reports, but correspondents provided no justification for their decisions.
In three cases, the contact authors said the retractions occurred after their papers were published, and thus they would not take action, or it was unclear what they planned; in one the retraction was three years after the systematic review was published, in two cases some retractions took place 2-5 months before publications were submitted or published online.
In three cases, contacts replied that they did not have the resources to undertake a re-analysis.
Two contact authors planned to update their reviews removing retracted trial reports, but it was unclear when this would happen, and both existing reviews continued to be cited in 2021 without correction.
Although we expected coauthors to contact the lead author who had ultimate responsibility for the publication, for seven citing publications coauthors appeared to have made decisions in the absence of the contact author.
In one case, the contact author indicated no intention of taking responsibility for assessing issues, and in another the journal editor did not think he had a responsibility for the citing publication.In three cases, editors appeared to lead discussions on corrective action.
For one clinical guideline, judged to be likely substantially impacted, with over 1500 citations, including 37 in other guidelines, the contact author asked to be removed from the mailing list and gave no other response.

Discussion
Twenty-seven clinical trial reports published 1997-2012 and retracted 2015-2019 have been disseminated in 88 systematic reviews and clinical guidelines.None of the quality assessments published in the citing publications identified any of the retracted trials as concerning.We judged that the findings of 45 (51%) of these citing publications were likely to change after retracted trial reports were removed.For 39/45 (87%) effect sizes were likely to be substantially different.One year after notifying authors and/or journals that their publications had cited retracted trial reports, only 9 had published any kind of public notice, and only four of the 39 publications judged most likely to be substantially affected had alerted readers.Although copying e-mails of notification to more than one author was more likely to elicit a reply, we found no effect of including other authors on addressing concerns in those authors' systematic reviews or guidelines.Nor did copying in editors or contacts for guideline organizations provoke significantly more responses or alter the frequency of publication of correction notices.Guideline organizations were no more likely to reply or update their readers than other publications.
Whether guidelines' and systematic reviews' findings were judged likely to change by removal of the retracted trial, the size of the likely change, or the need for data reanalysis, had no demonstrable effect on outcomes.However, our trial may have been underpowered to detect differences.Our trial data reflect not only systematic reviews and guidelines in osteoporosis care but also the evidence base for older people with dementia, stroke, and Parkinson's disease.Authors in other fields might respond differently, but we suspect this is unlikely.
In a recent estimate of the epistemic effect of retractions on meta-analyses, Fanelli, Wong, and Moher (2021) examined the potential impact of retracted publications on 50 meta-analyses published between 2013-2016, mostly in clinical journals.The effect size across all meta-analyses without retracted studies was not statistically different from the effect size including those studies.Fanelli, Wong, and Moher (2021) indicated that the impact of retractions was likely to be variable and context specific.To some extent, meta-analyses and systematic reviews by their methodology should be protected from the influence of retraction of a single study, and all but one of the 50 meta-analyses they examined each included only one retracted report.However, in the case of retracted trial reports from Sato and Iwamoto, comparable clinical trials from other authors contributing evidence were absent or few, with small sample sizes.Sato and Iwamoto often published several trial reports of similar interventions in high risk populations (e.g., patients with low vitamin D status, Parkinson's disease, or stroke), resulting in frequent instances of more than one retracted trial report in citing publications.Thus, the potential for influence is much greater than found by Fanelli, Wong, and Moher (2021).
In a study of 98 retracted randomized controlled trials, Kataoka et al. (2022) found none of the 77 systematic reviews or clinical guidelines including trials retracted beforehand in their evidence synthesis subsequently made corrections.Only 5% (6/130) of evidence syntheses in systematic reviews and 11% (2/18) of clinical guidelines made corrections when trials were retracted after article publication.Three of the publications making corrections were prompted by our trial.Kataoka et al. (2022) demonstrate that failure to correct systematic reviews and clinical guidelines extends beyond the retracted work of Sato and Iwamoto.
There is already a lengthy delay in alerting readers to publications with integrity issues (Grey, Avenell, and Bolland 2021).Inactivity in correcting systematic reviews and clinical guidelines that cite retracted publications exacerbates potential harm to patients, and may mislead clinicians, policy makers and other researchers.While prompter retractions, or at least notifications of concern, would likely reduce misinformation arising from publications with integrity issues, system-wide changes are needed to truly stem that adverse impact.
Our trial suggests that automated e-mails, generated for example from bibliographic citation databases, that notify retractions to either authors of citing publications or publishers (Hamilton 2019; van der Vet and Nijveen 2016), will not elicit prompt corrections or updates to those publications.
Although our e-mails arrived unannounced and did not come from these sources, the details they provided were greater than could be provided by automated systems.Our e-mails may have been treated as junk mail, although this seems unlikely, given that they came from an academic e-mail address.That we received no answer or saw no update to citing publications does not mean that publication of notices will not occur in the future, but if a response does not occur within a year, it seems unlikely to expect that one will eventuate.
What needs to change?Firstly, when researchers write papers it is essential that the bibliographic databases and popular search engines (van der Vet and Nijveen 2016) they consult have up-to-date, accessible, information on retractions, expressions of concern and corrections.Whatever the notice, it must be clearly linked to the affected publication, rather than as a disconnected update, as happened with four of the nine notifications from this trial.Consistent terminology to mark affected publications needs to be visibly and rapidly adopted, and used throughout the publishing process (Schneider et al. 2021).Wider use of Crossref's CrossMark system by publishers would also allow readers to review retractions and corrections when viewing articles (Cosentino and Veríssimo 2016).
Secondly, at manuscript submission authors need to check that they have not cited retracted work, and confirm this in the submission process (Sox and Rennie 2006).The International Committee of Medical Journal Editors has long had guidance for this (Sox and Rennie 2006;International Committee of Medical Journal Editors 2022); journals should require confirmation at the submission stage.Confirmation that retracted papers have been excluded from systematic reviews should be stated in the publication and could be included in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist for reporting of systematic reviews (PRISMA 2020).Authors should also confirm at submission of systematic reviews and guidelines that they will publish a note updating their publication if cited references in their evidence syntheses are retracted.
Thirdly, publishers should improve quality control of their publications by checking submitted and published articles for citations to retracted work, using software packages coupled with Retraction Watch's database of retractions (Retraction Watch's website 2022), in much the same way they may use software to help detect plagiarism (Cosentino and Veríssimo 2016).Publishers could support the maintenance and oversight of a comprehensive database of retractions.
Fourthly, as we have shown, automated alerts from citation databases or publishers to authors are unlikely to lead to changes alerting readers to retractions, except in a minority of cases, and then not promptly.Although they are not the entire solution, automated alerts should become part of a system-wide approach to mitigate the impact of retractions, because even if only a minority of authors act on these alerts, that will still provide readers with an assurance that the paper remains reliable after the retracted citations are removed.
Finally, and most important, we would argue that readers must be alerted at least to the presence of retracted citations in the evidence syntheses of systematic reviews and clinical guidelines as soon as possible.Until authors publish a note, attesting to the influence or lack of influence of retracted citations, citing publications should be visibly marked on the front page.These "reliability flags" (Fu and Schneider 2020) ought to be the responsibility of publishers, who should alert citing authors to their presence.Such flags would alert readers before authors provide a notice of impact, and in cases where citing authors do not feel it is their responsibility to address these issues.The Cochrane Library has adopted similar processes: flagging retracted studies in Cochrane reviews with an editorial note until the review is updated, whether or not the systematic review's findings are changed (Cochrane Editorial and Publishing Policy Resource 2022).Given the importance of rapidly updating readers, one Cochrane Group asks authors to turn round revised reviews within 6 weeks (Alfirevic et al. 2021).
In the current analyses, the majority of the retracted trials in the citing publications were assessed as low or moderate risk of bias, or moderate to high quality.In only one instance did an author's e-mail reply indicate that there had been some concerns about the retracted trial reports at the time of undertaking their systematic review.Existing quality assessment tools are not configured to identify concerns about publication integrity.Those are better identified using specific tools such as the REAPPRAISED checklist.This is best applied during evaluation of the individual trials at the time of their initial reporting or where there are concerns during meta-analysis (Grey et al. 2020).
Many of these issues and solutions to dealing with cited retractions were raised back in 2006 by Sox and Rennie (2006), but remarkably little progress is evident.With the advent of Retraction Watch's database and better information systems, it must be possible to provide solutions, if all sectors of academic publishing are prepared to work together.Until that time, patients are at risk from the unrecognized and/or uncorrected effects of retracted publications in influential clinical guidance documents.Unless correcting systems are put in place, policymakers who use affected clinical guidelines could unintentionally put patients at risk of harm.

88 systematic reviews and guidelines (with 10,917 citations themselves) citing 27 retracted RCT reports Contact author n=40 Up to 3 authors n=42 Editor n=44 No editor n=44 One publication aware before emails sent n=41 emails sent One study retracted before emails sent n=39 emails sent One study retracted before emails sent n=43 emails sent One publication aware before emails sent
conditions: the prevention or management of osteoporotic fractures, prevention of falls, preventing or managing stroke and transient ischemic attack, provision of nutrition support, managing limb muscle dysfunction in patients with chronic obstructive pulmonary disease, and preventing vitamin D deficiency.By August 2021, these 88 citing publications had themselves been cited 213 times by clinical guidelines or position statements.By August 2021, 72 citing publications listed by WoS themselves had 10,493 citations, and 10 unlisted by WoS had 424 citations on Scopus.Of these 82 citing publications, 67 (82%) had at least one citation in 2021.Six citing publications were not listed by WoS or Scopus, four of which were clinical guidelines.Two citing publications, clinical guidelines dating from 2008 and 2014, relating to stroke and transient ischemic attack, had more than 1500 citations each [Kernan et al. 2014; The European Stroke Organisation (ESO) Executive Committee and the ESO Writing Committee 2008], with numerous citations in 2021.
n=43 emails sent Figure 1.Flow chart for the results of the search.medical

Table 1 .
Assessment of impact on citing publications.
Six publications had only one author and were not, therefore, eligible to be randomized to copying in other authors. *