Hydroxychloroquine use during the first COVID-19 wave: a case study highlighting the urgent need to enhance research practices within the publication ecosystem

Abstract

The rapid dissemination of scientific findings through media and social networks can profoundly impact public health policies and behaviors. However, the reliability of such data is crucial, as evidenced by significant cases like the retracted study on hydroxychloroquine (HCQ) during the COVID-19 pandemic. This paper examines the retraction of a widely publicized study by Pradelle et al., which concluded that HCQ was associated with an excess of 16,990 deaths during the pandemic’s first wave. This finding was heavily influenced by a meta-analysis that did not robustly support its conclusions, particularly regarding the dose-response relationship of HCQ. Our analysis identified significant methodological flaws, including the misapplication of effect sizes and a lack of sensitivity analyses, rendering the study results unreliable. The retraction process, however, lacked transparency, failing to adequately describe in details the reasons for the study flaws to the public. This case underscores the broader challenges in scientific publishing, including the robustness of the peer-review process, the rise of fraudulent practices, and the erosion of trust in scientific institutions. We advocate for reforms to enhance transparency, improve data verification, and incentivize thorough peer review to maintain public trust and ensure the accuracy of scientific literature.

Text box 1. Contributions to the literature
• Highlighting methodological weaknesses: This paper uncovers significant methodological flaws in the retracted study by Pradelle et al., particularly the use of effect sizes without dose-subgroup analyses and the absence of robust sensitivity checks, which undermine the reliability of their conclusions about hydroxychloroquine and mortality during COVID-19
• Emphasis on transparency and integrity: The paper stresses the critical need for transparency in the retraction process. It critiques the lack of public disclosure about the specific reasons for retraction, suggesting that this opacity impedes comprehensive understanding and public trust in scientific research
• Call for publication ecosystem reforms: It advocates for systemic changes to enhance peer review processes, promote data verification, and combat issues like predatory publishing practices, thereby reinforcing the importance of reliability and reproducibility in scientific studies

Introduction

The widespread availability of scientific findings through various media and social networks means that any data of public health interest can rapidly and significantly influence not only policy decisions but also public opinion and individual behaviors. In light of the current rapid dissemination of any kind of information, it is essential that scientific studies adhere to rigorous methodological standards, given the potential clinical, behavioral, and societal impacts of their findings [1]. To illustrate this point, it is notable that a considerable number of studies are retracted each year due to issues related to the accuracy and reliability of their data, with over 10,000 retractions reported in 2023 [2]. This represents a significant waste of resources, and once erroneous information is disseminated, the consequences for the public can be severe and difficult to reverse. Ensuring the accuracy and reliability of the scientific literature is therefore crucial, as evidenced by notable cases of COVID-19-retracted studies that have had profound societal impacts [3].

A striking example of the consequences of inadequate scientific reporting is the “Lancet Gate” which concerns the retraction of a paper published in The Lancet during the COVID-19 pandemic [4]. This study aimed to assess the safety and efficacy of hydroxychloroquine (HCQ) with or without a macrolide as a treatment for COVID-19 using a multinational registry. Initially, the paper received widespread media coverage and influenced health policies globally. Governments and health organizations either recommended or halted the use of HCQ based on these findings. However, subsequent scrutiny revealed significant flaws in the data and methodology, leading to the paper’s retraction [4]. In an open letter addressed to the authors and the editors, 180 scientists raised several critics, outlining significant concerns about the study’s statistical analysis and data integrity. This example of retraction further emphasize the importance of data verification and the ethical responsibility of researchers to ensure the accuracy of their findings. Indeed, the influence of this publication extended beyond academia, having affected clinical practices and policy decisions during a critical period of the pandemic.

The societal consequences of inadequate scientific reporting are profound. Misinformation can lead to public health risks, as seen in the HCQ case, where individuals may have been exposed to unnecessary risks or deprived of potentially effective treatments. Furthermore, the erosion of public trust in scientific institutions can have long-term effects, making it more challenging to convey critical information during future health crises or to adhere to prevention policies such as vaccination. The dissemination of flawed data can also lead to financial implications, as governments and organizations may allocate resources based on incorrect information.

The analysis of HCQ as a treatment for COVID-19 has subsequently been subject to extensive scrutiny and heated debate, with conflicting results emerging from various studies with defenders of the pro- and the cons-. This debate has spread well beyond the scientific sphere and has even taken a political turn in many countries.

In January 2024, a manuscript published by Pradelle et al. in Biomedicine & Pharmacotherapy provided an estimate of the deaths induced by compassionate use of HCQ during the first COVID-19 wave, also led to a huge worldwide media coverage [5]. The authors reported that HCQ might have been associated with an excess of 16,990 deaths during the first wave of the COVID-19 pandemic in countries for which data were available [5]. The outcome of the study was of major health public impact, especially in the concerned countries, i.e. Belgium, France, Italy, Spain, Turkey, and the USA. However, the article contained multiple methodological issues, which were rapidly highlighted by numerous Letters to the Editor and Correspondence from readers [6]. End of January 2024, in light of the extent and depth of the criticism about this article, the Editor-in-Chief of Biomedicine & Pharmacotherapy made the decision to handle this matter through the Committee on Publication Ethics (COPE) process for corrections to the scientific record. This finally led to the retraction of this article in August 2024. According to the Editor-in-Chief, the retraction was due to two major issues (i) the reliability of the data and choice of the data, in particular, the Belgian dataset was found to be unreliable based on estimates, and (ii) the assumption that all patients that entered the clinic were being treated the same pharmacologically was incorrect [5]. We submitted a Letter to the Editor which contained additional analysis and we highlighted important weaknesses of the original paper which were supposedly taken into consideration for the decision of retraction. Although this article has been retracted due to critical errors in data reliability and assumptions, it remains publicly accessible without any accompanying publication of Letters to the Editor or Correspondences that specifically highlight these flaws. This lack of transparency hinders a comprehensive understanding of the study shortcomings. The aim of this article is to address the significant concerns surrounding the transparency and integrity of scientific publishing, particularly in the context of the retracted article by Pradelle et al. and the connected papers and to point out weaknesses of the current publication ecosystem to prevent misinformation and maintain public trust in scientific institutions.

The concern

Several trials of different designs were conducted to investigate the efficacy and safety of HCQ for the prevention and/or the treatment of COVID-19 patients [7]. As the pandemic unfolded, conflicting findings emerged from clinical trials and observational studies. Some early studies indicated potential benefits, while others raised issues about the drug’s safety and efficacy [8]. Indeed, concerns about potential adverse drug reactions, including heart rhythm abnormalities and increased mortality [9, 10], had prompted regulatory agencies to caution against its use outside of controlled settings [10]. The Wolrd Health Organization (WHO) and other health authorities reversed their recommendations of the use of HCQ in COVID-19 patients based on accumulating evidence [11,12,13].

Pradelle et al. estimated the in-hospital mortality attributable to HCQ during the first wave of COVID-19 by combining the mortality rate, HCQ exposure, number of hospitalized patients, and the increased relative risk of death with HCQ [5]. The main finding of their study was that HCQ might have been associated with an excess of 16,990 deaths during the first wave of the COVID-19 pandemic in the 6 countries for which data were available. Such attributable risk analysis is associated with many limitations, some of which being identified by the authors [5]. In our letter, we pointed out that that their study did not adequately address dose-subgroup and sensitivity analyses which precludes any overall firm conclusions on in-hospital mortality attributable to HCQ.

To allow a better understanding of these limitations, it is important to remind that Pradelle et al. utilized an odds ratio (OR) reported in another study, a meta-analysis published by Axfors et al., which encompasses 14 published and 15 unpublished trials, as the estimator for HCQ-related mortality [10]. This meta-analysis reported an OR of 1.11 (95% confidence interval (CI) 1.02; 1.20, see below for a discussion between confidence and compatibility interval) and was based on 4,316 patients treated with HCQ and 5,696 controls. The outcomes reported by Pradelle et al. were entirely influenced and extrapolated from this effect size [5] but the significance of this effect size of 1.11 needed to be interpreted with caution. Indeed, two studies, namely WHO SOLIDARITY and RECOVERY, contributed to 88.9% of the weights in their overall model [10]. In essence, the pooled OR obtained from the meta-analysis was heavily influenced by these two specific trials. As highlighted by the authors themselves, RECOVERY and WHO SOLIDARITY employed HCQ in comparatively higher doses than all other trials, which may explain the increased OR observed while including them in the model.

To provide a nuanced analysis of the impact of this aspect on the overall results, we reiterated the meta-analysis of Axfors et al. by using the same effect sizes as the one reported by the authors for each individual study and we conducted a dose-subgroup analysis by stratifying studies according to administered dose [10]. Since the publication of Axfors et al. [10], some included studies have been retracted [14] or corrected [15] but as the aim of our exercise was to point out the flaws of the previous investigations, using different datasets would have complicated the comparisons. In our subgroup analysis, we investigated whether using lower doses of HCQ (e.g., ≤ 2400 mg/5 days or ≤ 4800 mg/5 days) also significantly increased the risk of mortality across trials. The ‘low-dose’ HCQ regimen (2400 mg in total over 5 days) was recommended at least in Belgium and used as a reasonable regimen for hospitalized patients [16, 17]. Our analyses revealed that when pooling studies employing HCQ doses ≤ 2400 mg/5 days (i.e., k = 12, n patients treated with HCQ = 947, n controls = 745), an OR of 0.94 (95%CI 0.56; 1.59) was found (Fig. 1A), indicating no clear evidence of a mortality benefit or harm. The wide confidence interval indicates substantial uncertainty, with effects ranging from a 44% reduction to a 59% increase in mortality risk that reasonably agree with the observed data.‘. Clinically, our analysis does not support the use of HCQ doses ≤ 2400 mg/5 days for mortality reduction, though it does not strongly suggest harm eitherThe same observation held true when pooling studies employing HCQ doses ≤ 4800 mg/5 days (i.e., k = 25, n patients treated with HCQ = 1,672, n controls = 1,479) with an OR of 0.97 (95%CI 0.73; 1.29). Only high-dose regimens of HCQ were associated with a significant and potentially clinically relevant increase in mortality (Fig. 1A & Fig. 1B, OR of 1.12, 95%CI 1.01; 1.25). Unlike lower-dose regimens, where no clear effect was observed, this analysis suggests that HCQ at high doses is associated with a 12% increase in mortality, with most plausible effects ranging from a minimal increase (1%) to a more substantial rise (25%). Given that the entire 95% compatibility interval is above 1.0, these findings align with previous concerns regarding the safety profile of high-dose HCQ and support current recommendations against its use in COVID-19 treatment.

Besides this methodological concern of applying an effect size found exclusively for high-dose studies to all patients, regardless of the dose they might have received, this OR of 1.11 has not been demonstrated to be robust. Indeed, Axfors et al. [10] did not conduct a leave-one-out analysis despite this sensitivity analysis is considered as a crucial methodological step to assess the robustness of a model. Interestingly, upon excluding either the WHO SOLIDARITY or the RECOVERY study from the model in leave-one-out analysis, the significance of the results is annulled (omitting WHO SOLIDARITY: OR 1.08 (95%CI:0.99; 1.19), omitting RECOVERY: OR 1.11 (95%CI:0.95; 1.30), plots available in Open Science Framework https://osf.io/ewudy/). The robustness of a meta-analytic model should be ensured through sensitivity analyses, and the significance of an effect size should not be attributable to solely one single trial. Furthermore, Axfors et al. [10] ran additional sensitivity analyses to assess the robustness of their results across four different meta-analytic approaches (reported in their Appendix) [5]. From these results, it is noteworthy that only one of the meta-analytic approaches tested (i.e., the Hartung-Knapp-Sidik-Jonkman (HKSJ) model with the Paule-Mandel estimator for tau²) yielded to a statistically significant OR of 1.11, while the three other statistical approaches failed to demonstrate the statistical significance of the effect size [10].

When embarking on a study of such public health interest, Pradelle et al. [5] should have ensured that the main effect size on which they based their analysis [10], and which was consistently employed across their models to estimate the number of excess deaths, reflected the use of HCQ in the concerned countries, was robust and unbiased. Our reanalysis points out that this is not the case [5].

A important point for consideration is that even at low doses HCQ regimen, no reduction in mortality was observed suggesting that, when it comes to mortality as the outcome, HCQ did not show a benefit in hospitalized patients suffering from COVID-19. This mainly justifies the past and still up-to-date recommendations and guidelines to not use HCQ in this indication.

Fig. 1

Random effect meta-analysis of mortality for the treatment of COVID-19 trials stratified by dose. 1A Random effect meta-analysis of mortality for the treatment of COVID-19 trials stratified by dose (≤ 2400 mg/5 days vs. >2400 mg/5 days). 1B Random effect meta-analysis of mortality for the treatment of COVID-19 trials stratified by dose (≤ 4800 mg/5 days vs. >4800 mg/5 days). The dose of HCQ received during the first 5 days of hospitalization was calculated from Table 2 of Axfors et al. presenting the group-level characteristics of each included randomized controlled trial. We used the random effect model of the Hartung-Knapp-Sidik-Jonkman (HKSJ) approach with the Paule-Mandel (PM) estimator for tau², as it was the statistical method used by Axfors et al. All analyses were performed on R, version 2023.09.0 + 463, using the “meta” package. Scripts related to this analysis is freely available on Open Science Framework (https://osf.io/ewudy/)

The importance of statistical versus clinical significance

While statistical significance is often used as a decision-making criterion in scientific research, it is crucial to distinguish statistical results from clinical significance [18, 19]. An effect size of 1.11, even if statistically significant, as reported by Axfors et al. and reused by Pradelle et al., does not inherently imply clinical relevance, as its impact depends on contextual factors that were not generalizable. Furthermore, our analysis has highlighted multiple methodological flaws in the Pradelle et al.. study, including the misapplication of effect sizes, lack of subgroup analyses, and the absence of robust sensitivity checks. These deficiencies ultimately led to conclusions that do not align with real-world clinical outcomes. As seen in multiple countries during the COVID-19 pandemic, the use of HCQ was highly variable in terms of dosage, patient selection, and co-administration with other treatments, factors that were inadequately accounted for in the meta-analysis driving the conclusions of Pradelle et al.. Therefore, the reported statistical associations do not necessarily reflect the true benefit or harm of HCQ in clinical practice, further reinforcing the need for rigorous methodological standards and cautious interpretation of statistical findings in shaping public health policies.

In addition to the limitations in clinical interpretation, it is interesting to reconsider the way statistical intervals are reported. Traditional confidence intervals may be misinterpreted as defining strict boundaries of certainty, whereas they could be understood as compatibility intervals, i.e., representing a range of effect sizes that remain reasonably consistent with the observed data, given the assumptions of the statistical model. As emphasized in recent epidemiological and statistical literature, compatibility intervals provide a more accurate representation of the uncertainty surrounding an estimate, rather than implying a definitive measure of confidence in a specific value [18, 20, 21]. In the case of the Pradelle et al. study, the reported OR of 1.11 (95% confidence/compatibility interval: 1.02–1.20) was used as a central estimate to extrapolate the number of excess deaths attributed to HCQ. Given that the lower bound of this interval suggests a marginal increase in risk and the upper bound indicates only a modest effect, their assumptions should have been tempered accordingly. Relying on a point estimate without incorporating at least a 95% compatibility interval may have led to an overstatement of HCQ potential harm, reinforcing the need for a more cautious approach when translating statistical associations into real-world public health conclusions.

Furthermore, the interpretation of statistical findings should not only be based on p-values as binary thresholds of significance but rather as measures of compatibility between the observed data and the tested hypothesis. A p-value just below 0.05 does not necessarily confirm a meaningful effect, just as a p-value slightly above 0.05 does not necessarily imply the absence of an effect. Reframing p-values in terms of data compatibility rather than rigid statistical significance thresholds would have provided a more nuanced interpretation of the findings, preventing potential misrepresentations of real-world impact [19, 20]. Moreover, the conventional p < 0.05 threshold is arbitrary, and the choice of significance levels should be adapted based on the context and potential risks involved. For example, in high-stakes public health decisions, using a wider compatibility interval (e.g., 97%) could be more appropriate, as it accounts for a greater degree of uncertainty and reduce the risk of drawing premature conclusions from borderline statistical findings.

Also, one of the major concerns in contemporary scientific research is p-hacking, a practice where researchers selectively report analyses or manipulate statistical methods to achieve a p-value below 0.05, thus increasing the likelihood of false-positive findings. This can take various forms, including data dredging, post-hoc hypothesis testing, and selective reporting of statistically significant results, all of which contribute to inflated effect sizes and irreproducible findings [22, 23]. A notable example of potential statistical bias can be observed in the meta-analysis by Axfors et al. While they tested multiple statistical models, the only approach yielding a statistically significant result (OR = 1.11, 95% CI: 1.02–1.20, P < 0.05) was the Hartung-Knapp-Sidik-Jonkman (HKSJ) model with the Paule-Mandel estimator for tau², whereas all alternative meta-analytic methods failed to demonstrate statistical significance. This selective emphasis on a single model raises concerns about model dependency and the robustness of the reported effect size.

Furthermore, Pradelle et al. adopted this effect size without critically evaluating its methodological limitations, including the absence of leave-one-out sensitivity analyses, which would have demonstrated that excluding a single dominant study (e.g., WHO SOLIDARITY or RECOVERY) nullifies the statistical significance of the result. Such practices highlight the broader issue of overreliance on statistical thresholds without ensuring robustness and reproducibility, reinforcing the necessity for greater transparency in statistical methodology, model selection and evaluation of the compatibility between the observed data and the tested hypothesis.

Viewpoint and perspectives

The peer-review process, a cornerstone of scientific integrity, is currently facing significant challenges that threaten the quality of published research. The recent retraction of several high-profile papers, particularly those related to the use of HCQ in COVID-19, has highlighted systemic weaknesses in the publication ecosystem. These issues not only undermine trust in scientific literature but also raise concerns about the potential long-term impact on scientific advancement and public policy.

One of the main weaknesses in the current peer-review process is the issue of reproducibility. The anonymous and opaque nature of traditional peer-review has made it difficult to guarantee the reliability and reproducibility of published research. As noted in the editorial of Bommier in Ethics, Medicine and Public Health, the reluctance of major journals to confront scams such as data manipulation, p-hacking, and predatory publishing practices has significantly eroded trust in scientific publication [24]. The retraction of flawed studies during the COVID-19 pandemic illustrates the potential consequences of these systemic weaknesses. Another critical issue is the declining willingness of qualified reviewers to participate in the peer-review process. The increasing workload on academics, combined with the lack of tangible rewards for peer reviewers, has led to a situation where many reviewers are declining invitations to review manuscripts. A comprehensive survey of biomedical reviewers revealed that the primary reasons for declining reviews are conflicting workload demands and the tight deadlines imposed by journals [25]. The current peer-review system is also being compromised by the rise of fraudulent practices such as the use of “paper mills” defined by COPE as the process by which manufactured manuscripts are submitted to a journal for a fee on behalf of researchers with the purpose of providing an easy publication for them, or to offer authorship for sale [26]. These fraudulent activities contribute to a sharp increase in the number of retracted papers, with misconduct-related retractions quadrupling in the past two decades [2, 27]. This increase in retractions is not merely a reflection of improved detection methods but also indicates a growing prevalence of unethical behavior in scientific publishing. The situation is further exacerbated by the exploitation of special issues and guest editor roles in journals, which have been used to facilitate the rapid publication of low-quality or sham papers [2].

In addition to these challenges, the very nature of reviewing process is under threat. The traditional model of peer review is rooted in the spirit of cooperation and mutual support within the academic community. However, the rise of a more competitive, neoliberal academic environment is eroding these values, leading to a decline in collegial cooperation and a greater emphasis on individual success over collective responsibility [28]. This shift not only threatens the integrity of the peer-review process but also undermines the broader goals of scientific collaboration and knowledge advancement.

To address these challenges, there have been calls for significant reforms to the peer-review process. One proposed solution is to incentivize peer review by offering rewards such as Continuing Medical Education (CME) credits, public acknowledgment, offering of free open-access to publish their works and opportunities for professional advancement through editorial board positions [25]. These incentives could help to restore enthusiasm for a quality peer review process.

Furthermore, there is a growing recognition of the need for greater transparency and accountability in peer review. This could involve requiring journals to adopt open peer review models, where reviews and reviewer identities are made public, or implementing stricter guidelines for the verification of data and the methodologies used in submitted manuscripts [25]. Such measures would help to increase trust in the peer-review process and ensure that published research meets the highest standards of scientific rigor. On the contrary, it may lead to a shortage of qualified reviewers willing to undertake this task.

Finally, the adoption of open science practices could provide effective solutions to some of these challenges by promoting transparency and accountability. One of the key pillars of open science is data sharing, which involves, for example, making research data, statistical scripts, or analysis codes openly available to the scientific communauty. Recognizing the importance of transparency which specifies whether and how datasets or other supporting materials can be accessed by reviewers and readers.

To facilitate open data practices, several scientific repositories and platforms are existing, allowing researchers to deposit scientific related materials in a structured and publicly accessible manner. Platforms such as Open Science Framework (OSF), Zenodo, Dryad, and Figshare are examples of robust infrastructures ensuring that scientific material remain available for scrutiny, reanalysis, and further research.

Open peer review models, where reviewer reports and identities are disclosed, could also improve the quality of evaluations and develop a more constructive and accountable review process. Furthermore, post-publication peer review platforms offer a mechanism for continuous scrutiny of published studies, ensuring that errors can be corrected more efficiently than through traditional editorial channels [29].

In conclusion, the recent high-profile retractions, particularly related to COVID-19 studies, underscore systemic weaknesses within the publication ecosystem, including issues of reproducibility, declining reviewer participation, open access to datasets, scripts, and methodological details and the rise of fraudulent practices. These challenges not only erode trust in scientific literature but also threaten the long-term progress of scientific research and its influence on public policy. It is crucial to implement meaningful reforms that incentivize reviewer participation, enhance transparency, and enforce stricter standards for data verification and methodology.