Short Communication

Is CONSORT a Risk of Bias Tool for Experimental Studies: A Big Misunderstanding

Introduction

Risk of bias (RoB) assessment supports establishing transparency of evidence synthesis and promoting the dissemination of robust evidence through systematic reviews (SRs).¹ Some of the RoB tools are general tools and some are developed for assessing RoB in specific study designs i.e. case-control studies.^2-4

Regarding RoB, two other terms namely quality of conduct (QoC), and quality of reporting (QoR), are also used for appraising studies. Here we want to discuss a misunderstandings or lack of unanimous understanding about these three terms.⁴ Although these terms are highly associated, they are not necessarily the same. A study may have a high QoC, but low QoR and vice versa. There may also be situations in which a study with low RoB may suffer from poor QoR. Basically, QoC is closer to RoB than QoR, but even these two are not necessarily the same thing. Although experts often recommend a more robust approach to the assessment of study methods that focuses on the RoB rather than on quality per se,⁵ the most important issue would be to consider a suitable tool for assessing the issue of interest either quality or RoB.

Based on our experience it seems there is not a unanimous understanding of the three terms discussed above among some review researchers. QoR is commonly assessed by CONSORT tool.⁶ Although, assessing QoR with CONSORT alongside the use of RoB tools supports a more comprehensive appraisal process,^7,8 sometimes misunderstandings or at least impreciseness may occur. One such issue is to consider CONSORT as a Rob tool.

CONSORT improves the transparency of trial results. Misreporting avoidable issues has been demonstrated to impact the findings of SRs and meta-analyses, leading to research inefficiency.⁹ Thus this study has aimed to investigate how ‘CONSORT statement’, as a reporting quality of randomized controlled trials, has been misused in recent SRs as a ROB tool.

Methods: A mini-review was performed in international databases including PubMed, Google Scholar and hand-searches for published and unpublished literature from 2000 to 2021 and written in English. The following keywords: risk of bias, “Consolidated Standards of Reporting Trials”, CONSORT, “systematic review” were used. Citations were screened and those meeting our inclusion criteria were retained.

Results

Here we present some examples of such a misunderstanding in literature where some authors have inappropriately used CONSORT as a RoB tool (Table 1).

Other than these overt examples of the issue, there are also examples of ambiguity in RoB assessment methodology or tools.^20-28 Moreover, some studies have stated in their Abstract that RoB had been assessed using CONSORT. However, the information provided in the full text of the article was different from what stated in Abstract.^29,30

Discussion: Although the review audience can themselves judge how appropriate is it to use CONSOERT as a ROB tool, as some of these articles are published in good journals, it is quite essential to draw the attention of SR researchers, journal editors/reviewers as well as the reading audience to the fact that QoR and RoB are not the same and separate specific tools should be used for assessing each of them.

This mini-review tries to re-inform the scientific audience, that although the quality reporting of clinical trials is improved by the CONSORT tool, it may not be reasonable to use CONSORT as an alternative for available standard tools developed and validated for assessing RoB.

Conclusion

The CONSORT statement aims to improve clarity, consistency, and transparency in reporting randomized controlled trials. Researchers undertaking SRs should thoughtfully select tools aligned with their objectives to ensure reliability and foster trust in their published findings.

Competing Interests

The authors declare no conflict of interest.

Data Availability Statement

All data generated or analyzed for this study are included in this published article.

Ethical Approval

Not applicable.

Research Highlights

References

1. Higgins J, Chandler J, Cumpston M, Li T, Page MJ, Welch VJ, et al. Cochrane Handbook for Systematic Reviews of Interventions. Hoboken: Wiley; 2019. 10.1002/9781119536604
2. Farrah K, Young K, Tunis MC, Zhao L. Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols. Syst Rev 2019; 8(1):280. doi: 10.1186/s13643-019-1172-8 [Crossref] [ Google Scholar]
3. Jarde A, Losilla JM, Vives J. Methodological quality assessment tools of non-experimental studies: a systematic review. An Psicol 2012; 28(2):617-28. doi: 10.6018/analesps.28.2.148911 [Crossref] [ Google Scholar]
4. Ma LL, Wang YY, Yang ZH, Huang D, Weng H, Zeng XT. Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better? Mil Med Res 2020; 7(1): 7. what are they and which is better? Mil Med Res 2020; 7(1):what are they and which is better? Mil Med Res 2020; 7(1). doi: 10.1186/s40779-020-00238-8 [Crossref] [ Google Scholar]
5. Moher D, Jadad AR, Tugwell P. Moher D, Jadad AR, Tugwell PAssessing the quality of randomized controlled trialsCurrent issues and future directions. Int J Technol Assess Health Care 1996; 12(2):195-208. doi: 10.1017/s0266462300009570 [Crossref] [ Google Scholar]
6. Grant S, Mayo-Wilson E, Montgomery P, Macdonald G, Michie S, Hopewell S. CONSORT-SPI 2018 explanation and elaboration: guidance for reporting social and psychological intervention trials. Trials 2018; 19(1):406. doi: 10.1186/s13063-018-2735-z [Crossref] [ Google Scholar]
7. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med 2010; 152(11):726-32. doi: 10.7326/0003-4819-152-11-201006010-00232 [Crossref] [ Google Scholar]
8. Tikka C, Verbeek J, Ijaz S, Hoving JL, Boschman J, Hulshof C. Quality of reporting and risk of bias: a review of randomised trials in occupational health. Occup Environ Med 2021; 78(9):691-6. doi: 10.1136/oemed-2020-107038 [Crossref] [ Google Scholar]
9. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S. Reducing waste from incomplete or unusable reports of biomedical research. Lancet 2014; 383(9913):267-76. doi: 10.1016/s0140-6736(13)62228-x [Crossref] [ Google Scholar]
10. Lubans DR, Plotnikoff RC, Lubans NJ. Review: A systematic review of the impact of physical activity programmes on social and emotional well-being in at-risk youth. Child Adolesc Ment Health 2012; 17(1):2-13. doi: 10.1111/j.1475-3588.2011.00623.x [Crossref] [ Google Scholar]
11. Cattuzzo MT, Dos Santos Henrique R, Ré AH, de Oliveira IS, Melo BM, de Sousa Moura M. Motor competence and health related physical fitness in youth: a systematic review. J Sci Med Sport 2016; 19(2):123-9. doi: 10.1016/j.jsams.2014.12.004 [Crossref] [ Google Scholar]
12. Foguet-Boreu Q, Fernandez San Martin MI, Flores Mateo G, Zabaleta Del Olmo E, Ayerbe García-Morzon L, Perez-Piñar López M. Cardiovascular risk assessment in patients with a severe mental illness: a systematic review and meta-analysis. BMC Psychiatry 2016; 16:141. doi: 10.1186/s12888-016-0833-6 [Crossref] [ Google Scholar]
13. Müller AM, Alley S, Schoeppe S, Vandelanotte C. The effectiveness of e-& mHealth interventions to promote physical activity and healthy diets in developing countries: a systematic review. Int J Behav Nutr Phys Act 2016; 13(1):109. doi: 10.1186/s12966-016-0434-2 [Crossref] [ Google Scholar]
14. Leye Benoist F, Niang SO, Faye B, Sarr M, Seck A, Ndiaye D. Treatment of dentin hypersensitivity: a systematic review of randomized clinical trials. J Dent Oral Care Med 2016; 2(2):204. doi: 10.15744/2454-3276.2.204 [Crossref] [ Google Scholar]
15. Fang T, Jozsa F, Al-Chalabi A. Nonmotor symptoms in amyotrophic lateral sclerosis: a systematic review. Int Rev Neurobiol 2017; 134:1409-41. doi: 10.1016/bs.irn.2017.04.009 [Crossref] [ Google Scholar]
16. Al-Sowygh ZH. Does surgical crown lengthening procedure produce stable clinical outcomes for restorative treatment? A meta-analysis. J Prosthodont 2019; 28(1):e103-9. doi: 10.1111/jopr.12909 [Crossref] [ Google Scholar]
17. Spitz A, Christovam IO, Marañón-Vásquez GA, Masterson DF, Adesse D, Maia LC. Global gene expression profile of periodontal ligament cells submitted to mechanical loading: a systematic review. Arch Oral Biol 2020; 118:104884. doi: 10.1016/j.archoralbio.2020.104884 [Crossref] [ Google Scholar]
18. Tan T, Rutges J, Marion T, Hunn M, Tee J. Cyanoacrylate dermal closure in spine surgery: systematic review and pooled analysis. Global Spine J 2020; 10(4):493-8. doi: 10.1177/2192568219861619 [Crossref] [ Google Scholar]
19. Pagaduan JC, Chen YS, Fell JW, Wu SS. Can Heart rate variability biofeedback improve athletic performance? A systematic review. J Hum Kinet 2020; 73:103-14. doi: 10.2478/hukin-2020-0004 [Crossref] [ Google Scholar]
20. Annis A, Freitag MB, Evans RR, Wiitala WL, Burns J, Raffa SD. Construction and use of body weight measures from administrative data in a large national health system: a systematic review. Obesity (Silver Spring) 2020; 28(7):1205-14. doi: 10.1002/oby.22790 [Crossref] [ Google Scholar]
21. Slotte S, Sääkslahti A, Kukkonen-Harjula K, Rintala P. Fundamental movement skills and weight status in children: a systematic review. Balt J Health Phys Act 2017; 9(2):115-27. doi: 10.29359/bjhpa.09.2.11 [Crossref] [ Google Scholar]
22. Kolahi J, Khazaei S. Assessment of blinding success among dental implant clinical trials: a systematic review. Dent Hypotheses 2015; 6(4):129-33. doi: 10.4103/2155-8213.170636 [Crossref] [ Google Scholar]
23. López-Medina MD, Linares-Abad M, López-Araque AB, López-Medina IM. López-Medina MD, Linares-Abad M, López-Araque AB, López-Medina IMDry care versus chlorhexidine cord care for prevention of omphalitisSystematic review with meta-analysis. Rev Lat Am Enfermagem 2019; 27(3):e3106. doi: 10.1590/1518-8345.2695.3106 [Crossref] [ Google Scholar]
24. Gupta AK, Daigle D. The use of low-level light therapy in the treatment of androgenetic alopecia and female pattern hair loss. J Dermatolog Treat 2014; 25(2):162-3. doi: 10.3109/09546634.2013.832134 [Crossref] [ Google Scholar]
25. Ebrahimi A, Tayebi N, Fatemeh A, Akbarzadeh M. Investigation of the role of herbal medicine, acupressure, and acupuncture in the menopausal symptoms: an evidence-based systematic review study. J Family Med Prim Care 2020; 9(6):2638-49. doi: 10.4103/jfmpc.jfmpc_1094_19 [Crossref] [ Google Scholar]
26. Rathod S, Wanikar I, Paralikar A. Soft and hard-tissue changes after the placement of smart blood derivative platelet-rich fibrin into intrabony defects: a systematic review and meta-analysis with at least 9-month follow-up. J Int Clin Dent Res Organ 2018; 10(2):49-58. doi: 10.4103/jicdro.jicdro_10_18 [Crossref] [ Google Scholar]
27. Zhai X, Wang Y, Mu Q, Chen X, Huang Q, Wang Q. Methodological reporting quality of randomized controlled trials in 3 leading diabetes journals from 2011 to 2013 following CONSORT statement: a system review. Medicine (Baltimore) 2015; 94(27):e1083. doi: 10.1097/md.0000000000001083 [Crossref] [ Google Scholar]
28. McFarland LV, Goh S. Preventing pediatric antibiotic-associated diarrhea and Clostridium difficile infections with probiotics: a meta-analysis. World J Metaanal 2013; 1(3):102-20. doi: 10.13105/wjma.v1.i3.102 [Crossref] [ Google Scholar]
29. Alkhawaldeh JM, Soh KL, Mukhtar FB, Ooi CP. Effectiveness of stress management interventional programme on occupational stress for nurses: a systematic review. J Nurs Manag 2020; 28(2):209-20. doi: 10.1111/jonm.12938 [Crossref] [ Google Scholar]
30. Tallarico M, Caneva M, Meloni SM, Xhanari E, Omori Y, Canullo L. Survival and success rates of different shoulder designs: a systematic review of the literature. Int J Dent 2018; 2018:6812875. doi: 10.1155/2018/6812875 [Crossref] [ Google Scholar]