Carátula del artículo
Risk of bias in dentistry-related randomized controlled trials in spanish language journals
Conchita Martin
Universidad Complutense de Madrid, Spain
Isabel Delgado
Universidad Complutense de Madrid, Spain
Rosa Rojo
Universidad Rey Juan Carlos, Spain
Rafael Souper
Universidad de Chile, Chile
Universidad de Chile, Chile
Carola Veliz
Universidad de Chile, Chile
Universidad de Chile, Chile
Josefina Salazar
Hospital Clínico San Borja-Arriarán, Chile
Marlene Diaz
Iberoamerican Cochrane Center, Spain
Javier Cuellar
Universidad de Chile, Chile
Matías Dallaserra
Universidad de Chile, Chile
Universidad de Chile, Chile
Xavier Bofill
Hospital Clínico San Borja-Arriarán, Chile
Josefina Bendersky
Universidad de Chile, Chile
Hospital Clínico San Borja-Arriarán, Chile
Sebastian Zapata
Universidad de Chile, Chile
Universidad de Chile, Chile
Julio Villanueva javm@uchile.cl
Universidad de Chile, Chile
Universidad de Chile, Chile
Iberoamerican Cochrane Center, Spain
International journal of interdisciplinary dentistry, vol. 16, no. 2, pp. 137-141, 2023
Sociedad de Periodoncia de Chile Implantología Rehabilitación Odontopediatria Ortodoncia
Received: 17 July 2022
Revised document received: 26 October 2022
Accepted: 19 November 2022
INTRODUCTION
Randomized controlled trials (RCTs) are considered the best approach to assess the effects, benefits, and harms of therapeutic intervention, drug, device, or technique in human beings1-3. RCTs are also the foundation of systematic reviews and other evidence synthesis documents4,5. Evidence-based hierarchies place RCTs just below systematic reviews as the highest form of evidence6,7 that could be achieved from an in vivo/clinical trial. RCTs are widely accepted as the ‘‘gold standard’’ for obtaining unbiased estimates of treatment effects3. However, the reliability of individual test conclusions depends largely on internal validity, based on the quality of the research methodology and execution 6. Therefore, high-quality reporting of the details of such research is essential1,8,9.
RCTs can have weaknesses in the design or analyses that could lead to underestimating or overestimating the intervention; this is known as bias and can affect in any direction10. It is usually impossible to know the extent to which biases have affected a particular study’s results. Consequently, to draw reliable conclusions, reviewers should consider the possible limitations of the studies, which relate to the extent to which their design, conduct, analysis, and presentation were appropriate to answer your research question7,11. Therefore, assessing the validity of included studies is a fundamental component of any systematic review and should influence the review’s analysis, interpretation, and conclusions.
Many tools have been proposed to assess RCTs’ quality in the context of systematic reviews or other evidence syntheses. Most of them are scales in which several quality components are scored and combined to obtain a summary score; or checklists, in which specific questions are asked11-13. The use of these scales is explicitly discouraged. As well as suffering from the generic problems of scales, they have a strong emphasis on reporting rather than conducting and do not cover one of the most important potential biases in randomized trials, namely allocation concealment. The Cochrane Collaboration published and adopted a new method in February 2008, “the risk of bias tool” (RoB), a domain-based evaluation to rate the validity of the included studies5,14, which is also used in many non-Cochrane reviews. This RoB tool is based on the following principles: It does not use quality scales or scores. It focuses on internal validity, assessing the risk of bias in the RCTs findings, not on the report’s quality. The risk of bias assessment requires a judgment based on the trial method to know if this method gives rise to bias and chooses domains based on theoretical and empirical considerations. It focuses on the risk of bias in the data represented in the trial and not as originally reported. Finally, reports specific assessments of risk of bias results15.
The assessment of RCTs’ methodology and its effects on the results has been carried out in medicine and dentistry during the last few years1,6,9,16,17. However, despite the relevance of the elements mentioned and the important number of articles published on the subject, the evaluation of risk of bias in randomized clinical trials published in the Spanish language in the field of dentistry is largely unknown. Knowing in detail the quality of these publications will help editors, evaluators of research funds and clinical investigator communities in the region decide the planning, execution, and publication of RCTs. Therefore, this study aimed to evaluate RCTs’ risk of bias in dental journals published in Spain and Latin America in the Spanish language.
METHODS
Study Design
We conducted a systematic retrospective survey of all randomized clinical trials published from 1980 to 2019 in dentistry-related journals published in the Spanish language in Latin America and Spain. The data collection method for this sample was published previously18. It contained Controlled Clinical trials, but we chose only the RCTs.
Data Source
We used the RCTs identified through the manual search of Spanish and Latin American dental journals up to 201418, entered in BADERI, Database of Iberoamerican Clinical Trials and Journals, by its initials in Spanish19. This search was updated twice, in 201820 and 2019. The manual search methods are published in several previous articles21-24.
Eligibility criteria
We included only randomized clinical trials specific in dentistry with a recoverable full text published between 1980 and 2019. We excluded quasi-randomized clinical trials or nonrandomized clinical trials. We also excluded articles that report conference proceedings, pilots, or feasibility studies, that conduct secondary analysis on RCTs, or are translations.
Data extraction
For each of the journals, a form was filled out detailing information on the name of the journal, ISSN, the total number of articles, as well as the number of possible clinical trials detected, all classified according to the year and month of publication, thus making it possible to know at all times the issues and volumes that had been reviewed and those that were still pending. To be eligible, a journal had to be published periodically and the original research had to be published, regardless of whether it was active at the time of the search.
Three pairs of investigators extracted the data (MD- MD, RS-CT- ID-CM). The principal investigators (CM-JV) were responsible for resolving discrepancies during the extraction and analysis process. Each pair of reviewers extracted the data for each article independently and in parallel.
Outcomes
The first risk of bias tool developed by Cochrane Collaboration was used to assess bias in the selected RCTs, since at the start of this study, the “RoB 2” tool was not available14. Our outcomes were the 7 domains considered in the risk of bias (RoB) analysis described in the Systematic Reviews’ Handbook 14. These domains refer to 6 types of biases: (1) Selection Bias: The generation of random sequences refers to the systematic differences between the base characteristics of the groups being compared 10. (2) Performance Bias: Blinding of participants and staff may reduce the risk that knowledge of the intervention received, rather than the intervention itself, will affect outcomes or measurements of outcomes. (3) Detection bias: blinding of outcome evaluation. This refers to the systematic differences between the groups in the way the results are determined. Blinding (or masking) of outcome evaluators may reduce the risk that knowledge of the intervention received, rather than the intervention itself, will affect the measurement of outcomes. (4) Attrition bias: incomplete outcome data refers to systematic differences between groups in terms of dropout from a study. (5) Reporting bias: Selective information bias refers to systematic differences between reported and unreported data. (6) Other biases: only relevant in certain settings. These are mainly related to particular trial designs (e.g., split-mouth, cross-traffic, and recruitment bias in cluster-randomized trials)14,25. These domains are listed in Table 1. For each domain, one of the three assessments was given (low, uncertain or high risk of bias). At the end of bias assessment, any disagreement was resolved by open discussion or by involving the fourth reviewer (ID, MD, JS, MD) with two senior reviewers (CM, JV). Authors of the included trials were not contacted to get additional information prior to giving risk of bias assignments. The overall risk of bias was ascertained for each selected trial using the 7 domains. Responses were entered into RevMan (Review Manager (RevMan) [Computer Program]. Version 5.4. The Cochrane Collaboration, 2020., n.d.) to elaborate the risk of bias graphs and their subsequent analysis.
Table 1
The seven domains of Cochrane Collaboration’s tool (Higgins JPT, Altman DG, Sterne JAC (editors), n.d.) for assessing risk of bias.
Statistical analysis
Only a descriptive analysis with summary statistics was performed, given that the entire population of RCTs was included; therefore, statistical inference techniques were not performed. We provide the risk of bias graphs performed with the RevMan program.
We used the Review Manager (RevMan) [Computer program]. Version 5.4. The Cochrane Collaboration, 2020 for graphics and analysis.
RESULTS
Identification of RCTs
The search for RCTs was carried out in all Spanish-speaking countries (Spain and Latin America), and clinical trials (CT) were sought in the journals of Argentina, Bolivia, Chile, Colombia, Costa Rica, Cuba, Dominican Republic, Ecuador, El Salvador, Guatemala, Honduras, Mexico, Nicaragua, Panama, Paraguay, Peru, Spain, Uruguay and Venezuela. In most of these countries it was not possible to locate any clinical trials. RCTs were identified in the following countries: Argentina, Chile, Colombia, Spain, Mexico, Peru, Uruguay and Venezuela. Table 2 shows the number of journals identified by country and the number of CCTs and RCTs found. The distribution of clinical trials is published in a previous article 18.
Table 2
Details of the number of journals identified by country, number of articles reviewed, and number of CCTs and RCTs found
We reviewed a total of 25,810 articles in 114 journals. We excluded 25.423 because they corresponded to case reports, narrative reviews, in vitro studies, animal studies or observational studies. Only 387 (1,5%) of the published articles were potentially eligible for inclusion. After the full-text review, 292 (1,13%) RCTs were included for analysis, published in 51 of the reviewed journals, whereas 95 (0,4%) Controlled Clinical Trials (CCTs) were excluded. Figure 1 presents the flow diagram for the article selection process and details of the number of articles reviewed and excluded. Of the included RCTs, 152 were published in Spain (52%), 63 in Chile (22%), 25 in Colombia (11,9%), 22 in Mexico (8%), 12 in Peru (4%), 8 in Argentina (3%), 8 in Venezuela (3%), and 2 in Uruguay (1%). Spain and Chile were the countries with the highest number of clinical trials found Chile had the highest number of RCTs per number of articles reviewed (2.09%), followed by Spain (1,39%).
Most of the journals identified and reviewed are not indexed in any of the main databases and do not have an impact factor. In fact, only one journal from Spain is indexed in PubMed and Journal Citation Report.
Figure 1
Flow diagram for the article selection process
Risk of Bias (RoB) analysis
Figure 2 presents the distribution of RoB assessment by domain across studies. Among the domains that were most evaluated as unclear or high risk of bias are those assessing selection bias: sequence generation with 164 (56%) and 18 (6.1%) studies, respectively, and allocation concealment with 213 (72.7%) and 6 (2%) studies, respectively. Furthermore, the blinding of outcome assessment was evaluated in 138 (47.1%) studies as unclear and in 57 (19.5%) studies as high risk of bias.
Figure 2
Risk of bias graph: review authors’ judgments about each risk of bias item presented as percentages across all included studies.
For the domain of blinding of participants and personnel, 118 (40.3%) studies were assessed as low risk of bias, 110 (37.5%) as unclear, and 64 (21.8%) as high risk.
The best evaluated domains were incomplete outcome data, selective reporting, and other biases, with 223 (76.1%), 263 (89.8%), and 183 (62.5%) studies evaluated with low risk of bias, respectively.
RoB assessment over time
Figure 3 shows the number of RCTs found per year in the reviewed journals. When assessing changes in risk of bias evaluations over the years, the rates of low risk of bias across domains show an increasing trend (Figure 4A). Unclear (figure 4B) and high (Figure 4C) risk of bias assessment rates show a decreasing trend. There is a marked decrease in the high risk of bias rates in the blinding domains, but in turn a slight increase in the unclear risk of bias rates for these domains. (Figure 4).
Figure 3
RCTs published per
Figure 4a
Proportions of each RoB assessment across domains over the years Figure 4A: Low risk of bias domain rates
Figure 4b
Proportions of each RoB assessment across domains over the years. Figure 4B: Unclear risk of bias domain rates
Figure 4c
Proportions of each RoB assessment across domains over the years. Figure 4C: High risk of bias domain rates
DISCUSSION
In this study, 292 RCTs published in 51 dental journals of Spanish language were identified. Surprisingly, less than half (42,1%) of the dental journals that publish articles in Spanish language issue RCTs. This aspect might be explained by the low number of Spanish speaking journals indexed in main databases, discouraging authors of these countries from publishing their RCT in these sources, making them resort to foreign English language journals with a higher impact factor. To achieve an improvement, Spanish-language journals must have editorial committees that are better prepared to achieve a better level of peer review that will lead them to be included in indexed databases. In this sense, researchers and reviewers should be trained and professionalized. Obtaining these points will lead to studies such as RCTs being published in Spanish-language journals, thus bringing the evidence closer to a public that reads scientific literature and is limited by the English language. In this way, the importance of RCTs in clinical decision making will be known and, therefore, have a direct impact on the oral health of the population.
A risk of bias evaluation was performed for all the included RCTs using the RoB Tool developed by the Cochrane Collaboration14. As previously mentioned, the items with lower risk of bias were Incomplete Outcome Data, Selective Reporting, and Other Biases. Conversely, the ones most weakly evaluated were related to blinding, either for the participants and personnel or the evaluators, in which 21,8% and 19,5% of the studies reported high risk of bias, respectively. This data reveals higher difficulties in the design and elaboration phase of the study than in the posterior reporting stage. One possible explanation for this phenomenon is the implementation of the CONSORT statement, improving and facilitating the quality of reporting, as stated in the Turner’s systematic review7,26.
It is important to state that to achieve the maximum quality of RCTs, it is essential to improve the items regarding allocation concealment and blinding, as evidence has proved that they are directly related to the treatment effect10,27. However, our findings show an increase in the rates of low risk of bias evaluations in all domains, as well as a decreasing proportion of studies judged as having high risk of bias over the years (Figure 4). This tendency in the low RoB evaluations shows a clear breakpoint around the year of 2010, increasing significantly. This phenomenon could also be explained by the implementation of the new version of the CONSORT guideline of that year, leading to the production of studies of higher quality26.
Despite the low proportion of RCTs published in Spanish language journals, the number is increasing over the years. The findings of this study show a low steady publication rate until 1998, heterogeneously increasing until a peak incrementation is produced in 2008, surrounding the mean of 25 studies published per year (Figure 3).
These findings are consistent with previous research regarding the RoB of RCTs published in the Cochrane Databases of Systematic Reviews28.
Our study has certain limitations. Firsthand, our search was developed only in dentistry related journals. Therefore, studies on this discipline published in journals from other specialties could have been left out. Moreover, we only reviewed journals from Latin America and Spain that publish in Spanish language, leaving aside articles written by authors residing in these countries of possible higher quality, published in English or in foreign journals. Finally, the assessment of the RoB was performed from the publication alone, without considering protocols, web materials or other useful data for a more complete evaluation and analysis.
As for the main strengths in our study, we highlight the exhaustive comprehensive hand search in journals published in Spanish language in Latin America and Spain. Furthermore, the data extraction and RoB assessments were performed in duplicate and independently by two authors. To our knowledge, this is the first published study that evaluates the RoB of RCTs in Spanish speaking dental journals.
CONCLUSIONS
A low percentage of Spanish-language dental journals issue RCTs. Our assessment of these RCTs’ RoB suggests that there are higher difficulties in the design and conduction phase of the study than in the posterior reporting stage.
However, our findings suggest an increase in the proportion of the low risk of bias assessments across domains over the years. We encourage researchers and editors to improve the quality of the design, conduction, and reporting of RCTs, to reduce potential biases and their impact on the certainty of the findings. A low percentage of dental journals in Spanish publish RCTs. Our evaluation of the RB of these RCTs suggests that there are greater difficulties in the study design and conduct phase than in the subsequent reporting phase.
However, our findings suggest an increase in the proportion of low risk of bias evaluations in all domains over the years. We encourage investigators and publishers to improve the quality of RCT design, conduct, and reporting to reduce potential biases and their impact on the certainty of results.
References
Alamri HM, Alharbi F. Quality Assessment of randomized clinical trials reporting in endodontic journals: An observational study from 2012 to 2017. J Endod. 2018;44(8):1246-1250. doi:10.1016/j.joen.2018.05.011
Bonfill X, Ballesteros M, Gich I, Serrano MA, García López F, Urrútia G. Description of the protocols for randomized controlled trials on cancer drugs conducted in Spain (1999-2003). Stemmer SM, ed. PLoS ONE. 2013;8(11):e79684. doi:10.1371/journal.pone.0079684
Hulley SB, ed. Designing Clinical Research. 4th ed. Wolters Kluwer/Lippincott Williams & Wilkins; 2013.
Dickersin K, Scherer R, Lefebvre C. Systematic reviews: identifying relevant studies for systematic reviews. BMJ. 1994;309(6964):1286-91. doi:10.1136/bmj.309.6964.1286
Fleming PS, Buckley N, Seehra J, Polychronopoulou A, Pandis N. Reporting quality of abstracts of randomized controlled trials published in leading orthodontic journals from 2006 to 2011. Am J Orthod Dentofacial Orthop. 2012;142(4):451-458. doi:10.1016/j.ajodo.2012.05.013
Turner L, Shamseer L, Altman DG, et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Methodology Review Group, ed. Cochrane Database of Systematic Reviews. Published online November 14, 2012. doi:10.1002/14651858.MR000030.pub2
Cairo F, Sanz I, Matesanz P, Nieri M, Pagliaro U. Quality of reporting of randomized clinical trials in implant dentistry. A systematic review on critical aspects in design, outcome assessment and clinical relevance. J Clin Periodontol. 2012;39:81-107. doi:10.1111/j.1600-051X.2011.01839.x
Elangovan S, Prakasam S, Gajendrareddy P, Allareddy V. A risk of bias assessment of randomized controlled trials (RCTs) on periodontal regeneration published in 2013. J Evid Based Dent Pract. 2016;16(1):30-40. doi:10.1016/j.jebdp.2015.03.016
Wood L, Egger M, Gluud LL, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ . 2008;336(7644):601-605. doi:10.1136/bmj.39465.451748.AD
Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists. Control Clin Trials. 1995;16(1):62-73. doi:10.1016/0197-2456(94)00031-W
Crowe M, Sheppard L. A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed. J Clin Epidemiol. 2011;64(1):79-89. doi:10.1016/j.jclinepi.2010.02.008
Juni P. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ . 2001;323(7303):42-46. doi:10.1136/bmj.323.7303.42
Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, In: Higgins JPT, (editors). Assessing risk of bias in a randomized trial. In: Cochrane Handbook for Systematic Reviews of Interventions Version 6.1. https://www.training.cochrane.org/handbook
Higgins JPT, Green S, Cochrane Collaboration, eds. Cochrane Handbook for Systematic Reviews of Interventions . Wiley-Blackwell; 2008.
Al-Namankany AA, Ashley P, Moles DR, Parekh S. Assessment of the quality of reporting of randomized clinical trials in paediatric dentistry journals. Int J Paediatr Dent. 2009;19(5):318-324. doi:10.1111/j.1365-263X.2009.00974.x
Oomens MAEM, Heymans MW, Forouzanfar T. Risk of bias in research in oral and maxillofacial surgery. Br J Oral Maxillofac Surg. 2013;51(8):913-919. doi:10.1016/j.bjoms.2013.05.004
Villanueva J, Delgado I, Saldarriaga JR, et al. Identification and description of controlled clinical trials in Spanish language dental journals. Health Info Libr J. 2018;35(3):192-201. doi:10.1111/hir.12214
Pardo-Hernandez H, Urrútia G, Barajas-Nava LA, et al. BADERI: an online database to coordinate handsearching activities of controlled clinical trials for their potential inclusion in systematic reviews. Trials. 2017;18(1):273. doi:10.1186/s13063-017-2023-3
Bachelet VC, Carrasco VA, Bravo-Córdova F, et al. Quality of reporting for randomised clinical trials published in Latin American and Spanish journals: A protocol for a systematic survey of three clinical specialities. BMJ Open. 2020;10(6):e036148. doi:10.1136/bmjopen-2019-036148
Bonfill X, Osorio D, Posso M, et al. Identification of biomedical journals in Spain and Latin America. Health Info Libr J . 2015;32(4):276-286. doi:10.1111/hir.12110
Gutarra-Vilchez RB, Pardo-Hernandez H, Arévalo-Rodríguez I, Buitrago D, Bonfill X. Identification and description of controlled clinical trials published in Spanish Gynaecology and Obstetrics journals and risk of bias assessment of trials on assisted reproductive techniques. Eur J Obstet Gynecol Reprod Biol. 2016;203:5-11. doi:10.1016/j.ejogrb.2016.04.039
Loezar C, Madrid E, Jahr C, et al. Identification and description of controlled clinical trials published in Spanish Ophthalmology Journals. Ophthalmic Epidemiol. 2018;25(5-6):436-442. doi:10.1080/09286586.2018.1503688
Turrillas M, Sitjà-Rabert M, Pardo H, et al. Identification and description of controlled clinical trials published in Physiotherapy journals in Spain. J Eval Clin Pract. 2017;23(1):29-36. doi:10.1111/jep.12441
Chan AW, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ . 2005;330(7494):753. doi:10.1136/bmj.38356.424606.8F
Schulz KF, Altman DG, Moher D, for the CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ . 2010;340(mar23 1):c332-c332. doi:10.1136/bmj.c332
Pildal J, Chan AW, Hróbjartsson A, Forfang E, Altman DG, Gøtzsche PC. Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study. BMJ . 2005;330(7499):1049. doi:10.1136/bmj.38414.422650.8F
Reveiz L, Chapman E, Asial S, Munoz S, Bonfill X, Alonso-Coello P. Risk of bias of randomized trials over time. J Clin Epidemiol . 2015;68(9):1036-1045. doi:10.1016/j.jclinepi.2014.06.001
Notes
Notes
FUNDING There was no funding available for this study.
ETHICS APPROVAL No ethical approval was required for this study.
Conflict of interest declaration
CONFLICTS OF INTREREST The authors have no conflicts of interest to declare
Author notes
* Corresponding author: Julio Vallanueva | Email: javm@uchile.cl
