Several organisations have been working to improve the way practice guidelines are developed through defining standards, establishing common methods of rating the quality of the evidence and the strength of recommendations, and defining which criteria should be used in the assessment of guideline quality.

In their document issued in 1990, the Institute of Medicine (IOM) defined practice guidelines as «systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances» (Institute of Medicine, 1990, p. 38). In 2011, the IOM published a revised definition stating that «Clinical practice guidelines are statements that include recommendations intended to optimize patient care, informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options» (Institute of Medicine, 2011, p. 4). This publication was followed by eight standards for trustworthy practice guidelines, including: transparency of the process and funding, management of conflicts of interests, multidisciplinary composition of the guidelines panel with experts and other stakeholders, using systematic reviews of existing evidence, establishing evidence foundations and rating the strength of recommendations, clear articulation of recommendations, external review of the guidelines, and updating.

A set of standards similar to the one developed by Institute of Medicine was published by the Guidelines International Network (GIN) a year later. It highlighted the importance of guideline development processes that are both rigorous and feasible even for modestly funded groups to implement, and initiated an effort to generate a consensus regarding minimum standards for high-quality guidelines. The GIN proposed a set of key components for guideline development which address panel composition, the decision-making process, conflicts of interests, guideline objectives, development methods, evidence reviews, bases for recommendations, ratings of that evidence and those recommendations, guideline reviews, updating processes, and funding (Qaseem et al., 2012).

The GRADE (Grading of Recommendations, Assessment, Development and Evaluation) Working Group, which was established in 2000, developed consensus – on the systematic and transparent approach – regarding the rating of the quality of evidence and the strength of recommendations (Guyatt et al., 2008), which has become a standard in the development of the guidelines with over 100 organisations following those methods. In the GRADE approach, the evidence is rated using explicit criteria (domains) such as risk of bias, imprecision, inconsistency, indirectness, publication bias, magnitude of effect, dose-response, and effect of plausible residual confounding.

The starting point in this approach is the overall methodology of the studies, i.e., for randomized trials, the default estimate of the effect is high confidence, while for observational studies, it is low. Finally, evidence quality is classified in one of four categories: high, moderate, low, or very low (Guyatt et al., 2008). In the GRADE approach, recommendations can be strong or weak depending on the confidence regarding balance between desirable and undesirable effects of an intervention. The criteria which are taken into account were summarised in the Evidence to Decision (EtD) framework and include evidence on benefits, harms and burdens, and its certainty, values and preferences, costs, equity, acceptability, and feasibility (Alonso-Coello et al., 2016a; 2016b).

The work on the instruments to assess the quality of practice guidelines closely followed the works on setting standards for their development. In 2003, an international group of developers and researchers – AGREE Collaboration – published the first version of a tool to assess guideline quality (AGREE, 2003), which was replaced by the updated version in 2010 (Brouwers et al., 2010). It comprises 23 items in six quality domains, such as scope and purpose (specifying objectives, questions, and population covered); stakeholder involvement (multidisciplinary panel, views and preferences of the target population); rigour of development (methods used for gathering and synthesizing the evidence for guideline development, formulation of the recommendations, and the process for updating the guideline); clarity of presentation (specific, unambiguous and easily identifiable recommendations); applicability (focus on the likely barriers to and facilitators of implementation, strategies to improve uptake, and resource implications of applying the guideline); and editorial independence (unbiased formulation of recommendations and competing interests).

Recently, a new tool – AGREE-REX – has been developed. It is a complement to the AGREE II tool (an updated version of AGREE) that addresses three factors that must be considered to ensure high quality guideline recommendations. Those factors are three domains: clinical applicability, values and preferences (those of target users and guideline developers), and implementability in local contexts (Brouwers et al., 2020). Both tools can be used to support the reliable and valid evaluation of guidelines and their development and reporting.

The AGREE and AGREE II instruments were used to assess the quality of clinical practice guidelines. Thus, in an overview of reviews which evaluated guidelines published between 1980 and 2007, Alonso-Coello et al. (2010) showed acceptable quality (≥60 %) in the scope and purpose and clarity of presentation domains, moderate for rigour of development, and low for all other analysed domains. A total of 62 % of those documents were recommended with or without provisions. The authors also observed improvements over time in the quality of guidelines across domains, except in the editorial independence domain. In another overview of reviews evaluating guidelines published seven years later and covering the guidelines issued between 1990 and 2014 (Armstrong et al., 2017) the authors observed improvements in the quality scores across all domains and 82 % of the documents were recommended for use with or without modifications (Figure 1).

Figure 1
Mean or median scores in each of the AGREE domains and percentage of clinical guidelines recommended in the overall assessment.
Based on: Alonso-Coello et al. (2010), Armstrong et al. (2017) and Rabassa et al. (2018).

NUTRITIONAL GUIDELINES – PROBLEMS RAISED

Nutritional guidelines have multiple purposes, from promoting health and wellbeing to protecting a sustainable food system. As highlighted by Bero, Norris, and Lawrence (2019), the current approach in developing nutritional guidelines has been adapted from methods to establish clinical practice guidelines. Of course, these methods are useful and applicable to some types of nutritional guidelines – e.g., setting dietary or nutrient reference intake values – but not necessarily for studying dietary patterns or nutrient interactions when a mainly observational design is used. Rigorous guidelines are usually based on systematic reviews and their development is based and optimized for randomized trials with single component interventions. Dietary guidelines should regard a broader context than clinical recommendations. This leads to consider qualitative evidence, studies with observational design, and the need to incorporate complexity into systematic reviews, i.e., a broader context than clinical recommendations (like emphasising equity, human rights, or sociocultural acceptability).

Thus, nutrition polices that deal with dietary risk factors roughly fall into two broad categories: nutrition-specific (i.e., focusing on the immediate causes of malnutrition: e.g., fortification), and nutrition-sensitive (i.e., focusing on the causes of malnutrition: e.g., agricultural supporting programs). A predominance of nutrition-specific data over nutrition-sensitive data may affect nutrition policies, while both are important, as they provide data that complement each other.

Up to 2015, reviews about nutrition constituted about 8 % of all evaluations published in the Cochrane Database¹. The profile of those reviews is dominated by nutrition-specific interventions and half of them are reviews about nutrient supplementation alone. A similar predominance was reported in the profiles of nutrition policies and guidelines found in the WHO e-Library of Evidence for Nutrition Actions (eLENA) and with the implementation of nutrition actions listed in the WHO Global database on the Implementation of Nutrition Action (GINA) (Lawrence et al., 2016).

Another difficulty is related to the funding of nutrition studies, often connected with industry stakeholders, which may affect topics of produced evidence, i.e., prioritizing products that can be commercialized and marketed (Fabbri et al., 2018). In 2017 the National Academy of Sciences published two comprehensive reports that described concerning deficiencies in the process of dietary guideline development in the USA. The problems were associated with opacity in the formation of guideline committees, methodology of data analysis, and lack of transparency of the overall process of guideline development and update (National Academies of Sciences & Medicine, 2017a, 2017b).

Guidelines’ quality in the nutritional field has also been recently addressed in the literature. The systematic review done by Erickson, Sadeghirad, Lytvyn, Slavin, and Johnston (2017) analysed the scientific basis of guideline recommendations on sugar intake published between 1995 and 2016. The authors assessed the quality of all included guidelines (.=9) using the AGREE II tool. Moreover, the authors used the GRADE tool to rate the quality of the evidence that underpinned each recommendation. Among all the domains in the AGREE II tool, the lowest scores for most of the guidelines were assigned to the following domains: rigour of development, applicability, and editorial independence, which was similar to the assessment published for guidelines in general. The latter was the weakest point in all the guidelines. The quality of the evidence supporting the included recommendations, assessed with the GRADE tool, was low to very low.

Also, Rabassa et al. (2018) performed an overview of reviews evaluating nutritional guidelines using the AGREE Instrument. Most of the 67 included guidelines had been published between 2008 and 2012 by European or North American organisations, mostly public institution or scientific societies. The included nutritional guidelines covered various nutrition topics, such as nutrition and disease management, allergy, malnutrition, nutrition, health, and wellness. The quality of the nutritional guidelines was acceptable (≥60 %) in two domains: scope and purpose and clarity and presentation, while it was low in all other domains (Figure 1). Only 43 % of the guideline documents received a score of at least 60 % in at least three domains (including rigour of development). Finally, they did not observe significant changes in quality over time.

Another study by Blake, Durao, Naude, and Bero (2018) examined the methods that were used to synthesize evidence and grade recommendations among FAO’s food-based dietary guidelines (FBDGs) from 2010 to 2015. Most of the 32 included FBDGs (eleven were from countries in Latin America and the Caribbean, ten from Europe, seven from Asia and the Pacific, two from Africa, and two from North America), and 72 % were updates of previously published guidelines (Figure 2).

Figure 2
Type of evidence used to underpin recommendations in 32 food-based dietaries guidelines.
Source: Blake et al. (2018)

The authors examined in detail the methods used to conduct the evidence review process for the guidelines. Only 9 % of included guidelines defined the question set, while about 13 % reported identifying and searching for evidence: only two listed databases that were searched, and just one reported a search for unpublished data. Reporting of methods used to extract data was described in the case of two documents: Nutrition Evidence Library (NEL) methodology, and dual coding (visual and verbal) and World Cancer Research Fund (WCRF) methodology.

Only three out of 32 guidelines (9 %) reported methods used to assess the risk of bias for included sources (individual studies or reviews). The methods mentioned were: the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) tool, used for systematic reviews; the NEL Bias Assessment Tool, used for individual studies; and WHO levels of evidence and Australian National Health and Medical Research Council levels of evidence hierarchy (NHMRC). About 16 % of included guidelines reported on methods used to rate the overall quality of evidence and only two guidelines described methods used to synthesize the data. There was only one document that described all steps of guidelines preparation (defining the research question, identifying and searching for evidence, data extraction, evaluating the quality of the gathered evidence, and synthesis of the evidence), while 84 % of the documents did not describe any of those steps.

The main method for formulating recommendations was consensus (87 % of guidelines). Moreover, 22 % of those guides involved experts or health professionals, while 12 % of the documents provided no information about it. None of the guidelines reported grading the strength of recommendations through consensus or a structured GRADE process. Some guidelines reported grading the recommendations with WHO levels of evidence for study design – which categorize the evidence as convincing, probable, or possible – or methods used by the World Cancer Research Fund – which categorize the evidence as convincing, probable, unlikely, or limited to determine causality.

In terms of conflict of interests management reported in the guidelines, the documents included: a policy for dealing with them (9 %), reporting of funding sources for guidelines (31 %), reporting conflicts of interests of the members of the working group (12 %), either in the guideline document or in supporting documents (Blake et al., 2018). Overall the study concludes that despite a progress in evidence-based methods used for the creation of FBDGs, there are variations and differences in the tools that are used to approach evidence and its quality, the methods for grading recommendations, the trust in results obtained from observational studies that may be prone to higher risk of bias, or conflict of interest management.

Comparing the results of this study to standards published by IoM in 2011, several deficiencies were identified: in many guidelines, an opacity could be noted in the whole process and there was insufficient information on funding, together with insufficient transparency in the management of conflicts of interests. In the same way, in many cases guidelines did not use systematic reviews of existing evidence as the basis of their recommendations and did not specify the process for rating the strength of their recommendations.

NutriRECS initiative

The deficiencies described above can be overcome by establishing a systematic and transparent process. Such processes have been proposed by the NutriRECS (Nutritional Recommendations and accessible Evidence summaries Composed of Systematic reviews) project (Johnston et al., 2018), which follows internationally accepted methodological standards including the GRADE approach for developing trustworthy nutritional guidelines.

The panel in the NutriRECS guidelines includes key stakeholders, e.g., methodologists, patients, and members of the general community, who will ensure that chosen outcomes are important. It also uses a strict policy of conflicts of interests and their management for panel members and is not limited to financial ones, but also includes intellectual and other conflicts of interest, e.g., conflicts related to committed dietary behaviour that may have impact on interpretation of the results. To ensure a high quality for the systematic reviews and meta-analyses, the Cochrane Handbook guidance is used, including study protocols with pre-specified methods registered in PROSPERO (Zeraatkar et al., 2017). The values and preferences of the target population are also incorporated by using the results of systematic reviews on values and preferences related to specific foods or dietary patterns and including members of the general community in the guidelines panel. The certainty of the evidence for each outcome is rated using GRADE methods.

Subsequently, the guideline panel uses structured and transparent GRADE Evidence to Decision frameworks (Alonso-Coello, 2016a; 2016b) to record and report the judgments during the formulation of recommendations. This approach includes considering the following criteria: evidence on benefits, harms and burdens, and their certainty, together with values and preferences, costs, equity, acceptability, and feasibility (Johnston et al., 2018).

This approach was used to develop the first set of recommendations on the consumption of red and processed meat (Johnston et al., 2019). The recommendations are underpinned by five systematic reviews that covered health outcomes and values and preferences outcome. The work focused on the health impact of a reduction to three servings of red and processed meat per week and the GRADE certainty of evidence from all reviews was considered low to very low, with small or very small effect sizes.

The set of presented articles that are summarized and concluded in the form of recommendations can serve as an example of how the NutriRECS approach works in practice. As the authors of the recommendation on consumption of red and processed meat have highlighted (Johnston et al., 2019), the work focused on exclusively on health issues, considering that issues related to animal welfare and a potential environmental impact to be outside the scope of their recommendations, which constitutes a limitation of their study.

The publication of these guidelines was associated with an extensive coverage, comments, and critique in public media and scientific literature (Qian, 2020; Rubin, 2020), because the conclusions challenged an accepted statement about red and processed meat and its apparent causal relation with a range of critical health outcomes like cancer, heart disease, or type 2 diabetes. Criticism also discussed the fact that the leading author did not disclose potential conflicts of interest regarding funding as part of recruitment to the Department of Nutrition at Texas A&M University, although this was after the guideline recommendations were completed, and for a 2016 paper funded by the International Life Sciences Institute (ILSI) on the quality and scientific basis of the guidelines addressing sugar intake. However, the ILSI funding ended before the three-year International Committee of Medical Journal Editors reporting period. This funding information was subsequently added as a correction to the red and processed meat guidelines article. The authors also replied to the critique in the pages of Annals of Internal Medicine (Johnston et al., 2020; Zeraatkar et al., 2020).

This and other issues show the growing importance of continuing to acquire reliable means to agree on criteria and to base nutritional recommendations on the highest-quality evidence, as well as to recognise the limitations of low certainty evidence. The different tools described in this article constitute the basis to evaluate and improve the quality of nutritional guidelines.

The GRADE Working Group, which was established in 2000, developed a consensus regarding the rating of the quality of evidence and the strength of recommendations, which has become a standard in the development of guidelines with over a 100 organisations following those methods.

In 2003, an international group of developers and researchers – the AGREE Collaboration – published the first version of a tool to assess guideline quality. Recently, a new tool, AGREE-REX, has been developed. It is a complement that addresses three factors that must be considered to ensure high quality guideline recommendations: clinical applicability, values and preferences of the target users and guideline developers, and implementability in local contexts.

References

AGREE. (2003). Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: The AGREE project. BMJ Quality & Safety in Health Care, 12(1), 18–23. https://doi.org/10.1136/qhc.12.1.18

Alonso-Coello, P., Irfan, A., Sola, I., Gich, I., Delgado-Noguera, M., Rigau, D., Tort, S., Bonfill, X., Burgers, J., & Schunemann, H. (2010). The quality of clinical practice guidelines over the last two decades: A systematic review of guideline appraisal studies. BMJ Quality Safety in Health Care, 19(6), e58. http://doi.org/10.1136/qshc.2010.042077

Alonso-Coello, P., Oxman, A. D., Moberg, J., Brignardello-Petersen, R., Akl, E. A., Davoli, M., Treweek, S., Mustafa, R. A., Vandvik, P. O., Meerpohl, J., Guyatt, G. H., & Schünemann, H. J. (2016a). GRADE Evidence to Decision (EtD) frameworks: A systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. BMJ, 353, i2089. https://doi.org/10.1136/bmj.i2089

Alonso-Coello, P., Schünemann, H. J., Moberg, J., Brignardello-Petersen, R., Akl, E. A., Davoli, M., Treweek, S., Mustafa, R. A., Rada, G., Rosenbaum, S., Morelli, A., Guyatt, G. H., & Oxman, A. D. (2016b). GRADE Evidence to Decision (EtD) frameworks: A systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ, 353, i2016. https://doi.org/10.1136/bmj.i2016

Armstrong, J. J., Goldfarb, A. M., Instrum, R. S., & MacDermid, J. C. (2017). Improvement evident but still necessary in clinical practice guideline quality: A systematic review. Journal of Clinical Epidemiology, 81, 13–21. https://doi.org/10.1016/j.jclinepi.2016.08.005

Bero, L. A., Norris, S. L., & Lawrence, M. A. (2019). Making nutrition guidelines fit for purpose. BMJ, 365, l1579. https://doi.org/10.1136/bmj.l1579

Blake, P., Durao, S., Naude, C. E., & Bero, L. (2018). An analysis of methods used to synthesize evidence and grade recommendations in food-based dietary guidelines. Nutrition Reviews, 76(4), 290–300. https://doi.org/10.1093/nutrit/nux074

Brouwers, M. C., Spithoff, K., Kerkvliet, K., Alonso-Coello, P., Burgers, J., Cluzeau, F., Férvers, B., Graham, I., Grimshaw, J., Hanna, S., Kastner, M., Kho, M., Qaseem, A., Straus, S., & Florez, I. D. (2020). Development and validation of a tool to assess the quality of clinical practice guideline recommendations. JAMA Network Open, 3(5), e205535. https://doi.org/10.1001/jamanetworkopen.2020.5535

Brouwers, M. C., Kho, M. E., Browman, G. P., Burgers, J. S., Cluzeau, F., Feder, G., Fervers, B., Graham, I. D., Grimshaw, J., Hanna, S. E., Littlejohns, P., Makarski, J., & Zitzelsberger, L. (2010). AGREE II: Advancing guideline development, reporting and evaluation in health care. CMAJ, 182(18), E839–842. https://doi.org/10.1503/cmaj.090449

Erickson, J., Sadeghirad, B., Lytvyn, L., Slavin, J., & Johnston, B. C. (2017). The scientific basis of guideline recommendations on sugar intake: A systematic review. Annals of Internal Medicine, 166(4), 257–267. https://doi.org/10.7326/M16-2020

Fabbri, A., Lai, A., Grundy, Q., & Bero, L. A. (2018). The influence of industry sponsorship on the research agenda: A scoping review. American Journal of Public Health, 108(11), e9-e16. https://doi.org/10.2105/AJPH.2018.304677

Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008). GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ, 336(7650), 924–926. https://doi.org/10.1136/bmj.39489.470347.AD

Institute of Medicine. (1990). Clinical practice guidelines: Directions for a new program. National Academies Press. https://doi.org/10.17226/1626

Institute of Medicine. (2011). Clinical practice guidelines we can trust. National Academies Press. https://doi.org/10.17226/13058

Johnston, B. C., Alonso-Coello, P., Bala, M. M., Zeraatkar, D., Rabassa, M., Valli, C., . . . Vandvik, P. O. (2018). Methods for trustworthy nutritional recommendations NutriRECS (Nutritional Recommendations and accessible Evidence summaries Composed of Systematic reviews): A protocol. BMC Medical Research Methodology, 18(1), 162. https://doi.org/10.1186/s12874-018-0621-8

Johnston, B. C., Zeraatkar, D., Han, M. A., Vernooij, R. W. M., Valli, C., El Dib, R., Marshall, C., Stover, P. J., Fairweather-Taitt, S., Wójcik, G., Bhatia, F., de Souza, R., Brotons, C., Meerpohl, J. J., Patel, C. J., Djulbegovic, B., Alonso-Coello, P., Bala, M. M., & Guyatt, G. H. (2019). Unprocessed red meat and processed meat consumption: Dietary guideline recommendations from the nutritional recommendations (NutriRECS) consortium. Annals of Internal Medicine, 171(10), 756–764. https://doi.org/10.7326/M19-1621

Lawrence, M., Naude, C., Armstrong, R., Bero, L., Covic, N., Durao, S., Ghersi, D., Macdonald, G., MacLehose, H., Margetts, B., Tovey, D., Volmink, J., & Young, T. (2016). A call to action to reshape evidence synthesis and use for nutrition policy. Cochrane Database of Systematic Reviews, 11, ED000118. https://doi.org/10.1002/14651858.ED000118

National Academies of Sciences, Engineering, & Medicine. (2017a). Optimizing the process for establishing the dietary guidelines for Americans. The National Academies Press.

National Academies of Sciences, Engineering, & Medicine. (2017b). Redesigning the process for establishing the dietary guidelines for Americans. The National Academies Press.

Qaseem, A., Forland, F., Macbeth, F., Ollenschlager, G., Phillips, S., & van der Wees, P. (2012). Guidelines international network: Toward international standards for clinical practice guidelines. Annals of Internal Medicine, 156(7), 525–531. https://doi.org/10.7326/0003-4819-156-7-201204030-00009

Qian, F., Riddle, M. C., Wylie-Rosett, J., & Hu, F. B. (2020). Red and processed meats and health risks: How strong is the evidence? Diabetes Care, 43(2), 265–271. https://doi.org/10.2337/dci19-0063

Rabassa, M., Garcia-Ribera Ruiz, S., Sola, I., Pardo-Hernandez, H., Alonso-Coello, P., & Martinez Garcia, L. (2018). Nutrition guidelines vary widely in methodological quality: An overview of reviews. Journal of Clinical Epidemiology, 104, 62–72. https://doi.org/10.1016/j.jclinepi.2018.08.018

Rubin, R. (2020). Backlash over meat dietary recommendations raises questions at corporate ties to nutrition scientists. JAMA, 323(5), 401–404. https://doi.org/10.1001/jama.2019.21441

Zajac, J., Agarwal, A., Barberato, S., Bartoszko, J., Hitchcock, C., Leenus, A., Lo, C., Cruz Lopes, L., Bergamaschi, C. C., Zea, A. M., Alonso-Coello, P., Guyatt, G., & Johnston, B. (2017). Red meat and health outcomes: A systematic review. PROSPERO 2017 CRD42017074074. https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=74074

Zeraatkar, D., Guyatt, G. H., Alonso-Coello, P., Bala, M. M., Rabassa, M., Han, M. A., Vernooij, R. W. M., Valli, C., El Dib, R., & Johnston, B. C. (2020). Red and processed meat consumption and risk for all-cause mortality and cardiometabolic outcomes. Annals of Internal Medicine, 172(7), 511–512. https://doi.org/10.7326/L20-0070

Notes

Notas de autor