Comparison of methodological quality between the 2007 and 2019 Canadian dietary guidelines

Abstract

Objective:

With significant shifts in the dietary recommendations between the 2007 and 2019 Canadian dietary guidelines, such as promoting plant-based food intake, reducing highly processed food intake and advocating the practice of food skills, we compared their differences in guideline development methods.

Design:

Two reviewers used twenty-five guided criteria to appraise the methods used to develop the most recent dietary guidelines against those outlined in the 2014 WHO Handbook for Guideline Development.

Setting:

Canada.

Participants:

2007 and 2019 dietary guidelines.

Results:

We found that the 2019 guidelines were more evidence-based and met 80 % (20/25) of the WHO criteria. For example, systematic reviews and health organisation authoritative reports, but not industry reports, constituted the evidence base for the dietary recommendations. However, recommendations on food sustainability and food skill practice were driven primarily by stakeholders’ interests. By contrast, less information was recorded about the process used to develop the 2007 guidelines, resulting in 24 % (6/25) consistency with the WHO standards.

Conclusions:

Our analysis suggests that a more transparent and evidence-based approach is used to develop the 2019 Canadian dietary guidelines and that method criteria should support further incorporation of nutrition priorities (food sustainability and food skills) in future dietary guideline development.

National food-based dietary guidelines are the foundation of food and nutrition policies. They guide the public in making food choices to maintain well-being and prevent diet-related non-communicable diseases. Dietary guidelines are updated periodically to reflect the current body of evidence as well as changes in priorities for public health nutrition⁽¹⁾. In 2019, Health Canada published its updated food-based dietary guidelines⁽²⁾. Compared with the 2007 version, the significant changes include increased plant-based food intake, limiting highly processed food and sugar-sweetened beverage intakes and encouraging food skill practice, such as home cooking and eating together. With these differences seen between two versions of the guidelines, we wanted to understand whether the methods and processes involved in developing the guidelines might also differ.

Earlier studies have suggested that the methods used to develop dietary guidelines vary across countries. For instance, a global review of national food-based dietary guidelines has reported inconsistencies across the guidelines in retrieving evidence, assessing the quality of evidence and synthesising the evidence when formulating the recommendations⁽³⁾. Additionally, many guidelines did not disclose conflicts of interest and funding sources⁽³⁾, raising a question of the food industry’s influence in dietary recommendations.

On a per-country basis, previous studies have compared the contents of dietary recommendations between versions of the dietary guidelines, such as in Canada (1992 v. 2007)⁽⁴⁾ and Brazil (2006 v. 2014)⁽⁵⁾. However, it remains unclear how methods used to update dietary guidelines evolve within a country. Therefore, the objective of this case study was to compare the differences in guideline development methods between the 2007 and 2019 Canadian dietary guidelines, aiming to inform potential areas for improvement in future dietary guideline development.

Methods

Document sources

All documentation for both of the Canadian guidelines is available online. This includes the dietary guidelines for 2007⁽⁶⁾ and the dietary guideline development history⁽⁷⁾, as well as the dietary guidelines for 2019⁽²⁾ and its accompanying evidence⁽⁸⁾, consultation^(9,10) and revision processes⁽¹¹⁾. We also referred to a published perspective by Bush et al.⁽⁴⁾ to extract relevant data for the 2007 guidelines. Additionally, we contacted Health Canada for assistance when there was uncertainty during the appraisal of the guideline methods.

Assessment of guideline methods

We adopted twenty-five criteria (Table 1) outlined in the 2014 WHO Handbook for Guideline Development⁽¹²⁾ to appraise the methods used to develop each version of the Canadian dietary guidelines. We identified relevant information and rated the methods used in each guideline against the description of each criterion as Yes, No or Unclear. If ‘Yes’ was rated, the verbatim text was extracted to substantiate the rating. ‘No’ was selected when there was a direct statement of a specific method that was not employed. In cases where no clear explanations were mentioned to substantiate the respective criterion, ‘Unclear’ was rated. We then calculated the proportion of those rated ‘Yes’ as being consistent with the WHO criteria for each guideline.

Table 1.

Appraisal of methodological rigour of the Canada dietary guidelines against the 2014 WHO Handbook for Guideline Development

Process and method domains	Process and method criteria	Description of each criterion	2007	2019
I. Guideline development group
Was each of the following accounted for when creating the guideline development group?	(1) Discipline representation	Information about the composition, discipline and relevant expertise of the guideline development group should be provided.	Yes	Yes
	(2) Diversity representation	Information about gender, diversity, across the life-course, subject to different gender norms and belonging to different income and education groups of the guideline development group.	Unclear	Yes
	(3) Stakeholder input	Stakeholders such as non-governmental organisations, advocacy groups, funders, target audiences and service-users may be invited to ensure transparency of the processes and facilitate implementation.	Yes	Yes
II. Conflicts of interest
Was each of the following steps taken regarding conflicts of interest?	(4) Conflicts of interest disclosure	Is there an explicit statement that all group members have declared whether they have any competing interests?	Unclear	Unclear
	(5) Conflicts of interest managed	Members’ declaration of interests must be reported to the steering group. Potential candidates for membership who have major conflicts of interest, be they financial or nonfinancial, cannot be appointed to the Guideline Development Group (GDG). Minor conflicts of interest can be managed at the individual level (e.g. by restricting participation in parts of the GDG meeting) or at the group level.	Unclear	Unclear
	(6) Disclosure of funders and the role of funders in the guideline development process and recommendations	Is there an explicit statement of the funder of the guideline and the role of funders in the final guideline recommendations?	Unclear	Yes
III. Systematic review methods
	(7) Formulation of key questions for the evidence review in PICO, PICOT or PEO format	Key questions are framed in a way that enables a systematic search of the literature and delineates inclusion and exclusion criteria for the body of evidence to formulate the research questions for the recommendations in such format.	Unclear	Yes
	(8) Choosing (finalising) priority outcomes for systematic review	List high-priority key questions and the outcomes to formulate recommendations.	Unclear	Yes
	(9) Systematic methods to search for evidence	Details of the strategy used to search for evidence should be provided, including search terms used, sources consulted and dates of the literature covered. Sources may include electronic databases, hand searching journals, reviewing conference proceedings and other guidelines.	Unclear	Yes
	(10) Evidence retrieval	Process of data from eligible studies are extracted, and search strategy and results should be carefully documented.	Unclear	Yes
	(11) Evidence quality assessment	Each study included in a systematic review should be assessed for risk of bias (e.g. use the Cochrane risk of bias tool, Quality assessment tolls project report, etc.)	Unclear	Unclear
	(12) Evidence synthesis	The findings of the systematic review may be synthesised in a narrative manner or quantitatively in a meta-analysis. The review should describe how data were handled and why a given approach to synthesis was taken for each outcome.	Unclear	Yes
IV. Transparency of evidence substantiation
If evidence is explicitly linked to the recommendation, what type of evidence is reported?	(13) Are recommendations explicitly linked to substantiating evidence?	An explicit link between the recommendations and the evidence on which they are based should be included in the guideline. The guideline user should be able to identify the components of the body of evidence relevant to each recommendation.	Yes	Yes
	(a) Primary research	Primary individual studies	No	No
	(b) Systematic reviews	Systematic reviews of clinical trials/observational studies	No	Yes
	(c) Summary of the evidence	Summary of evidence table	No	No
	(d) GRADE evidence profiles	GRADE summary of evidence table	No	No
	(e) Evidence to a decision table		No	No
	(f) Evidence to other documents		Yes	Yes
V. Recommendation development: Factors that determine the direction and strength of a recommendation
	(14) Was a consensus process clearly described?	A description of the methods used to formulate the recommendations and how final decisions were arrived at should be provided. For example, methods may include a voting system, informal consensus and formal consensus techniques. Areas of disagreement and methods of resolving them should be specified.	Yes	Yes
	(15) Was a method employed to determine the strength and/or certainty of the recommendation?	Is there a method provided to influence the direction and strength of a recommendation (e.g. use GRADE framework and others)	Unclear	Unclear
	(16) The priority of the problem: Is the problem a priority of the recommendation?	The problem’s priority is determined by its importance and frequency (i.e. the burden of disease, disease prevalence or baseline risk)—the greater the importance of the problem, the greater the likelihood of a strong recommendation.	Unclear	Yes
	(17) The quality of the evidence: What is the overall quality of the body of evidence?	Is there a method provided to grade the quality of the body of evidence to assess the strength of the recommendation (e.g. GRADE and others)	Unclear	Yes
	(18) The certainty of evidence (e.g. confidence in effect estimates)	The quality of the evidence – the degree of confidence in the estimates of effect. This is a key factor in determining the strength of a recommendation.	Unclear	Yes
	(19) Benefits and harms: How substantial are the desirable and/or undesirable anticipated effects/associations?	The balance between an intervention’s or exposure’s benefits and harms. Did the guideline development group consider the magnitude of the effects and the relative importance of the outcomes, including any disadvantages or inconveniences associated with the intervention?	Unclear	Yes
	(20) Balance: Does the balance between desirable and undesirable effects support the recommendation?	Does the balance between desirable and undesirable effects favour the intervention or the comparison?	Unclear	Unclear
	(21) Outcome importance: Is there important uncertainty about or variability in how much people value the main outcome?	Is there important uncertainty about or variability in how much people value the main outcomes, including adverse effects and burden of the test and downstream outcomes of clinical management that is guided by the test results?	Unclear	Yes
	(22) Equity: What would be the impact on health equity?	What would be the impact on health equity?	Yes	Yes
	(23) Acceptability: Is the option acceptable to key stakeholders?	A strategy to address concerns about acceptability during implementation will be included in the guideline with the recommendations. Acceptability is affected by several factors, such as who benefits from an intervention and who is harmed by it; who pays for it or saves money on account of it and when the benefits, harms and costs occur.	Unclear	Yes
	(24) Feasibility: Is the option feasible to implement?	Feasibility is influenced by the resources available, programmatic considerations, the existing and the necessary infrastructure and training, etc.	Yes	Yes
VI. Peer review process
	(25) Was the guideline/recommendation reviewed by an external review group?	Is there an explicit statement about the peer review of the final draft guideline? The external review group is composed of persons interested in the subject of the guideline as well as individuals who will be affected by the recommendations.	Unclear	Yes
Criteria met (%)			24	80

Extraction of recommendations

Additionally, we extracted the key recommendations to understand the methods linked with the respective version of the guidelines. Recommendations are not always clearly indicated. To support consistent extractions, we used the 2014 WHO Handbook⁽¹²⁾ to define recommendation, which ‘tells the intended end-user of the guideline what he or she can or should do in specific situations to achieve the best health outcomes possible, individually or collectively’. We also adopted a set of criteria⁽¹³⁾ to identify recommendations. These include ‘consistent semantic and formatting indicators’, ‘a summary section to facilitate identification of recommendations’, using ‘decidable and executable wording’, and ‘avoiding embedding recommendation text within long paragraphs’.

Data extraction

Two reviewers (Z.D. and C.M.K.) independently extracted the data and critically appraised the methods and processes used to develop the guidelines. Disagreements were resolved by consensus with L.B.

Results

Table 1 describes how methods were used to develop the 2007 and 2019 Canadian guidelines according to the WHO criteria. Data extraction in this regard is listed in online Supplemental Tables 1 and 2. Overall, the guideline development process appears to be more transparent in the 2019 guidelines than the 2007 guidelines. We found that the 2019 guidelines met 80 % (20/25) of the criteria, while the 2007 guidelines met 24 % (6/25). However, neither guidelines clearly described disclosure or management of conflicts of interest among the stakeholders involved in the guideline development.

A notable difference between the two guidelines is in the evidence used to substantiate the recommendations. The 2007 dietary recommendations only include a few authoritative reports, including the WHO/FAO joint report, US DRI and the 2005 US Dietary Guidelines Advisory Committee Report⁽⁴⁾. In contrast, the evidence underpinning the 2019 recommendations consists of systematic reviews and a summary of evidence from various health organisation authoritative reports. This evidence supports the recommendations on the influence of dietary patterns, saturated fats, processed meat, Na and ‘free sugars’ in overweight/obesity, metabolic risk and dental decay^(2,8). However, for both guidelines, it was unclear whether a ‘risk of bias’ assessment (‘11. Evidence quality assessment’ in Table 1) was applied to rate the individual studies.

Furthermore, the 2019 guidelines met most of the criteria in the domain of ‘Recommendations development.’ It administered two phases of online public consultation after the draft of the recommendations^(9–11) and explicitly stated that industry commissioned reports were excluded from the evidence base to substantiate the dietary recommendations⁽²⁾. Interestingly, the recommendation of food skills in the 2019 guidelines (home cooking and eating together) was formulated primarily based on ‘broad stakeholder interest’, as there was insufficient consistent evidence⁽²⁾.

Online Supplemental Table 3 compares the main dietary recommendations between the guidelines. Both guidelines target those 2 years and older and include a brief recommendation on physical activity, although the 2007 version also includes specific recommendations for a particular age group. Furthermore, the 2019 version has shifted towards increased plant-based food intakes, more specific recommendations on limiting highly processed food and sugar-sweetened beverage consumption (e.g. ‘Replace sugary drinks with water.’), and greater emphasis on food skills and the cultural aspect of food practice. Additionally, no recommendations are made for food serving sizes, dairy intake, acceptable macronutrient distribution range and vitamin D supplement intake for adults over 50 years in the 2019 guidelines.

Discussion

Both the 2007 and 2019 Canadian dietary guidelines have emphasised the importance of healthy food consumption to prevent non-communicable disease in the Canadian context. It is encouraging to see that the 2019 guideline recommendations are primarily based on systematic reviews and have met 80 % of the 2014 WHO criteria for public health guideline development. Additionally, the exclusion of industry reports as a source of evidence for the 2019 guidelines suggests a stricter approach to reduce conflicts of interest.

Several shifts in the methods used in developing the 2019 guidelines may explain the differences in the dietary recommendations. For example, the 2019 guidelines used systematic review methods to synthesise evidence and formulate the recommendations based on consistent findings, including those pertaining to dietary fibre, wholegrains, meat and meat alternatives in association with cardiometabolic risk⁽⁸⁾, as well as those pertaining to dietary unsaturated fats^(14,15), highly processed food intake and sugar-sweetened beverages^(16,17). By contrast, only three documents were cited in the 2007 guidelines to support the recommendations⁽⁴⁾. Second, we recognise that the body of evidence and nutrition priorities may shift during different time points. For example, dietary impact on the planet sustainability is a new addition to the 2019 guidelines, drawing on emerging evidence from dietary changes in green-house gas emissions and land and water use in recent years⁽¹⁸⁾. Also, even though the impact of improving food skills on human health was drawn on systematic reviews of observational studies⁽¹⁹⁾, they are ‘highly relevant’⁽²⁾ to public health nutrition, as stated in the 2019 guidelines.

However, the removal of the previous 2007 dairy food group in the new guidelines remains unclear, even though osteoporosis was one of the health outcomes included in the evidence review for the 2019 guidelines⁽⁸⁾. From the literature, we have seen consistent findings in the effect of dairy products on bone growth in children^(20,21) and on lowering the risk of osteoporosis and fractures in older adults⁽²²⁾. Future work is needed to understand the impact of the removal of dairy recommendations on bone health across different age groups.

From the methodological standpoint, we believe that it is appropriate to use the criteria outlined in the 2014 WHO Handbook to appraise both the 2007 and 2019 Canadian guidelines. As a newer guideline, we acknowledge that the 2019 guidelines describe the most recent methods used to develop the dietary guidelines, but think that it is important to determine whether the methods used to develop both guidelines have remained static or changed over the years before drawing conclusions about future improvements. On another note, guideline development standards, such as using systematic review methods and editorial independence, have been evolving since 2003⁽²³⁾. Because less information was recorded about the process used to develop the Canadian dietary guidelines before and during 2007⁽⁷⁾ than the 2019 guidelines, over three-quarters of the items were marked ‘Unclear’ for the criteria in the 2007 guidelines. This suggests that lower transparency of the guideline development process could be an issue for the 2007 guidelines. Consequently, we were unable to rule out whether these processes marked as ‘Unclear’ were a case of not being mentioned or of the methods not being implemented.

We note that various organisations, such as the FAO/WHO⁽²⁴⁾ and the European Food Safety Authority⁽²⁵⁾, have proposed methods to develop food-based dietary guidelines, using a scientific process to establish the diet–disease relationship and formulate the dietary recommendations. Because our study focused on evaluating the rigour of guideline development methods, we chose to use the 2014 WHO standards, an internationally recognised guideline development method for developing clinical and public health guidelines. Such standards cover compressively from the representation of guideline development group to peer review process, with emphasis on systematic review methods to search evidence (e.g. methods for assessing quality and analysis), transparency of evidence used (e.g. type of evidence and use of evidence tables), equity, feasibility and rating of the importance of outcomes⁽¹²⁾. These criteria ensure credible guideline recommendations that accurately reflect the balance of potential benefits and harms, and they are similar to those listed in The Appraisal of Guidelines for Research and Evaluation II related to guideline development methods, but not reporting⁽²⁶⁾.

However, methods in the 2014 WHO Handbook⁽¹²⁾ are not tailored specifically for formulating nutritional guidelines and recommendations. Therefore, using these standards to rate dietary guideline methods may be viewed as a limitation. Further, the 2014 WHO standards seem to lack adequate methods to evaluate evidence relevant to the shifts in nutrition priorities, such as food sustainability and food practice, including eating together and home cooking^(18,19). Because current evidence-based methods primarily rely on clinical trial findings⁽²⁷⁾, we propose using approaches such as the logical maps to guide research questions in dietary guideline development. This may ensure evidence collection and evaluation are driven by the research questions rather than by the available methods, such as the hierarchy of evidence⁽²⁷⁾. Therefore, future methods in developing dietary guidelines could consider allowing high-quality observational studies to address emerging food and nutrition priorities, which are not always suitable for randomised controlled trials.

Generalisability of the findings from two recent Canadian dietary guidelines may be limited in the North American context as a high-income Western country. However, such limitations may primarily affect the specific types of foods recommended rather than guideline development methods. Although two reviewers independently extracted and appraised the data, we cannot rule out the possibility of overlooking certain information during the process of rating the guidelines, even after consulting with the guideline development authority.

Conclusion

In summary, this case study suggests that the 2019 Canadian dietary guidelines have adopted greater transparency and a more evidence-based approach in formulating the recommendations. Our analysis also suggests new method criteria should support further incorporation of nutrition priorities such as food sustainability and food skills in future dietary guideline development.