Methodological quality of public health guideline recommendations on vitamin D and calcium : a systematic review protocol

Abstract

Introduction

Current recommendations for vitamin D and calcium in dietary guidelines and bone health guidelines vary significantly among countries and professional organisations. It is unknown whether the methods used to develop these recommendations followed a rigourous process and how the differences in methods used may affect the recommended intakes of vitamin D and calcium. The objectives of this study are (1) collate and compare recommendations for vitamin D and calcium across guidelines, (2) appraise methodological quality of the guideline recommendations and (3) identify methodological factors that may affect the recommended intakes for vitamin D and calcium. This study will make a significant contribution to enhancing the methodological rigour in public health guidelines for vitamin D and calcium recommendations.

Methods and analyses

We will conduct a systematic review to evaluate vitamin D and calcium recommendations for osteoporosis prevention in generally healthy middle-aged and older adults. Methodological assessment will be performed for each guideline against those outlined in the 2014 WHO handbook for guideline development. A systematic search strategy will be applied to locate food-based dietary guidelines and bone health guidelines indexed in various electronic databases, guideline repositories and grey literature from 1 January 2009 to 28 February 2019. Descriptive statistics will be used to summarise the data on intake recommendation and on proportion of guidelines consistent with the WHO criteria. Logistic regression, if feasible, will be used to assess the relationships between the methodological factors and the recommendation intakes.

Ethics and dissemination

Ethics approval is not required as we will only extract published data or information from the published guidelines. Results of this review will be disseminated through conference presentations and peer-reviewed publications.

PROSPERO registration number

CRD42019126452

Strengths and limitations of this study.

• To the best of our knowledge, this will be the first study that critically appraises the methodological quality of guideline recommendations for vitamin D and calcium intakes.

• This study will provide insights to address potential limitations in guideline development and identify areas for improvement in developing vitamin D and calcium recommendations.

• Eligible guidelines published in English only may potentially limit the sample size and regional coverage of the guideline recommendations included in our analysis.

• Information required to assess methodological quality of guidelines may be missing, particularly when other guideline development standards (eg, the Institute of Medicine standards for trustworthy clinical guidelines) rather than the WHO Handbook for Guideline Development were used to develop the public health guidelines.

Introduction

Current recommendations for vitamin D and calcium in dietary guidelines and bone health guidelines vary significantly among countries and professional organisations.^1–4 Several factors may have contributed to such variation: dietary sources of vitamin D and calcium are different among countries and regions, with some but not all fortifying the nutrients in the food products as an example; some guidelines may consider supplement use as part of the recommendations, while others recommend sunlight exposure as a source of vitamin D. For the latter, race and skin tone also contribute to the appropriate length of time of sun exposure to achieve certain vitamin D levels. Another possible reason for these varied recommendations is that evidence on the efficacy of vitamin D and/or calcium supplementation^{3 5–14} in the prevention of osteoporosis, particularly in fracture prevention, is conflicting; and their adverse effects in cardiac risks^{15 16} and compromised renal function^{10 12} must also be taken into account. Further, what defines vitamin D deficiency measured by serum 25(OH)D is debatable and varies among the general populations.^{14 17–19} This variation of optimal vitamin D level further contributes to the inconsistent findings in randomised control trials testing the effects of different dosages of vitamin D and/or calcium supplementation in fracture prevention.^{13 14}

Additionally, inconsistencies exist in the guideline development processes used to retrieve, appraise and synthesise relevant evidence, as well as in reporting conflicts of interest and funding sources in national dietary guidelines.²⁰ This can potentially further affect the discordance in the recommended intakes of vitamin D and calcium in guideline recommendations. For example, findings from a global review of food-based dietary guidelines suggest that social and economic equity and cultural factors need to be incorporated in guideline development in order to recommend appropriate food intakes among populations with different backgrounds. Further, there are significant regional differences in dairy intake recommendations across different dietary guidelines.²¹ As dairy is the main source of dietary calcium and vitamin D in some but not all populations,^{22 23} recommendations about dietary sources need to consider the ethnic and cultural contexts. Taken together, guideline development methods should include, but be not limited to, evidence identification, evaluation and synthesis; as well as incorporating stakeholders’ positions, feasibility and acceptability of the recommendations.

The objective of this study is to compare recommendations for vitamin D and calcium intakes and their associated parameters (eg, sun exposure for vitamin D synthesis and serum 25(OH)D level to define vitamin D status), and the methods used in formulating these recommendations for middle-aged and older adults in public health guidelines. We will further assess whether the similarity or differences in the vitamin D and calcium recommendations can be explained by the guidelines’ methodological quality. Findings from this study will illustrate methodological rigour and potential limitations in current public health guidelines for vitamin D and calcium recommended.

Methods

Overview

We will include public health guidelines or policy statements related to vitamin D/ calcium and bone health for generally healthy adults aged 40 years and above. Because middle-aged and older adults are individuals at risk to develop osteoporosis, we intend to include those who may experience menopause as young as 40 years to ensure the coverage of all age groups at risk in the included guidelines. We will include both food-based dietary guidelines and guidelines for osteoporosis (including fracture) prevention. We will use the definition described in the 2014 WHO Handbook for Guideline Development to define guidelines and recommendations, that is, ‘any document containing recommendations for clinical practice or public health policy. A recommendation tells the intended end-user of the guideline what he or she can or should do in specific situations to achieve the best health outcomes possible, individually or collectively’.²⁴

Inclusion criteria

1. Most recent version of national food-based dietary guidelines.

2. Most recent version of national guidelines, policy statements or standards for osteoporosis prevention

We will only include national guidelines that have been developed by a nationally or internationally recognised government authority or by a medical/academic society or organisation. This is to ensure consistency between the food-based dietary guidelines and the bone health guidelines at country (state) level, as food-based dietary guidelines are typically a state government document. In addition to the guideline documents, we will include supporting documents such as those provide details for the methodology used and evidence underpinning the recommendations. For instance, guideline committee’s reports, in which we can locate methodology and supporting evidence will be included. An example is the ‘Scientific Report of the 2015 Dietary Guidelines Advisory Committee’, which describes the development of the dietary guideline and supporting evidence for the ‘Dietary Guidelines for Americans 2015–2020’. Similarly, ‘a review of the evidence to address targeted questions to inform the revision of the Australian Dietary Guidelines’²⁵ as well as the Nutrient Reference Values document for Australia and New Zealand are companion documents with evidence supporting the recommendations for the ‘Australian Dietary Guidelines 2013’.²⁶ If there are multiple versions of a national guideline from the same country or authority, only the most recent version will be included. Similarly, if an updated bone health guideline is based on the previous documents that describe the process of the recommendation development, these documents will be included to locate the information on methods and evidence used to support the recommendations.

Exclusion criteria

1. Food guides such as food pyramids, food plates or simple designed pictorial or graphic representation

2. Bone health guidelines regarding vitamin D and calcium recommendations in the management of osteoporosis, secondary osteoporosis (eg, osteoporosis due to rheumatoid arthritis) or for a particular group of population (eg, pregnant women) or those with health condition (eg, patients with cancer, cirrhosis and so on)

We will not include food guides, because they lack substantial materials to document the guideline development process. Guideline recommendations on clinical treatment of any bone disorders, or guidelines targeted to a particular group of populations such as those with HIV or cancer patients or pregnant or lactating women or a particular type of osteoporosis (eg, glucocorticoid-induced osteoporosis) will be excluded. This is because the focus of this study is to review recommended vitamin D and calcium intakes for generally healthy populations to maintain bone health or to prevent osteoporosis.

Search strategy

We will search guidelines or policy statements that are published from 1 January 2009 until 28 February 2019 in the following electronic databases: MEDLINE (via OVID), EMBASE (via OVID), CINAHL (via EBSCO) and Practice-Based Evidence in Nutrition. Additionally, the following sources which include guidelines specifically will be searched: National Guideline Clearinghouse, National Institute for Health and Care Excellence and Guidelines International Network. We will only include documents published in English but no geographic regions are restricted. We will use a combination of controlled vocabulary and free-text terms for vitamin D, sunlight, calcium, dairy, vegetable, seafood, fortified food (as these are the good dietary sources for vitamin D and/or calcium), dietary patterns, osteoporosis/fracture and guideline. The search strategy for Medline via Ovid is described in online supplementary material. Similar search strategies with appropriate syntax will be applied to EMBASE and CINAHL. We will also search the grey literature via the Food and Agricultural Organization website for relevant food-based dietary guidelines²⁷ and the International Osteoporosis Foundation²⁸ for bone health guidelines from national government agencies or organisations. Additionally, we will consult leading experts in the field of bone health to avoid any oversights.

Data extraction process

Recommendations for vitamin D and calcium

Verbatim text of qualitative and quantitative recommendations on dietary intake of vitamin D/calcium, vitamin D/calcium containing foods, a healthy dietary pattern beneficial to bone health, supplementation dosage for vitamin D/calcium, nutrient reference intakes for vitamin D and calcium, timing and length of sun exposure for vitamin D synthesis, and serum 25(OH)D level to define vitamin D status will be extracted from each included guideline. Because there is no standard for wording of recommendation across and within guidelines,^{29 30} we will adopt the criteria described in the report by Woolf and colleagues for the presentation and formulation of recommendations.³¹ These criteria include ‘consistent semantic and formatting indicators’, ‘a summary section to facilitate identification of recommendations’, ‘decidable and executable wording’ and ‘avoiding embedding recommendation text within long paragraphs’. We will not adopt ‘evidence quality and recommendation strength in proximity to each recommendation’, as an objective in this review is to assess the quality of evidence underpinning the recommendations. For example, in bone health public guidelines, the following would be considered as eligible recommendations: ‘general practitioners should recommend that postmenopausal women and older men maintain a diet high in calcium to meet the Australian recommended dietary intake’ or ‘general practitioners should recommend the following important lifestyle choices for all postmenopausal women and older men: adequate but safe exposure to sunlight as a source of vitamin D’.³² Statements or text mentioning vitamin D or calcium as knowledge-based information or as a rationale to support an argument will be excluded as a recommendation. For example, ‘soy beverages fortified with calcium, vitamin A and vitamin D, are included as part of the dairy group, because they are similar to milk based on nutrient composition and in their use in meals’.³³

We will use a pilot-tested data extraction form to capture vitamin D and calcium recommendation intakes and categorise as ‘yes’ or ‘no’ according to criteria described above.³¹ Verbatim text will be extracted, if rated as yes, including numerical values and/or recommendations without numerical values. Data extraction will be conducted independently by two reviewers via Research Electronic Data Capture, an electronic data capture tools hosted at the University of Sydney.³⁴ Any discrepancies in the data extracted will be resolved through discussion between the reviewers; otherwise, further discussion with the senior author will be carried out to resolve the disagreement through consensus. Additionally, we will contact the guideline authors to obtain all relevant materials during the data extraction to avoid missing documents.

Methodological processes

We will appraise the guideline recommendation development processes against the criteria outlined in the second edition of the 2014 WHO Handbook for Guideline Development,²⁴ a ‘gold standard’ for public health guideline development. The reasons we have chosen the 2014 WHO handbook include it was developed by the primary international public health agency; it is more recent compared with other standards; and it incorporates the most comprehensive domains and elements for a rigorous guideline development.^{35 36} Compared with the Appraisal of Guidelines for Research and Evaluation II (AGREE II),³⁷ the WHO handbook criteria cover the same domains with more extensive details regarding the guideline development process. For example, for conflicts of interest, the 2014 WHO guideline handbook includes both disclosure and management of conflicts of interest among the guideline development group members and funders,²⁴ while the AGREE II instrument addresses conflicts of interest among the guideline development group members only.³⁷

The items outline in the WHO handbook with description of the criteria are described in table 1, which includes the following domains: guideline development group, conflicts of interest, review methods, transparency of evidence supporting the recommendations, recommendation development and peer review process. We will record whether each included guideline recommendation is consistent with each of the WHO criteria and classify as yes, no or unclear. If yes is rated, verbatim text will be extracted from the guideline to substantiate per the recommended item. No is referred to those which explicitly state none for the items to be appraised. ‘Unclear’ is referred to those that neither explicitly state none nor those with relevant statements supporting the criterion. Two reviewers will perform the critical appraisal and data extraction independently. Discrepancies will be first discussed and resolved through consensus among the reviewers, and with the senior author if it remains unresolved after the first attempt.

Table 1.

Appraisal of the processes and methods used in the recommendations for vitamin D and calcium in public health guidelines

Process and method domains	Process and method criteria	Description	Examples where to look
I. Guideline development group (GDG, including members of steering group, research team and individuals involved formulating the final recommendations)
Were each of the following considered in the formation of the guideline development group?	1. Discipline representation	Information about the composition, discipline and relevant expertise of the guideline development group should be provided.	Information can be located in methods, guideline panel member list, acknowledgements and appendices.
	2. Diversity representation	Information about gender, diversity, across the life-course, subject to different gender norms and belonging to different income and education groups of the guideline development group.
	3. Stakeholder input	Stakeholders such as non-governmental organisations, advocacy groups, funders, target audiences and service-users may be invited to ensure transparency of the process and facilitate implementation.
II. Conflicts of interest
Were each of the following steps addressed regarding conflicts of interest?	4. Disclosure of conflicts of interest obtained (extract verbatim text of COI for each member).	Is there an explicit statement that all group members have declared whether they have any competing interests?	Paragraphs/chapters describing the guideline development group or acknowledgements section in methods, conflicts of interest, guideline panel and appendix.
	5. Conflicts of interest managed.	Members declaration of interests must be reported to the steering group. Potential candidates for membership who have major conflicts of interest, be they financial or nonfinancial, cannot be appointed to the GDG. Minor conflicts of interest can be managed at the individual level (eg, by restricting participation in parts of the GDG meeting) or at the group level.
	6. Disclosure of funders of the guideline obtained and disclose funder’s role in influencing the guideline development process and recommendations.	Is there an explicit statement of funder of the guideline and the role of funders in the final guideline recommendations?
III. Systematic review methods
Were methods for each of the following addressed in the guideline?	7. Formulation of key questions for the evidence review in PICO, PICOT or PEO format (also extract the key questions in such format).	Key questions are framed in a way that enables a systematic search of the literature and delineates inclusion and exclusion criteria for the body of evidence to formulate the research questions for the recommendations in such format.	Examine the paragraphs/chapters describing the guideline development process in methods, literature search strategy and appendices.
	8. Choosing (finalising) priority outcomes for systematic review.	List high-priority key questions and the outcomes to formulate recommendations.
	9. Systematic methods to search for evidence.	Details of the strategy used to search for evidence should be provided including search terms used, sources consulted and dates of the literature covered. Sources may include electronic databases, hand searching journals, reviewing conference proceedings and other guidelines.
	10. Evidence retrieval (screening and selection of eligible studies).	Process of data from eligible studies are extracted and search strategy and results should be carefully documented.
	11. Evidence quality assessment.	Each study included in a systematic review should be assessed for risk of bias (eg, use the Cochrane risk of bias tool, quality assessment tolls project report and so on).
	12. Evidence synthesis	The findings of the systematic review may be synthesised in a narrative manner or quantitatively in a meta-analysis. The review should describe how data were handled and why a given approach to synthesis was taken for each outcome.
IV. Transparency of evidence supporting the recommendations
If evidence was explicitly linked to recommendation, what type of evidence was reported?	13. Are recommendations explicitly linked to substantiating evidence?	An explicit link between the recommendations and the evidence on which they are based should be included in the guideline. The guideline user should be able to identify the components of the body of evidence relevant to each recommendation.	Define and examine the recommendations in the guideline and the text describing the body of evidence that underpins them. Examples of commonly labelled sections or chapters in a guideline where this information can be found include: recommendations and key evidence.
	a. Primary research.	Primary individual studies.
	b. Systematic reviews.	Systematic reviews of clinical trials/observational studies.
	c. Summary table of the evidence	Summary of evidence table.
	d. GRADE evidence profiles	GRADE summary of evidence table.
	e. Evidence to decision table
	f. If evidence is explicitly linked to recommendation, what is the citation information, if applicable?	List the citations of the studies underlying the recommendations.
V. Recommendation development: factors that determine the direction and strength of a recommendation
Was each of the following items considered when developing the recommendation? (also communicate with guideline authority whether other documentation may provide such information if they cannot be located in the main guideline reports).	14. Was a consensus process clearly described for developing recommendations.	A description of the methods used to formulate the recommendations and how final decisions were arrived at should be provided. For example, methods may include a voting system, informal consensus and formal consensus techniques. Areas of disagreement and methods of resolving them should be specified.	Examples of commonly labelled sections or chapters in a guideline where this information can be found include methods and guideline development process or in appendices.
	15. Was a method employed to determine strength and/or certainty of the recommendation?	Is there a method provided to influence the direction and strength of a recommendation (eg, use GRADE framework and others).
	16. Priority of the problem: Is the problem a burden of disease?	The problem’s priority is determined by its importance and frequency (ie, burden of disease, disease prevalence or baseline risk). The greater the importance of the problem, the greater the likelihood of a strong recommendation.
	17. Quality of the evidence: Is higher quality of the body of evidence included to support the recommendation?	Is there a method provided to grade the quality of body of evidence to assess the strength of the recommendation (eg, GRADE and others).
	18. Certainty of evidence: Does the recommendation include consistent body of evidence (eg, confidence in effect estimates)?	The quality of the evidence—the degree of confidence in the estimates of effect. This is a key factor in determining the strength of a recommendation.
	19. Benefits and harms: Are evaluations performed on the net benefit or net harm associate with an intervention or exposure?	The balance between an intervention’s or exposure’s benefits and harms. Did the guideline development group consider the magnitude of the effects and the relative importance of the outcomes, including any disadvantages or inconveniences associated with the intervention.
	20. Balance: Does the balance between desirable and undesirable effects support the recommendation?	Does the balance between desirable and undesirable effects favour the intervention or the comparison?
	21. Outcome importance: Is there important uncertainty about or variability in how much people value the main outcome?	Is there important uncertainty about or variability in how much people value the main outcomes, including adverse effects and burden of the test and downstream outcomes of clinical management that is guided by the test results?
	22. Equity: Does the evidence used reduce inequalities, improve equity or contribute to the realisation of one of several human rights defined under the international legal framework?	What would be the impact on health equity?
	23. Acceptability: Is the option acceptable to key stakeholders?	A strategy to address concerns about acceptability during implementation will be included in the guideline with the recommendations. Acceptability is affected by several factors, such as who benefits from an intervention and who is harmed by it; who pays for it or saves money on account of it; and when the benefits, harms and costs occur.
	24. Feasibility: Is the option feasible to implement?	Feasibility is influenced by the resources available, programmatic considerations, the existing and the necessary infrastructure and training and so on.
VI. Peer review process
	25. Was the guideline/recommendation reviewed by an external review group?	Is there an explicit statement about the peer review of the draft final guideline? The external review group is composed of persons interested in the subject of the guideline as well as individuals who will be affected by the recommendations.	Information can be located in methods, guideline panel member list, acknowledgements and appendices.

GRADE, Grading of Recommendations, Assessment, Development and Evaluations.

Other information to be extracted

Guideline title, country of origin, guideline developing authority or organisation, publication year, age group of the population, gender of the population (men, women or both) and funding body will be extracted. Further, we will extract the information of the evidence underpinning the recommendations including the types of evidence (primary research, systematic review, or summary of evidence table: see details in ‘transparency of evidence supporting the recommendations’ in table 1) and their citation information.

Patient and public involvement

No patient involved.

Main outcomes

A binary outcome will be created based on whether a recommendation exists in a public health guideline for the following: vitamin D intakes, vitamin D containing food consumption (such as fortified dairy or other fortified beverage and seafood), a healthy diet for bone health, sun exposure for vitamin D synthesis, supplement use of vitamin D, serum 25(OH)D level to define vitamin D status, calcium reference intakes, calcium containing food consumption (such as dairy and dark-green leafy vegetables) and supplement use of calcium. If quantitative recommendations (those with amount per day) are available, we will categorise the numerical values into different groups and present the distribution of the recommended intake values.

Data synthesis

Using the information extracted from the included guidelines, we will summarise the recommendation (those with values or recommendation text) for vitamin D and calcium, their food sources, dietary patterns and sun exposure for vitamin D, as well as serum level of 25(OH)D to define vitamin D status, by types of guidelines (food-based dietary guidelines vs bone health guidelines), by continent (Asia, Australia, Europe, North America, South America and Africa), by gross national income per capita (low, middle and high) and by disclosure of conflict of interest (yes, no). Also, we will present the proportion of the guidelines that are consistent with each of the criteria outlined in the WHO handbook for all guidelines, and separately for the dietary guidelines and bone health guidelines. We will also use descriptive statistics (eg, frequency and proportion for categorical variables) to summarise the characteristics of each included guideline.

If feasible, we will perform logistic regression analysis to examine the associations between each methodological factor (yes vs none (combining no and unclear)) in the six domains of guideline development methods (see table 1) and a positive recommendation (defined as yes for the recommendation) for dietary vitamin D/calcium, supplemental vitamin D/calcium, a healthy diet for bone health, sun exposure (for vitamin D synthesis) and optimal vitamin D level, where each of the recommendations will be considered as a binary outcome. Also, we may perform a multinomial logistic regression analysis for the association between each of the methodological factors (those in table 1) and categories of the recommended values for vitamin D/calcium (ie, dietary intake values/supplemental intake values and optimal 25(OH)D level on an ordinal scale), after adjustment for key guideline characteristics. The reason that a multinomial logistic regression is proposed is because that recommended intakes for vitamin D/calcium and optimal vitamin D level in public health guidelines are often clustered or provided as a range. For example, vitamin D recommendation in a guideline could be 400–800, 600–800, 800–1000, 1500–2000 IU/day; and calcium recommendation could be 600, 700–800, 1000, 1000–1200, ≥1000 mg/day and so on. Therefore, a logistic regression analysis is likely more suitable in these analyses.

Discussion

To the best of our knowledge, this will be the first study to critically appraise methodological quality regarding guideline recommendations for dietary and supplemental vitamin D and calcium intakes, their food sources, a healthy diet pattern and sun exposure for vitamin D synthesis. This review will advance our knowledge on how guideline development methods and processes may affect the similarity or differences of the intake recommendations. These findings will further address potential limitations in public health guidelines for the recommended intakes of vitamin D and calcium in middle-aged and older adults.

As we will only include guidelines or statements published in English, this may reduce the sample size and limit the coverage of non-English speaking countries if their guidelines/statement reports are not published in English. Although we believe that the criteria outlined in the 2014 WHO Handbook for Guideline Development cover the most comprehensive process for guideline recommendation development, we understand that some guideline authorities may adopt other standards. For example, the Institute of Medicine standards are commonly used to develop trustworthy clinical guidelines.³⁵ Therefore, we might experience guidelines with missing data regarding our stringent and comprehensive methodological criteria according to the WHO Handbook for Guideline Development. Further, recent concerns have been raised about possible over consumption of phosphorous from meat and dairy sources and highly processed foods. Because the amount of phosphorus additives in processed food products are generally not accounted for, current nutrition databases assume that phosphorous level remains similar for the same types of foods (eg, natural beef and processed beef products), this would potentially underestimate the actual intake of phosphorous in the populations,^{38 39} and result in a lower recommended intake of calcium in the guidelines.⁴⁰ Due to the scope and feasibility of this study, we will not further account for such underestimation of phosphorous intake at the population level, which could be a potential limitation of this review to address the appropriate recommendations for calcium intake in the included guidelines.

Further, assessment of the quality of the evidence underpinning the recommendations for vitamin D and calcium is out of the scope in this study protocol, as the focus here is to appraise the methods used to develop the public health guidelines. However, we will extract information about the types of evidence cited to support each included recommendation. In a follow-up study, we will further assess the evidence quality (eg, risk of bias) of the cited studies and systematic reviews.

Ethics and dissemination

Ethics approval is not required as we will only extract published data or information from the published guidelines. We will seek to present our findings at international academic conferences and report our findings in a peer-reviewed medical journal article. We also plan to present our findings to key stakeholders in public health authorities and with public health advocates for bone health and osteoporosis prevention.

Conclusions

Currently, there are no studies that have comprehensively appraised methodological rigour in guideline development methods and processes used to develop vitamin D and calcium recommendations. Due to global ageing and a rapid rise of osteoporosis, this review will provide a timely assessment of guideline recommendations for vitamin D and calcium, and help to address potential limitations and identify areas for improvement in developing future guideline recommendations for vitamin D and calcium.