Communication of scientific uncertainty: international case studies on the development of folate and vitamin D Dietary Reference Values

Abstract

Objective

Transparent evidence-based decision making has been promoted worldwide to engender trust in science and policy making. Yet, little attention has been given to transparency implementation. The degree of transparency (focused on how uncertain evidence was handled) during the development of folate and vitamin D Dietary Reference Values was explored in three a priori defined areas: (i) value request; (ii) evidence evaluation; and (iii) final values.

Design

Qualitative case studies (semi-structured interviews and desk research). A common protocol was used for data collection, interview thematic analysis and reporting. Results were coordinated via cross-case synthesis.

Setting

Australia and New Zealand, Netherlands, Nordic countries, Poland, Spain and UK.

Subjects

Twenty-one interviews were conducted in six case studies.

Results

Transparency of process was not universally observed across countries or areas of the recommendation setting process. Transparency practices were most commonly seen surrounding the request to develop reference values (e.g. access to risk manager/assessor problem formulation discussions) and evidence evaluation (e.g. disclosure of risk assessor data sourcing/evaluation protocols). Fewer transparency practices were observed to assist with handling uncertainty in the evidence base during the development of quantitative reference values.

Conclusions

Implementation of transparency policies may be limited by a lack of dedicated resources and best practice procedures, particularly to assist with the latter stages of reference value development. Challenges remain regarding the best practice for transparently communicating the influence of uncertain evidence on the final reference values. Resolving this issue may assist the evolution of nutrition risk assessment and better inform the recommendation setting process.

Historic public health scares (e.g. in the UK following the Phillips inquiry into variant CJD/new variant CJD, the human prion disease caused by the BSE crisis in 1996) contributed to a general commitment for transparency, openness and evidence-based (information indicating whether a belief or proposition is true or valid) decision making in food-related science and policy^{( 1 )}. This resulted in the formal separation of science and policy activities and adoption of the risk analysis framework^{( 2 )}.

The risk analysis framework was first adapted for use in the food area following a joint FAO/WHO expert consultation in 1995^{( 3 )}. The framework has been comprised of three interconnected activities: (i) risk assessment (scientific evaluation of health effects); (ii) risk management (policy decisions to minimise risk); and (iii) risk communication (exchange of risk information in/outside the framework). A principle has been to maintain the separation of activities while recognising the interaction between activities (e.g. separation can vary from different organisations to different groups/tasks within the same organisation responsible for different activities)^{( 3 )}.

Central to implementation of the risk analysis framework has been the encouragement of transparency and disclosure of uncertainty in the evidence underlying scientific advice (risk assessment) or political decision making (risk management)^{( 2 , 4 )}. The transparent handling of uncertainty may refer to ‘explicit recognition of any uncertainty either in the current state of knowledge or in the adequacy of the available data’ (e.g. accuracy/quality or degree of incomplete/quantity of data)^{( 5 )}. Transparency and openness have been promoted to achieve greater accountability and credibility during risk analysis, improved communication between stakeholders (e.g. science, policy, industry, health practice) and safeguard the abuse of public organisation power or resources^{( 2 , 6 – 8 )}.

Policies of transparency can be found across various public health disciplines responsible for risk analysis activities, such as the European Food Safety Authority (EFSA), the National Institute for Health and Care Excellence (NICE) and FAO/WHO^{( 2 , 4 , 9 )}. Implementation of these policies requires the process and outcome of a risk analysis activity to be evident and obvious, i.e. documented and understandable (transparent), as well as accessible (open)^{( 7 , 8 , 10 , 11 )}. This may be related to the organisation responsible for a risk analysis activity (e.g. disclosure of organisation role and responsibility, membership, confidentiality signed agreements, declaration of interest) or the technical content of the activity (e.g. disclosure of non-published requests, methods, findings and conclusions via websites, reports and manuscripts, meeting minutes and agendas, open consultations and open meetings, release of data and study results, etc.).

A commitment to transparency and openness may seem the correct thing to do in view of due diligence towards scientific practice^{( 8 )}. Nevertheless, the implementation of transparency per se may be limited or represent certain challenges^{( 11 )}. Detailing and providing access to all elements of the risk analysis process may be challenging in terms of resource, technical, confidentiality or wider trust issues (e.g. articulating the nature of risk assessment or inherent uncertainty of scientific endeavour; detailing contradictory views during commonplace expert discussions while maintaining unanimity in the final scientific advice)^{( 12 , 13 )}.

Nutrition risk assessment to derive micronutrient Dietary Reference Values (DRV) is an area where greater transparency and openness has been encouraged^{( 14 )}. DRV are developed in the first two steps of the risk assessment process. The identification of a nutrient-related hazard in a food/food group is established in step one (nutrient-related hazard identification). The qualitative/quantitative evaluation of adverse health effects associated with a nutrient is established in step two (nutrient-related hazard characterisation). This constitutes the development of DRV. In a full risk assessment, DRV (steps one and two) would be combined with a nutrient intake/exposure assessment (step three) to assess exposure in relation to the DRV (step four)^{( 15 )}. Once completed, the scientific advice from the risk assessment is used to inform development of recommendations in risk management. Thus, the development of DRV can be considered a precursor to the development of micronutrient recommendations. Recommendations are defined as population targets of micronutrient intake necessary for adequate growth, function and health throughout the human lifespan. These are widely used in monitoring and evaluating population intakes as well as in developing public health policies, interventions and dietary risk communications (e.g. food-based dietary guidelines to help the population meet recommended intakes)^{( 14 , 16 )}.

The widespread use of DRV (directly or indirectly via recommendations) belies the complexity and uncertainty with which they are developed. The first stage of DRV development has been referred to as the ‘nutrition problem formulation stage’. During problem formulation risk managers and assessors are required to establish a shared understanding of the problem and the purpose of the risk assessment^{( 10 )}. Previously, the primary focus of DRV development was to devise intakes related to nutrient deficiency or overconsumption. More recently the remit of DRV has expanded to also encompass intakes for health benefits (including intakes to delay the onset of disease). Yet, the evidence base surrounding the role of nutrition on health has been far from certain. Hence, prevailing scientific knowledge is used to handle limitations and uncertainty in the evidence (unknown, unreliable or indefinite evidence), develop plausible assumptions and complete the DRV development process^{( 14 , 17 )}.

Guidelines have been produced to assist with the development of DRV (e.g. the Institute of Medicine’s Dietary Reference Intakes (DRI) for the USA and Canada; the EFSA Panel on Dietetic Products, Nutrition and Allergies’ Scientific Opinion on principles for deriving and applying DRV)^{( 16 , 18 )}. Yet, no standardised approach or agreed best practice has been used to set DRV. Furthermore, disparity has been observed in DRV developed by different national or international bodies (e.g. vitamin D DRV). The lack of agreed best practice and disparity in DRV has led to confusion among inter/national policy decision makers, health professionals, the food industry and consumers^{( 19 )}.

An open and transparent DRV process has the potential to assist with understanding why values differ between countries by clearly detailing what, why and how decisions were made, particularly regarding the degree of uncertainty in the evidence and how this was handled or influenced the strength of the final risk assessment conclusions/resultant reference values^{( 19 , 20 )}. This would inform the debate on DRV development best practice. Furthermore, increased documentation, understanding and accessibility to the DRV process could contribute to the responsible use of DRV throughout risk management and risk communication activities^{( 2 , 14 )}.

The European Commission-supported Network of Excellence EURRECA (EURopean micronutrient RECommendations Aligned, FOOD-CT-20006-36196, 2007–2012) was tasked with reviewing the methodologies used to derive DRV, assessing the reasons for value disparity and the potential for methodological alignment across national and international DRV development. Early EURRECA findings highlighted the importance of the nutrition problem formulation stage to ensure risk assessment activities remained achievable while also fulfilling the requirements of risk managers^{( 20 )}. Further research suggested DRV disparity was unlikely to be explained by differences in concepts, definitions or defined population groups. Instead it was considered likely to be due to different interpretations or assumptions taken while accounting for uncertainty and limitations during evidence evaluation^{( 19 )}. Finally, research identified variation in the transparency of different types of bodies in relation to how risk managers integrated DRV with other types of advice to develop recommendations^{( 20 )}. The current study built upon these previous findings. Transparency and openness during the DRV risk assessment process was explored, with a particular focus on how uncertainty in the evidence base was handled. Three areas of interest, likely to demonstrate handling of uncertainty in the evidence base, were identified during DRV development: (i) the request to develop DRV; (ii) the process of DRV evidence evaluation; and (iii) the integration of evidence to develop final DRV.

Method

Design

Qualitative in-depth case studies explored the handling of uncertain evidence and the transparency of vitamin D and folate DRV development in six countries/regions: (i) Australia and New Zealand (ANZ); (ii) the Netherlands (NL); (iii) the Nordic countries (Denmark, Finland, Iceland, Norway and Sweden; NOC); (iv) Poland (PL); (v) Spain (ES)^{( 21 )}; and (vi) the United Kingdom (England, Northern Ireland, Scotland and Wales; UK).

Each country/region produced a descriptive report based upon semi-structured interviews with at least two advisory committee members together with desk research (Table 1). A common protocol was followed throughout. The interview schedule, desk research and final case study report all centred upon the transparency and handling of uncertain evidence in three predefined areas: (i) the request to develop DRV, e.g. source of request, scope of request, reasons for the request; (ii) the process of DRV evidence evaluation, e.g. selection and evaluation of evidence; and (iii) integration of evidence to develop final DRV, e.g. formation of quantitative numbers and advisory committee report.

Table 1.

Country/region, number of interview samples, response rate, and references and associated updates for the folate and vitamin D DRV

Country	No.	% response rate	Folate	Vitamin D
ANZ	4	57	NHMRC 2006( 38 )	NHMRC 2006( 38 )
NL	4	67	HCN 2003( 27 ), HCN update 2008( 39 )	HCN 2000( 40 ), HCN update 2008( 41 )
NOC	4	80	NNR 2004( 42 )	NNR 2004( 42 ), *NNR 2012 published subsequent to data collection( 26 )
PL	4	100	Jarosz and Bułhak-Jachymczyk 2008 ( 43 )	Jarosz & Bułhak-Jachymczyk 2008 ( 43 )
ES	3	100	Moreiras 2011( 44 ), Ortega 2011( 45 ), Serra Majem 2001 (SENC)( 46 ), Martínez 2010 (FESNAD)( 47 )	Moreiras 2011( 44 ), Ortega 2011( 45 ), Martínez 2010 (FESNAD)( 47 )
UK	2	33	COMA 1991( 48 ), SACN folate 2006( 49 )	COMA 1991( 48 ), SACN vitamin D update 2007( 50 ); *new vitamin D DRV expected 2014

DRV, Dietary Reference Values; ANZ, Australia and New Zealand; NL, Netherlands; NOC, Nordic countries (Denmark, Finland, Iceland, Norway and Sweden); PL, Poland; ES, Spain; UK, United Kingdom; NHMRC, Australian National Health and Medical Research Council; HCN, Health Council of the Netherlands (Gezondheldsraad); NNR, Nordic Council of Ministers’ Nordic Nutrition Recommendations; SENC, Spanish Society of Community Nutrition; FESNAD, Federación Española de Sociedades de Nutrición, Alimentación y Dietética; COMA, Committee on Medical Aspects of Food Policy; SACN, Scientific Advisory Committee on Nutrition.

^*Recommendations under development at time of study data collection.

Transparency and openness was qualitatively judged via the availability of accessible, understandable documentation on the DRV process (e.g. downloadable/upon request meeting agendas, minutes and key discussion documents; ability to attend open meetings and consultations and documentation surrounding DRV; documented risk analysis policy, organisation role and responsibility, membership, declaration of interest; stakeholder involvement; etc.).

Vitamin D and folate were selected due to their prioritisation as micronutrients of interest in Europe^{( 22 )}. Countries/regions were originally selected from Europe with a subsequent opportunity taken to also collect data in Australia and New Zealand. The countries/regions sampled represented diversity in the age and method of developing DRV as well as in geographical location, socio-cultural factors and institutional infrastructure. Data sets were collected from December 2010 to June 2011.

Interviews

Interviewees were recruited due to their involvement in the advisory committee and/or development of vitamin D or folate reference values. This included members of the advisory committee and/or members of relevant advisory committee working subgroups. The response rate ranged from 33 % (UK) to 100 % (ES and PL; Table 1). Consent was obtained from each participant, with interviews recorded and later transcribed verbatim. All data were made anonymous and held in accordance with the local data privacy laws (e.g. UK Data Protection Act)^{( 23 )}. The semi-structured interview schedule was piloted and devised by the research team with questions focused upon the transparency and handling of uncertainty in the aforementioned three areas of DRV development.

Desk research

Sourced documents related to the development of vitamin D or folate DRV in each country/region. Keyword searches were conducted of advisory committee websites, publication websites (e.g. PubMed Central) and search engines (e.g. Google). Documents referred to during the interviews and manual searches of references elicited several additional documents. Further information was obtained in the UK case study via observation at two open advisory committee meetings (14 February 2011 and 7 June 2011). Desk research key search terms included variants of the nutrient name (e.g. folate and folic acid), variants of the word DRV (e.g. micronutrient recommendations, nutrient intake values, nutritional objectives) and/or the advisory committee name (e.g. Health Council for the Netherlands). Excluded documents were not relevant to vitamin D or folate DRV, the specific country/region, human nutrition or healthy populations or were duplications from previous searches. Desk research was led and organised by questioning the transparency and handling of uncertainty in the aforementioned three areas of DRV development.

Analysis

Initial interview and desk research analysis was conducted in the native language using template analysis and a skeleton coding structure created and modified by partners during preliminary analyses^{( 24 )}. Case studies consisted of an English-translated summary of interview- and/or desk research-identified themes with illustrative quotes organised by the three areas (value request, evidence evaluation and final values). Construct validity was upheld by limiting the subjective collection of data and performing traceable literature searches. Internal validity was upheld by encouraging several interpretations of the data to be considered. Multiple sources of information (triangulation between interview, desk research and DRV reports where possible) were used to cross-reference (corroborate/dispute findings). The uniform framework of case study reporting allowed further analysis where cross-case synthesis was used to describe and identify any differences or similarities across countries/regions in the transparency and handling of uncertainty^{( 25 )}. One research team member conducted the cross-case synthesis. The qualitative software NVivo version 9 (2010) was used to organise case study data. A subsection of the data was cross-coded by another research team member. One fewer theme was identified by the second coder. This was subsequently dropped from further analysis. Agreement on remaining themes and sub-themes was 91%. Interpretation of cross-case analysis results was reflected back to case study authors and amended where necessary.

Data presentation

Interview quotes have been displayed in the tables using double quotation marks and italics. Verbatim quotes have been modified in publication for improved readability. Case study/desk research text has been displayed in the tables using single quotation marks. The term ‘micronutrient recommendation’ was used in the interview schedule and throughout the present study. This term could be translated and understood across the cases sampled as pertaining to either micronutrient DRV or the use of values to make recommendations.

Note that quotes have been unaltered in this respect and differences in the terminology can be seen as cases refer to their respective micronutrient recommendations, e.g. ‘The term “Nordic Nutrition Recommendations” refers to a set of dietary reference values (DRVs) for essential nutrients that includes the average requirement (AR), recommended intake (RI), upper intake level (UL), lower intake level (LI), and reference values for energy’^{( 26 )}.

Results

Transparency and openness of the DRV process (i.e. documentation, understanding and accessibility) varied across the countries/regions (cases) and different areas of the DRV development process studied. Results have been presented below in the three areas studied: (i) the request to develop DRV; (ii) the process of DRV evidence evaluation; and (iii) the integration of evidence to develop final DRV.

Request to develop DRV

In ES, where multiple sets of DRV were developed by different individuals, teams and organisations (Spanish Society of Community Nutrition (SENC), Federación Española de Sociedades de Nutrición, Alimentación y Dietética (FESNAD), Universidad Complutense de Madrid), minimal documentation could be found to clarify details on the source, scope or reason for the request to develop DRV (meeting agendas/minutes, discussion documents, press releases, website pages, final report details, journal publications). However, the Spanish interview data were more informative and interviewees openly discussed details surrounding the request for previous DRV. In contrast, greater information (via desk research and interview data) on the request to develop DRV was available in countries/regions with an official set of DRV and a clear body tasked with their development, such as in ANZ (National Health and Medical Research Council (NHMRC)/Ministry of Health (MOH)), NL (Health Council of the Netherlands (HCN)), NOC (Nordic Council of Ministers (NCM)), PL (National Food and Nutrition Institute (Instytutu Żywności i Żywienia; IŻŻ)) and UK (Committee on Medical Aspects of Food Policy (COMA)/Scientific Advisory Committee on Nutrition (SACN); Tables 1 and 2).

Table 2 .

Request to develop DRV

Value request	Quotes: case study text or interview [case study/participant reference]
Source of request	“No, it has not been the Government as in other countries... there have been private initiatives but nothing coming from the Government.” [ES]
	“…items of work can come to the SACN in three ways: they can come from Government, requests from Government; from the Committee itself; and from… anybody just writing to the Committee. The reality is the Committee’s workload means it’s mainly driven by Government.” [UK]
Reason for request	“All nutrients are reviewed one way or the other every eight year and then they put special focus on some of them every time. And vitamin D has been discussed very much in the last years, also in America, so that is why vitamin D is one of the chosen nutrients.” [NOC]
	“Concerning the folate, firstly quite huge discrepancies between Polish recommendations and those in other countries. Secondly, elevated blood homocysteine level and quite big percentage of neural tube defects in Polish population. Also the health policy, how to improve the situation (…).” [PL]
Scope of request	“What we did is making the question more precise and operationalised it, like it is called, in cooperation with the MHWS. Let’s say, the original questioning is a bit adapted in such a way that we could better provide answers.” [NL]
	‘The Chair emphasised the need for a broad risk assessment, which would inform government whether the newly available data on vitamin D has implications for UK public health policy.’ [UK]

DRV, Dietary Reference Values; ES, Spain; SACN, Scientific Advisory Committee on Nutrition; UK, United Kingdom; PL, Poland; MHWS, Ministry of Health, Welfare and Sport; NL, Netherlands.

“Italics and double quotation marks denote interview quotes”; ‘single quotation marks denote case study/desk research text’.

Regarding the source of the request to develop DRV, Spanish interviewees cited the majority of activity was undertaken by independent and academic institutions. On the other hand, in the five cases of ANZ, NL, NOC, PL and UK the requests were predominantly sourced from the government or within the advisory committee as a set programme of work.

Across all six cases the reasons provided for the request to develop DRV ranged from acknowledged discrepancy between local DRV and those of other countries, DRV age, the emergence of new data, a set time cycle for the programme of work, information provision to assist policy option selection and triggers from monitoring practices, such as clinical health outcomes or markers of chronic disease risk (e.g. neural tube defects, LDL). However, details on the final trigger to initiate a request for particular nutrient DRV to be reviewed or developed were not always transparent, i.e. the influence of lobbying (professional, academic, commercial or special interest group) on initiating or suppressing any development of DRV. Similarly, the criteria for judging when DRV were ‘too old’, when there was ‘sufficient’ new evidence to initiate renewing DRV or at what point disease incidence constituted a public health issue to address were not always transparently documented.

The scope of request differed between cases. In the ANZ, PL and ES cases the scope of the request centred upon the utilisation and adaptation of international DRV to each respective country/region (predominantly the US/Canadian DRV in ANZ and PL; various international DRV in ES). This was clearly detailed in ANZ. The NHMRC was asked to assess the body of evidence used to establish the US/Canadian DRV (and any relevant literature that had subsequently been released) and provide an opinion on one of four courses of action: whether they should adopt, adopt with minor changes, adopt with substantial changes or reject the US/Canadian DRV in ANZ.

The remaining three cases (NL, NOC and UK) all referred to the importance of clarifying the scope of the request and iterative interactions between government representatives and advisory committee members during the nutrition problem formulation (e.g. to demarcate risk assessment/risk management activities or develop values for deficiency/adequacy/optimal/toxicity). Problem formulation was deemed necessary to ensure the request would be both appropriate and achievable (e.g. scientific limits of knowledge and resource constraints such as expertise, finance, time). Transparency appeared to be the greatest regarding the current UK vitamin D DRV development process. Here, the public was allowed to attend an open meeting where the scope of the request was discussed (also known as ‘terms of reference’ in the UK) and online access was made available for downloading detailed meeting minutes and discussion documents surrounding request clarification.

Process of DRV evidence evaluation

Several discussions were undertaken on how to evaluate and interpret evidence that contained methodological and theoretical uncertainties (e.g. folate assay method, folate equivalents, uncertainty factors, bioavailability factors, vulnerable population groups, mechanisms of action, multiple nutrient interactions). These discussions were present in the cases which adopted/adapted existing international DRV as well as those which developed their own DRV (Table 3). For example, the ANZ, PL and ES cases made reference to the evidence evaluation previously conducted in the original DRV as well as the additional evidence evaluation required to ensure adapted DRV were up-to-date and relevant to their respective country/region.

Table 3.

Process of DRV evidence evaluation

Evidence evaluation	Quotes: case study text or interview [case study/participant reference]
Protocols for evidence evaluation	‘The NHMRC explains the reviewers were directed to take the body of evidence and key papers used to establish the USA/Canadian dietary reference intakes and provide an analysis of any vital missing documents and/or documents which were published after the Canadian dietary reference intakes were established, using the NHMRC’s six levels of evidence.’ [ANZ]
	“Historically we have had recommended intakes in Spain, as well as food composition tables, but it was not clear how they were made (...).” [ES]
	‘In the 2006 reconsiderations the committee applied a rating system for weighing quality of scientific evidence based on CBO’s “evidence based guideline development” and SIGN grading system, in which the highest level of evidence include only systematic reviews of good quality [2 & 6, appendix C & D resp.]. Despite, there were various uncertainties related to scientific evidence.’ [NL]
	‘Systematic literature reviews will be performed to minimise potential reporting bias through comprehensive and reproducible searches using clearly defined search strategies together with clearly defined and described selections and reporting protocols. This means that the reviewing of literature will be more systematic in connection with NNR5 than it has been in earlier rounds.’ [NOC]
	‘The consultation document [Code of Practice for Scientific Advisory Committees, published by the Government Office for Science 17th Sep 2010] was circulated to members, along with a short paper “How SACN operates” to describe SACN practices relevant to the specific areas covered by the consultation questions. This was appended with a copy of the Committee’s framework for evaluation of evidence.’ [UK]

DRV, Dietary Reference Values; NHMRC, Australian National Health and Medical Research Council; ANZ, Australia and New Zealand; ES, Spain; CBO, Dutch Institute for Healthcare Improvement; SIGN, Scottish Intercollegiate Guidelines Network; NL, Netherlands; NNR5, Nordic Council of Ministers’ Nordic Nutrition Recommendations, 5th edition; NOC, Nordic countries (Denmark, Finland, Iceland, Norway and Sweden); SACN, Scientific Advisory Committee on Nutrition; UK, United Kingdom.

“Italics and double quotation marks denote interview quotes”; ‘single quotation marks denote case study/desk research text’.

To increase the transparency and scientific rigour of the evidence evaluation process, four cases (ANZ, NL, NOC, UK; Table 4) used protocols to guide evidence evaluation, address uncertainty in the data and limit interpretation bias. These evidence evaluation protocols differed slightly. Yet, all four cases provided guidance on the systematic reviewing of literature (including meta-analyses), the assessment of individual study quality/risk of bias and accounting for uncertainty when weighting or deciding the strength of the evidence base.

Table 4.

Protocols used to assess study quality/risk of bias across cases (PL/ES unknown)

	ANZ (NHMRC 2000)( 51 )	NL (CBO 2007( 52 ) and SIGN 50 2007( 53 ))	NOC (NNR5 Working Group 2011)( 54 )	UK (SACN 2008)( 55 )
Assessment of study quality/risk of bias	Quality checklist	Quality checklist	Quality checklist (total quality score)	Quality checklist
	Six evidence categories;	Six evidence categories;	Three evidence categories;
	I: RCT systematic review	A1: Systematic reviews (2 grade A2 studies+)	A: Low level of bias. Considered valid. High quality. Valid estimation of nutrient exposure. Control for confounders in design and analyses.
	II: ≥1 RCT	A2: Randomised, double-blind comparative intervention study (good quality/size)	B: Bias not sufficient to invalidate results. Do not meet all A category criteria. May be missing information, thus difficult to assess limitations
	III-1: Pseudo-RCT	B1: Systematic reviews of good quality (2 grade B2 studies+)	C: Significant bias that may invalidate the results
	III-2: Comparative studies with concurrent controls, cohort, case–control or interrupted time series	B2: Comparative (not A2), good quality cohort or patient case studies
	III-3: Comparative studies with historical control, ≥2 single arm/interrupted time series without parallel control group	C: Non-comparative studies
	IV: Case series, post/pre test	D: Opinion of the committee
Strength of evidence summary	Narrative summary	Four categories;	Four categories;	Narrative summary
		Convincing: 1 grade A1 or ≥2 grade A1	Convincing (high): Causal/absence of relationship
		Probable: 1 grade B1 or ≥2 grade B2	Probable (moderate): Causal relationship
		Insufficient: 1 grade A2 or B2 or grade C	Limited–suggestive (low): Suggestive of effect direction
		Insufficient: Based upon the committee’s opinion (grade D)	Limited–no conclusion (insufficient): Limited quantity, inconsistent effect direction

PL, Poland; ES, Spain; ANZ, Australia and New Zealand; NHMRC, Australian National Health and Medical Research Council; NL, Netherlands; CBO, Dutch Institute for Healthcare Improvement; SIGN, Scottish Intercollegiate Guidelines Network; NOC, Nordic countries (Denmark, Finland, Iceland, Norway and Sweden); NNR5, Nordic Council of Ministers’ Nordic Nutrition Recommendations, 5th edition; UK, United Kingdom; SACN, Scientific Advisory Committee on Nutrition; RCT, randomised controlled trial.

ANZ, NL and NOC used a series of checklists to assess study quality and a number of categories to differentiate between results from different study designs. The UK also utilised a series of checklists but discouraged the use of numerical grading. It was clearly acknowledged that a degree of subjective judgement was still required with the application of these protocols, especially when moving from the quality assessment of each study to collating the strength of the total evidence. Nevertheless, the following of a protocol for evidence evaluation and reporting of results was seen by these four cases as important steps towards a standardised approach to the evidence evaluation decision-making process.

Integration of evidence to develop final DRV

Interviewees across all cases acknowledged that evidence evaluation did not constitute the end of the DRV development process. Evidence evaluation findings needed to be appraised in the context of the original request to form the final DRV. Expert judgement was required to take account of the underlying certainty of the evidence, decide the strength of evidence and conclude with specific reference values. This stage of the process appeared to be less standardised with few transparent protocols available to aid the expert deliberation and consensus seeking decision-making process (Table 5).

Table 5.

Integration of evidence to develop final DRV

Final values	Quotes: case study text or interview [case study/participant reference]
Reporting of strength and uncertainty in evidence	‘Published throughout the 2006 NRV document are justifications which state how each nutritional recommendation was established. The NHMRC states that a decision should be made about what is feasible and appropriate in a given situation and the extent to which reasonable standards have been met by the available body of evidence’. [ANZ]
	‘Although the NRV are evidence-based wherever possible, the data to establish recommendations is generally very limited (…) Therefore, the Working Party had to rely on subjective professional judgements to estimate recommendations from alternative age, gender or life stage categories.’ [ANZ]
	‘NNR 2004 where there is a chapter for each micronutrient where recommendation are stated followed by discussion of the evidence and uncertainty.’ [NOC]
	“Well, maybe not all uncertainties are published. In other words, it is clear that there have been discussions, comments, questions that have not been published (…).” [ES]
	“(…) the exact discussion that takes place and the names of the committee members who said what, this is confidential.” [NL]

DRV, Dietary Reference Values; NRV, Nutrient Reference Values; NHMRC, Australian National Health and Medical Research Council; ANZ, Australia and New Zealand; NNR, Nordic Council of Ministers’ Nordic Nutrition Recommendations; NOC, Nordic countries (Denmark, Finland, Iceland, Norway and Sweden); ES, Spain; NL, Netherlands.

“Italics and double quotation marks denote interview quotes”; ‘single quotation marks denote case study/desk research text’.

The four cases which employed evidence evaluation protocols (ANZ, NL, NOC, UK) appended these protocols to reports and used them to guide descriptions of the strength and degree of certainty in the evidence underlying each DRV. In addition, prior to the publication of DRV reports, the standard practices of peer review (NL, NOC, PL) and invited or open consultation (ANZ, NL, NOC, PL, UK) provided an element of transparency and credibility to the process. Nevertheless, from the interviews it was clear that considerable discussions occurred between the evidence evaluation stage and the creation of the final DRV, even in relation to the wording or terminology used to describe DRV concepts (particularly PL) or the strength of evidence/degree of uncertainty (particularly UK). However, the details from many of these discussions were rarely communicated in the final report.

Two interviewees (ES, NL) specifically mentioned that the exact discussions undertaken by those developing DRV were not commonly reported. Reasons cited for this included a concern for the confidentiality of those who developed the recommendations as well as a belief that it was unnecessary to report every aspect of discussions that occurred during the deliberations before a consensus was reached. An exception was observed in NL when a disagreement of an advisory committee member could not be resolved during the formation of the folate DRV^{( 27 )}. Consequently, a footnote was added to the Dutch report detailing a minority position of disagreement regarding the expression of folate v. folic acid as well as folate equivalent selection.

Discussion

Transparency and openness of the DRV process was not universally adopted across countries or areas of the DRV development process studied. Implementation of transparency policies may be limited by a lack of dedicated resources and best practice procedures, particularly to assist with the latter stages of reference value development.

Findings aligned with previous research regarding transparency in risk management activities^{( 20 )}. The countries with dedicated advisory committees for risk assessment activities (ANZ, NL, NOC, PL and UK) demonstrated greater transparency of the DRV process than ES, where coordinated action at a national level was in its infancy and DRV had been developed by dedicated individuals. Arguably, NL, NOC and UK appeared to display the most transparency (also potentially ANZ, although the underpinning US/Canadian DRV setting process was not studied). Similarly to the above, this could be explained by a greater legal and clarified role of the relevant advisory committees and their responsibilities (demarcation between risk assessors, risk managers and risk communicators), a higher political priority or greater dedication of resources for nutrition and a longer tradition of DRV development. Furthermore, both NOC and UK were currently updating DRV at the time of study. Thus, the nature of transparency observed may have reflected the current increased calls for transparency and been associated with the most recent micronutrient DRV setting processes.

All cases demonstrated differences in transparency (in general and specific to the handling of uncertain evidence) across the three a priori areas studied (value request, evidence evaluation and final values). Based upon these findings, the status quo for current best practice regarding transparent and open handling of uncertainty in nutrition risk assessment has been discussed below.

Regarding the request to develop DRV, transparency was increased via documentation and accessibility to the iterative discussions between risk assessors, risk managers and others in problem formulation. These discussions acknowledged the uncertainty and limitations of the evidence base available to ensure the feasibility of the risk assessment task. This transparent detailing of the source, scope and reasons for any DRV request corresponded with guidance seen in a number of international risk assessment procedural documents (e.g. the nutrition problem formation stage before nutrition risk assessment activity in CODEX Alimentarius; defining the problem in the EURRECA framework for setting micronutrient recommendations)^{( 10 , 14 )}.

Transparency in risk assessment evidence evaluation is designed to minimise the probability of bias^{( 28 )}. Best practice for transparent handling of uncertainty in the cases studied referred to the documented protocols designed to guide and standardise the process of sourcing and interpreting evidence. Bias was reduced by employing an independent body to conduct the review and reproducibility was increased by employing standardised systematic literature review processes. Difficulties were still seen regarding the handling of uncertainty during evidence evaluation and communicating how this influenced the quantification/qualification of risk (e.g. the merits of grading different types of evidence). In addition, whether restricting a review to evidence from randomised controlled trials or meta-analyses limited the scope of the review or the utility of the final advice/reference values. However, the constant updating and disclosure of evidence evaluation procedures in the most recent cases reflected an ability to detail the overall strength of evidence reviewed.

Few protocols were available to assist with the articulation of, or guide the transition from, evidence evaluation to the integration of evidence and development of quantitative values/final DRV. This has been highlighted previously in the health-care area, where difficulties have been observed translating evidence evaluation into scientific advice for use in recommendation/guideline development^{( 29 , 30 )}. These findings support the requirement for further initiatives such as the EFSA consultation exercise in 2013 on how to conduct consistent procedures for expert elicitation during risk assessment^{( 31 )}.

Cases demonstrated that consensus seeking was an essential stage of the DRV development process, designed to increase validity and reliability of decisions rather than relying upon an individual’s judgement. However, it was difficult to identify the best practice for transparently developing quantitative DRV; values that require a certainty not always apparent in the underlying data. The production of a transparent DRV report documenting the complexity of evidence-based expert decision making, the strength of advice and the areas of uncertainty remained a challenge. This had the potential to pose subsequent problems for the interpretation of DRV by those outside the advisory committee (e.g. responsible for setting micronutrient recommendations/relevant toxicology risk assessment activities)^{( 32 – 34 )}.

The present study was based upon qualitative case studies. Therefore, the selection of micronutrients, countries and the three a priori defined areas of study may have limited external validity and the degree to which these findings can be transferred outside the sample studied. The authors recognise the study would have benefited from including analysis of other countries that develop micronutrient DRV, e.g. the USA/Canada. The exploratory nature of the present study justified the use of a qualitative design and steps were taken to limit biased interpretation. A common protocol and the three a priori defined areas of study were employed to maximise study rigour via clarity of the research goal and the consistent method of data collection, analysis and reporting. This also enabled the combination of data across countries. Construct validity was upheld by limiting the subjective collection of data, regular monthly research team meetings and performing traceable literature searches. Internal validity was upheld by encouraging several interpretations of the data to be considered. To maintain the cultural context and authenticity of the data, the majority of qualitative interpretation was conducted in the native language and information from interviews, observation or desk research provided the ability to cross-reference findings. The case studies (data collection, analysis and reporting) were led by the three a priori defined areas and did not represent the totality of the DRV development process. Nevertheless, they have provided an insight into the nature of transparency, in particular regarding the handling of uncertain data, available in an evidence-based decision-making process across multiple countries/regions.

Debate shall continue regarding the best practice for nutrition risk assessment, particularly handling uncertainty in the evidence surrounding nutrition, diet, lifestyle and health and rating the overall evidence underpinning DRV^{( 35 )}. Future discussions may focus on adaptations to the risk analysis framework to facilitate increasing requirements for nutrition risk–benefit assessment in the development of DRV^{( 26 , 36 )}. Furthermore, how to meet the challenge of incorporating developing evidence bases into risk assessment, such as those related to individual differences (e.g. metabolomics), non-randomised controlled trial study designs or whole-diet approaches (e.g. epidemiology)^{( 15 )}.

It may not be possible or necessary for all countries/regions to follow an aligned DRV risk assessment procedure nor for all reference values to be identical. However, transparency, as well as increasing the accountability and credibility of DRV development, can facilitate the sharing of best practice to inform the evolution of nutrition risk assessment. Therefore, the recent transparency initiatives from the EFSA, such as proposals to promote public access to risk assessment technical data, are to be welcomed^{( 37 )}.

Conclusion

Implementation of transparency policies may be limited by a lack of dedicated resources and best practice procedures, particularly to assist with the latter stages of reference value development. Challenges remain regarding the best practice for transparently communicating the influence of uncertain evidence on the final reference values. Resolving this issue may assist the evolution of nutrition risk assessment and better inform the recommendation setting process.