Skip to main content
. 2023 Jan 20;21(1):e07704. doi: 10.2903/j.efsa.2023.7704
Question Rating Explanation for expert judgement

1. Did the study design or analysis account for important confounding?

Key question

++

There is direct evidence that appropriate adjustments or explicit considerations were made for primary covariates and confounders in the final analyses through the use of statistical models to reduce research‐specific bias including standardisation, matching, adjustment in multivariate model, stratification, propensity scoring, or other methods that were appropriately justified. Acceptable consideration of appropriate adjustment factors includes cases when the factor is not included in the final adjustment model because the author conducted analyses that indicated it did not need to be included,

AND there is direct evidence that primary covariates and confounders were assessed using valid and reliable measurements,

AND there is direct evidence that other exposures anticipated to bias results were not present or were appropriately measured and adjusted for.

Note: This applies to:
  • Studies which characterised exposure though serum/plasma selenium and accounted for age, sex and BMI as potential confounders
  • Studies which characterised exposure though dietary questionnaire (e.g. FFQ) and accounted for age, sex, BMI as potential confounders, and total energy intake (the latter, for the purpose of reducing measurement errors)
+

There is indirect evidence that appropriate adjustments were made, OR it is deemed that not considering or only considering a partial list of covariates or confounders in the final analyses would not appreciably bias results.

AND there is evidence (direct or indirect) that primary covariates and confounders were assessed using valid and reliable measurements, OR it is deemed that the measures used would not appreciably bias results (i.e., the authors justified the validity of the measures from previously published research),

AND there is evidence (direct or indirect) that other co‐exposures anticipated to bias results were not present or were appropriately adjusted for, OR it is deemed that co‐exposures present would not appreciably bias results.

Note: This applies to studies which characterised exposure though dietary questionnaire (e.g. FFQ) and accounted for age, sex, BMI as potential confounders, but did not control for total energy intake

NR

There is insufficient information provided about the distribution of known confounders,

OR there is indirect evidence that primary covariates and confounders were assessed using measurements of unknown validity,

OR there is insufficient information provided about the measurement techniques used to assess primary covariates and confounders,

OR there is insufficient information provided about co‐exposures in occupational studies or studies of contaminated sites where high exposures to other chemical exposures would have been reasonably anticipated

There is indirect evidence that the distribution of potential confounders differed between the groups and was not appropriately adjusted for in the final analyses

OR there is indirect evidence that primary covariates and confounders were assessed using measurements of unknown validity,

OR there is indirect evidence that there was an unbalanced provision of additional co‐exposures across the primary study groups, which were not appropriately adjusted for

− −

There is direct evidence that the distribution of primary covariates and known confounders differed between the groups, confounding was demonstrated, and was not appropriately adjusted for in the final analyses,

OR there is direct evidence that primary covariates and confounders were assessed using non valid measurements,

OR there is direct evidence that there was an unbalanced provision of additional co‐exposures across the primary study groups, which were not appropriately adjusted for.

2. Were outcome data complete without attrition or exclusion from analysis? ++

There is direct evidence that loss of subjects (i.e., incomplete outcome data) was adequately addressed and reasons were documented when subjects were removed from a study.

Acceptable handling of subject attrition includes:
  • Very little missing outcome data (less than 10%, considering N of subjects included over N of subjects eligible (after exclusion of prevalent cases, participants with missing variables) for the analysis);
  • Reasons for missing subjects unlikely to be related to outcome (for survival data, censoring unlikely to be introducing bias);
  • Missing outcome data balanced in numbers across study groups, with similar reasons for missing data across groups (i.e. unlikely to be related to exposure),

OR missing data have been imputed using appropriate methods and characteristics of subjects lost to follow up or with unavailable records are described in identical way and are not significantly different from those of the study participants.

+

There is indirect evidence that loss of subjects (i.e., incomplete outcome data) was adequately addressed and reasons were documented when subjects were removed from a study,

OR it is deemed that the proportion lost to follow‐up would not appreciably bias results (i.e. losses are not expected to be related to both exposure and outcome, considering N of subjects included over N of subjects eligible for the analysis). This would include reports of no statistical differences in characteristics of subjects lost to follow up or with unavailable records from those of the study participants.

Generally, the higher the ratio of participants with missing data to participants with events, the greater potential there is for bias.

For studies with a long duration of follow‐up, some withdrawals for such reasons are inevitable.

NR There is insufficient information provided about numbers of subjects lost to follow‐up
There is indirect evidence that loss of subjects (i.e., incomplete outcome data) was unacceptably large (greater than 20% in each group, Genaidy et al., 2007) and not adequately addressed
− −

There is direct evidence that loss of subjects (i.e., incomplete outcome data) was unacceptably large and not adequately addressed.

Unacceptable handling of subject attrition includes:
  • Reason for missing outcome data likely to be related to true outcome, with either imbalance in numbers or reasons for missing data across study groups (i.e. likely to be related to the exposure);
  • Or potentially inappropriate application of imputation.
3. Did selection of study participants result in appropriate comparison groups? ++

Cohort studies: There is direct evidence that subjects (both exposed and non‐exposed) were similar (e.g., recruited from the same eligible population, recruited with the same method of ascertainment using the same inclusion and exclusion criteria, and were of similar age and health status), recruited within the same time frame

Case–control studies: There is direct evidence that cases and controls were similar (e.g., recruited from the same eligible population including being of similar age, gender, ethnicity, and eligibility criteria other than outcome of interest as appropriate), recruited within the same time frame, and controls are described as having no history of the outcome.

Note: A study will be considered low risk of bias if baseline characteristics of groups differed but these differences were considered as potential confounding or stratification variables

+

Cohort studies: There is indirect evidence that subjects (both exposed and non‐exposed) were similar (e.g., recruited from the same eligible population, recruited with the same method of ascertainment using the same inclusion and exclusion criteria, and were of similar age and health status), recruited within the same time frame

OR differences between groups would not appreciably bias results.

Case–control studies: There is indirect evidence that cases and controls were similar (e.g., recruited from the same eligible population, recruited with the same method of ascertainment using the same inclusion and exclusion criteria, and were of similar age), recruited within the same time frame, and controls are described as having no history of the outcome,

OR differences between cases and controls would not appreciably bias results

NR

Cohort studies: there is insufficient information provided about the comparison group

Case–control studies: there is insufficient information provided about the appropriateness of controls

Cohort studies: There is indirect evidence that subjects (both exposed and non‐exposed) were not similar, recruited within very different time frames

Case–control studies: There is direct evidence that controls were drawn from a very dissimilar population than cases or recruited within very different time frames,

Note: A study will be considered low risk of bias if baseline characteristics of groups differed but these differences were considered as potential confounding or stratification variables

− −

Cohort studies: There is direct evidence that subjects (both exposed and non‐exposed) were not similar, recruited within very different time frames

Case–control studies: There is direct evidence that controls were drawn from a very dissimilar population than cases or recruited within very different time frames.

Note: A study will be considered low risk of bias if baseline characteristics of groups differed but these differences were considered as potential confounding or stratification variables

4. Can we be confident in the exposure characterisation?

Key question

++

There is direct evidence that exposure was consistently assessed (i.e., under the same method and time‐frame) using well‐established methods that directly measure exposure (e.g., measurement of the chemical in air or measurement of the chemical in blood, plasma, urine, etc.),

OR exposure was assessed using less‐established methods that directly measure exposure and are validated against well‐established methods.

Note: This applies to studies which characterised Se exposure through repeated measurements of plasma/serum selenium.

Graphite furnace atomic absorption spectrometric method and inductively coupled plasma tandem mass spectrometry are considered reliable methods

+

There is indirect evidence that the exposure was consistently assessed using well‐established methods that directly measure exposure,

OR exposure was assessed using indirect measures (e.g., questionnaire or occupational exposure assessment by a certified industrial hygienist) that have been validated or empirically shown to be consistent with methods that directly measure exposure (i.e., inter‐methods validation: one method vs. another).

Note: This applies to studies which characterised exposure through:
  • A single measurement of plasma/serum selenium at baseline
  • A semi‐quantitative FFQ and food composition data which are representative of the food supply of the study participants; i.e: lack of validation of the FFQ for selenium intake is deemed not to affect the result substantially
  • Toenails concentration with measures applied to avoid contamination of the samples
NR

There is insufficient information provided about the exposure assessment, including validity and reliability, but no evidence for concern about the method used

Note: This applies to studies which characterised exposure through toenails concentration but do not describe measures applied to avoid contamination of the samples

There is indirect evidence that the exposure was assessed using poorly validated methods that directly measure exposure,

OR there is direct evidence that the exposure was assessed using indirect measures that have not been validated or empirically shown to be consistent with methods that directly measure exposure (e.g., a job‐exposure matrix or self‐report without validation)

− −

There is direct evidence that the exposure was assessed using methods with poor validity,

OR evidence of exposure misclassification (e.g., differential recall of self‐reported exposure).

5. Can we be confident in the outcome assessment?

Key question

++

There is direct evidence that the outcome was assessed using well‐established methods

AND subjects had been followed for the same length of time in all study groups. Acceptable assessment methods will depend on the outcome, but examples of such methods may include: objectively measured with diagnostic methods, measured by trained interviewers, obtained from registries (Shamliyan et al. 2010),

AND there is direct evidence that the outcome assessors (including study subjects, if outcomes were self‐reported) were adequately blinded to the study group, and it is unlikely that they could have broken the blinding prior to reporting outcomes.

Note: This applies to studies in which incident cases of T2DM were identified based on systematic clinical screening of the participants (i.e. measures of fasting glucose, glucose at 2 h during OGTT, glycated haemoglobin for diagnostic purposes). Diagnostic based on single fasting glucose measurement is considered acceptable.

+

There is indirect evidence that the outcome was assessed using acceptable methods (i.e., deemed valid and reliable but not the gold standard)

AND subjects had been followed for the same length of time in all study groups [Acceptable, but not ideal assessment methods will depend on the outcome, but examples of such methods may include proxy reporting of outcomes and mining of data collected for other purposes]

OR it is deemed that the outcome assessment methods used would not appreciably bias results,

AND there is indirect evidence that the outcome assessors (including study subjects, if outcomes were self‐reported) were adequately blinded to the study group, and it is unlikely that they could have broken the blinding prior to reporting outcomes,

OR it is deemed that lack of adequate blinding of outcome assessors would not appreciably bias results, which is more likely to apply to objective outcome measures

Note: This applies to studies in which diagnosis of T2DM was based on self‐reporting (e.g. self‐report of a diabetes diagnosis or use of diabetes medication)
  • With/without confirmation of diagnosis based on medical records or validated questionnaire
  • With/without use of hospital discharge databases, prescription drug databases as complementary source of information

Differential underreporting of T2DM in ‘high’ selenium exposure groups is considered unlikely. Non‐differentiable underreporting across exposure groups is not expected to bias rate ratios.

NR There is insufficient information provided about blinding of outcome assessors

There is indirect evidence that the outcome assessment method is an insensitive instrument (e.g., a questionnaire used to assess outcomes with no information on validation),

OR the length of follow up differed by study group,

OR there is indirect evidence that it was possible for outcome assessors (including study subjects if outcomes were self‐reported) to infer the study group prior to reporting outcomes

− −

There is direct evidence that the outcome assessment method is an insensitive instrument,

OR the length of follow up differed by study group,

OR there is direct evidence for lack of adequate blinding of outcome assessors (including study subjects if outcomes were self‐reported), including no blinding or incomplete blinding.

++: Definitely low RoB; +: probably low RoB; NR: not reported; −: probably high RoB; − −: definitely high RoB.