Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Apr 1.
Published in final edited form as: J Acad Nutr Diet. 2015 Feb 12;115(4):519–527.e2. doi: 10.1016/j.jand.2014.12.016

Reproducibility and Intermethod Reliability of a Calcium Food Frequency Questionnaire for use in Hispanic, Non-Hispanic Black, and Non-Hispanic White Youth

Nicholas J Ollberding 1, Vicente Gilsanz 2, Joan M Lappe 3, Sharon E Oberfield 4, John A Shepherd 5, Karen K Winer 6, Babette S Zemel 7, Heidi J Kalkwarf 8
PMCID: PMC4380535  NIHMSID: NIHMS652052  PMID: 25683819

Abstract

Background

A dietary assessment instrument designed for use in a nationally representative pediatric population was required to examine associations between calcium intake and bone mineral accrual in a large, multi-center study.

Objective

To determine the reproducibility and intermethod reliability of a youth calcium food frequency questionnaire (FFQ) in a multiracial/ethnic sample of children and adolescents.

Design

Reproducibility (n=69) and intermethod reliability (n=393) studies were conducted by administering repeat FFQs and three unannounced 24-h dietary recalls to stratified random samples of individuals participating in the Bone Mineral Density in Childhood Study (BMDCS).

Participants/setting

Children and adolescents ages 5–21 years.

Main outcome measures

Calcium intake estimated from the FFQ and 24-h dietary recalls.

Statistical analysis

Reproducibility was assessed by the intraclass correlation coefficient (ICC). Intermethod reliability was assessed by deattenuated Pearson correlations between the FFQ and 24-h recalls. Attenuation factors and calibration corrected effect estimates for bone density were calculated to determine the potential impact of measurement error on associations with health outcomes.

Results

The ICC (ICC=.61) for repeat administrations and deattenuated Pearson correlation between the FFQ and 24-h recalls (r=.60) for all subjects indicated reproducibility and intermethod reliability (Pearson r = 0.50–0.74 across sex and age groups). Attenuation factors were ≤ 0.50 for all sex and age groups and lower for non-Hispanic blacks (λ = 0.20) and Hispanics (λ = 0.26) than for non-Hispanic whites (λ = 0.42).

Conclusions

The BMDCS calcium FFQ appears to provide a useful tool for assessing calcium intake in children and adolescents drawn from multiracial/ethnic populations and/or spanning a wide age range. However, similar to other FFQs, attenuation factors were substantially less than one indicating the potential for appreciable measurement error bias. Calibration correction should be performed and racial/ethnic differences in performance considered when analyzing and interpreting findings based on this instrument.

Keywords: diet, questionnaires, calibration, diet recall, nutrition assessment

INTRODUCTION

Food frequency questionnaires (FFQ) have been used extensively to obtain information on usual intake of foods and nutrients, including calcium, and to examine estimates of calcium intake in relation to health outcomes.1–6 Advantages of FFQs for measuring dietary calcium intake, especially in large studies, include low cost, ease of administration, and focus on usual consumption of calcium-containing foods and supplements. Despite these strengths, nutrient intakes estimated from FFQs are subject to measurement error due to the limited number of food items queried, reliance on generic memory, potential reporting biases, and inaccuracies in food and nutrient databases. To properly interpret findings from a newly developed FFQ, reproducibility (intramethod reliability) and validation studies must be conducted. Ideally, the extent of error is determined from objective measures of true dietary intake; however, such measures are rare and confined to a select few recovery biomarkers.7 Therefore, intermethod reliability studies are generally conducted that compare the performance of the new instrument with an accepted method. These studies are often referred to as relative validation studies, although, an accurate measure of dietary intake is required to determine validity and therefore the term intermethod reliability is preferable.8 Multiple, unannounced 24-h dietary recalls or dietary records are commonly used to assess intermethod reliability under the assumption that these methods provide an unbiased measure of dietary intake for a single day.9–12 Regression calibration provides a means to correct for measurement error in the primary instrument (FFQ) and to obtain relatively unbiased measures of association between a food or nutrient and health outcome when the assumptions for proper reference instrument are met.10,13,14

The Bone Mineral Density in Childhood Study (BMDCS) was a large, national cohort of U.S. children and adolescents designed to develop reference data for bone mass and density and to investigate the determinants of bone accretion.15,16 To examine associations between calcium intake and bone mineral accrual in this cohort a dietary assessment instrument designed for use in a nationally representative pediatric population was required. Therefore, a FFQ developed from dietary data collected as part of the third National Health and Nutrition Examination Study (NHANES III) and tailored for pediatric use was designed by NutritionQuest (Berkeley, CA) and used as the primary instrument to assess calcium intake. The purpose of this study was to examine the reproducibility and intermethod reliability of the BMDCS calcium FFQ for estimating usual dietary calcium intake in representative subsets of children and adolescents participating in the BMDCS. In addition, regression calibration was performed to assess potential measurement error bias when using the BMDCS calcium FFQ and to obtain correction factors that may be used to inform future studies. An example is provided investigating the impact of measurement error when estimating associations of calcium intake with bone mineral content (BMC) and areal bone mineral density (aBMD) at the spine to highlight the extent to which measurement error may impact effect estimates (i.e. regression coefficients) and whether calibration correction may provide results more in line with calcium supplementation trials.17–19

MATERIALS AND METHODS

Study population

The BMDCS was comprised of approximately 2,000 participants recruited from five clinical centers in the U.S.: Children’s Hospital of Los Angeles (Los Angeles, CA), Cincinnati Children’s Hospital Medical Center (Cincinnati, OH), Creighton University (Omaha, NE), Children’s Hospital of Philadelphia (Philadelphia, PA), and Columbia University (New York, NY). The initial cohort consisted of a multiracial/ethnic sample of girls 6–15 years of age and boys 6–16 years of age who were enrolled between July 2002 and November 2003 and followed annually for six years.15 A second recruitment period, including individuals 5 and 19 years of age, occurred between August 2006 and November 2007 to increase the number of younger and older participants.16 Healthy, normally developing children and adolescents were enrolled into the study. Participant recruitment and enrollment criteria have been described previously.15,16 Informed consent was obtained from participants 18 years of age or older. Informed consent was obtained from the participant’s parent or guardian and assent obtained from the participant for those less than 18 years of age. The study protocol was approved by the institutional review boards of all five centers.

Two substudies were conducted (reproducibility and intermethod reliability), each using a stratified, random sample of BMDCS participants 5–21 years of age at enrollment into the substudies. The goal of the reproducibility substudy was to assess the test-retest performance of the FFQ. Recruitment of participants (n = 69) occurred from 2006–07 by randomly selecting participants within strata defined by sex, age (≤ 13 y and > 13 y), and center. The first FFQ (FFQ1) was administered during the visit. The questionnaire was completed by study participants > 13 years of age and by the parent and child together for those ≤ 13 years of age. The second FFQ (FFQ2) was mailed to participants one to two weeks after the study visit and returned within one month. Visual aids including pictures with glasses and bowls and lines indicating volumes were provided to aid participants in assessing food quantity.

The goal of the intermethod reliability substudy was to assess the accuracy of the FFQ in relation to three unannounced, interviewer administered 24-h dietary recalls. Recruitment of participants (n = 430) occurred from 2006–09. Participants were randomly selected within strata defined by sex, age (≤ 13 y and > 13 y), and center. The sampling frame provided a subsample generally representative of the BMDCS cohort. The FFQ was administered during a clinical visit occurring from 2006–09. The three unannounced 24-h recalls were conducted via telephone by study dietitian within one month of completing the FFQ. Visual aids including pictures with glasses and bowls and lines indicating volumes were provided to assist in portion size estimation. The FFQ and 24-h recalls were completed by study participants > 13 years of age and by the parent and child together for those ≤ 13 years of age. A total of 393 participants completed all three recalls and were included in the present analyses. Compensation of $10 was provided for participation in a substudy.

Development of the FFQ

The BMDCS FFQ was designed to measure food items contributing calcium to the diets of children and adolescents over the previous seven days. The FFQ was developed by NutritionQuest (Berkeley, CA) in accordance with procedures for the Block dietary questionnaires and screeners.20–22 The initial list of calcium-containing foods was obtained from 24-h recalls collected as part of the NHANES III.23 Individual food items were then aggregated into categories based upon similarities in food type, nutrient content, and culinary usage, and rank-ordered on contribution to total calcium intake. The minimum set of food categories that could capture ≥ 90% of calcium intake in each of nine pre-specified age-race/ethnicity groups (6–10 y, 11–13 y, 14–19 y; Hispanic, non-Hispanic black, non-Hispanic white) were included on the FFQ. The final questionnaire included queries on 45 distinct food and beverage categories (Supplemental Table 1). Five responses were provided to assess how often the foods and beverages were consumed over the preceding week (1 day/week, 2 days/week, 3–4 days/week, 5–6 days/week, everyday). In addition, four portion size options were provided for each category to capture usual amounts consumed. Dietary calcium intake (mg/d) was estimated from the questionnaire using food composition tables developed and maintained by NutritionQuest. For the present analysis, instrument performance for calcium intake from foods alone was examined, as quantitative information on supplement use for participants in the substudies was not collected on the FFQ. The CaFFQ used in the BMDCS is available from NutritionQuest (http://nutritionquest.com/).

Administration of the 24-h recalls

Three non-consecutive, unannounced 24-h recalls were conducted via telephone by trained dietitians from Cincinnati Children’s Hospital Medical Center (CCHMC) and Children’s Hospital of Philadelphia (CHOP). CCHMC was responsible for interviews with participants in Cincinnati, Creighton, and the male participants in Los Angeles. CHOP was responsible for interviews with participants in Philadelphia, New York, and the female participants in Los Angeles. The dietitian asked the participant about the foods and beverages consumed over the previous 24-h employing a multiple pass method to enhance detail and limit omissions. Quantities were expressed in terms of typical bowls, cups, etc. Study participants were encouraged to consult the food-model pamphlet, distributed at the study visit, to assist in assessing portion sizes. The recalls were planned to occur on two weekdays and one weekend day. The recalls were collected and analyzed using the Nutrition Data System for Research (NDSR) software versions 2006–2009, developed by the Nutrition Coordinating Center, University of Minnesota.

Bone densitometry

Dual-energy x-ray absorptiometry (DXA) scans were performed using a dedicated Hologic, Inc. (Bedford, MA) bone densitometer (QDR4500A, QDR4500W, Apex, and Delphi A models) at each clinical center. Scans were performed for each participant according to the manufacturer’s guidelines to obtain spine BMC and aBMD. Calibration stability across sites and over time was maintained using traveling and site-specific phantoms as previously described.15 All scans were analyzed by the DXA Core Laboratory at the University of California, San Francisco using Hologic software release 12.3 to obtain BMC and aBMD values for each participant. Data from the lumbar spine scans were used in the present analysis as they have shown associations with calcium intake in the BMDCS cohort24, and the lumbar spine is a recommended site for pediatric bone health assessment.25

Additional measures

Weight (0.1 kg) was measured on a digital scale and height (0.1 cm) was measured using a stadiometer. Sexual maturation was assessed by physical exam of testes volume (males) and breast size (females) by a pediatric endocrinologist or nurse practitioner. Sexual maturity was defined according to the Tanner criteria. Usual weight-bearing physical activity (PA, hours per week) was assessed with the modified self-report tool.26 The questionnaire listed 37 weight-bearing activities, such as walking, basketball, dance, etc., in which children and adolescents are likely to participate. Participants indicated the time spent in activities in which they participated during the previous week.

Statistical analysis

Reproducibility

The mean difference and 95% confidence interval (CI) for calcium intake estimated from FFQ1 and FFQ2 and the intraclass correlation (ICC) were used to assess reproducibility (intramethod reliability). High ICC values, denoting low within-subject variation, reflect a high degree of reproducibility. Results were calculated for all participants and for younger (5–13 y) and older (14–21 y) participants separately. ICCs of ≥ 0.6 were considered a priori to reflect appreciable reproducibility.

Intermethod reliability

The mean difference and 95% confidence interval (CI) for calcium intake estimated from the average of three 24-h recalls and the FFQ, deattenuated Pearson correlations, and attenuation factors were used assess intermethod reliability. Deattenuation corrects for random measurement error (i.e., day-to-day variability in the 24-h recalls) in the observed correlation.7 To calculate the deattenuated Pearson correlation (rt), the following formula was used: rt = ro √(1 + [S2w/S2b]/nx) where ro is the observed correlation, S2w/S2b is ratio of the within- to between-subject variances in calcium intake estimated from repeat measures, and nx is the number of 24-h recalls. Deattenuated correlations in the range of r=0.4–0.7 were considered a priori to reflect intermethod reliability.9,27 Attenuation factors, calculated as the slope of the regression of the average of the three 24-h recalls on the FFQ, provide a measure of the expected bias due to measurement error in the FFQ.28 The attenuation factor (λ) is a multiplicative scalar that operates on the true value biasing it to the extent denoted by λ; where, for example, true value = observed value/λ. Attenuation factors closer to 1 indicate minimal bias; whereas, lower values denote greater expected attenuation when estimating associations between dietary intake and health parameters. Calcium intakes were natural log transformed where necessary to better meet model assumptions. Results were obtained for all participants and for predefined sex-age and race/ethnic groups.

Calibration correction

Uncorrected (referred to herein as naïve) and calibration corrected multiple linear regression was used to examine associations between usual calcium intake and BMC and aBMD at the spine adjusted for age (continuous), sex, race (black vs. non-black), height (continuous), total physical activity (continuous), tanner stage, and clinical center. Calibration was performed using the method described by Spiegelman et al.29 for linear regression. Calcium intake assessed via the FFQ was assumed to be measured with error and the average of the three 24-h recalls was used as the reference instrument. All covariates were assumed to be measured without error and all regressions were performed on the log-log scale to better meet model assumptions.

RESULTS

The mean age of participants in the BMDCS reproducibility and intermethod reliability substudies was 14.5 y (min=5 y, max=21 y) and 13.1 y (min=5 y, max=21 y), respectively at the time of data collection (Table 1). For the reproducibility subsample, non-Hispanic whites comprised the largest proportion of participants (52%), followed by Hispanics, non-Hispanic blacks, and Asian Americans with the majority of participants at Tanner stage 4 or 5. For the intermethod reliability subsample, non-Hispanic whites comprised the largest proportion of participants (49%), followed by non-Hispanic blacks, Hispanics, and Asian Americans. The greatest proportions of participants were Tanner stage 1 and 5 with fewer at intermediate stages. Approximately 45% of participants in both samples were male with mean heights of 159 cm and 151 cm in the reproducibility and intermethod reliability subsamples, respectively.

Table 1.

Characteristics of participants in the Bone Mineral Density in Childhood Study (BMDCS) calcium food frequency questionnaire reproducibility and intermethod reliability substudies

Characteristic Reproducibility (n=69) substudy Intermethod reliability (n=393) substudy
 Age y, mean (SD) 14.5 (3.9) 13.1 (5.2)
 Males, n (%) 31 (44.9) 184 (46.8)
 Race/ethnicity, n (%)
  Asian 8 (11.6) 20 (5.1)
  Non-Hispanic black 10 (14.5) 81 (20.6)
  Hispanic 12 (17.4) 68 (17.3)
  Non-Hispanic white 36 (52.2) 194 (49.4)
  Other 3 (4.4) 30 (7.6)
 Height cm, mean (SD) 158.5 (17.8) 150.5 (22.3)
 Tanner stage, n (%)
  1 8 (11.6) 125 (31.8)
  2 4 (5.8) 12 (3.1)
  3 10 (14.5) 18 (4.6)
  4 12 (17.4) 38 (9.7)
  5 33 (47.8) 170 (43.3)
  Missing 2 (2.9) 30 (7.6)
Open in a new tab

Abbreviations: SD, standard deviation.

Note: Values may not sum to total due to missing data.

Reproducibility

The mean difference in calcium intake estimated with repeat administrations of the FFQ (FFQ1 - FFQ2) was 17 mg/d (95% CI: −89, 123) (Table 2). The intraclass correlation for calcium intake (log scale) indicated reproducibility of the FFQ (ICC = 0.61). In analyses stratified on age, there was no difference in mean calcium intake upon repeat administration in both age groups; however, the intraclass correlation was lower for younger (5–13 y) when compared to older (14–21 y) participants. The low correlation for younger children was highly influenced by one outlying value (Cook’s D = 2.2) and improved to ICC = 0.52 upon removal.

Table 2.

Reproducibility of calcium intake for repeat administrations of the BMDCS calcium FFQ

FFQ1
Median (25th, 75th)		 FFQ2
Median (25th, 75th)		 Difference FFQ1 - FFQ2
Mean (95% CI)		 ICCa
Calcium (mg/d)
 All subjects (n=69) 655 (485, 883) 685 (417, 979) 17 (−89, 123) 0.61
 Ages 5–13 y (n=31) 744 (474, 909) 826 (557, 984) −87 (−235, 61) 0.31
 Ages 14–21 y (n=38) 636 (484, 739) 539 (398, 907) 102 (−48, 252) 0.75
Open in a new tab

Abbreviations: CI, confidence interval; ICC, intraclass correlation coefficient; FFQ, food frequency questionnaire; BMDCS, Bone Mineral Density in Childhood Study.

Note: FFQ2 was completed and returned ~ 1 month after FFQ1.

a

Calculated for the natural log transformed values.

Intermethod reliability

The FFQ underestimated calcium intake relative to the average of three 24-h recalls (Table 3). The mean difference for calciumrecalls – calciumFFQ was 72 mg/d (95% CI; 12, 131) for all subjects indicating a systematic bias at the group level as expected based on the FFQ development protocol. Confidence intervals for the mean difference between the FFQ and 24-h recalls were wide for all age-sex and race sub-groups, but reached statistical significance for all females, females 10–14 y, females 15–21 y, and non-Hispanic whites. The deattenuated Pearson correlation coefficient for the FFQ and average of three 24-h recalls was r = 0.60 for all subjects indicating intermethod reliability and ranged from r = 0.50 to r = 0.74 across sex and age groups. Correlations were modestly higher for females than for males and for participants 5–7 years of age. In race/ethnic-specific analyses, the correlations were lowest for non-Hispanic blacks (r = 0.48) followed by Hispanics (r = 0.54) and non-Hispanic whites (r = 0.62). The attenuation factor for the FFQ relative to the average of three 24-h recalls was λ = 0.36 for all subjects indicating the extent to which associations between calcium intake estimated from the FFQ and health outcomes may be biased towards the null in univariate analyses. Attenuation factors were ≤ 0.50 for all sex and age groups and λ = 0.20 for non-Hispanic blacks, λ = 0.26 for Hispanics, and λ = 0.42 for non-Hispanic whites.

Table 3.

Intermethod reliability of the BMDCS calcium FFQ to measure calcium intake when compared to the average of three 24-h diet recalls

Calcium (mg/d) Diet Recalls
Median (25th, 75th)		 FFQ
Median (25th, 75th)		 Difference recalls - FFQ
Mean (95% CI)		 Deattenuated Pearson rac Attenuation factor (λ)bc
All subjects (n=393) 871 (652, 1160) 773 (520, 1145) 72 (12, 131) 0.60 0.36
 Males (n=184) 924 (741, 1327) 891 (624, 1269) 39 (−70, 149) 0.55 0.35
  Ages 5–7 y (n=56) 940 (756, 1112) 997 (689 1188) 0 (−100, 100) 0.74 0.37
  Ages 10–14 y (n=62) 1000 (812, 1330) 917 (669, 1262) 2 (−133, 138) 0.54 0.29
  Ages 15–21 y (n=66) 845 (656, 1424) 776 (553, 1352) 108 (−161, 376) 0.50 0.37
 Females (n=209) 831 (605, 1116) 650 (458, 993) 100 (42, 157) 0.61 0.34
  Ages 5–7 y (n=57) 916 (637, 1157) 821 (542, 1127) 59 (−26, 145) 0.69 0.50
  Ages 10–14 y (n=73) 874 (671, 1111) 639 (491, 964) 126 (20, 232) 0.54 0.27
  Ages 15–21 y (n=79) 743 (538, 1020) 593 (385, 905) 105 (3, 207) 0.61 0.30
 Black (n=81) 778 (633, 900) 706 (478, 925) 28 (−84, 140) 0.48 0.20
 Hispanic (n=68) 835 (602, 1132) 700 (467, 1075) 42 (−99, 183) 0.54 0.26
 White (n=194) 963 (744, 1280) 872 (557, 1172) 113 (21, 205) 0.62 0.42
Open in a new tab

Abbreviations: CI, confidence interval; FFQ, food frequency questionnaire; ln, natural log; ro, observed Pearson correlation; BMDCS, Bone Mineral Density in Childhood Study.

Notes: All values for the 24-h diet recall are for the three day average.

a

Deattenuated for random measurement error in the 24-h diet recalls: ro √(1 + [S2w/S2b]/3).

b

λ = slope of the regression of the average of the three 24-h diet recalls on the FFQ.

c

Calculated for the natural log transformed values.

Calibration correction

Uncorrected and calibration corrected estimates for the association of calcium intake with BMC and aBMD at the spine are provided in Table 4. There were no statistically significant associations overall, or for non-Hispanic black or Hispanic participants; however, the uncorrected results were biased towards no association to the degree expected by the attenuation factors. For non-Hispanic whites, in uncorrected models, a 10% increase in calcium intake was associated with a 0.54% increase in BMC (p = 0.02) and a 0.41% increase in aBMD (p = 0.03). Adjusted for measurement error, a 10% increase in calcium intake was associated with a 1.52% increase in BMC (p = 0.03) and a 1.16% increase in aBMD (p = 0.04). In our sample, an increase in calcium consumption of roughly 300 mg/day, equivalent to a cup of milk or the low range of supplemental calcium provided in previous trials, corresponded to an approximate increase of 40% in daily intake. In calibration corrected models this translates into an approximate increase of 2.13% and 1.62% in BMC and aBMD, respectively when estimated within the range of the data among non-Hispanic whites. Overall, attenuation factors obtained from the multivariable models were generally similar to those obtained for the univariate model presented in Table 3 (Supplemental Table 2; data shown for all subjects only).

Table 4.

Associations of calcium intake with bone mineral content and areal bone mineral density at the spine for participants in the BMDCS intermethod reliability substudy

Bone Mineral Content Bone Mineral Density
β (SE) p-value β (SE) p-value
All subjects (n = 358)
 FFQ
  Uncorrected 0.016 0.014 0.27 0.011 0.012 0.36
  Calibration correcteda 0.051 0.047 0.28 0.035 0.038 0.36
Black (n = 76)
 FFQ
  Uncorrected 0.018 0.030 0.55 0.001 0.025 0.97
  Calibration correcteda 0.110 0.192 0.57 0.006 0.153 0.97
Hispanic (n = 64)
 FFQ
  Uncorrected −0.055 0.028 0.05 −0.032 0.022 0.15
  Calibration correcteda −0.228 0.144 0.11 −0.134 0.105 0.20
White (n = 175)
 FFQ
  Uncorrected 0.054 0.023 0.02 0.041 0.019 0.03
  Calibration correcteda 0.152 0.069 0.03 0.116 0.057 0.04
Open in a new tab

Abbreviations: SE, standard error; BMDCS, Bone Mineral Density in Childhood Study; FFQ, food frequency questionnaire.

Notes: Associations for natural log-log models. All models adjusted for age (continuous), sex, race (black vs. non-black), height (continuous), total physical activity (continuous), tanner stage, and clinical center in the calibration and health parameter models where appropriate. Subjects who refused Tanner staging (n=30) or did not undergo bone mineral content/density testing (n=5) were excluded from analyses.

a

Calibration correction performed using the method described by Spiegelman et al.27 for linear regression.

DISCUSSION

These data indicate that the BMDCS calcium FFQ provides reasonably reproducible and reliable estimates of calcium intake in representative samples of children and adolescents participating in the BMDCS when compared to the average of three 24-h dietary recalls. Calcium intake was generally underestimated by the FFQ relative to the 24-h recalls; however, correlational analyses demonstrated deattenuated associations consistent with our a priori criterion for intermethod reliability. Despite reasonable agreement between instruments, attenuation factors were ≤ 0.50 across all subgroups highlighting the potential for substantial measurement error bias when estimating associations between calcium intake and health outcomes. Nutrition professionals utilizing the BMDCS calcium FFQ are therefore recommended to obtain calibration corrected estimates, in addition to naïve estimates that assume no measurement error, when assessing associations between calcium intake and health outcomes to better understand the extent to which measurement error may have impacted findings.

The BMDCS FFQ was able to reproducibly measure dietary calcium intake. Previous research assessing the reproducibility of both paper and web-based FFQs designed specifically to measure dietary calcium intake in children and adolescents have reported correlations for test-retest performance ranging from r=0.68–0.79.6,30–32 The modestly weaker correlation in our sample was likely due to differences in the time frame queried by the FFQ. Unlike previous FFQs designed to assess calcium intake over the preceding month, the BMDCS calcium FFQ queries intake occurring over the preceding week. This decision was made to limit cognitive difficulties in estimating the usual frequency and quantity of foods items consumed over longer time spans in an attempt to improve accuracy. Thus, the repeat administrations in our study do not reflect dietary intake occurring over the same time period with an expectant reduction in the ICC due to incorporating true within-person variation in calcium intake. The reason for the discrepancy in the ICC for younger (5–13 y) when compared to older (14–20 y) participants is unknown, but could reflect a more varied diet of calcium containing foods in younger participants or simply sampling variation. Differences in test-retest performance according to age in previous reports have been inconsistent.6,30,31

Compared to the average of three unannounced 24-h recalls, the BMDCS FFQ underestimated dietary calcium intake on average. This was expected as the FFQ was designed to capture ≥ 90% of calcium intake in select age-race/ethnicity groups based upon 24-h recall data. Previous studies have reported lower point estimates for mean calcium intake estimated from a short 10-item FFQ relative to the average from 24-h recalls31 and higher mean calcium intake estimated from a 41-item FFQ relative to a seven-day weighted food record.6 Numerous factors influence the assessment of absolute calcium intake on a FFQ including the length of the questionnaire, food list, response options, nutrient calculations, and subject instructions7 and must be considered along with the study objectives and participant burden when designing the questionnaire. The deattenuated correlation for the BMDCS FFQ and 24-h recalls of r=.60 was similar to, or better than, correlations with reference measures for other FFQs designed to measure calcium intake in children and adolescent in the U.S.6,30–33 and other countries34–38; despite the wide age range and multiracial/ethnic sample. This highlights the potential utility of this instrument for assessing calcium intake in future studies drawn from multiracial/ethnic populations or those spanning a wide age range. The higher correlation for females than for males in our sample has not been consistently observed in previous reports, and the lack of uniformity across age groups is in agreement with earlier studies.6,30,31 A noteworthy finding was that despite developing the FFQ from items providing the greatest contribution to dietary calcium intake in Hispanic, non-Hispanic black, non-Hispanic white children and adolescents, correlations between the FFQ and 24-h recalls were lower for non-Hispanic blacks and Hispanics than for non-Hispanic whites. Similar differences have been reported previously.6 The reason for this discrepancy is unknown and highlights unresolved challenges in culturally tailoring aspects of the questionnaire beyond the list of foods to elicit more accurate responses in specific racial/ethnic groups.

Attenuation factors obtained from the slope of the regression of the 24-h recalls on the FFQ indicated the potential for appreciable measurement error bias when estimating associations between calcium intake and health outcomes. Examples for the association of dietary calcium intake with BMC and aBMD at the spine demonstrated naïve associations were attenuated by approximately two-thirds relative to calibration corrected results. Measurement error of this magnitude has the potential to severely distort associations between calcium intake and health outcomes with the direction of the bias uncertain in multivariable statistical models.28,39,40 Furthermore, attenuation factors were lower for Hispanics and non-Hispanic blacks than for non-Hispanic whites indicating that naïve effect estimates may be more biased for these groups. Measurement error also impacts study power; reducing effective sample size by a factor inversely proportional to λ2.28 Thus, the ability to detect statistically significant associations in Hispanics and non-Hispanics blacks, relative to non-Hispanic whites, would also be reduced. Calcium supplementation trials in children and adolescents have generally reported modest increases (1%–6%) in BMC and BMD during active supplementation, although it is unclear whether improvements are confined to children with low baseline levels or persist beyond the end of supplementation.17–19 In calibration corrected models, a ~300 mg/day increase in calcium intake was related to a 2.11% and 1.64% increase in BMC and aBMD in non-Hispanic whites; more in line with results obtained from calcium supplementation trials. Calibration correction using multiple 24-h recalls as the reference instrument may allow for more accurate results, as effect estimates have been, in part, corrected for measurement error. A measure of true usual calcium intake, however, would be required to eliminate bias due to measurement error. While traditional regression calibration provides a means to correct naïve effect estimates (e.g. regression coefficients), it does not recapture lost study power, improve tests for statistical significance, and is impacted by error in the assessment of model covariates.

Regression calibration also relies on the assumptions that the error in the reference instrument is unbiased and contains only within-person error that is uncorrelated with errors in the primary instrument.9 Comparisons of self-report instruments with recovery biomarkers have demonstrated both intake-related and correlated person-specific biases violating both assumptions of a proper reference instrument.13,14 To the extent that measurement error in the 24-h dietary recall method is associated with error in the BMDCS FFQ, correlational analyses (i.e. intermethod reliability) and attenuation factors will have been overestimated and the true attenuation for estimated effects greater than that reported. Despite these limitations, regression calibration using the 24-h recall as the reference instrument is preferable to performing no adjustment, as improvements in parameter estimates have been shown for both univariate and multivariable methods.41 Nutrition professionals using the BMDCS calcium FFQ are therefore recommended to report the naïve estimates that assume no measurement error, as well as calibration corrected results when possible.

Strengths of this study include the rigorous methodology employed in the development of the FFQ, the large sample size for the intermethod reliability substudy, the dietitian administered 24-h dietary recalls, and the multiracial/ethnic composition of the substudies contributing to the generalizability of the results. There were also limitations. First, as previously discussed, correlations and attenuation factors for the BMDCS FFQ presented here may be overly optimistic relative to true usual calcium intake due to correlated errors in the primary and reference instruments. However, this limitation is common to all dietary intermethod reliability studies in which an unbiased measure of dietary intake is unavailable. Second, the study power to detect small and small-to-moderate mean differences between instruments was limited for the intermethod reliability and reproducibility substudies, respectively. Third, timing with respect to the administration of the FFQ and 24-h recalls may have modestly reduced correlations between measures since they did not assess dietary intake occurring over the exact same period of time. Fourth, we were unable to examine whether calcium intake from supplements had any material impact on measures of reproducibility and intermethod reliability. Fifth, Tanner staging was not calibrated across clinicians; however, Tanner stage had little impact on the regression calibration or health outcome models and any misclassification is therefore not expected to have materially impacted the study findings. In addition, the BMDCS calcium FFQ was developed using 24-h recalls collected during the third NHANES (1988–94) warranting examination of the generalizability of the tool for capturing contemporary food and beverage intakes. The food items included on the FFQ remained capable of capturing >90% of calcium intake among U.S. children ages 8–17 years based 24-h recall data collected during the 2005–2006 NHANES (personal communication, NutritionQuest). Thus, similar performance of the instrument is expected for contemporary nutrition research studies.

CONCLUSION

The BMDCS calcium FFQ appears to provide a useful tool for assessing dietary calcium intake in children and adolescents. The ability to administer a single, short instrument with demonstrated performance across a wide age range and in racially/ethnically heterogeneous populations makes the instrument an appealing option for nutrition professionals. While meeting our a priori criterion for intermethod reliability in all racial/ethnic groups examined, the performance of the questionnaire was found to vary. Therefore, consideration with respect to potential racial/ethnic differences in measurement error bias needs to be given when analyzing and interpreting findings from naïve analyses based on this instrument. When possible, internal calibration studies should be conducted to facilitate the calculation of calibration corrected effect estimates. In the absence of an internal calibration study, nutrition professionals may also consider using the present findings to conduct external calibration sensitivity analyses to account for measurement error.

Supplementary Material

Acknowledgments

We acknowledge and thank the research dieticians at Cincinnati Children’s Hospital Medical Center and Children’s Hospital of Philadelphia for their expert service and dedication to this project. We also thank the study members and their families for their participation.

Funding:

This research was supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) contracts (N01-HD-1-3228, -3329, -3330, -3331, -3332, - 3333), the National Center for Research Resources (UL1 RR026314 and UL1 RR024134), and the General Clinical Research Grants (M01-RR-000240 and M01-RR-08084).

Footnotes

Conflicts of Interest

Nicholas J. Ollberding – I have no conflicts of interest to declare.

Vicente Gilsanz – I have no conflicts of interest to declare.

Joan M. Lappe – I have no conflicts of interest to declare.

Sharon E. Oberfield – I have no conflicts of interest to declare.

John A. Shepherd – I have no conflicts of interest to declare.

Karen K. Winer – I have no conflicts of interest to declare.

Babette S. Zemel – I have no conflicts of interest to declare.

Heidi J. Kalkwarf – I have no conflicts of interest to declare.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Contributor Information

Nicholas J. Ollberding, Email: nicholas.ollberding@cchmc.org, Assistant Professor, Division of Biostatistics and Epidemiology, at the Cincinnati Children’s Hospital Medical Center, Cincinnati, OH. Phone: (513) 803-4432.

Vicente Gilsanz, Email: vgilsanz@chla.usc.edu, Professor, Division of Radiology, Children’s Hospital of Los Angeles, Los Angeles, CA. Phone: 323-361-4571.

Joan M. Lappe, Email: jmlappe@creighton.edu, Professor, Division of Endocrinology, at Creighton University, Omaha NE. Phone: 402-280-4646.

Sharon E. Oberfield, Email: seo8@columbia.edu, Professor, Division of Pediatric Endocrinology Diabetes and Metabolism, Columbia University, New York, NY. Phone: 212-305-6559.

John A. Shepherd, Email: john.shepherd@ucsf.edu, Associate Adjunct Professor, Department of Radiology, University of California, San Francisco, San Francisco, CA. Phone: 415-891-7437.

Karen K. Winer, Email: winerk@exchange.nih.gov, Director, Pediatric Endocrinology Program, Eunice Kennedy Shriver National Institute of Child Health and Human Development, Bethesda, MD. Phone: 301-435-6877.

Babette S. Zemel, Email: zemel@email.chop.edu, Professor, Division of Gastroenterology, Hepatology, and Nutrition, at the Children’s Hospital of Philadelphia, Philadelphia, PA. Phone: 215-590-1669.

Heidi J. Kalkwarf, Email: heidi.kalkwarf@cchmc.org, Professor, Division of General and Community Pediatrics, at the Cincinnati Children’s Hospital Medical Center, Cincinnati, OH. Phone: (513) 636-3803.

References

  • 1.Feskanich D, Willett WC, Colditz GA. Calcium, vitamin D, milk consumption, and hip fractures: a prospective study among postmenopausal women. Am J Clin Nutr. 2003;77(2):504–511. doi: 10.1093/ajcn/77.2.504. [DOI] [PubMed] [Google Scholar]
  • 2.Park SY, Murphy SP, Wilkens LR, Stram DO, Henderson BE, Kolonel LN. Calcium, vitamin D, and dairy product intake and prostate cancer risk: the Multiethnic Cohort Study. Am J Epidemiol. 2007;166(11):1259–1269. doi: 10.1093/aje/kwm269. [DOI] [PubMed] [Google Scholar]
  • 3.Xiao Q, Murphy RA, Houston DK, Harris TB, Chow WH, Park Y. Dietary and supplemental calcium intake and cardiovascular disease mortality: the National Institutes of Health-AARP diet and health study. JAMA Intern Med. 2013;173(8):639–646. doi: 10.1001/jamainternmed.2013.3283. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Black RE, Williams SM, Jones IE, Goulding A. Children who avoid drinking cow milk have low dietary calcium intakes and poor bone health. Am J Clin Nutr. 2002;76(3):675–680. doi: 10.1093/ajcn/76.3.675. [DOI] [PubMed] [Google Scholar]
  • 5.Rockett HR, Wolf AM, Colditz GA. Development and reproducibility of a food frequency questionnaire to assess diets of older children and adolescents. J Am Diet Assoc. 1995;95(3):336–340. doi: 10.1016/S0002-8223(95)00086-0. [DOI] [PubMed] [Google Scholar]
  • 6.Zemel BS, Carey LB, Paulhamus DR, Stallings VA, Ittenbach RF. Quantifying calcium intake in school age children: development and validation of the Calcium Counts! food frequency questionnaire. Am J Hum Biol. 2010;22(2):180–186. doi: 10.1002/ajhb.20975. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Willett W. Nutritional epidemiology. 3. Oxford: Oxford University Press; 2013. [Google Scholar]
  • 8.White E, Armstrong BK, Saracci R, Armstrong BK. Principles of exposure measurement in epidemiology: collecting, evaluating, and improving measures of disease risk factors. 2. Oxford; New York: Oxford University Press; 2008. [Google Scholar]
  • 9.Subar AF, Thompson FE, Kipnis V, et al. Comparative validation of the Block, Willett, and National Cancer Institute food frequency questionnaires: the Eating at America’s Table Study. Am J Epidemiol. 2001;154(12):1089–1099. doi: 10.1093/aje/154.12.1089. [DOI] [PubMed] [Google Scholar]
  • 10.Stram DO, Hankin JH, Wilkens LR, et al. Calibration of the dietary questionnaire for a multiethnic cohort in Hawaii and Los Angeles. Am J Epidemiol. 2000;151(4):358–370. doi: 10.1093/oxfordjournals.aje.a010214. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Horn-Ross PL, Lee VS, Collins CN, et al. Dietary assessment in the California Teachers Study: reproducibility and validity. Cancer Causes Control. 2008;19(6):595–603. doi: 10.1007/s10552-008-9124-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Carithers TC, Talegawkar SA, Rowser ML, et al. Validity and calibration of food frequency questionnaires used with African-American adults in the Jackson Heart Study. J Am Diet Assoc. 2009;109(7):1184–1193. doi: 10.1016/j.jada.2009.04.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Kipnis V, Carroll RJ, Freedman LS, Li L. Implications of a new dietary measurement error model for estimation of relative risk: application to four calibration studies. Am J Epidemiol. 1999;150(6):642–651. doi: 10.1093/oxfordjournals.aje.a010063. [DOI] [PubMed] [Google Scholar]
  • 14.Kipnis V, Midthune D, Freedman LS, et al. Empirical evidence of correlated biases in dietary assessment instruments and its implications. Am J Epidemiol. 2001;153(4):394–403. doi: 10.1093/aje/153.4.394. [DOI] [PubMed] [Google Scholar]
  • 15.Kalkwarf HJ, Zemel BS, Gilsanz V, et al. The bone mineral density in childhood study: bone mineral content and density according to age, sex, and race. J Clin Endocrinol Metab. 2007;92(6):2087–2099. doi: 10.1210/jc.2006-2553. [DOI] [PubMed] [Google Scholar]
  • 16.Zemel BS, Kalkwarf HJ, Gilsanz V, et al. Revised reference curves for bone mineral content and areal bone mineral density according to age and sex for black and non-black children: results of the bone mineral density in childhood study. J Clin Endocrinol Metab. 2011;96(10):3160–3169. doi: 10.1210/jc.2011-1111. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Lanou AJ, Berkow SE, Barnard ND. Calcium, dairy products, and bone health in children and young adults: a reevaluation of the evidence. Pediatrics. 2005;115(3):736–743. doi: 10.1542/peds.2004-0548. [DOI] [PubMed] [Google Scholar]
  • 18.Winzenberg T, Shaw K, Fryer J, Jones G. Effects of calcium supplementation on bone density in healthy children: meta-analysis of randomised controlled trials. BMJ. 2006;333(7572):775. doi: 10.1136/bmj.38950.561400.55. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Huncharek M, Muscat J, Kupelnick B. Impact of dairy products and dietary calcium on bone-mineral content in children: results of a meta-analysis. Bone. 2008;43(2):312–321. doi: 10.1016/j.bone.2008.02.022. [DOI] [PubMed] [Google Scholar]
  • 20.Block G, Hartman AM, Dresser CM, Carroll MD, Gannon J, Gardner L. A data-based approach to diet questionnaire design and testing. Am J Epidemiol. 1986;124(3):453–469. doi: 10.1093/oxfordjournals.aje.a114416. [DOI] [PubMed] [Google Scholar]
  • 21.Block G, Coyle LM, Hartman AM, Scoppa SM. Revision of dietary analysis software for the Health Habits and History Questionnaire. Am J Epidemiol. 1994;139(12):1190–1196. doi: 10.1093/oxfordjournals.aje.a116965. [DOI] [PubMed] [Google Scholar]
  • 22.Block G, Gillespie C, Rosenbaum EH, Jenson C. A rapid food screener to assess fat and fruit and vegetable intake. Am J Prev Med. 2000;18(4):284–288. doi: 10.1016/s0749-3797(00)00119-7. [DOI] [PubMed] [Google Scholar]
  • 23.Centers for Disease Control and Prevention (CDC). National Center for Health Statistics (NCHS) National Health and Nutrition Examination Survey Data. Hyattsville, MD: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention; 1988–1994. [Accessed June 24, 2014]. http://www.cdc.gov/nchs/nhanes/nh3data.htm. [Google Scholar]
  • 24.Lappe JM, Watson P, Gilsanz V, et al. The Longitudinal Effects of Physical Activity and Dietary Calcium on Bone Mass Accrual Across Stages of Pubertal Development. J Bone Miner Res. 2014 Jul 31; doi: 10.1002/jbmr.2319. [E-pub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Gordon CM, Leonard MB, Zemel BS International Society for Clinical D. 2013 Pediatric Position Development Conference: executive summary and reflections. J Clin Densitom. 2014;17(2):219–224. doi: 10.1016/j.jocd.2014.01.007. [DOI] [PubMed] [Google Scholar]
  • 26.Slemenda CW, Miller JZ, Hui SL, Reister TK, Johnston CC., Jr Role of physical activity in the development of skeletal mass in children. J Bone Miner Res. 1991;6(11):1227–1233. doi: 10.1002/jbmr.5650061113. [DOI] [PubMed] [Google Scholar]
  • 27.Thompson FE, Byers T. Dietary assessment resource manual. J Nutr. 1994;124(11 Suppl):2245S–2317S. doi: 10.1093/jn/124.suppl_11.2245s. [DOI] [PubMed] [Google Scholar]
  • 28.Schatzkin A, Kipnis V, Carroll RJ, et al. A comparison of a food frequency questionnaire with a 24-hour recall for use in an epidemiological cohort study: results from the biomarker-based Observing Protein and Energy Nutrition (OPEN) study. Int J Epidemiol. 2003;32(6):1054–1062. doi: 10.1093/ije/dyg264. [DOI] [PubMed] [Google Scholar]
  • 29.Spiegelman D, McDermott A, Rosner B. Regression calibration method for correcting measurement-error bias in nutritional epidemiology. Am J Clin Nutr. 1997;65(4 Suppl):1179S–1186S. doi: 10.1093/ajcn/65.4.1179S. [DOI] [PubMed] [Google Scholar]
  • 30.Jensen JK, Gustafson D, Boushey CJ, et al. Development of a food frequency questionnaire to estimate calcium intake of Asian, Hispanic, and white youth. J Am Diet Assoc. 2004;104(5):762–769. doi: 10.1016/j.jada.2004.02.031. [DOI] [PubMed] [Google Scholar]
  • 31.Harnack LJ, Lytle LA, Story M, et al. Reliability and validity of a brief questionnaire to assess calcium intake of middle-school-aged children. J Am Diet Assoc. 2006;106(11):1790–1795. doi: 10.1016/j.jada.2006.08.014. [DOI] [PubMed] [Google Scholar]
  • 32.Wong SS, Boushey CJ, Novotny R, Gustafson DR. Evaluation of a computerized food frequency questionnaire to estimate calcium intake of Asian, Hispanic, and non-Hispanic white youth. J Am Diet Assoc. 2008;108(3):539–543. doi: 10.1016/j.jada.2007.12.006. [DOI] [PubMed] [Google Scholar]
  • 33.Rockett HR, Breitenbach M, Frazier AL, et al. Validation of a youth/adolescent food frequency questionnaire. Prev Med. 1997;26(6):808–816. doi: 10.1006/pmed.1997.0200. [DOI] [PubMed] [Google Scholar]
  • 34.Taylor RW, Goulding A. Validation of a short food frequency questionnaire to assess calcium intake in children aged 3 to 6 years. Eur J Clin Nutr. 1998;52(6):464–465. doi: 10.1038/sj.ejcn.1600575. [DOI] [PubMed] [Google Scholar]
  • 35.Bertoli S, Petroni ML, Pagliato E, et al. Validation of food frequency questionnaire for assessing dietary macronutrients and calcium intake in Italian children and adolescents. J Pediatr Gastroenterol Nutr. 2005;40(5):555–560. doi: 10.1097/01.mpg.0000153004.53610.0e. [DOI] [PubMed] [Google Scholar]
  • 36.Huybrechts I, De Bacquer D, Matthys C, De Backer G, De Henauw S. Validity and reproducibility of a semi-quantitative food-frequency questionnaire for estimating calcium intake in Belgian preschool children. Br J Nutr. 2006;95(4):802–816. doi: 10.1079/bjn20051671. [DOI] [PubMed] [Google Scholar]
  • 37.Magkos F, Manios Y, Babaroutsi E, Sidossis LS. Differences in the quantitative and qualitative performance of a calcium-specific food frequency questionnaire across age and sex. J Hum Nutr Diet. 2006;19(5):331–342. doi: 10.1111/j.1365-277X.2006.00718.x. [DOI] [PubMed] [Google Scholar]
  • 38.Moore M, Braid S, Falk B, Klentrou P. Daily calcium intake in male children and adolescents obtained from the rapid assessment method and the 24-hour recall method. Nutr J. 2007;6:24. doi: 10.1186/1475-2891-6-24. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Fuller WA. Measurement error models. Hoboken, N.J: Wiley-Interscience; 2006. [Google Scholar]
  • 40.Carroll RJ, Carroll RJ. Measurement error in nonlinear models: a modern perspective. 2. Boca Raton, FL: Chapman & Hall/CRC; 2006. [Google Scholar]
  • 41.Freedman LS, Schatzkin A, Midthune D, Kipnis V. Dealing with dietary measurement error in nutritional cohort studies. J Natl Cancer Inst. 2011;103(14):1086–1092. doi: 10.1093/jnci/djr189. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

NIHMS652052-supplement-supplement_1.pdf (42.8KB, pdf)

RESOURCES