The Predictive Value of Self-Report Questions in a Clinical Decision Rule for Pediatric Lead Poisoning Screening

SYNOPSIS

Objective

We derived a clinical decision rule for determining which young children need testing for lead poisoning. We developed an equation that combines lead exposure self-report questions with the child's census-block housing and socioeconomic characteristics, personal demographic characteristics, and Medicaid status. This equation better predicts elevated blood lead level (EBLL) than one using ZIP code and Medicaid status.

Methods

A survey regarding potential lead exposure was administered from October 2001 to January 2003 to Michigan parents at pediatric clinics (n=3,396). These self-report survey data were linked to a statewide clinical registry of blood lead level (BLL) tests. Sensitivity and specificity were calculated and then used to estimate the cost-effectiveness of the equation.

Results

The census-block group prediction equation explained 18.1% of the variance in BLLs. Replacing block group characteristics with the self-report questions and dichotomized ZIP code risk explained only 12.6% of the variance. Adding three self-report questions to the census-block group model increased the variance explained to 19.9% and increased specificity with no loss in sensitivity in detecting EBLLs of ≥10 micrograms per deciliter.

Conclusions

Relying solely on self-reports of lead exposure predicted BLL less effectively than the block group model. However, adding three of 13 self-report questions to our clinical decision rule significantly improved prediction of which children require a BLL test. Using the equation as the clinical decision rule would annually eliminate more than 7,200 unnecessary tests in Michigan and save more than $220,000.

The purpose of testing for elevated blood lead levels (EBLLs) in children is to identify cases that would require appropriate treatment and/or environmental intervention. The 1989 Centers for Medicare & Medicaid Services requirement to test all children enrolled in Medicaid is costly and requires extensive outreach.¹ It also created a need for clinical practice guidelines and decision rules to determine which young children are at high enough risk to require testing for lead poisoning.

In 1997, the Centers for Disease Control and Prevention (CDC)^2,3 developed such clinical practice guidelines and defined high risk of symptoms of lead poisoning as having a blood lead level (BLL) of ≥10 micrograms per deciliter (μg/dL).⁴ CDC encouraged state health departments to develop plans to identify all children at high risk of lead poisoning based on local data concerning BLLs and selected risk factors.

The CDC guidelines were based on several criteria, including age of housing and percentage of population with incomes below the federal poverty level (FPL) in the ZIP code in which the child resides. As part of its targeting, CDC also recommended using information obtained from five self-report questions on lead exposure (Figure).³ The Michigan Department of Community Health (MDCH) replaced question #5 with the following question: “Does the child's family use any home remedies that may contain lead?” and provided a list of such remedies. At the time, MDCH followed Medicaid rules requiring that all children enrolled in Medicaid be tested, although not all such children were actually tested. In addition, CDC and MDCH recommended that a child be tested if the child's caregiver answered “yes” or “I don't know” to any of the self-report questions, or if the child was Medicaid-enrolled or lived in a high-BLL-risk ZIP code.

Figure.

CDC and MDCH recommended screening questions for pediatric lead poisoning

CDC = Centers for Disease Control and Prevention

MDCH = Michigan Department of Community Health

While these self-report questions have been widely used, studies have not found them to be good predictors of EBLL.^5–8 Moreover, some of these questions are poorly worded and may not be understood by respondents. Therefore, there is a need to both improve the wording of these questions and assess the predictive validity of the improved questions.

Previous research in Michigan⁹ and elsewhere^10,11 has led to a refined geographic approach to predicting the BLL of children. Unlike previous studies that use ZIP codes and census tracts, this research relied heavily on the characteristics of the census-block group (the smallest geographic unit for which detailed data were available). In Michigan, census-block groups explained substantially more variance in BLL than census tracts or ZIP codes. This method yielded a prediction equation with better sensitivity and specificity than ZIP code and Medicaid status, which would have identified more high-risk children, while saving more than $150,000 during the four-year period 2002–2005.⁹

This equation, derived from our previous work,⁹ predicted BLL from a weighted linear combination of the following characteristics of the census-block group in which the child lived: percentage of housing built before 1940 (HSNG_PRE1940), percentage of housing built during 1940–1949 (HSNG_1940-49), percentage of population with income <185% FPL (INC_185%_POV), percentage who did not graduate from high school (HS_DROPOUT), percentage black, and percentage Latino; as well as the following characteristics of the child: Medicaid status, race/ethnicity (BLACK_CHILD or not), age, and year tested.

The optimal prediction equation for 2002–2005 was Ln(BLL − 0.5) = −0.487 + 0.694 × HSNG_PRE1940 + 0.0119 × HSNG_1940-49 + 0.212 × INC_185%_POV + 0.400 × %_BLACK + 0.556 × %_LATINO + 0.206 × BLACK_CHILD + 0.172 × MEDICAID + 0.109 × MEDICAID × HSNG_PRE1940 + 0.171 × MEDICAID × HSNG_1940-49.

This prediction equation also includes coefficients of dummy variables that adjust for the child's age and year of the BLL test. Additionally, it includes empirical Bayesian residuals generated by Hierarchical Linear Modeling.¹² These residuals estimate the degree to which the prediction equation over- or underestimates the BLL in each census-block group.

While prior literature suggests that these self-report exposure questions are not adequate predictors of BLL,^5–8 it is possible that some of them are worth adding to the census-block group prediction equation. The present study expands our previous work by including modified MDCH-CDC self-report questions and other self-report questions suggested in the literature. The purpose is to create the most cost-effective method of targeting BLL testing, which can then be disseminated by public health professionals to clinicians and parents.

METHODS

Two datasets were linked for this study. One was our survey on sources of lead exposure, which was administered from October 2001 to January 2003 to more than 4,000 parents and caregivers of young children at 36 pediatric clinics in Michigan, including 18 county health departments, two urban health systems, and 16 small clinics serving migrant workers. The survey contained questions that modified the five MDCH-CDC items, plus a question about lead smelters and questions about additional sources of lead exposure (e.g., pacifier use⁵ and water that came through lead pipes).¹³

Because respondents were disproportionately of low education, the current survey modified the CDC questions by dividing sentences with several clauses or phrases into simpler sentences. It also divided some of the CDC questions into several questions. CDC question #1 in the Figure was divided into three separate questions asking (1) whether the residence was built before 1950, (2) whether it had peeling paint, and (3) whether the child had visited a house with peeling paint. Our questionnaire had a Flesch-Kincaid reading grade level of 4.1, whereas the original CDC questionnaire has a reading grade level of 7.8. We subsequently created variables whose meaning closely corresponded to the original CDC questions (Table 1).

Table 1.

Descriptive statistics for key questions in a survey of BLLs^a and items in the MDCH clinical BLL database: Michigan, October 2001–January 2003

^aWhile many of the survey questions were based on CDC questions, the exact wording was often different from the original CDC wording.

^bAll variables, except pacifier use, are dichotomous, and the mean is the percentage of those with nonmissing responses who answered affirmatively. Pacifier use is coded 0 for “never or almost never” (74.7% of respondents), 0.5 for “sometimes” (8.0% of respondents), and 1 for “daily or almost daily” (16.0% of respondents).

^cThese compound questions were each originally asked as several separate questions and the answers were then combined for data analysis.

BLL = blood lead level

MDCH = Michigan Department of Community Health

CDC = Centers for Disease Control and Prevention

We pilot-tested the survey with 15 parents to ascertain whether parents were willing to complete it and readily understood the questions. All parents completed the survey. Only one parent had any difficulty with reading the questionnaire, and clinic staff described her as barely able to read.

After piloting, the survey was administered by trained staff at participating clinics. Of those invited to complete the survey, the participation rate exceeded 90%. As an incentive, respondents were given a long-distance phone card. The survey took less than 10 minutes to complete, and respondents filled it out while waiting at the clinic.

The second dataset consisted of a clinical registry of all BLL tests in Michigan maintained by MDCH, with identifiers removed. If a child had more than one BLL test, we analyzed the highest venous result, if available, and the highest capillary result, if not available. Only 26% of the tests were venous, although 65% of the EBLLs were venous.

Of the 4,194 survey responses, 92.4% (n=3,876) could be linked to the child's BLL test. The final number of cases (3,396) was primarily due to cases lost from BLL records that lacked a valid address. Other cases were excluded because the child was older than 5 years of age or the age was missing.

Methods for missing data

CDC recommended that missing data on any question regarding risks of lead exposure be treated as having answered “yes” to that question.² We followed this recommendation as one of our methods, with several qualifications. For example, because we asked questions about housing built before 1950 and the presence of peeling paint separately, we could have missing data on one of those questions. If the respondent answered “no” to either question, then even if there were missing data on the other question, we considered the answers tantamount to a “no” on the original CDC question #1. When using this method, we also recoded missing values of race to black, Medicaid eligibility to yes, and pacifier use to never, as these values are associated with higher BLL.

The aforementioned CDC method tends to produce biased coefficient estimates. To obtain unbiased estimates without list-wise deletion, we used 10 random imputations of missing data as an alternative method.¹⁴ The values of the dependent variable were used to impute missing values of the independent variables.¹⁵ The imputations were performed using SPSS® version 18.0, using the automatic imputation method.¹⁶

RESULTS

The range of BLLs was 1 to 164 μg/dL. Consistent with previous findings,^9,17 BLL was normally distributed after logarithmic transformation. The transformation that minimizes both skew and heteroscedasticity of variances is Ln(BLL − 0.5), where Ln is the logarithm to base e (≈2.718). This function was the dependent variable in all regressions. The minimum BLL recorded in the MDCH database was 1.0. Therefore, many results recorded as 1.0 might actually have been lower.

Comparing BLLs for survey cases with the full MDCH database

We compared BLL data for the survey cases with those from the full MDCH database of tests in 2002 (the year in which 83% of the BLL tests of children of survey respondents were conducted). As shown in Table 2, the prediction equation explained a much smaller proportion of the variance in Ln(BLL − 0.5) in the survey cases than in the full MDCH database. It also compared the conditional standard deviations (SDs) of the same dependent variable for the two datasets. These SDs were very similar in the two datasets, indicating that the dependent variable was predicted almost equally well¹⁸ in both.

Table 2.

Comparison of BLL results for survey data cases from October 2001–January 2003 with full MDCH BLL database for 2002

BLL = blood lead level

MDCH = Michigan Department of Community Health

SD = standard deviation

The survey data cases have a smaller mean for the dependent variable than does the full MDCH data. This smaller mean occurred because the survey data underrepresented some high-risk groups, such as low-income families in Detroit. Almost all of the self-report questions provided three response alternatives: “yes”, “no”, and “don't know.” Table 1 shows response distributions.

Regression analyses

The census-block group prediction score by itself explained 18.1% of the variance in Ln(BLL − 0.5), while the self-report questions plus dichotomized ZIP code risk explained only 12.6% of the variance. Thus, the self-report questions should not replace the census-block group information.

The major research aim was to estimate how much additional predictive value the self-report variables contributed to the estimation of BLL, above that provided by the previous census-block group prediction equation. We entered the linear combination of variables from the optimal census-block group prediction equation⁹ followed by the self-report variables.

For both the CDC “missing data implies risk”² and the multiple imputation methods, we added 13 self-report variables as predictors to the regression containing the census-block group equation. Twelve of these variables are in Table 1. The 13th variable was the product of the self-report response regarding whether the child lived in a house with interior peeling paint and the proportion of housing built before 1940 in the respondent's census-block group.

For the CDC method, the adjusted R² from adding all 13 predictors was 0.192, while for random imputation it was 0.200 (data not shown). For both methods, the same three added predictors had statistically significant coefficients, while the other predictors (not shown) were nonsignificant.

The next step was to conduct a regression analysis using only the census-block group equation and the three significant predictors. As shown in Table 3, the adjusted R² values, with only these three self-report predictors, were within 0.001 of the aforementioned adjusted R² values from all 13 predictors.

Table 3.

Unstandardized regression coefficients predicting Ln(BLL – 0.5) from census-block group prediction equation and self-report variables, assuming that missing responses are tantamount to “yes” (i.e., risk suggestive) and by multiple random imputation: Michigan, October 2001–January 2003^a

^aData on BLL and all variables contributing to the prediction score are from the Michigan Department of Community Health clinical registry. Other data are from the researchers' survey.

^bThe BLL multiplier is the anti-logarithm of the regression coefficient. It is equal to e^B, where e is the base of natural logarithms.

^cp<0.001

^dp<0.01

BLL = blood lead level

CDC = Centers for Disease Control and Prevention

SE = standard error

To aid interpretation, we present the anti-logarithms of the regression coefficients in Table 3. For both data analysis methods, all other things being equal, daily pacifier use resulted in a predicted BLL that was 81% of the BLL of a child who did not use one. For both methods, a child living in a house with peeling paint in a census-block group in which 100% of the housing was built before 1940 had a predicted BLL that was approximately 40% higher than that of an otherwise identical child in a house without peeling paint or a census-block group with no pre-1940 housing. Using the CDC method, children who had a sibling with an EBLL had a predicted BLL that was 13% higher than that of an otherwise identical child without such a sibling. Using random imputation, children with an EBLL sibling had a predicted BLL that was 46% higher.

Identifying the most cost-effective decision rule

The most cost-effective decision rule would correctly identify all children with EBLLs (100% sensitivity) without misidentifying any without EBLLs (100% specificity). We compared cost-effectiveness by computing the sensitivity and specificity for three BLL prediction equations: first by using the census-block group prediction equation by itself, and then by including the three significant self-report lead questions. This comparison was made for both the CDC's missing data implies risk strategy and the multiple imputation method. While no BLL is known to be safe,¹⁹ consistent with CDC, we defined EBLL as ≥10 μg/dL. A good clinical decision rule requires a well-constructed risk score,²⁰ and we used the predicted value of Ln(BLL−0.5). The Centers for Medicare & Medicaid Services requirement that all children on Medicaid be tested, and the CDC guidelines calling on state health departments to develop screening plans for all children at high risk, indicate that high sensitivity, i.e., correctly identifying all children with EBLLs, is more important than high specificity. We chose 0.3 as our risk score cutoff point because this number achieves high sensitivity.

Table 4 compares the prediction methods. The sensitivity of the three methods was identical (0.992). However, adding the three self-report questions to the census block group equation gave higher specificity. Adding these questions via CDC's method would eliminate 153 (673 – 520) unnecessary tests. Adding these questions via random multiple imputation would eliminate 186 (706 – 520) unnecessary tests.

Table 4.

Sensitivity and specificity of BLL scores^a in predicting BLLs of ≥10 micrograms per deciliter from census-block group prediction equation and adding three best self-report exposure predictors, by two different methods: Michigan, October 2001–January 2003^b

Note: Of 3,396 children, 23 had an EBLL.

^aThe score used for the census-block group prediction equation is derived from the BLL prediction equation found in: Kaplowitz SA, Perlstadt H, Post LA. Comparing lead poisoning risk assessment methods: census block group characteristics vs. zip codes as predictors. Public Health Rep 2010;125:234-45. This equation is derived from census-block group variables plus race/ethnicity, Medicaid status, and age. The risk scores for the CDC and random multiple imputation methods were derived from the regression coefficients in which the self-report variables were used as predictors in addition to the census-block group prediction equation. The testing cutoff point for these calculations is 0.3, which means that we would test only those whose risk score was at least 0.3.

^bData on BLL and all variables contributing to the prediction score are from the Michigan Department of Community Health clinical registry. Other data are from the researchers' survey.

^c(specificity) × (number of non-elevated tests)

^d(1 − sensitivity) × (number of elevated tests)

BLL = blood lead level

EBLL = elevated blood lead level

CDC = Centers for Disease Control and Prevention

Brown and Chattopadhyay found that the average cost of a private-sector BLL venous sample test was $31, including the cost of the blood draw.²¹ Adding the self-report questions would have resulted in a total savings of $31 × 153 = $4,743 for the CDC method and $31 × 186= $5,766 for the random multiple imputation method. This analysis was based on 3,396 cases, approximately 2% of the more than 160,000 BLL tests performed each year in Michigan from 2007 to 2009. Extrapolating suggests that using the three self-report questions statewide would annually eliminate more than 7,200 unnecessary tests and save more than $220,000 using the CDC method and almost $270,000 using random multiple imputation. This savings would occur with no loss in sensitivity.

DISCUSSION

In an effort to improve prior predictions of children's BLLs, we used a combination of survey and clinical data. Three of 13 self-report questions made statistically significant contributions to predicting BLL. These three were having a sibling with an EBLL, using a pacifier (i.e., using it more often is better), and having peeling paint inside a child's residence (when multiplied by the percentage of pre-1940 housing in the census-block group).

By comparison, the CDC-based question, “Has the child lived in a house built before 1950 with peeling paint inside?” and not multiplying by the census information had no significant predictive value, even when the interaction term involving this question was excluded from the model. This question may have had little value because parents who rent their housing often do not know when the house was built, and more than half of the respondents answering this question indicated that they rented. Similarly, the nonsignificant predictive value of the question on lead pipes may have been a result of parents not knowing whether or not their residence had such pipes.

The observed negative association between pacifier use and BLL is surprising, given that Dalton et al.⁵ found a significant positive association. Their results are consistent with the assumption that if pacifiers fall on the floor and pick up lead dust, children may then inhale or ingest lead upon insertion. However, our results suggest that a pacifier may act as a barrier that prevents lead dust from entering the mouth and/or reduces the need to suck on paint chips or lead-painted toys. We also note that our sample size (n=3,396) was much larger than Dalton's (n=463) and that we estimated the effect of pacifier use on BLL while controlling for many other variables, whereas Dalton's analyses were strictly bivariate.

Limitations

This study was subject to several limitations. For one, these data were from one state and represented a limited time period (late 2001 through early 2003). Also, the survey was not representative geographically, as the sample consisted of patients at the participating pediatric clinics in the state. Furthermore, some of the elevated capillary tests were not followed up by more accurate venous tests. Therefore, it is possible that some of the EBLL tests in the clinical records represented false-positives. Hence, we reran the analysis, eliminating elevated capillary BLL results. We found that all estimated regression coefficients were within 5% of the values presented in Table 3 and that none of our conclusions would be brought into question.

CONCLUSIONS

Prior clinical decision rules for targeting lead testing have used the criteria of Medicaid eligibility, residing in a high-risk ZIP code, or answering “yes” or “don't know” to one of the recommended self-report questions. These rules have a much lower predictive validity than an equation that is a linear combination of census-block group characteristics and several individual characteristics.⁹ As shown previously, adding the three self-report questions increased the predictive value of the census-block group equation.

CDC recently noted that since its 1997 recommendations were developed, all 42 CDC-funded childhood lead poisoning prevention programs in 37 states have developed data-driven targeted screening criteria. CDC went on to recommend that because “state and local officials are more familiar than federal agencies with local risk for elevated BLLs … that these officials have the flexibility to develop blood lead screening strategies that reflect local risk for elevated BLLs.”¹

Our clinical decision rule adheres to this guidance and has the potential to better target BLL testing in children, thereby reducing testing costs while simultaneously identifying more cases of EBLL. In fact, our previous research⁹ shows that using a census-block group equation in Michigan, rather than ZIP codes, would have identified more cases of BLL >10 μg/dL, while saving more than $150,000 in four years.

This study shows that adding the three self-report questions to the equations increases specificity with no loss in sensitivity and should increase the monetary savings with no loss in the number of elevated cases identified.

The purpose of the prediction equation is to create a clinical decision rule that provides medical providers and public health departments with a better approach to determine which children should undergo BLL testing. Such decision rules are derived from rigorous quantitative research^22–25 and are developed for clinically important conditions that are prevalent and for which current diagnostic testing practices vary widely and are inefficient.^9,25

While the census-block group equation has much greater predictive validity for EBLL than ZIP codes, it is fairly easy to know the ZIP code and, at first glance, using the prediction equation appears more difficult and time-consuming. However, a Web tool has been developed to perform the computations in this equation, and online access is available at http://midata.mwbc.info/bll/msu-mdch_program.htm. Clinic staff or parents can enter the child's Michigan address, Medicaid status, race/ethnicity, age, and the answers to the three self-report questions that are useful predictors. The Web tool processes these responses using the prediction equation and returns a recommendation as to whether or not the child needs a BLL test.

Clinical decision rules similar to the one presented in this article can be developed for other states. However, it will require developing a prediction algorithm using existing statewide BLL test data and U.S. Census Bureau information. We have described the type of statistical analyses required in this article and our previous article.⁹ A prediction equation would be the core of an online program that could be easily used by providers and parents alike. It would also facilitate cost-effective testing.^21–24