Inconsistency in the Self-report of Chronic Diseases in Panel Surveys: Developing an Adjudication Method for the Health and Retirement Study

Abstract

Objectives

Chronic disease data from longitudinal health interview surveys are frequently used in epidemiologic studies. These data may be limited by inconsistencies in self-report by respondents across waves. We examined disease inconsistencies in the Health and Retirement Study and investigated a multistep method of adjudication. We hypothesized a greater likelihood of inconsistences among respondents with cognitive impairment, of underrepresented race/ethnic groups, having lower education, or having less income/wealth.

Method

We analyzed Waves 1995–2010, including adults 51 years and older (N = 24,156). Diseases included hypertension, heart disease, lung disease, diabetes, cancer, stroke, and arthritis. We used questions about the diseases to formulate a multistep adjudication method to resolve inconsistencies across waves.

Results

Thirty percent had inconsistency in their self-report of diseases across waves, with cognitive impairment, proxy status, age, Hispanic ethnicity, and wealth as key predictors. Arthritis and hypertension had the most frequent inconsistencies; stroke and cancer, the fewest. Using a stepwise method, we adjudicated 60%–75% of inconsistent responses.

Discussion

Discrepancies in the self-report of diseases across multiple waves of health interview surveys are common. Differences in prevalence between original and adjudicated data may be substantial for some diseases and for some groups, (e.g., the cognitively impaired).

Chronic disease data from longitudinal health interview surveys have been widely used in aging and epidemiological research (Hodes & Suzman, 2007; Okura, Urban, Mahoney, Jacobsen, & Rodeheffer, 2004). These panel surveys, in which self-reported chronic disease information is collected in repeated interviews with the same study respondents, can monitor trends in health and function (Ferraro, 1980; Ferraro & Farmer, 1999). Practically, in the case of large, nationally representative data sources, health information that is self-reported is comparatively less difficult and expensive to collect and so can be obtained from a broader population sample.

Consequently, it is paramount that these surveys produce valid and reliable population estimates of chronic diseases and chronic disease risk. One indicator of the trustworthiness of self-reported chronic disease information in longitudinal surveys is the consistency in the responses of participants at reinterview and across multiple interview waves (Beckett, Weinstein, Goldman, & Yu-Hsuan, 2000) These data may be limited by discrepancies in the self-report of diseases by respondents across interview waves.

Longitudinal patterns of chronic disease self-reports that are clinically implausible (respondents switching their wave-by-wave answers of whether or not they have been diagnosed with a disease) present methodological challenges to researchers using these data. By definition, chronic diseases, once diagnosed, persist throughout an individual’s life span, despite treatment of the underlying pathophysiology and management of symptoms. The chronic diseases commonly asked about in health interview surveys are, presently, incurable. The consensus has been to view these queried diseases, once diagnosed, as the individual having the diseases in perpetuity (Fisher, Faul, Weir, & Wallace, 2005). Yet because inconsistency is apparent only in records that are longitudinal, its presence and extent may not be manifest when examining cross-sectional concordance with administrative data.

The result of repeated health interviews is a longitudinal record of diseases for each respondent. Most of these longitudinal records have expected consistent patterns. For example, most participants report never having been diagnosed with a chronic disease for the duration of their participation in the survey. In contrast, some report having been diagnosed with a chronic disease at their first interview, and they respond affirmatively to that disease diagnosis question in all subsequent interviews. Last, there are participants who initially report not having been diagnosed with a chronic disease but who are later diagnosed with the disease and thereafter respond affirmatively to the disease diagnosis question. Problems arise when the longitudinal records of self-reported diseases have patterns that are not clinically reasonable. For example, a respondent reports in 1998 of having been diagnosed with a chronic disease but reports at the following interview in 2000 of never being diagnosed with the disease.

Much has been theorized about middle-aged and older adults’ health beliefs and their conceptualization of illness and disease. The illness representations model suggests that the perceptions of individuals about and understanding of their health relates to their knowledge of diseases, the knowledge and experiences of their lay networks of family and friends, and their own experiences with the severity, chronicity, and symptomatology of their diseases (Leventhal & Crouch, 1997). The confluence of information about disease in others and their personal experiences with symptoms generates illness representations that guide them in their decisions in coping with and managing disease. Individuals who experience intermittent symptoms of emphysema, for example, may question and inconsistently report whether or not they have been diagnosed with emphysema (Halm, Mora, & Leventhal, 2006). In contrast, diseases with clearer symptomatology and event history, such as stroke, may lead adults to greater consistency in reporting this diagnosis.

Inconsistency in the self-report of chronic diseases may be concentrated in certain population subgroups. Individuals with cognitive deficits may not have the memory or the understanding to be consistent in their responses over time (Alzheimer’s Association, 2015; Hugo & Ganguli, 2014; Prince et al., 2013). A population-based survey of older adults will include substantial numbers of respondents with dementing illnesses in which memory impairment is an early hallmark. Likewise, individuals with poor health literacy or less education may have a limited understanding of chronic illness. There is an extensive literature on chronic disease differences related to ethnicity, education, socioeconomic status, and access to health care, with the causal mechanisms underlying these differences found to be complex and interwoven (e.g., environment and social support;Hayward, Miles, Crimmins, & Yang, 2000; Link & Phelan, 1995; Woolf & Braveman, 2011). The factors contributing to chronic illness differences may not be identical to the factors contributing to inconsistency in chronic disease reporting; nonetheless, the evidence underlying chronic illness differences can inform the problem of inconsistency in the self-report of diseases, and examining the relationship of these identified factors to inconsistency is an initial step.

The overall aim of this study was to investigate consistency in the self-reporting of chronic diseases across multiple waves of a longitudinal health interview survey, the Health and Retirement Study (HRS). An associated aim was to develop a method to adjudicate the inconsistencies, our innovation being the use of evidence from prior waves to assist in adjudicating responses in later waves. We hypothesized that inconsistences were more likely for respondents with cognitive impairment and for respondents of underrepresented race/ethnic groups, having lower education, or having less income/wealth.

Method

Data and Study Design

We performed secondary analysis of chronic disease data from Waves 1995–2010 of the HRS. The HRS is a population-based biennial longitudinal health interview survey of over 26,000 adults aged 51 years and older in the United States (Heeringa & Connor, 1995). Designed to study the health and economic consequences of aging, the survey is based on a multistage area probability sample of households that is nationally representative, enabling results to be generalized to the U. S. population. The HRS is sponsored by the National Institute on Aging and performed by the Institute for Social Research at the University of Michigan.

Our study investigated respondents from five HRS study cohorts: the original HRS cohort of adults born from 1931 to 1941; the Study of Assets and Health Dynamics Among the Oldest Old (AHEAD) cohort, born prior to 1924; the Children of the Depression (CODA) cohort, born 1924 to 1930; the War Baby (WB) cohort, born 1942 to 1947; and the Early Baby Boomer (EBB) cohort, born 1948 to 1953.

The HRS was approved by the University of Michigan Health Sciences Institutional Review Board. The data used in this analysis are publicly available and contain no unique identifiers, thus assuring respondent anonymity.

Study Population

We included all age-eligible respondents (51 years and older at entry) who participated in at least two interview waves, yielding a population sample of 24,156, representing 84,281,604 nationally. Respondents included adults living in the community and those residing in long-stay nursing facilities. When the respondent was unable to be interviewed for a survey wave (e.g., due to medical and/or cognitive problems), a proxy respondent, most often the spouse, answered questions for that respondent according to HRS protocol.

Variables and Their Measurement

Chronic Diseases

Each HRS biennial core survey wave provides self-report information on seven chronic diseases. Respondents first report whether or not a physician has diagnosed them with each disease. If the respondent indicates in that wave that he has the disease, he then answers follow-up questions about that disease (e.g., disease activity/severity). We used certain of these follow-up questions as “evidence for conclusively having the disease” and so to resolve inconsistencies in respondent reporting about having the disease in later waves:

• hypertension: requiring medication;

• heart disease: requiring medication or having congestive heart failure, angina, a previous heart attack, or previous heart surgery;

• chronic lung disease: requiring medication or other treatment;

• diabetes: requiring oral medication or insulin;

• cancer (excluding minor skin cancers): requiring surgery, chemotherapy, or radiation as treatment for the cancer or specifying the year of diagnosis of the cancer;

• stroke: requiring medication or having remaining problems; and

• arthritis (unspecified type): requiring medication or other treatment.

Cognitive Impairment and Depressive Symptoms

The HRS assesses for cognitive impairment in one of two ways (Herzog & Wallace, 1997). For self-respondents, the presence of cognitive impairment is determined using a validated performance-based measure, a modified version of the Telephone Interview for Cognitive Status (TICS). We defined cognitive impairment as a score of 0–11 on the 27-point cognitive scale. For respondents unable to complete the interview, we made use of an 11-point scale composed of the proxy’s assessment of the respondent’s memory, the proxy’s assessment of the respondent’s instrumental activities of daily living difficulties, and the interviewer’s assessment of the respondent’s cognitive impairment. We defined cognitive impairment as a score of 3–11 (Cigolle, Kabeto, Lee, & Blaum, 2013; Crimmins, Kim, Langa, & Weir, 2011). We defined having depressive symptoms as reporting 4 or more items of the 8-item Center for Epidemiologic Studies Depression (CES-D) measure (Steffick, 2000).

Healthcare access

For cross-wave analyses, we categorized insurance coverage as none (no coverage at any wave), intermittent (coverage at some waves), and continuous (coverage at all waves); we categorized physician visits as none (no visits reported at any wave), intermittent (visits reported at some waves), and continuous (visits reported at all waves). For longitudinal analyses, we included wave-specific, binary indicators of insurance status (any coverage vs. no coverage) and physician visits (any vs. none in the prior 2 years).

Demographic characteristics

Demographic variables included age, gender, ethnicity/race (Caucasian, African American, and Hispanic), marital/partner status, educational attainment, income, and wealth (net worth).

HRS Question Format

The HRS asks respondents about each of their chronic diseases using the following format:

• If a new interview respondent: “Has a doctor ever told you that you have _____?”

• If a reinterview respondent and if respondent reported in the last interview that he/she had _____: “Our records (from your last interview in [respondent’s last interview in month, year)] show that you have had _____.”

• If a reinterview respondent and if respondent did not report in the last interview that he/she had _____: “Since we talked last in [respondent’s last interview in month, year], has a doctor told you that you have _____?”

Respondents have the following options to reply:

• “1. Yes”

• “3. Disputes previous wave record, but now has condition”

• “4. Disputes previous wave record, does not have condition”

• “5. No”

• “8. Don’t know”

• “9. Refuse”

Thus, for respondents for whom the present interview is a reinterview, there is the opportunity to dispute a previous “Yes” response, that is, to dispute their “Yes” response to the same item in a previous interview wave. Respondents indicate that their previous “Yes” response is incorrect and that they now have the disease (Respondent Option 3) or that they now (still) do not have the disease (Respondent Option 4). For respondents for whom the present interview is a reinterview, there is no opportunity to dispute a previous “No” response.

Types of Inconsistencies in Self-Report of Chronic Diseases

We classified inconsistencies in respondents self-reporting chronic diseases across interview waves into four categories (Figure 1):

Figure 1.

Types of inconsistencies in self-report of chronic diseases.

^aType One Inconsistency: Response in the present wave disputes “Yes” response in the previous wave (Respondent Options 3 and 4). ^bType Two Inconsistency: “Don’t Know” or “Refused” response in the present wave (Respondent Options 8 and 9). ^cType Three Inconsistency: “No” response in the present wave is preceded by a “Yes” response in the previous wave (Respondent Option 5 preceded by Respondent Option 1). ^dType Four Inconsistency: “Yes” response in the present wave is followed by a “Disputes” response in the following wave(Respondent Option 1 followed by Respondent Options 3 or 4).

• Type One: Interviewee’s response in the present wave disputes his “Yes” response to the chronic disease question from the prior wave (Respondent Options 3 and 4).

• Type Two: Interviewee responds to the disease question in the present wave with “Don’t know” or “Refuse to answer” (Respondent Options 8 and 9).

• Type Three: Interviewee responds to the disease question in the present wave with “No” (Respondent Option 5). However, he responded to the same question in the prior wave with “Yes” (Respondent Option 1). (Here, respondents should have disputed their responses in the prior wave using Options 3 or 4, but did not).

• Type Four: Interviewee responds to the disease question in the present wave with “Yes” (Respondent Option 1). However, he disputed this “Yes” response in his reply to the same question in the following wave (using Respondent Options 3 or 4). (Type Four Inconsistencies are the counterpart to Type One inconsistencies).

Patterns of Responses Across Interview Waves

There are three patterns of responses across interview waves that are reasonable and nonproblematic:

• All “Yes” responses (all Respondent Option 1s). The respondent always had the disease.

• All “No” responses (all Respondent Option 5s). The respondent never had the disease.

• One or more “No” responses followed by “Yes” responses (one or more 5s followed by 1s). The respondent originally did not have the disease, but later was diagnosed with the disease.

All other patterns of responses are problematic or inconsistent.

Clarification About Terminology

We use problematic and inconsistent and discrepant as broad and inclusive terms to refer to responses that that do not logically follow across interview waves or that indicate uncertainty about having a disease. We reserve the term dispute for the subset of cases in which respondents explicitly disagree with their responses to the disease questions from the prior wave (Respondent Options 3 and 4).

Adjudication Methodology

In its biennial core survey interviews, the HRS includes follow-up questions about each chronic disease that are answered by all respondents who answered “Yes” to having the disease in that wave (see Variables and Their Measurement: Chronic Diseases). We used the responses to these follow-up questions to form algorithms to adjudicate inconsistencies in the self-reported disease data across HRS waves. For example, if a respondent ever reported having heart surgery, he is deemed to always have heart disease, even in waves where he denied having heart disease. Similarly, if a respondent ever reported using insulin for diabetes, he is deemed to always have diabetes, even in waves where he denied having diabetes.

We developed levels of adjudication that address each type of inconsistency in respondents’ self-report of chronic diseases:

• Adjudication Level One: If the respondent disputed the prior wave record but now replies that he has the disease, we adjudicated his response as “Yes”. If the respondent disputed the prior wave record and replies that he does not now have the disease, we examined his prior wave responses for evidence of having the disease. If evidence was present, we adjudicated his response as “Yes”; if evidence was not present, we adjudicated his response as “No.”

• Adjudication Level Two: For “Don’t know” and “Refuse to answer” responses, we examined the respondent’s prior wave responses for evidence of having the disease. If evidence was present, we adjudicated his response as “Yes”; if evidence was not present, we adjudicated his response as “No”.

• Adjudication Level Three: For a “No” response in the current wave being preceded by a “Yes” response in the prior wave, we examined the respondent’s prior wave responses for evidence of having the disease. If evidence was present, we adjudicated his response as “Yes”; if evidence was not present, we adjudicated his response as “No.”

• Adjudication Level Four: When a “Yes” response in the current wave was followed by a dispute of this “Yes” response in the succeeding wave, we examined the respondent’s current wave responses for evidence of having the disease. If evidence was present, we adjudicated his response as “Yes”; if evidence was not present, we adjudicated his response as “No.”

Adjudication Levels 1 through 4 are conservative correctives, based on the logic of and the phrasing within the HRS questionnaires themselves. (Fisher et al., 2005) Levels 1 through 4 address inconsistencies in chronic disease reporting by respondents between adjacent waves (that the respondent participated in). To restate, Levels 1 through 4 draw on evidence from the same wave or the immediately preceding wave.

In contrast, we developed an Adjudication Level 5, which is more ambitious, in that it carries evidence for having the disease from a preceding wave forward to all succeeding waves. Thus, Adjudication Level 5 addresses inconsistencies in chronic disease reporting by respondents across all waves that the respondent participated in. Adjudication Level 5 draws on evidence from any previous wave, not just the immediately preceding wave (Levels 1, 2, and 3) or the current wave (Level 4).

Statistical Analysis

We used HRS respondent population weights, specific to each core survey wave, to adjust for the complex sample design of the HRS, differential probability of selection into the sample, and differential nonresponse/respondent loss. If a respondent’s weight was missing for a particular wave, we used that respondent’s weight from the adjacent wave, per HRS recommendations. We used weighted data for those results that have import on a population basis: Supplementary Table 1 (respondent characteristics), Table 1 (predictors of inconsistency), Table 3 (effects of adjudication on disease prevalence), and Supplementary Table 4 (effects of adjudication on disease prevalence among the cognitively impaired). We used unweighted data for results involving actual counts of inconsistencies and adjudications (Table 2, Supplementary Table 2, Supplementary Table 3, and Table 4).

Table 1.

Association of Cognitive Impairment and Proxy Status With Inconsistency in Self-report of Chronic Disease

	Odds Ratio (95% CI)a
	Model 1b	Model 2c	Model 3d	Model 4e
Cognitive impairmentf	2.46 (2.28–2.66)	1.88 (1.72–2.05)	1.76 (1.61–1.92)	1.66 (1.49–1.84)
Use of proxyg	2.20 (2.02–2.41)	2.09 (1.91–2.28)	2.06 (1.88–2.26)	1.91 (1.68–2.18)
Number of wavesh	1.11 (1.10–1.13)	1.12 (1.10–1.14)	1.12 (1.11–1.14)	1.13 (1.11–1.14)
Agei (years)		1.02 (1.02–1.03)	1.02 (1.02–1.03)	1.02 (1.02–1.03)
Genderj
Female		0.98 (0.92–1.04)	0.97 (0.91–1.02)	0.96 (0.91–1.02)
Race/ethnicityk
African American		1.00 (0.91–1.10)	0.93 (0.85–1.02)	0.93 (0.85–1.03)
Hispanic		1.30 (1.15–1.47)	1.21 (1.07–1.36)	1.21 (1.08–1.37)
Other		0.98 (0.75–1.27)	0.94 (0.71–1.23)	0.94 (0.71–1.24)
Marital/partner statusl
Married/Partnered		0.92 (0.87–0.98)	0.99 (0.93–1.05)	0.99 (0.93–1.06)
Educationm (years)
12			0.98 (0.89–1.07)	0.97 (0.89–1.07)
>12			0.95 (0.87–1.04)	0.94 (0.86–1.03)
Wealthn (quintiles)
Second			0.84 (0.77–0.91)	0.84 (0.77–0.91)
Third			0.83 (0.75–0.92)	0.83 (0.76–0.92)
Fourth			0.71 (0.65–0.78)	0.72 (0.66–0.78)
Fifth			0.76 (0.69–0.84)	0.77 (0.70–0.84)
Insurance coverageo			1.05 (0.89–1.24)	1.05 (0.89–1.24)
Physician visitp			1.12 (1.00–1.27)	1.12 (0.99–1.26)
Cognitive impairment–proxy interaction				1.19 (1.01–1.42)

Note. The unit of analysis for the findings in Table 1 is the individual respondent. CI = confidence interval.

^aLongitudinal two-level random-intercept logistic regression model, incorporating wave-by-wave sampling weights.

^bAdjusted for number of waves.

^cAdjusted for number of waves and four demographic characteristics.

^dAdjusted for number of waves, four demographic characteristics, education, wealth, and health care access.

^eAdjusted for number of waves, four demographic characteristics, education, wealth, healthcare access, and cognitive impairment-proxy interaction.

^fCognitive impairment at present or previous waves. Referent group: normal cognition.

^gUse of proxy at present wave. Referent group: no proxy used.

^hNumber of waves (participated in) at present wave.

ⁱAge at present wave.

^jReferent group: male.

^kReferent group: Caucasian.

^lMarital/partner status at present wave. Referent group: unmarried/unpartnered.

^mReferent group: <12 years.

ⁿWealth at present wave. Referent group: first (lowest) quintile.

^oHad insurance in the prior 2 years. Referent group: no insurance coverage in the prior 2 years.

^pHad one or more physician visits in the prior 2 years. Referent group: no physician visits in the prior 2 years.

Table 3.

Effect of Adjudication on Prevalence of Chronic Diseases

	Wave	Prevalence of Disease (%)		Difference Between Adjudicated and Unadjudicated Prevalence (%)		p Value
		Unadjudicated	Adjudicated	Absolutea	Relativeb
Hypertension	1998	45.51	45.91	0.40	.88	<.001
	2010	60.61	62.79	2.18	3.60	<.001
Heart	1998	22.18	22.55	.37	1.67	<.001
	2010	26.21	28.53	2.32	8.84	<.001
Lung	1998	8.83	9.07	.24	2.68	<.01
	2010	10.87	12.11	1.24	11.43	<.001
Diabetes	1998	13.27	13.25	−.02	−.14	.82
	2010	22.37	22.91	0.53	2.38	<.001
Cancer	1998	10.88	11.02	.14	1.32	<.001
	2010	15.67	16.91	1.24	7.91	<.001
Stroke	1998	6.37	6.61	.24	3.82	<.001
	2010	7.49	8.50	1.01	13.49	<.001
Arthritis	1998	51.05	51.16	0.11	0.22	.25
	2010	61.21	63.99	2.77	4.53	<.001

Note. The unit of analysis for the findings in Table 3 is each individual response to each individual chronic disease. Weighted percentages for the prevalences were derived using HRS respondent population weights to adjust for differential probability of selection into the sample and differential non-response. Weights are specific to the wave of the analysis, 1998 and 2010, respectively. HRS = Health and Retirement Study.

^aAbsolute difference: difference between the unadjudicated prevalence and adjudicated prevalence.

^bRelative difference: absolute difference divided by the unadjudicated prevalence.

Table 2.

Inconsistent Responses and the Results of Their Adjudication, Across All Waves by Chronic Disease (1998–2010)

	Inconsistent Responses		Responses With a Status Change following Adjudication
	Sum	Percentage (%) ( of All Responses for Indicated Disease)	New “Yes” Responses	New “No” Responses	Sum	Percentage (%) (of All Responses for Indicated Disease)
Hypertension	3,511	2.98	2,374	689	2,903	2.47
Heart	2,441	2.07	1,942	429	2,267	1.93
Lung	1,864	1.58	1,100	304	1,322	1.12
Diabetes	1,433	1.22	472	215	591	0.50
Cancer	952	0.81	852	209	979	0.83
Stroke	874	0.74	935	131	1,001	0.85
Arthritis	4,219	3.59	1,554	2,150	2,457	2.09

Note. The unit of analysis for the findings in Table 2 is each individual response to each individual chronic disease.

Addressing the extent of inconsistency in longitudinal data requires several methodologic decisions at the outset. First, we chose to use the first standardized wave of the HRS—1998—as the first longitudinal wave in our analyses. However, for respondents in the AHEAD and the original HRS cohorts, we pulled data from 1995 and 1996 waves, respectively, when that data were informative (i.e., provided evidence that the respondent had a disease). Second, analyses of the reporting of individual chronic diseases by individual respondents can be performed from different perspectives or using different units of analysis. For much of our work, the unit of analysis was each response to each chronic disease (Tables 2 and 3, Supplementary Tables 2–4). We performed these analyses for each disease for each survey wave; we then summed the number of inconsistencies and adjudications for each disease across all waves to derive a total for each disease, producing the results in these five tables. In contrast, for other analyses, we were interested in the findings from the perspective of the individual respondents (Tables 1 and 4, Supplementary Table 1). Here, we specified that a respondent had inconsistency in self-reporting a disease if the respondent had at least one inconsistency for that disease, recognizing that some respondents had multiple inconsistencies for a single disease.

Table 4.

Respondents With Inconsistent Responses Across Waves and Their Adjudication, by Chronic Disease

	Preadjudication		After Adjudication Steps 1–4		After Adjudication Steps 1–5
	Respondents With Inconsistent Responses		Inconsistent Respondent Cases That Have Been Resolveda		Inconsistent Respondent Cases That Have Been Resolveda
	Number	Percentage (%)b	Number	Percentage (%)c	Number	Percentage (%)c
Hypertension	1,894	7.84	782	41.29	1,374	72.54
Heart	1,356	5.61	463	34.14	1,005	74.12
Lung	1,011	4.19	283	27.99	590	58.36
Diabetes	704	2.91	327	46.45	450	63.92
Cancer	532	2.20	54	10.15	378	71.05
Stroke	583	2.41	167	28.64	443	75.99
Arthritis	2,370	9.81	630	26.58	1,391	58.69

Note. The unit of analysis for the findings in Table 4 is the individual respondent.

^aA respondent case for a disease is resolved if, after applying the adjudication methodology, the respondent’s pattern of responses for that disease across interview waves became nonproblematic: all “Yes” responses, all “No” responses, or one or more “No” responses followed by “Yes” responses.

^bDenominator is all respondents (N = 24,156).

^cDenominator is all inconsistent respondent cases for the indicated disease.

We grouped the respondents by whether or not they provided any inconsistent responses over the survey period; we examined differences in respondent characteristics between the two groups using χ² for categorical variables, the Wald test for age and the number of waves participated in, and the Wilcoxon test of median difference for income and wealth (Supplementary Table 1; Lumley & Scott, 2013). We then performed a multivariable, longitudinal analysis of the association between respondent characteristics and the probability of providing an inconsistent response at each study wave (Table 1). We fit a two-level, random intercept, logistic regression model with study wave at Level 1 and respondent at Level 2. We adjusted for the complex sample design by incorporating sampling weights for the Level 2 individuals and the Level 1 time points derived from the HRS respondent sampling weights (Rabe-Hesketh & Skrondal, 2006) and computed robust (linearization-based) standard errors to account for clustering within primary sampling units (Heeringa, West, & Berglund, 2010). We constructed the model in four stages. In Stage 1, we regressed the binary indicator of inconsistent response (to any of the chronic disease questions) on cognitive impairment, use of a proxy, and the number of previous waves the respondent had participated in. In Stage 2, we added age, gender, race/ethnicity, and marital status. In Stage 3, we added education, wealth, insurance coverage, and physician utilization. Last, in Stage 4, we included an interaction term between cognitive impairment and the use of a proxy. With the exception of gender, race/ethnicity, and education, we allowed the value of predictors to vary across waves.

To assess the impact of the adjudication methodology, we tabulated the number of new “Yes” and new “No” responses and total numbers and percentages of changed responses for each disease across all waves (Table 2, Supplementary Table 3). We built on the existing HRS recommendation (Fisher et al., 2005) to recode Respondent Option 3 as “Yes” and Respondent Option 4 as “No”. We only counted an adjudicated response as a new “Yes” or a new “No” response if it differed from the HRS recommendation so as to not overstate the results for the adjudication methodology.

We report the difference between the unadjudicated prevalence and the adjudicated prevalence for each disease (Waves 1998 and 2010) in two ways (Table 3, Supplementary Table 4). The absolute difference is the percentage difference between the two prevalences. The relative difference is the absolute difference divided by the unadjudicated prevalence of the disease in that wave. We used McNemar’s test for paired proportions to test the difference between unadjudicated and adjudicated prevalences.

We defined a respondent case for a disease as resolved if, after applying the adjudication methodology, the respondent’s pattern of responses for that disease across interview waves became nonproblematic. That is, the respondent’s pattern of responses was all “Yes” responses, all “No” responses, or one or more “No” responses followed by “Yes” responses (Table 4).

Last, we encountered and developed solutions for three smaller data problems. First, Wave 1998 had a preload error for heart disease. (Some AHEAD respondents, who had previously reported having heart disease, were presented the 1998 heart disease question as if they did not have heart disease.) We systematically tracked these respondents through the 1998 and following waves, providing a correction for this error for these respondents across waves. (We found that the consequences of the error did not persist beyond the 2002 wave.) Second, the case of stroke is unique in that it includes an additional option (transient ischemic attacks) in self-reporting the disease; our adjudication for stroke accommodated this. Third, for all seven chronic diseases, the question format in Wave 1998 did not include an Option 4; rather Option 3 was a general dispute, “Disputes previous wave record.” We used evidence from the 1995 and 1996 waves to adjudicate Option 3 inconsistencies for the 1998 wave.

For all analyses, we utilized the statistical software STATA, version 14 (Stata Corp, College Station, Texas). To conduct the multilevel analysis, we employed the GLLAMM add-on program for Stata (Rabe-Hesketh, Skrondal, & Pickles, 2004).

Results

Nearly 30% of respondents had inconsistency in their self-report of chronic diseases across interview waves: 21.1% with inconsistency involving a single disease; 5.5%, inconsistencies involving two diseases; and 1.8%, inconsistencies involving three or more diseases. We found these inconsistencies to be associated with respondents’ demographic, interview, and health care access characteristics (Supplementary Table 1, weighted data). Compared with those with no inconsistencies, respondents with inconsistencies were older, women, of Hispanic ethnicity, unmarried, and less educated and had lower income and wealth. As expected, respondents who participated in more interview waves were more likely to have inconsistencies in disease reporting. For example, respondents in the AHEAD and the original HRS cohorts had more inconsistencies than those in the WB and EBB cohorts. Respondents who never had insurance coverage were less likely to have inconsistency and those with intermittent coverage were more likely. Similarly, respondents with no self-reported physician visits across interview waves were less likely to have inconsistency and those who reported intermittent physician visits were more likely.

In cross-sectional weighted analyses, respondents with cognitive impairment, a proxy, or both (10%–14% of respondents at each wave) accounted for 16.80% to 32.91% of respondents with inconsistency in disease reporting (p < .001 for each wave). Thus, the cognitive impairment and proxy contributions to inconsistency were sizable but not preponderant. Of the respondents with cognitive impairment, 10.91%–23.40% had inconsistency; of those with a proxy, 6.83%–25.22% had inconsistency. Thus, for cognitively impaired and/or proxied respondents, inconsistency in disease reporting was substantial. Respondents with depressive symptoms (15% of respondents at each wave) accounted for 16.08%–22.26% of respondents with inconsistency (p < .01 for five waves); of respondents with depressive symptoms, 6.11%–9.13% had inconsistency.

Table 1 (weighted data) examines respondent characteristics as predictors of inconsistency in the self-report of chronic disease. The odds ratios for the associations of cognitive impairment and of proxy status with inconsistency were statistically significant (Model 1). They remained so in models adjusting for demographic characteristics (Model 2) and for education, economic status, and health care access (Model 3). Testing the model for interactions, we found a statistically significant positive cognitive impairment–proxy status interaction (Model 4). Other respondent characteristics predictive of inconsistency were age, Hispanic ethnicity, and wealth. Gender, African American race, marital status, education, and health care access were not predictive of inconsistency.

Table 2 (unweighted data) shows, in its first two columns, the total number of inconsistent responses for each disease across the seven interview waves (1998–2010), and it shows the percentage of these inconsistent responses of all the responses for each disease across the waves. (The unit of analysis is each response to each chronic disease). Arthritis and hypertension most frequently had inconsistent responses, 3.59% and 2.98%, respectively. Stroke and cancer had the fewest inconsistent responses, 0.74% and 0.81%, respectively. Supplementary Table 2 provides additional detail, showing the numbers for each type of inconsistent response.

Table 2 shows, in its last four columns, the adjudication of the inconsistent responses for each disease across waves. Table 2 indicates the numbers of new “Yes” and new “No” responses, the total number of changed responses, and the percentage of changed responses (of all the responses for that disease across all waves). Diabetes, cancer, and stroke had the fewest changes (less than 1%); arthritis and hypertension, the most frequent (greater than 2%). Supplementary Table 3 provides additional detail, showing the numbers of successfully adjudicated responses for each type of inconsistent response.

Table 3 (with each response to each disease as the unit of analysis, as in Table 2) presents findings on how adjudication affects the prevalence of each chronic disease (weighted data). For example, hypertension in 2010 had an unadjudicated prevalence of 60.61% and an adjudicated prevalence of 62.79%. The absolute difference in prevalence between the unadjudicated and adjudicated prevalences was 2.18%; the relative difference was 3.60%. Table 3 presents data for each disease for Waves 1998 and 2010, not showing the findings for the intervening waves. As expected, absolute and relative differences in prevalence accumulate with succeeding interview waves, so that differences are smallest for 1998 and largest for 2010. By 2010, the disease with the largest absolute difference in prevalence is arthritis, 2.77%. However, two diseases have substantial increases in relative prevalence: lung disease, 11.43%, and stroke, 13.49%; both are diseases having lower prevalence in the older adult population, thereby more affected by any change in prevalence with adjudication.

We repeated the previous longitudinal analyses of inconsistent responses and their adjudication but this time included cognitive status and proxy status. Supplementary Table 4 (with each response to each disease as the unit of analysis, weighted data) replicates Table 3; it presents findings on how adjudication affects the prevalence of each chronic disease, but does so by cognitive status and proxy status. In general, responses by those with cognitive impairment alone or with both cognitive impairment and a proxy were more likely to be inconsistent. Heart disease, lung disease, cancer, and stroke had substantial increases (>20%) in relative prevalence.

Table 4 (unweighted data) summarizes findings about inconsistencies and their adjudication from the perspective of the respondents; (the unit of analysis is each individual respondent). The first column lists, for each chronic disease, the numbers of respondents with at least one inconsistent response for that disease across all waves in which the respondent participated; the adjacent column lists the percentage of respondents (of all the respondents) having at least one inconsistent response. For example, for hypertension, 1,894 respondents had at least one inconsistent response; these 1,894 respondents represent 7.33% of all respondents. The next columns show the outcomes after performing Adjudication Levels 1 through 4, that is, the number and percentage of respondent cases successfully adjudicated. For hypertension, 782 of the 1,894, or 41.29%, of respondent cases with at least one inconsistent response have been resolved. The final columns show the outcomes after performing Adjudication Levels 1 through 5. For hypertension, 1,374 of the 1,894, or 72.54%, of respondent cases with at least one inconsistent response have been resolved. Thus, Table 4 shows the effect of performing the more ambitious adjudication, compared with the more conservative adjudication. Table 4 also indicates that the adjudication steps are most successful for stroke (resolving 75.99% of inconsistent cases) and heart disease (74.12%) and least successful for lung disease (58.36%) and arthritis (58.69%).

Discussion

This study investigated consistency in the self-reporting of chronic diseases by respondents across multiple waves of a large longitudinal health interview survey. Our goal was to identify and characterize the extent of inconsistent responses and then to develop a method to adjudicate them. The key to our approach was making use of detailed chronic disease information from prior waves to inform and adjudicate inconsistencies in later waves. Using a stepwise method, we were able to adjudicate 60%–75% of the inconsistent responses. In so doing, we found that differences in disease prevalence between original and adjudicated data accumulate across succeeding waves, such that relative differences in prevalence may be substantial for some diseases. We also found that inconsistencies were concentrated in some subpopulations, particularly Hispanics, the less educated, the less wealthy, those with intermittent health care access, and the cognitively impaired. Upon further analysis, cognitive impairment, proxy status, age, Hispanic ethnicity, and wealth proved to be actual predictors of inconsistency.

One methodological (and philosophical) critique of adjudication is that it overrides respondents’ answers to health interview questions. An argument can be advanced that the most valid approach to self-report chronic disease data is to let the original responses stand, recognizing and accepting some degree of inconsistency. We acknowledge this viewpoint and consider it appropriate, especially for cross-sectional studies. Yet for cross-sectional studies and for longitudinal studies, our adjudication method, at minimum, serves as a sensitivity analysis, comparing findings using unadjudicated data to those using adjudicated data.

A related issue is what constitutes “truth.” For example, in our adjudication, in cases of multiple inconsistent responses by a respondent for a single disease, it may not be possible to specify which responses are correct and which are erroneous. The literature on this topic is broad. For chronic diseases, some researchers want to know how well respondents’ answers correspond to the truth as “objective fact.” (Fowler, 2014) (Did a physician tell the respondent that he has diabetes? Does the respondent have diabetes?) These facts can be verified via comparison to a gold standard (e.g., medical record). The challenge is that the gold standard data may be difficult and expensive to obtain. However, other researchers are interested in respondents’ self-report of diseases as truth as “subjective perception.” (What and how did the physician communicate to the patient? What did the patient hear? Does the patient have symptoms?) Here, Leventhal’s illness representations model provides a useful framework, by expanding the concept of disease beyond pathophysiology and clinical diagnosis to a patient’s understanding of a disease that is informed by its symptom burden and chronicity (Halm et al., 2006; Leventhal & Crouch, 1997).

The HRS itself acknowledges that researchers may handle the inconsistencies in the disease data differently depending on their research aims. The HRS user guide (Fisher et al., 2005) suggests one strategy: coding Option 3 responses (“Disputes previous wave record, but now has condition”) as “Yes” responses and Option 4 responses (“Disputes previous wave record, does not have condition”) as “No” responses. Our study builds on and substantially extends this strategy first by comprehensively assessing the extent of inconsistency; second, by addressing longitudinality between adjacent waves and across all waves; and, third, by proposing a comprehensive adjudication methodology.

The RAND Corporation, which provides cleaned and aggregated files for some HRS wave data, likewise has developed a strategy to address inconsistent responses to the disease questions. RAND files include “_E variables” that take into consideration previous and current wave responses (Fisher et al., 2005). If a respondent disputes a disease, RAND resets all prior wave responses to “No”. For example, if a disease in disputed in Wave 2002, it resets Waves 1998 and 2000 responses to “No”. This includes cases where a respondent gave information about treatment for a disease. If a respondent indicated receiving chemotherapy for cancer or using insulin for diabetes at these prior waves, the _E variables are still set to “No”. Thus, the RAND _E variables function in a direction counter to our adjudication methodology that uses prior evidence of treatment to change a subsequent inconsistent “No” response to “Yes”. A further distinction from RAND is that our methodology relies on a respondent twice indicating that he has a disease: first, in his response that he has been told by a physician that he has the disease and, second, in his response that he is being treated for the disease. In the understanding of disease as objective fact, the RAND methodology is more specific for each disease, and ours, more sensitive. The two methodologies may vary in appropriateness for different research questions, depending on the importance of disease sensitivity or specificity. (Fowler, 2014)

Inconsistencies in disease reporting could be explained by a change in the fact of a diagnosis, such as a misdiagnosis and/or a respondent obtaining a second opinion. Yet the vast numbers of inconsistency patterns do not fit a “Yes,” followed by a “No” with no other problematic responses. Similarly, inconsistencies could be explained by cure. Such an explanation involves untangling the chronic disease construct as both fact and perception. By definition, chronic diseases are not curable. For example, diabetes patients who make substantial behavior modifications can correct their hyperglycemia. Yet the underlying pathophysiology, nonfunctioning pancreatic islet cells (type 1) or insulin resistance (type 2), remains, and the ongoing behavior modifications are themselves nondrug treatments of the disease. Here, even though the disease persists (objective fact), the lack of symptoms and of medication interventions affects the respondent’s subjective perception of it. (Fowler, 2014; Leventhal & Crouch, 1997)

A methodological issue related to but not identical with consistency in longitudinal reporting is the concordance between self-reported disease information and medical record data or administrative data. Generally, researchers have found a substantial agreement here for many common chronic diseases, such as diabetes and hypertension (Weir, 2008; Okura et al., 2004; Singh, 2009). Low incidence diseases that represent seminal events, such as stroke, also have agreement between self-report and medical record. Thus, diseases with ongoing intervention or hard events fit the illness representations model (Leventhal & Crouch, 1997). However, the concordance literature is primarily limited to cross-sectional analyses or, at best, adjacent years of data to examine corroboration between data sources. Regarding the HRS itself, Wolinsky and colleagues investigated the concordance of self-report with Medicare claims data for the AHEAD cohort from 1991 to 2010, finding problems for hypertension, heart disease, and arthritis (Wolinsky, Jones, Ullrich, Lou, & Wehby, 2014). Our study, investigating not concordance but longitudinality, adds to their work in several ways: analyzing five HRS cohorts, examining the longitudinal consistency within individuals’ self-reports, and developing an adjudication methodology. It is in this area that the literature has been silent: recognizing and providing guidance on methods to handle respondents’ longitudinal records of disease reports that fluctuate in ways not consistent with the clinical course of chronic disease.

Multiple studies have shown that differences in the prevalence, management, and outcomes of chronic diseases are related to ethnicity, education, economic status, and access to health care (Hayward et al., 2000; Link & Phelan, 1995; Woolf & Braveman, 2011). Our study adds to this literature by examining the association of respondent characteristics with inconsistency in the self-report of chronic diseases over time. Cognitive impairment and the use of a proxy proved to be key predictors, suggesting that memory impairment has a strong effect on how individuals remember, understand, and report their diseases (Hugo & Ganguli, 2014). Our analyses found age, Hispanic ethnicity, and wealth to be additional predictors. There were no persistent associations with gender, African American race, marital status, education, insurance coverage, or frequency of physician visits.

Our study builds on the strengths of the HRS. It is a large, nationally representative, longitudinal survey that includes the middle-aged, the oldest-old, the community-dwelling, and those residing in long-stay nursing facilities. Many of the chronic disease questions are rich in detail, especially in the context of the multiple waves of the HRS. The HRS also includes detailed data on important covariates and outcomes, including demographic factors, spousal information, cognitive status, health care utilization, and mortality.

Our study is limited in that, although the information about the diseases is detailed, it is self-reported, and it is not exhaustive (e.g., no medication names). A key limitation is that the adjudication method works only in one direction; it can only adjudicate a “Yes” response followed by a “No” response. It is unable to address underdiagnosis or the initial timing of a diagnosis.

A chief aim of this research has been to do the detailed work of addressing inconsistencies in the HRS so that our findings and adjudication methodology may be of use to others. Our methods can serve as a sensitivity analysis for researchers using HRS data for cross-sectional and longitudinal investigations. Further, identifying inconsistency and remedying it is a foundational step when linking the HRS to the other HRS family data sets.

Our methods and findings have implications more broadly for the use of large longitudinal survey data sets. Responses to questions can be validated or adjudicated by utilizing additional evidence within the survey itself. Researchers can make use of the cascading or unfolding of evidence that respondents provide as they proceed deeper into specific interview topic areas. We believe that respondents are less likely to erroneously report a series of answers about a disease in contrast to a single answer to a single disease question. Thus, large longitudinal surveys enable researchers to weigh the preponderance of the evidence when adjudicating inconsistent responses across multiple waves.

Our findings also suggest possible considerations in the design of longitudinal health interview surveys. It may be helpful to ask all respondents, not only those reporting having a disease, follow-up questions about that disease. More detailed questioning of respondents who dispute a prior “Yes” response may reveal the reason for the dispute and provide information about the respondent’s understanding of the disease diagnosis. Caution, though, is warranted; the HRS Wave 1998 preload error for heart disease demonstrates that respondents modify their reports according to interviewer statements.

Diseases lacking objective diagnostic criteria are more subject to inconsistency in self-reporting, and diseases with low prevalence in the population may be more sensitive to that inconsistency. For all diseases, identification of incident cases is particularly challenging. HRS data are structured so that cumulative incidence (e.g., 2-year cumulative incidence) can be calculated. However, these estimations may be erroneous if the whole of each respondent’s longitudinal record is not taken into account.

Lastly, we found that certain respondent characteristics—cognitive impairment, use of a proxy, age, Hispanic ethnicity, and wealth—were predictors of inconsistency in chronic disease reporting. These findings suggest that researchers should use additional care in analyses of these population subgroups, probing the data for inconsistency. This caution is especially pertinent for older adults, given the increasing prevalence of cognitive impairment among older adults as they age. Nonetheless, cognitive impairment and/or having a proxy accounted for less than a third of inconsistencies. Researchers must be attuned to other contributors and explanations, from recall bias, to lack of understanding of what a chronic disease is and entails, to reluctance to acknowledge having a disease that is not curable, has ongoing morbidity, and requires the responsibility of self-management.

In sum, the study’s findings have methodological ramifications for research utilizing the HRS and other large longitudinal health interview surveys. The findings have methodological and substantive implications for the study of chronic disease, whether single diseases, combinations of diseases (comorbidity), or overall burden of diseases (multimorbidity), and for measures of disease (incidence, prevalence, and concordance with other data sources [laboratory data and administrative data]). Further, the findings identify adults with chronic disease who may be at risk for impaired understanding of disease and difficulty with self-management of their disease. (Garber, Nau, Erickson, Aikens, & Lawrence, 2004; Simpson et al., 2004).

Supplementary Material

Supplementary material is available at The Journals of Gerontology, Series B: Psychological Sciences and Social Sciences online.

Funding

The research presented is supported by the National Institute On Aging of the National Institutes of Health under award. The content is solely the responsibility of the authors and does not represent the official views of the National Institutes of Health. This work was supported by the National Institute on Aging at the National Institutes of Health (grant number R03AG048852 to A.R.Q. and C.T.C; 5K08AG031837 to C.T.C.); the Ann Arbor VA Geriatric Research, Education and Clinical Center (GRECC) to C.T.C.; the University of Michigan Claude D. Pepper Older Americans Independence Center to C.T.C. and J.L.; the National Hartford Centers of Gerontological Nursing Excellence to C.L.N.; the American Diabetes Association Career Development Award (grant number ADA 7-13-CD-08 to A.R.Q.); and the Summer Institute on Mentoring Researchers in Latino Health Disparities at San Diego State University (grant number NIH/NHLBI R25HL105430 to A.R.Q.).