A comparison of response rates in the English Longitudinal Study of Ageing and the Health and Retirement Study

Abstract

Survey response rates are an important measure of the quality of a survey; this is true for both longitudinal and cross-sectional surveys. However, the concept of a response rate in the context of a panel survey is more complex than is the case for a cross-sectional survey. There are typically many different response rates that can be calculated for a panel survey, each of which may be relevant for a specific purpose. The main objective of our paper is to document and compare response rates for two long-term panel studies of ageing, the English Longitudinal Study of Ageing (ELSA) and the Health and Retirement Study (HRS) in the United States. To guide our selection and calculation of response rates for the two studies, we use a framework that was developed by Peter Lynn (2005) and present several different types of longitudinal response rates for the two surveys. We discuss similarities and differences in the study designs and protocols and how some of the differences affect comparisons of response rates across the two studies.

Introduction

Response rates are often used to gauge the quality of a survey. They provide a single measure that is taken to reflect the representativeness of the respondents who participated and the overall quality of the survey. To maximize response rates, panel surveys work hard to retain participants over time and to limit the loss of statistical power from attrition out of the study. Some fieldwork strategies, like the use of incentives, are often used to encourage participation amongst specific groups, which helps to address concerns about the possible effect of non-response bias on some measures derived from the survey data.

The use of response rates as a basis for comparison across different studies has some key limitations. Despite recent efforts to develop guidelines (The American Association for Public Opinion Research, 2008) there is a lack of standardisation in the way that survey organizations and researchers use survey outcome data to calculate response rates. Second, particularly in the case of panel surveys, which involve more than one data collection round, there are many different ways that response rates can be calculated. In recognition of this, Lynn (2005) has developed a framework of best practice in the recording of outcomes and the estimation and presentation of response rates for surveys with multiple data collection events. Lynn proposes that no single response rate can summarize the overall level of response in a panel survey and that different response rates are relevant for different analytic and evaluative purposes. The following rates are outlined by Lynn (2005):

Longitudinal response rates

Longitudinal response rates are useful for analysts who make use of multiple data collection rounds for longitudinal analysis at the individual (or micro) level. The “complete” response rate is defined by Lynn as the proportion of sample members who participate in every data collection round, of those who were eligible for all rounds. This response rate gives an indication of the completeness or representativeness of the sample that is used in longitudinal analyses involving all waves. Longitudinal response rates may also be calculated for any subset of (two or more) waves.

Cross-sectional response rates

Cross-sectional response rates are needed for those who restrict their analysis to a single round of data collection and/or to a set of discrete waves. Response rates may either be unconditional (i.e. the proportion of sample members who respond in a given wave of all those who are eligible in that wave) or conditional on prior wave response (i.e. the proportion of sample members who respond in a given wave of those who responded in the immediately prior wave or some other prior wave).

The primary objective of this paper is to use Lynn’s framework to document response rates for two large and influential panel studies of ageing, the English Longitudinal Study of Ageing (ELSA) and the Health and Retirement Study (HRS) based in the United States. The ELSA study was designed to be comparable to the HRS from the outset and, as a result, the two studies share many core design features. The samples for both studies are nationally representative of community dwelling, middle-aged and older adults (age 50+ for ELSA and age 51+ for HRS).ⁱ Both are longitudinal panel surveys and conduct a core interview with the same study participants every two years. The questionnaires for the core interview contain substantial overlap on a wide range of topics including employment history, retirement experiences, plans and expectations, economic status, family and household composition, support transfers, health, disability and use of health services.

This is the first time that Lynn’s framework has been applied to these two studies. The paper focuses on variations in sample design and fieldwork protocol employed by each study, while at the same time discussing the utility and limitations of the Lynn framework. The paper does not discuss at any length the implications of response rate differences across studies, and no attempt is made to measure response bias. Both issues warrant further investigation, but go beyond the scope of this paper. The issue of response bias is addressed in several other papers in this Special Issue of the journal.

The paper first provides a description of the sample designs for the ELSA and HRS studies before moving to the eligibility criteria adopted for inclusion in the response calculations. Each type of response rate is then presented separately before leading to a discussion of cross-study differences.

Design of the ELSA and HRS studies

English Longitudinal Study of Ageing

The ELSA sample was designed to represent people aged 50 and over (persons born on or before 29 February 1952), living in private households in England at the time of wave 1. A total of 11,391 interviews were achieved with age-eligible sample members (or core members) at the first wave. The sample was selected from households that had previously responded to the Health Survey for England (HSE). The HSE uses a multi-stage stratified random sampling procedure. Three HSE years, 1998, 1999 and 2001 were selected as the sampling frame for ELSA wave 1. Each of these HSE years had a general population “core” sample that was nationally representative. HSE 1999 also had an ethnic minority boost sample but this was not followed up for ELSA. The HSE response rates are found to be relatively constant from year to year.ⁱⁱ

HSE households were only issued to the field at ELSA wave 1 if they included at least one individual who was age-eligible, and who according to administrative records remained alive and gave permission to be re-contacted in the future. No indication was given to respondents at the time of their HSE interview that they would be specifically approached for the ELSA study. The decision to follow-up participants from HSE resulted in ELSA inheriting a large degree of non-response prior to its first wave. In HSE cooperating households, age-relevant information had been collected in order to establish which households were eligible for ELSA, but this was not available for HSE non-cooperating households as no interview had taken place. As a result, for response rate calculations it was necessary to assume that the same proportion of people in HSE non-cooperating households would have been eligible for ELSA. Further detail on the ELSA sampling design can be found in Appendix A or the ELSA Technical Reports (Taylor et al 2007; Scholes et al 2008; Scholes et al 2009). Details about the HSE are also available from its Technical Reports (Erens and Primatesta 1999; Erens, Primatesta and Prior 2001; Prior et al 2003).

Table 1 illustrates the gap in time between the HSE interview and ELSA wave 1. The HSE interview represented the first time ELSA participants were approached and their HSE data can be linked by analysts to data collected as part of the ELSA study. The HSE interview is therefore treated as ELSA Wave 0 and is included as the first stage in the calculation of response rates. The first wave of ELSA fieldwork started in March 2002. Those sampled from HSE 1998 had the largest gap of four years between their HSE and ELSA interview. Those sampled from HSE 2001 had the smallest gap of one year, and therefore unsurprisingly had the highest household contact rate at ELSA wave 1 compared with HSE 1998 and 1999 (Taylor et al 2007).

Table 1.

Timing of HSE and ELSA waves

BIRTH COHORT		HSE			ELSA
		1998	1999	2001	2002	2003	2004
1900 – 1952	HSE 1998	*			*		*
1900 – 1952	HSE 1999		*		*		*
1900 – 1952	HSE 2001			*	*		*

Health and Retirement Study

The Health and Retirement Study (HRS) has a nationally representative sample of over 20,000 men and women over the age of 50 in the United States. The study began in 1992 as a longitudinal study of a pre-retirement cohort of individuals born in 1931–1941 and their spouses of any age. This birth cohort is referred to as the original HRS cohort. In 1993 a parallel study, the Study of Asset and Health Dynamics of the Oldest Old (AHEAD), was launched. The AHEAD sample was comprised of a cohort of persons born before 1924 and their spouses of any age. In 1998, the study design was modified to convert the HRS sample from a set of specific cohorts into a steady state sample that represents the community-dwelling U.S. population over age 50. This was achieved by combining the HRS and AHEADcohorts into a single data collection effort, based on a common questionnaire and common field protocols, and adding new cohorts in 1998 to fill in the age range over 50 (the CODA cohort consisting of persons born between 1924 and 1930 and the War Baby (WB) cohort born between 1942 and 1947). The steady state design is maintained by adding a new six-year cohort of persons in their early to mid-50s every six years (2004, 2010, etc).

Sample design

In 1992, a large household screening operation based on an area multi-stage probability sample design was undertaken to identify eligible sample members for the HRS, AHEAD and WB cohorts. A brief screening interview was attempted with approximately 69,500 households across the United States. The screening interview contained a listing of all adult members of the household, their year of birth, and partner status. About 14% of addresses in the sample were found to be unoccupied or non-residential and are considered non-eligible. Of the remaining addresses, screening interviews were completed with over 99% (Heeringa and Conner 1995).

The AHEAD sample was supplemented at the oldest ages (age 80+) with individuals selected from the Medicare Beneficiary list maintained by the then Health Care Financing Administration (HCFA), now the Center for Medicare and Medicaid Studies (CMS) (Heeringa 1995). The Medicare list provided roughly one-half of the sample with the remainder coming from the 1992 household screen. HRS had slightly better success recruiting respondents from the sample obtained through the household screener compared to the Medicare list frame (response rates for the two groups were 82% and 77%, respectively). The CODA sample (first interviewed in 1998) was drawn entirely from the Medicare list sample.

In 2004, a second household screening effort was undertaken with 38,385 households to identify eligible sample members for the Early Baby Boom (EBB) (born 1948–1953) and Middle Baby Boom (MBB) (born 1954–1959) cohorts. The EBB cohort was recruited into the study in the 2004 wave and the MBB cohort was added in 2010. The 2004 household screening interview was very similar to that used in 1992. 13% of households were determined to be vacant or non-residential (non-eligible) and, of the remaining households, a screening interview was completed with 91%.

In response rate calculations for the HRS, the screener response rate is factored into the baseline response rate for each entry cohort. Specifically, the baseline response rate is calculated as the product of the screener response rate and the interview response rate, where the interview response rate is the percentage of known eligible sample members who completed an interview. More detail on how eligibility is defined for each study is provided in a later section.

Minority individuals (Blacks and Hispanics) are oversampled in the HRS at a rate of about 2 to 1. In addition, the HRS, AHEAD and WB cohorts contain an oversample of Florida residents.

Table 2 identifies the years in which core interviews were conducted with each cohort. Each of the cohorts has been followed up at roughly two-year intervals since their introduction into the study. Although there is some variation across waves, since 1998 the fieldwork has generally started in February or March of the designated year and ended in January or February the following year.

Table 2.

Data collection years for the HRS, by study cohort

BIRTH COHORT	COHORT	DATA COLLECTION WAVE (1992–2006)
		92	93	94	95	96	97	98	99	00	01	02	03	04	05	06
1890 – 1923	AHEAD		*		*			*		*		*		*		*
1924 – 1930	CODA							*		*		*		*		*
1931 – 1941	Original HRS	*		*		*		*		*		*		*		*
1942 – 1947	War Baby							*		*		*		*		*
1948 – 1953	Early Baby Boom													*		*

Response rates: conceptualisation, calculation and comparison

Conceptual issues in comparisons of response rates across studies

Table 3 puts the two studies together and shows the years in which each study began and the cycle of data collection through to 2006. Subsequent rounds of data collection have taken place in 2008 and 2010, but those waves are not included in this paper.

Table 3.

Data collection years for ELSA and HRS

Year	ELSA	HRS
1992		HRS cohort (Wave 1)
1993		AHEAD cohort (Wave 1)
1994		HRS cohort (Wave 2)
1995		AHEAD cohort (Wave 2)
1996		HRS cohort (Wave 3)
1997
1998	Original cohort (Wave 0) (HSE Interview Years)	HRS cohort (Wave 4) AHEAD cohort (Wave 3) CODA & WB cohorts (Wave 1)
1999
2000		HRS cohort (Wave 5)
		AHEAD cohort (Wave 4)
		CODA & WB cohorts (Wave 2)
2001	Original cohort (Wave 0) (HSE Interview Year)
2002	Original cohort (Wave 1)	HRS cohort (Wave 6) AHEAD cohort (Wave 5) CODA & WB cohorts (Wave 3)
2003
2004	Original cohort (Wave 2)	HRS cohort (Wave 7) AHEAD cohort (Wave 6) CODA & WB cohorts (Wave 4) EBB cohort (Wave 1)
2005
2006	Original cohort (Wave 3) Refresher cohort (Wave 1)	HRS cohort (Wave 8) AHEAD cohort (Wave 7) CODA & WB cohorts (Wave 5) EBB cohort (Wave 2)

As illustrated by Table 3, the variations in survey design and fieldwork timing adopted by ELSA and HRS present difficulties around how best to calculate and compare response rates, and there are several different approaches that could be taken.

One option for response rate comparison would be to focus on a calendar year (or set of years) in which interviews were conducted in both studies, (i.e. 2002i.e. 2004 or 2006) and calculate response rates based on the group of respondents who were interviewed and eligible for interview in that year. Analysts might opt to use this approach in order to compare outcomes from each study at the same point in time. However, comparisons of response rates in this way are problematic, as the HRS is at a very different stage of its lifecycle than ELSA and participants have varying levels of study experience.

An alternative, and the one adopted in this paper, is to take a wave by wave approach, for example comparing the first wave of ELSA with the first wave of HRS, and repeating this for subsequent waves. It is then possible to track the success of each study in obtaining cooperation from respondents in the baseline wave, as well as in maintaining the panel of original sample members at each successive wave.

For ELSA, the full cohort of individuals age 50+ has been interviewed since the start of the study, making this approach fairly straightforward. (The refresher cohorts added in subsequent waves of ELSA are not included in response rate calculations for this paper.) However this is not the case with HRS. As noted previously (and shown in Tables 2 and 3), the HRS sample is made up of different birth cohorts, most of which entered the study in a different year. Thus, wave 1 for the original HRS sample occurred in 1992, whereas wave 1 for the Early Baby Boom cohort took place in 2004. Or, stated differently, 2004 was wave 1 for the EBB cohort, wave 4 for the CODA and WB cohorts, wave 6 for the AHEAD cohort, and wave 7 for the HRS cohort. In order to obtain comparable longitudinal response rates for the HRS, we first calculated wave-specific response rates (i.e. wave 1, wave 2, wave 3, etc) for each cohort, then took the weighted average of the wave-specific response rates (weighted by the sample size of each cohort) to get an overall response rate for each wave. This means that the response rates for different waves are based on different combinations of cohorts. For example, in Tables 5–8, response rates for waves 1 and 2 are based on all five cohorts, those for waves 3 to 5 are based on four cohorts (all except the EBB cohort, for which only two waves had been conducted through 2006), the response rate for waves 6–7 are based on only two cohorts (HRS and AHEAD), and that for wave 8 is based only on the original HRS cohort.

Table 5.

Unconditional cross-sectional response rates for each wave

Study	RR1	RR2	RR3	RR4	RR5	RR6	RR7	RR8
ELSA*	70.2%	46.5%	39.2%	36.1%
HRS	78.0%	72.5%	72.0%	70.3%	68.4%	69.1%	67.6%	66.4%

^*For ELSA, ₁=wave 0; ₂=wave1, etc.

Table 8.

Conditional longitudinal response rates (conditional on participation at baseline)

Study	RR1,2|1	RR1,2,3|1	RR1,2,3,4|1	RR1,2,3,4,5|1	RR1,2,3,4,5,6|1	RR1,2,3,4,5,6,7|1	RR1,2,3,4,5,6,7,8|1
ELSA*	81.5%	70.8%
HRS	92.6%	87.6%	83.3%	79.6%	74.7%	71.9%	68.6%

^*For ELSA, ₁=wave 1; ₂=wave 2, etc.

To further add to the complexity, it is also necessary to treat ELSA Wave 0 (HSE interview) as the first wave for comparison with the equivalent Wave 1 for HRS for some response rates. This is to take account of the fact that respondents had prior experience of the HSE interview, and therefore ELSA wave 1 could not be classified as the first contact attempt. It is important to note however that ELSA wave 1 is still referred to as the baseline wave of ELSA, as only households with at least one productive interview with an age-eligible sample member at wave 1 were followed up for interview at wave 2.

Eligibility criteria

Lynn’s (2005) framework highlights the importance of establishing clear and consistent definitional criteria before calculating response rates. A key issue relates to the survey outcome of interest (i.e. what is deemed a completed data collection event or “response”). This is made complicated when data collection events involve multiple components. For example, ELSA and HRS both have a biennial core interview followed by a self-administered questionnaire, but every four years ELSA also has a separate visit by a qualified nurse. In the calculation of response rates, should the data collection event therefore be considered “complete” only if all components have been successful? Also, if respondents start the interview but stop halfway, or if they decline to answer some questions, should these situations be treated as complete outcomes?

A second issue relates to eligibility status (i.e. who should be considered as part of the population of interest). In general, not all sample members of a survey may be eligible for all data collection rounds or events. HRS has carried out supplemental data collection exercises using postal or internet surveys to a sub-set of the main sample, so they may not all be eligible at the same point in time. Furthermore, deaths or geographical moves within the study area may affect eligibility status from one study wave to the next.

For each study, eligibility criteria were set in order to classify sample members according to their status at each wave. They were categorized as either respondents, non-respondents or ineligible. For analysis, respondents and non-respondents are always included in the response rate denominator, while those deemed ineligible are removed completely from the calculations. The definitional criteria used for each group are given below.

Respondents

Each study defines a “response” in a given wave as full or partial completion of a core interview or proxy interview, but not necessarily the supplemental components (e.g. self-completion or nurse visit).

Ineligible

As expected, both studies treat people who have died as ineligible. ELSA also classifies those who have moved outside of Britain as ineligible (due to moving out of the population of interest), where as HRS does not apply any geographical ruling to its eligibility criteria. Unlike HRS, ELSA also excludes those who have moved into an institution or care home from the response rate denominator, in order to allow consistent comparisons with earlier waves of ELSA. To explain this further, moves to an institution or care home were recorded by ELSA interviewers after wave 1, but the actual interview for those in care homes was first introduced at wave 3. ELSA Technical Reports also therefore treat those in care homes as ineligible in the published response rates.

The difference in eligibility criteria across studies described above is likely to have a negligible effect on response rates due to small numbers. By ELSA wave 3, only 89 age-eligible sample members interviewed at wave 1 were identified by interviewers to have moved out of Britain, and 76 had moved into an institution (most likely a residential/nursing home). In total this equates to only 1.4% of those who had successfully completed a wave 1 interview.

Non-respondents

This group consists mainly of those who have refused at a given wave or who could not be contacted. In addition, both studies include those who have asked to be removed from the sample, or who have moved and cannot be traced, as eligible for the study. For ELSA, only moves not known to be outside of Britain are classified as non-response.

Unknown eligibility

Each study has a sub-group of sample members whose eligibility is ‘unknown’ at a given wave due to non-contact or unsuccessful tracing. To compensate for this, ELSA estimated a proportion of cases with unknown eligibility to be ineligible at each wave using age-sex mortality rates and annual rates of moves into an institution. Those “unknown” cases not reclassified as ineligible remain as non-respondents. In contrast the HRS, for which the percentage with unknown status is typically very low, assumed all those with unknown status to be eligible in response rate calculations (so treat them as non-respondents). The justification for applying the age-sex mortality rates to “unknown eligibles” in ELSA is due to incomplete mortality information. Mortality checking is carried out for ELSA sample members who provided consent to linkage to the National Health Service Central Register at wave 1, but there are groups from HSE that are not covered by this process. For example, ELSA never established who resided in non-cooperating HSE 1998, 1999 or 2001 households, and as a result, no information is available to link to official mortality records. In addition, responding households at HSE that did not take part in ELSA wave 1 are not included in the mortality checking process (Taylor et al. 2007).

Response rate definitions

For ELSA, the response calculations are based on the original age-eligible sample members identified for wave 1 in 2002. The ELSA wave 3 refreshment sample of those aged 50–53 has not been included for ease of analysis and comparability with ELSA Technical Reports. For HRS, the response rate calculations are based on all age-eligible sample members at a given wave from all cohorts.

We focus on the following set of response rates outlined in Lynn’s framework: 1) unconditional cross-sectional response rates, 2) cross-sectional response rates conditional on completion in the prior wave, 3) unconditional longitudinal response rates, and 4) longitudinal response rates conditional on completion in the baseline wave. Each of these response rates is defined in Table 4.

Table 4.

Description of response rates from Lynn’s framework

Response rate	Description	Numerator	Denominator
RRi	Unconditional cross-sectional response rate for wave i	Response at wave i (i=1,…,k)	Eligible at wave i (i=1,…,k)
RRi|i−1	Conditional cross-sectional response rate for wave i, given completion in immediately prior wave (i−1)	Response at wave i (i=1,…,k)	Eligible at wave i (i=1,…,k) and response at wave i−1
RR{1,2,…,k}	Unconditional longitudinal response rate for any combination of waves 1,2,…,k	Response at every wave of interest (1,2,…,k)	Eligible at every wave of interest (1,2,…,k)
RR{1,2,…,k}|1	Conditional longitudinal response rate for any combination of waves 1,2,…,k, given completion in baseline wave (wave 1)	Response at every wave of interest (1,2,…,k)	Eligible at every wave of interest (1,2,…,k) and response at wave 1

The first two response rates are cross-sectional. As such, they represent the percentage of sample members who participated in a given wave of the survey. The unconditional cross-sectional response rate represents the percentage who participated in wave i, of all sample members who were eligible to participate in that wave (regardless of whether an interview was attempted). As such, it gives an indication of the percentage of the target population that is represented in the survey at each wave. The conditional cross-sectional response rate represents the percentage who participated in wave i, of those who were eligible to participate in wave i and who had participated in the previous wave (i−1). This is sometimes referred to as a “re-interview” response rate, and reflects the success of a survey at retaining respondents from one wave to the next. Cross-sectional response rates are most useful for analyses using a single wave of survey data.

The lower pair of response rates from Table 4, that we focus on in the paper, are longitudinal response rates. Longitudinal response rates represent the percentage of sample members who participated in multiple waves of the survey (minimum of two, up to the total number of waves that have been conducted), of those who were eligible in all of those waves. The unconditional longitudinal response rate represents the percentage of sample members who participated in each wave of a series of waves, of all sample members who were eligible to participate in each of those waves. As with the unconditional cross-sectional response rate, it gives an indication of the percentage of the target population that is represented in a sequence of waves in the survey. The conditional longitudinal response rate represents the percentage of sample members who participated in each of a series of waves, of those who were eligible in all of those waves and who participated in the baseline wave. As such, it reflects the success of the survey in retaining the original panel in subsequent waves. Generally longitudinal response rates correspond with a series of consecutive waves starting with the baseline wave (e.g., waves 1–4), but they could also be calculated for other sequences of waves (e.g. waves 1, 3, 5, or waves 4–8). Longitudinal response rates are most useful for analyses that make use of multiple waves of survey data; the rate that is most relevant for a given analysis will correspond with the waves used in the analysis.

Response rates for each study

Tables 5–8 present response rates for each of the four types described in Table 4. Appendix B includes examples based on ELSA, which illustrate how the different types of response rates were calculated.

Cross-sectional response rates

Table 5 presents unconditional cross-sectional response rates for ELSA and HRS for each wave. These rates represent the percentages of all eligible sample members in the designated wave who completed an interview in that wave. For HRS, the unconditional cross-sectional response rate for wave 1 (RR₁) is typically referred to as the baseline response rate, whereas for ELSA this rate corresponds with response at HSE (wave 0). As a result the RR₂ rate reported for ELSA actually represents that achieved at wave 1 (baseline).

For HRS the baseline response rate was 78%. At ELSA wave 0 the achieved rate was 70.2% and the baseline rate at wave 1 was 46.5%. The ELSA wave 1 rate includes non-cooperating HSE households in the denominator (see Appendix B for example calculation). Because non-responding households at wave 1 were not followed in either of the studies, response rates for each consecutive wave are necessarily lower than the baseline response rates. The unconditional cross-sectional response rate at wave 2 was 72.5% for HRS. By wave 8 of HRS, about two-thirds of the original target sample who were still eligible at wave 8 completed an interview in that wave.

Table 6 presents conditional cross-sectional response rates, which are based on those who responded in the previous wave. The numerator for each rate includes individuals who completed an interview in both waves i and waves i−1 and the denominator includes those who were interviewed in wave i−1 and still eligible in wave i.

Table 6.

Conditional cross-sectional response rates for each wave

Study	RR2|1	RR3|2	RR4|3	RR5|4	RR6|5	RR7|6	RR8|7
	W1|W0=	W2|W1=	W3|W2=
ELSA*	64.7%	81.5%	85.6%
HRS	92.6%	94.1%	94.5%	94.6%	95.1%	95.5%	95.6%

^*For ELSA, ₁=wave 0; ₂=wave1, etc.

The conditional cross-sectional response rates provide an indication of the success of each survey at retaining respondents from one wave to the next. For example, R_2|1 represents the percentage of baseline respondents who completed an interview at wave 2 (of those still eligible at wave 2). These rates are all considerably higher than the unconditional cross-sectional response rates in Table 5.

Focusing first on the wave 2 rates (R_2|1), HRS has the higher rate of 92.6%. ELSA’s conditional rate of 64.7% represents response at ELSA wave 1 conditional upon participation at wave 0 (HSE years 1998, 1999 or 2001). ELSA’s comparatively low rate is largely due to the inclusion of those who did not consent to re-contact after their HSE interview. These cases were in fact not issued to field at the start of ELSA wave 1 but still need to be included in the denominator (see discussion section). It is worth noting that the decision to sample from three different HSE years is unlikely to have affected the overall conditional R_2|1 rate for ELSA, as similar household response rates were reported for each HSE year (see ELSA wave 1 technical report).

For both ELSA and HRS, the conditional cross-sectional rates tend to increase over the length of the study, as the sample that was interviewed in the prior wave becomes increasingly selective of more cooperative individuals.

Longitudinal response rates

Table 7 presents unconditional longitudinal response rates. These rates are cumulative; they represent the percentage of respondents who completed an interview in a set of consecutive waves starting with the first wave (1 through k), among all sample members who were eligible in all of those waves (including those who were in the original sample but did not complete an interview at the first wave). As such they represent the proportion of the target population that has participated in all of the designated waves of the study.

Table 7.

Unconditional longitudinal response rates

Study	RR1,2	RR1,2,3	RR1,2,3,4	RR1,2,3,4,5	RR1,2,3,4,5,6	RR1,2,3,4,5,6,7	RR1,2,3,4,5,6,7,8
ELSA*	45.8%	38.6%	34.5%
HRS	70.4%	67.7%	64.3%	61.3%	60.7%	58.4%	56.1%

^*For ELSA, ₁=wave 0; ₂=wave1, etc.

The proportion of the target sample that was represented in both waves 1 and 2 in the surveys was 45.8% in England and 70.4% in the U.S. The rates decline across waves, though the decline is fairly gradual for both studies. The rates in bold represent the “complete” response rates based on all waves through 2006. For ELSA, the complete response rate indicates that about 35% of eligible respondents participated in the HSE and Waves 1–3 of ELSA. For HRS, 56% of eligible respondents participated in Waves 1–8 of the HRS.

Table 8 presents conditional longitudinal response rates. This rate represents the percentage of respondents, that participated at wave 1 and were eligible in all of the other waves, who completed an interview in a set of consecutive waves, starting with the baseline wave (1 to k). In other words, it is the proportion of the eligible baseline sample that completed an interview in all of the designated set of waves.

For the ELSA conditional longitudinal rate it made sense to adopt ELSA wave 1 as the first wave of the study (baseline) rather than wave 0, as this rate reflects how successful the study has been in maintaining the original panel who had actually completed an ELSA baseline interview.

The conditional longitudinal rates for waves 1 and 2 are, by definition, the same as the conditional cross-sectional response rates for wave 2 shown in Table 5 (i.e. conditional on wave 1 participation). For subsequent waves, however, the two sets of rates differ. As with the unconditional longitudinal rates, the conditional longitudinal rates gradually decline across waves. Still, the rates remain quite high, reflecting the success of the studies in retaining those who originally participated in the study in subsequent waves. Over 70% of the baseline respondents participated in each of the first three waves of ELSA (of those who were eligible for all three waves), and over two-thirds of the original HRS respondents participated in eight consecutive waves.

Discussion

The results in Tables 5–8 reflect substantial differences in response rates across the two studies. While Lynn provides a clear model for calculating different types of response rate, there are issues relating to sample design and fieldwork practice across HRS and ELSA which need to be carefully considered. Some of the key issues are noted below.

Study design features and protocols that may influence response rates

Sample design

In calculating response rates for ELSA and HRS, it became apparent that differences in study design complicate both the calculation and interpretation. Unconditional rates are dependent on the initial response to the survey, and as such are influenced by the sample frame and sampling procedures. This paper has highlighted the difficulties around trying to compare unconditional rates based on follow-up from another survey (ELSA) with other multi-stage sampling techniques employed in HRS.

HRS inherited very little non-response during recruitment for the original sample, due to a highly successful screening field effort. In contrast, ELSA carried forward a high proportion of non-response from HSE which then affected the overall unconditional response rates. The following sample breakdown of age-eligible sample members for HSE (Wave 0) shows the magnitude of this effect:

The estimate of the number of age-eligible individuals in HSE non-cooperating households accounts for nearly 25% of the denominator. Hence, there is a large group of people who did not themselves take part at HSE and so were not followed up for ELSA, but who are still considered eligible for response rate calculations. A small proportion of these are estimated to have died or moved into an institution prior to calculating rates for each wave, so the denominator is reduced accordingly (see eligibility section). It is important to bear in mind therefore that the denominators used in the ELSA response calculations represent estimated rather than actual eligibility. In contrast, the success of HRS household screening to obtain the initial sample has limited the amount of estimation required.

Furthermore, those age-eligible individuals who were interviewed for HSE but refused to be re-contacted after their HSE interview, are also still considered eligible in response calculations for ELSA, despite never being approached for the study. This needs to be kept in mind when evaluating ELSA’s cross-sectional rate for wave 1 conditional upon participation at wave 0. If the 1,681 age-eligible individuals who refused re-contact after wave 0 are excluded from the denominator, the rate would increase from 64.7% to 71.6% (see stage 5 in Appendix A).

Lynn’s framework is used to illustrate the proportion of sample members interviewed from the target population, but no account is taken of the impact of fieldwork management on response rate. All of the response rates presented in this paper include ‘all those eligible’ in the denominator, but in reality, a sizeable proportion of eligible sample members are not issued to field at the start of each wave. Fieldwork agencies tend to rely more on field response rates in order to track success of fieldwork efforts, as this rate is based on all those actually issued to field. Both ELSA and HRS have had to remove a number of cases from the sample due to refusal to be re-contacted, but they are still considered eligible for the rates presented in this paper. Generally there are cross-study differences in how non-respondents are managed, as HRS was less restrictive in their handling of prior wave refusals after the second wave. At wave 2, both studies chose to issue households from wave 1 with at least one productive interview with an age-eligible individual. However, at the start of wave 3 some element of subjectivity was introduced to the decision to issue prior wave refusals. For ELSA, 91% of those who completed a wave 1 interview and were still eligible at wave 3 were issued to field. For HRS, the comparable figure was over 99%.

Overall, conditional response rates seem to provide the most standardized basis for analysis of cross-study study performance. By limiting the denominator to those interviewed at the previous wave, it is possible to get a sense of how successful each study has been in maintaining its original panel of members.

The possible influence of some fieldwork practices across studies on response rate is covered below. Our understanding of study differences can be enhanced further by looking at how different types of non-respondents are handled across studies and how this may impact on interpretation of the overall rates.

Interview Mode

The core HRS questionnaire has been designed for administration either in person or by telephone. Up to 2002, follow-up interviews with all participants under age 80 were conducted by telephone and, since 2006, half of those under 80 (those not assigned to the enhanced face-to-face sample in that wave) complete the interview by telephone in each wave. The remainder are interviewed face-to-face. Although the ELSA questionnaire has similar content, it has only been administered face-to-face. In longitudinal studies, using the same mode, each wave helps to avoid potential mode effects across waves. The face-to-face mode has helped ELSA interviewers to establish a good rapport with sample members over time and has allowed the inclusion of some cognitive and physical measures which require the presence of an interviewer. For HRS, the decision to implement telephone interviewing for part of the sample was a cost-saving decision. However, the practice may have served to encourage participation amongst would-be refusers, who find the telephone mode more convenient or less invasive.

Incentives

There is strong support for the use of incentives in surveys, as incentives increase response rates in a linear fashion and may act as a motive in itself for participation (Singer 2002). Prepaid incentives are also found to be more effective than promised or contingent incentives (Jackle and Lynn 2007). With this in mind, differences in the incentive amounts offered to HRS and ELSA respondents may have had some influence on willingness to participate. ELSA offers £10 to sample members for completion of a face-to-face interview. In contrast, the amount offered to HRS participants for the core interview has increased over time, from $20 in 1992 to $40 in 2006. In HRS, an additional incentive of $40 is given to participants in the enhanced face-to-face sample (which includes physical measures, biomarkers and psychosocial self-administered questionnaire). An extra incentive of this kind was not offered to ELSA sample members who completed a follow-up nurse visit (wave 2) or self-completion (waves 1 to 3). Although both studies have employed the use of differential incentives amongst highly resistant respondents, the amounts differ substantially, with HRS offering up to $100 to this group in comparison to £20 offered by ELSA. The method of administering incentives also differs across the studies. In the baseline wave of HRS, incentives are paid at the time of interview, and in follow up waves, incentives are included with the initial contact letter, prior to scheduling the interview. In contrast, ELSA has only administered incentives on a conditional basis (after the interview), which may affect the willingness of participants to respond.

Proxy interviews

A further key difference that impacts on the response rates in the two studies, is the way proxy respondents are used. ELSA has a more restrictive policy regarding proxies in comparison to HRS. ELSA allows proxies if cognitive impairment, physical or mental ill health prevented a respondent from doing a face-to-face interview. Likewise if the respondent was away in hospital or temporary care throughout the whole fieldwork period, a proxy interview was permitted. HRS has a somewhat more lenient policy towards proxy interviews. In addition to health-related restrictions, which make up the bulk of reasons for proxy interviews in HRS, proxy interviews are accepted for respondents who are unavailable or unwilling to be interviewed but who grant permission for someone to complete the interview on their behalf. ELSA rates for complete proxy interviews were 1% and 2% at waves 2 and 3 respectively, whereas the rates in the HRS varied between 5% in 1992 and almost 14% in 1995 (AHEAD). Since combining the cohorts in 1998, the HRS proxy rate has been between 7 and 11%.

There are pros and cons to using proxy respondents in surveys. Accepting proxy interviews not only helps to improve response rates, but it may also reduce selection bias, as individuals who are interviewed by proxy tend to be different in important ways from those who complete a self-interview. This is particularly true for surveys of older adults, for whom poor health tends to be a key factor in non-participation. The paper by Weir, Faul and Langa in this Special Issue examines this in relation to measures of cognition. On the other hand, if proxy respondents answer questions differently to how respondents would themselves answer, the use of proxy interviews may increase measurement bias.

Between wave contacts

HRS and ELSA differ with respect to the number and types of contacts that are made with respondents between core interviews. HRS conducts supplemental postal and internet studies between core interview waves and most respondents receive at least one request to participate in a minimum of one supplemental study. In addition, HRS typically sends a newsletter to respondents shortly before the start of each round of core data collection. In contrast, ELSA does not send a newsletter between waves or conduct between-wave supplemental studies, although ELSA typically sends holiday greeting cards and a newsletter prior to the start of a new wave. Between wave contacts are often thought to be beneficial for keeping participants engaged and interested in the study. At a minimum, sending something to respondents, whether it is a card, newsletter or questionnaire, can be useful for identifying potential movers, based on mail that is returned as undeliverable. However, whether benefits extend beyond that is unclear. In an experimental study based on an Internet panel survey in the Netherlands (the LISS panel), investigators found that sending participants different types of materials containing the information about and/or highlights of findings from the study (e.g. newsletters, post-cards, e-cards, ring binders) had no effect on participation in subsequent waves of the study (Scherpenzeel and Vis 2010). With regard to additional survey components (supplemental studies), analysis of interview outcomes in HRS suggests that the mode and content of the supplemental requests may have a role in influencing continued participation in the core interview (Ofstedal and Couper 2008). Although the HRS supplemental studies were not assigned experimentally and results should be interpreted with some caution, respondents who were invited to participate in the internet survey and the diabetes mail survey had higher response rates in the next core interview wave than those who were not asked to participate. In contract, HRS participants who were invited to participate in the Consumption and Activities Mail Survey (CAMS), which focuses primarily on household expenditures, were less likely to participate in the next core interview wave.

Tracking movers

Both ELSA and HRS use similar fieldwork protocols for locating or tracking participants who move between waves. Both studies make use of information collected in a previous wave on contact persons and information obtained from public, commercial and/or administrative databases, such as telephone and address listings. A number of these methods are carried out proactively (i.e. before the start of fieldwork for a given wave) by centralized staff, whereas others are carried out during data collection by field interviewers and/or staff who are specifically trained in tracking methods (Couper and Ofstedal 2009). The percentage of participants who are not located (lost to tracking) in each wave depends on the level of mobility among sample members, as well as the quality of the resources available for tracking movers. Both of these factors may differ across ELSA and HRS. For the general population, mobility rates tend to be somewhat higher in the United States compared to most countries in Western Europe (Couper and Ofstedal 2009). Nevertheless, in both studies, the fraction of respondents who are not located each wave is very small. In the 2006 wave of HRS, 0.6% of total core sample members and 5.6% of core non-respondents were not successfully traced. These figures are similar to ELSA which at wave 3 (2006) had 1.4% of total eligible sample members that had moved and could not be traced (equivalent to 7% of wave 3 non-respondents issued to field). Rather, most of the non-response at each wave (between 75% and 80%) is due to refusal.

Conclusion

The primary purpose of this paper was to document response rates for ELSA and HRS using the framework for longitudinal studies proposed by Lynn (2005). In doing so, this paper has shown that even with a specified framework, a strict comparison of response rates across studies can be problematic without considering differences in sample design, eligibility criteria and fieldwork protocol. As noted in the discussion, the higher response rates observed in HRS can be explained, at least in part, by differences in sample design, respondent incentives, protocols relating to the use of proxy respondents, and interview mode.

For purposes of comparison across studies, it may be of practical interest to supplement the rates proposed by Lynn (2005) with other types of response rates. For example, the impact of fieldwork management can be represented by using the field response rate, based only on those cases actually issued to field in a given wave. The conditional cross-sectional response rate in Lynn’s framework (presented in Table 6) is most similar to the field response rate, except that it excludes sample members who did not complete an interview in the prior wave and, thus, will always be higher than the field response rate. However, a downside of the conditional cross-sectional rates in this paper (conditional on participation in the immediately prior wave) is that they do not capture respondents’ movement in and out of the study. A critical element of longitudinal studies is bringing people back in after they missed a wave (e.g. see the Kapteyn et al article in this Special Issue) and this could be tracked by using yet another response rate: the cross-sectional response rate conditional on baseline response.

Response rates provide only part of the picture with regard to selection bias; the other part depends on the extent to which non-respondents differ from respondents on characteristics of interest (Groves and Couper 1998). Our exclusive focus on response rates is, thus, a limitation of this paper. Where they are possible, analyses comparing non-respondents and respondents would help inform the degree to which respondents are representative of the target population and the extent to which non-response bias is likely to be an issue. Such comparisons are typically not feasible for the baseline wave, as information on non-respondents tends to be extremely limited or absent altogether. However, a key strength of panel surveys is that they allow for comparisons of those who drop out versus continue to participate in subsequent waves, and several of the papers in this Special Issue address such comparisons. Additional research along these lines is needed in order to make informed judgments about quality within and across studies.

Table 9.

Breakdown of age-eligible sample members from HSE households

Total number of productive HSE individual interviews	18,651
Total number of non-responding individuals in HSE cooperating households	1,270
Estimate of non-responding individuals in HSE non-cooperating households	6,630
Total individuals (response rate denominator)	26,551

APPENDIX A

Detailed sampling design for ELSA study

Creation of the ELSA wave 1 sample from the Health Survey for England (HSE) is best described in five stages:

• Stage 1: 31,051 households were issued at the start of fieldwork across HSE 1998, 1999 and 2001.

• Stage 2: In the early stages of the HSE interview, all responding households were asked to provide the date of birth for every resident regardless of whether each went on to complete a full individual HSE interview. This meant that all age-eligible individuals could be identified in responding households. In contrast, non-responding households were not included in the ELSA sampling frame because there was no available information about residents that would have made it possible to identify those who were aged 50+ at the time of ELSA wave 1.

  A sampling frame was constructed from the HSE responding households using information about the residents at the time of HSE interviewing. Overall, 23,132 households responded to HSE 1998, 1999 and 2001 and so formed the foundation of the ELSA sample while a further 7,919 households did not respond to HSE and so were not included in the sampling frame.

• Stage 3: From the available HSE information two sample member types were identified for the ELSA wave 1 interview in 13,203 households.

  • First, potential age-eligible sample members (SM) were identified. These were defined as individuals who were living within an HSE responding household and were born before 1 March 1952. In total 19,924 sample members were identified.

  • Second, potential younger partners (YP) were defined as the cohabiting younger spouses/partners of sample members, who were living within the household at the time of the HSE interview and were born after 29 February 1952. 1,269 younger partners from HSE were identified.

  9,929 households that responded to HSE were not eligible for inclusion in the final ELSA sample because they did not contain an age-eligible individual.

• Stage 4: A mortality check was conducted for those potential sample members and younger partners who gave their permission (95%) to be ‘flagged’ with the National Health Service Central Register (NHSCR) run by the Office for National Statistics (ONS). This register keeps track of registrations with general practitioners but also with official death registrations and with people who leave the UK health system. No check was conducted on the HSE 2001 sample as little time had passed since that interview. 401 households were dropped as a result of deaths between HSE and ELSA wave 1.

• Stage 5: Potential sample members and younger partners were not included in the final ELSA sample if all HSE respondents aged 50 years or older within the household had refused, when asked, to being re-contacted in the future. Even though these people had not directly refused to take part in ELSA (they would not have been aware of the study at the time of HSE) it would have been unethical to have re-contacted them. Overall, 1,224 of the 12,802 eligible HSE households were removed on this basis (9.6%). This equated to a loss of 1,681 age-eligible individuals.

To summarise, the ELSA wave 1 sample was only selected from households that responded to HSE (Stage 2). Furthermore, households were only issued to field if they included at least one age-eligible individual (Stage 3) who, according to administrative records, remained alive (Stage 4) and gave permission to be re-contacted in the future (Stage 5).

APPENDIX B

Example response rate calculations for the ELSA study

Unconditional cross-sectional response rate (Table 5)

ELSA wave 1 (W1)

In order to derive unconditional response rates it was necessary to classify age-eligible ELSA sample members according to their status at HSE (W0). A distinction is made between those from HSE cooperating and HSE non-cooperating households.

HSE cooperating households, respondents in W0

Respond in W1 = 11,205

Non-respond in W1 = 6,125

Ineligible in W1 = 1,321

Total = 18,651

HSE cooperating households, individual non-respondents in W0

Respond in W1 = 186

Non-respond in W1 = 1,027

Ineligible in W1 = 57

Total = 1,270

HSE non-cooperating households

Non-respond in W1 = 5,947

Ineligible in W1 = 683

Total = 6,630

The number of productive outcomes in wave 1 was 11,391. The number estimated to be eligible was 11,205 + 186 + 6,125 + 1,027 + 5,947 = 24,490. Hence, as shown in Table 5, the estimated unconditional response rate in wave 1 was 11,391 / (11,391 + 6,125 + 1,027 + 5,947) = 0.465 × 100 = 46.5%.

For the calculation of ELSA unconditional response rates we have included non-cooperating HSE households. However, if we base the calculation solely on cooperating HSE households the unconditional rate increases to 61.4% (11,391) / (11,391+7,152) = 0.614 × 100.

Also it is worth bearing in mind that included in the non-response figures for wave 1 are individuals who were not issued at wave 1 because they refused to be re-contacted after their HSE interview (see ELSA sample design section). This group therefore had no opportunity to be interviewed at wave 1, but still need to be included in the denominator for unconditional rates because they were part of the original target population.

Conditional cross-sectional response rate (Table 6)

ELSA wave 1

HSE denominator

Total productive interviews at W0 = 18,651.

Total ineligible by time of W1 = 1,321.

Total denominator for W1 response calculation = 17,330.

ELSA wave 1

Productive interviews completed at W0 and W1 = 11,205.

Productive interview at W0 only = 6,125.

As shown in Table 6, the estimated cross-sectional response rate in wave 1 conditional on successfully responding in wave 0 was 11,205/17,330 = 0.647 × 100 = 64.7%.

Unconditional longitudinal response rate (Table 7)

ELSA wave 1

The denominator for the wave 1 (longitudinal) unconditional response rate focused on those original age-eligible sample members in waves 0 and 1 (irrespective of their outcome status at either wave or whether issued to field in wave 1). The numerator focused on those eligible sample units that responded in both waves 0 and 1. The response rate, therefore, indicates the proportion of eligible sample units that responded in every wave up to and including wave 1.

Productive interviews completed at W0 and W1 = 11,205.

Total number estimated to be eligible for interview atW0 and W1 = 24,490.

As shown in Table 7, the estimated (longitudinal) unconditional response rate in wave 1 is 11,205/24,490 = 0.458 × 100 = 45.8%.

Conditional longitudinal response rate (Table 8)

ELSA wave 3

The wave 3 longitudinal response rate (defined for respondents in waves 1, 2 and 3) conditional upon having successfully responded in wave 1 was calculated as follows:

• Number who successfully responded in ELSA waves 1–3 = 7,168.

• Number who took part at wave 1 and were estimated to be eligible for interview in waves 2 and 3 = 10,126.

As shown in Table 8, the estimated longitudinal response rate in wave 3 conditional upon response in wave 1 was 7,168/10,126 = 0.708 × 100 = 70.8%.