Impact of Definitions of Loss to Follow-up on Estimates of Retention, Disease Progression, and Mortality: Application to an HIV Program in Mozambique

Abstract

Patient retention is critical to the management of chronic diseases such as human immunodeficiency virus (HIV); hence, accurate measures of loss to follow-up (LTF) are important. Many different LTF definitions have been proposed. In a cohort of 9,692 HIV-infected patients initiating antiretroviral therapy in Mozambique from 2006 to 2011, we investigated the impact of the definition of LTF on estimated rates of LTF, acquired immunodeficiency syndrome (AIDS)-defining events, and death by applying 17 different definitions of LTF gleaned from HIV literature. We further investigated the impact of 4 specific components of the LTF definitions. Cumulative incidences of LTF and AIDS-defining events were estimated by treating death as a competing risk; Kaplan-Meier techniques and variations to account for informative censoring were used to estimate rates of mortality. Estimates of LTF 2 years after treatment initiation were high and varied substantially, from 22% to 84% depending on the LTF definition used. Estimates of 2-year mortality varied from 11% to 16%, and estimates of 2-year AIDS-defining events varied from 6% to 8%. As seen here, the choice of LTF definition can greatly affect study conclusions and program evaluations. Selection of LTF definitions should be based on the study outcome, available data on clinical encounters, and the patients' visit schedules; we suggest some general guidelines.

The burden of chronic diseases, such as heart disease, cancer, and diabetes, is high in low- and middle-income countries and is forecasted to increase with population aging and urbanization (1). One example is human immunodeficiency virus (HIV)/acquired immunodeficiency syndrome (AIDS), a chronic disorder manageable in the era of antiretroviral therapy (2). Properly managed AIDS patients take antiretroviral therapy for life, much as other patients might take statins, antihypertensives, or insulin (3).

Reducing loss to follow-up (LTF) is among the most important challenges in chronic disease care worldwide. Retention is essential to optimize patient outcomes, and, for HIV, to help limit the spread of disease (4, 5). However, high rates of LTF have been reported in HIV treatment programs in low- (6, 7), middle- (7–9), and high-income (10–12) settings. Rates of LTF also vary substantially across treatment programs, even in similar settings (6). In studies of chronic diseases with routine clinical data, LTF is not only a nuisance encountered in analysis but also an important study outcome itself. Accurate measures of LTF are valuable for understanding processes and programmatic details, and they are critical to evaluating clinical outcomes.

Disparate definitions of LTF have been applied by HIV investigators and program evaluators. A review of patient retention in antiretroviral therapy programs from 33 cohorts in sub-Saharan Africa incorporated 8 definitions of LTF (6). Multisite cohort collaborations often define LTF as no visit within 1 year of the study close (8, 13). Empirical data from 111 facilities in Africa, Asia, and Latin America were used to recommend adopting >180 days since the last clinic visit as a standard LTF definition (14). Additional definitions of patient retention have been proposed that capture constancy of care (15).

Several components comprise the definition of LTF. These include the following: 1) length of time without a visit (e.g., 60, 180, or 365 days) or number of missed consecutive visits (e.g., 2 or 3 visits); 2) what type of encounters count as visits (e.g., clinic visits, pharmacy pickups, or any encounter); 3) whether to define LTF retrospectively (e.g., patient seen within 180 days of study close) or prospectively (e.g., patient not seen at clinic for 180 days); and 4) from which date to start counting (e.g., time from last visit or from missed visit).

Using data from a single cohort of HIV-infected patients starting antiretroviral therapy in Mozambique, we studied the impact of different LTF definitions on estimates of LTF, AIDS-defining events, and death. First, we applied 17 definitions of LTF found in HIV/AIDS literature (13–44). Second, we varied components of the LTF definition to assess the impact on study estimates.

MATERIALS AND METHODS

The Friends in Global Health Mozambican cohort has been described elsewhere (45). Briefly, Friends in Global Health is a Vanderbilt-affiliated nongovernmental organization supporting the Mozambican Ministry of Health in the provision of HIV care and treatment at government health centers in 12 rural districts of Zambézia Province, a north-central region of 4.2 million persons with a 2010 estimated adult HIV prevalence of 12% (45). Districts began enrolling patients into antiretroviral therapy-based care in June 2006, Friends in Global Health support began in February 2007, and routine data entry began in March 2007. This study uses data from 10 districts receiving Friends in Global Health support; 2 districts did not have electronic databases installed at the time of analysis. Districts included Alto Molocue, Gile, Ile, Inhassunge, Lugela, Maganja, Mopeia, Murrumbala, Namacurra, and Pebane. Analyses included data from each district seat's hospital/health center and the following health centers: Nauela (Alto Molucue), Alto Ligonha (Gile), Mugulama (Ile), Gonhane (Inhassunge), Namagoa (Lugela), and 7 Abril (Pebane). Paper forms designed for clinical record documentation by the Ministry of Health were completed by clinicians, laboratory technicians, pharmacists, and counselors. Data were entered daily by Friends in Global Health staff. Data quality audits occurred every 6 months. Patients could be tracked across Friends in Global Health–supported clinics.

Adult (≥15 years) patients who started antiretroviral therapy before July 1, 2011, were included in this study. Regular clinic visits were scheduled at least once per 3 months; typically they were more frequent (approximately once per month), especially during the first year after antiretroviral therapy initiation. Medication pickup was scheduled at least once per month and may or may not have coincided with the clinic visit.

Primary study outcomes were LTF (defined below), AIDS-defining events, and all-cause mortality. AIDS-defining events were defined as progression to World Health Organization (WHO) stage IV. Stage of disease was to be recorded at each clinic visit. Analyses of AIDS-defining events were limited to patients who started antiretroviral therapy with WHO stages I–III. Analyses of time to LTF and death used data from all included patients.

To study the definition of LTF, we considered the following 4 components:

1. Length of time to consider a patient lost: 60, 90, 180, or 365 days.

2. Type of qualifying visit: clinic (including laboratory), pharmacy, or both (labeled any encounter).

3. Retrospective or prospective assessment of patient loss (see below).

4. Time measured from last visit versus from missed visit.

The retrospective definition classifies patients as LTF if they do not attend a qualifying visit within x days prior to the database/study close (July 1, 2011, in this study). A patient who has a large gap in care but has a qualifying visit (or dies) within x days prior to the database close is therefore not LTF with the retrospective definition. The prospective definition defines a patient as LTF if they do not have a qualifying visit for x days. Thus, with the prospective definition, a patient who has a large gap (>x days) is considered LTF, regardless of return (or death) before the database/study close.

Most definitions of LTF comprise various combinations of the above 4 components. Mozambique national guidelines define LTF as no contact for >60 days from the last scheduled clinic visit or medication pickup (60 days, any encounter, prospective, time from last visit) (45). A recently suggested “universal definition” defined LTF as no clinic visit within 180 days prior to a specified date (180 days, clinic visit, retrospective, time from last visit) (14). The WHO definition of LTF (or ‘drop’) is >90 days from the missed clinical or drug-pickup appointment without any follow-up contacts (90 days, any encounter, prospective, time from missed visit) (46). Multisite cohort collaborations have defined LTF as no encounter within 365 days of study close (365 days, any encounter, retrospective, time from last visit) (8, 13).

The cumulative incidence of LTF and AIDS-defining events were estimated by treating death as a competing risk, acknowledging the fact that a patient who dies can no longer be LTF or have a future AIDS-defining event (43, 47). Mortality was computed by using standard Kaplan-Meier techniques. For mortality and AIDS-defining event analyses, patients who were LTF during the first 2 years of antiretroviral therapy were censored at the last pre-LTF visit date.

Standard estimation of AIDS-defining events and mortality requires the assumption of noninformative censoring—that patients LTF had an incidence of AIDS-defining events/death similar to that of patients not LTF. Such an assumption is improbable for our rural Mozambican data. To account for potential informative censoring, we also computed the 1-year incidence of death by using a nomogram approach developed by other HIV investigators (42) with the calculator provided at www.iedea-sa.org. This approach uses tracing data from other sub-Saharan Africa cohorts (48) to estimate the proportion of those LTF who died, and it incorporates this estimate into overall mortality estimates.

Analyses were performed at the Vanderbilt Institute for Global Health by using R statistical software, version 2.13.1 (available at http://www.R-project.org; R Foundation for Statistical Computing, Vienna, Austria). Analysis scripts are available at http:/biostat.mc.vanderbilt.edu/MozambiqueLTFU. The study was approved by the Mozambican Ministry of Health bioethics committee and the Vanderbilt University Institutional Review Board.

RESULTS

Our study includes 9,692 HIV-infected adults initiating antiretroviral therapy in 10 Mozambican districts. There were 98,223 patient visits (including laboratory) and 140,006 pharmacy visits. The median time from antiretroviral therapy initiation to last visit was 399 days (interquartile range, 122–803). The median age was 33 years (interquartile range, 27–40), and 60% were women. Among patients with available data at antiretroviral therapy initiation, 19% were classified as WHO stage IV, and the median CD4+ cell count was 184 cells/µL (interquartile range, 102–279). The WHO stage and CD4 count were not documented within 90 days before and 14 days after antiretroviral therapy initiation for 44% and 50% of subjects, respectively. Our AIDS-defining event analysis includes 4,390 HIV-infected adults classified as WHO stage I–III at antiretroviral therapy initiation.

Application of 17 LTF definitions

We estimated the incidence of LTF, AIDS-defining events, and mortality 2 years after antiretroviral therapy initiation, applying 17 definitions of LTF found in the HIV/AIDS literature (Figure 1) (13–44). Estimates of the cumulative incidence of LTF after 2 years ranged from 22% (retrospective definition of >18 months since last contact (44)) to 84% (prospective definition of >1 month late for scheduled encounter (16–18)).

Figure 1.

Seventeen definitions of lost to follow-up used in the literature applied to data from a single, rural Mozambican cohort, 2006–2011. Subfigures demonstrate the following: cumulative incidence of LTF (A); cumulative incidence of AIDS-defining events (ADE) (B); and probability of death at 24 months for each definition (C). AIDS, acquired immunodeficiency syndrome; LTF, lost to follow-up.

The choice of definition for LTF also affected estimates of the cumulative incidence of AIDS-defining events and mortality. Estimates of 2-year incidence of AIDS-defining events ranged from 6% (using >2 months late for scheduled encounter definition (22, 23)) to 8% (combination of missed 2 consecutive visits or no visit in past 3 months (19)). Estimates of 2-year mortality ranged from 11% (using >1 month late for scheduled encounter definition (16–18)) to 16% (visit constancy definition: ≥2 visits separated by >3 months during a 1-year period (15)).

Impact of components of LTF definition

In Figure 2, we demonstrate the impact of various components of the LTF definition. Estimates are centered at a definition that classifies patients as LTF if they go 180 days from their last visit without any encounter (i.e., 180 days, any encounters, prospective, time from last visit). As expected, rates of LTF monotonically increased as the qualifying number of days to be LTF decreased (Figure 2, row 1, column 1), ranging from 29% (365 days) to 87% (60 days).Varying the number of days also impacted estimates of the cumulative incidence of AIDS-defining events and mortality, although these relationships were less pronounced and curves crossed (Figure 2, row 1, columns 2 and 3).

Figure 2.

Impact of varying specific components of the lost to follow-up definition, Mozambican cohort, 2006–2011. We investigated the impact of various components of the definition of LTF on estimates. All estimates are centered at a definition that classifies patients as LTF if they go 180 days from their last visit without any encounter (180 days, any encounters, prospective, time from last visit). The rows demonstrate the sensitivity of results to variation in these choices: The top row varies the length of time, the second row (from the top) varies what qualifies as a clinic visit, the third row compares prospective and retrospective definitions, and the bottom row compares time from last visit with time from missed visit. The columns correspond to the different study outcomes: The first column contains the cumulative incidence of LTF, the second column contains the cumulative incidence of AIDS-defining events (ADE), and the third column contains the probability of death. AIDS, acquired immunodeficiency syndrome; ART, antiretroviral therapy; LTF, lost to follow-up.

If only clinic (including laboratory) visits were incorporated, the cumulative incidence of LTF after 2 years tended to be much higher (70%) than the incidences based on definitions incorporating only pharmacy visits (51%) and incorporating both clinic and pharmacy visits (44%) (Figure 2, row 2, column 1). Counting only clinic encounters also led to higher mortality estimates because many of the people considered lost were still alive (and visiting the pharmacy) but were censored and treated as those who were actively remaining (and dying) in the study (Figure 2, row 2, column 3). Results were similar for the AIDS-defining events endpoint.

Estimates changed further if one altered the definition from prospective to retrospective (Figure 2, row 3). The incidence of LTF was higher when the prospective definition was used, because a patient was considered lost if he/she did not have a visit for 180 days at any point during the 2-year follow-up period. In contrast, with the retrospective definition, the 180-day gap must have occurred at the end of study. Therefore, patients considered LTF with the prospective definition had an opportunity to return to follow-up with the retrospective definition. The estimated incidences of AIDS-defining events and death were similar under the retrospective and prospective definitions (Figure 2, row 3, columns 2 and 3).

If LTF required 180 days from the last visit versus 180 days from the missed visit, the rate of LTF was only slightly higher (Figure 2, row 4, column 1). In our setting, visits occurred approximately monthly (including pharmacy visits), so the 180-day period from the missed visit is similar to the 210-day period from the last visit.

Informative censoring

Standard estimates of mortality are shown at 1 year for different definitions of LTF making the noninformative-censoring assumption (Figure 3B) and instead using a nomogram approach that assumes a proportion of those lost died on the basis of tracing data from elsewhere in sub-Saharan Africa (42) (Figure 3C). (Note, the nomogram approach was developed for 1 (not 2)-year mortality; Figure 3A shows the corresponding 1-year estimates of LTF to facilitate comparisons.) The estimates based on the nomogram were higher and more variable than standard estimates (11%–18% vs. 9%–11%, respectively). The ordering of point estimates across definitions of LTF was approximately the same when using the nomogram and standard approaches.

Figure 3.

Correcting 1-year mortality for lost to follow-up, Mozambican cohort, 2006–2011. Subfigures demonstrate the following: cumulative incidence of LTF at 12 months (A); probability of death at 12 months (B); and nomogram-adjusted probability of death at 12 months (C). LTF, lost to follow-up.

DISCUSSION

The definition of LTF has a major impact on estimates and interpretations of LTF, and it can affect estimates of the incidence of death and other events. Applying the different definitions of LTF found in the literature to a single cohort in Mozambique, we obtained estimates of LTF in the first 2 years of antiretroviral therapy ranging from 22% to 84%. Estimated probabilities of 2-year mortality varied from 11% to 16%. We posit that much of the heterogeneity in the incidence of LTF seen in the literature and in some of the differences between estimates of mortality is due to differences in LTF definitions, which may not represent true programmatic outcome differences.

The large impact of the LTF definition on results provides strong motivation to establish a uniform definition across cohorts and studies to allow more equitable comparisons (14). However, we believe that there are reasons why one might require several standardized definitions depending on cohort characteristics and intended study outcomes. In addition to summarizing the impact of specific components of LTF definitions on estimates, Appendix Table 1 contains general recommendations for selecting the definition of LTF based on study outcomes.

If the study outcome is mortality, we recommend using the retrospective definition based on any encounter (clinic or pharmacy). If a patient has a long period without care but then returns (or dies) near the end of the study, then vital status is known, and the patient need not be counted as lost. In contrast, if incidence of a soft endpoint or clinical event (e.g., AIDS-defining events) is the primary outcome, it is preferable to define LTF by using a prospective definition based on clinic encounters only. For example, it is possible for someone to have an AIDS-defining event during a gap in care and then to return to care; a retrospective definition for LTF would implicitly assume that any AIDS-defining event during a gap in care would be accurately reported to providers upon a patient's return, which is not necessarily true. Similarly, because HIV-related diagnoses are made during clinic visits, pharmacy visits should not be incorporated into the LTF definition when the primary outcome is a clinical event.

If incidence of LTF itself is the primary outcome, then the definition should largely depend on what the investigator wants to capture. There are many ways to think about being LTF, for example, lost to the clinic, not retained in care, vital status unknown. Different definitions serve different goals, each of which might be valuable in certain contexts (15). With that said, a reasonable choice could be a prospective definition counting any encounter (clinic or pharmacy) as a qualifying visit, with patients defined as lost at the time of their missed visit. From a clinician's perspective, a patient is first considered lost at the time of his/her missed visit (prior to that time, the patient was in care as actively as anyone who then attended his/her regular scheduled visit), and such a definition could be useful to prompt tracing of patients recently deemed LTF.

We have not recommended a specific number of days for the LTF definition, because it should depend on the frequency of visits and other components of the definition. For example, if patients are scheduled at clinic once every 3 months, defining LTF as 90 days without a clinic visit will overclassify patients as lost. However, for a clinic with monthly visits, or if any encounter qualifies and there are monthly pharmacy pickups, then defining LTF as >90 days from the last visit may be reasonable.

The challenge then becomes how to compare results across cohorts. Chi et al. (14) empirically selected a retrospective LTF definition of 180 days from last visit, counting only clinic encounters. Their definition is applicable to many circumstances, especially since some sites do not record pharmacy pickup or missed visits. However, their definition was empirically selected to minimize the misclassification of being truly LTF, defined as not being seen during a 1-year period. One could therefore argue that Chi et al.'s (14) “gold standard” LTF definition was actually 365 days without any encounter. Furthermore, their definition is impractical in developed country settings where scheduled visits may be as infrequent as once per 6 months. For example, in a subsequent paper combining data from the Swiss cohort and Zambia, the authors, including many from the same group of researchers advocating the universal definition, used yet another definition for LTF (>14 months without a visit) (43). This suggests that the optimal length of time should be a function of the frequency of visits. LTF definitions that incorporate both numbers of missed visits and length of time (e.g., >180 days without a visit and at least 1 missed visit) may be most valid in combined cohort studies with heterogeneous visit schedules. Multisite cohorts, such as the Antiretroviral Therapy in Lower Income Countries (ART-LINC), the Antiretroviral Therapy Cohort Collaborative (ART-CC), and the Caribbean and Central and South American Network for HIV Research (CCASAnet), have used a retrospective LTF definition of >365 days without any encounter (8, 13). Such definitions may be useful for evaluating mortality; HIV-related clinic visits are more frequent than once per year, which avoids problems of length of time being shorter than the frequency of visits. However, these definitions are insensitive when the study outcomes are retention and AIDS-defining events, and they may still result in poor cross-cohort mortality comparisons if rates of patient tracking and death ascertainment differ across cohorts.

The impact of the LTF definition on estimates of death in our Mozambican cohort was less substantial than the impact on estimates of LTF, but it still resulted in estimates of 2-year mortality ranging from 11% to 16%. Whether this represents a large range depends on one's perspective. In our study, estimates of mortality tended to increase with decreasing estimates of LTF (Figure 1). This suggests that Mozambican patients who were counted as LTF using definitions resulting in higher estimates of LTF were more likely to have died than those remaining in the cohort and not classified as LTF. Most analyses of AIDS-defining events or death, including those presented in Figure 1, B and C, and in Figure 2, assume that the rate of AIDS-defining events/death among patients who are lost is similar to the rate of AIDS-defining events/death among patients remaining in the study. Note that the choice of LTF definition can affect the plausibility/impact of this noninformative censoring assumption.

The noninformative censoring assumption is often unreasonable. In sub-Saharan Africa HIV care settings, patients who are LTF tend to be less healthy than those remaining in care (42, 48), whereas in other settings (e.g., upper income nations), patients who are LTF are often more healthy than those remaining in care (8). Statistical approaches that account for informative censoring using patient covariates exist and can yield unbiased estimates under certain (often unrealistic) assumptions (49, 50). Other methods that adjust overall mortality for LTF advocate tracing a randomly sampled subset of lost patients or using vital registration data (not available in Mozambique) to estimate mortality among those LTF (51–53). In the absence of tracing data, a simple approach is to assume that a proportion of those LTF had the event and to investigate the sensitivity of results to this assumed event rate (10). An analogous approach advocated by Egger et al. (42) and applied here assumes that the rate of mortality among those LTF is similar to that seen elsewhere in sub-Saharan Africa, where sites with lower LTF rates had higher mortality rates among actively traced LTF patients (48). Given that we tended to observe lower mortality rates when using definitions that yielded higher rates of LTF, when applied to our data, this nomogram approach not only yielded higher 1-year mortality estimates but also magnified differences in estimates across LTF definitions. The approach advocated by Egger et al. (42) appears reasonable in our rural impoverished setting and may be the best we can do without an ability to trace a subset of lost patients. However, it is unlikely that rates of mortality among those who are lost can be accurately imputed across cohorts, particularly considering the impact of the definition of LTF on initial estimates of LTF and death used to develop the nomogram.

Our study applies different LTF definitions to a single cohort; some findings may not hold when definitions are applied to other cohorts or study settings. Study weaknesses include our inability to ascertain deaths or trace lost patients. Our study assessed antiretroviral therapy retention, but we acknowledge that LTF also occurs pre–antiretroviral therapy initiation where LTF definitions may not be identical (e.g., there are no regular pharmacy visits pre–antiretroviral therapy). Interpretation of the AIDS-defining event analyses is limited, as more than half of the patients were excluded from this analysis, many because of missing information on clinical stage of disease at initiation of antiretroviral therapy. Finally, we censored all lost patients at their last visit, which could bias mortality estimates because time after the last visit was therefore included only in the subset of observed deaths. Some authors have addressed this potential bias by adding follow-up time to lost patients (30), although the best approach to address this potential bias warrants further study. Study strengths include the relatively large number of patients and our application of these methods with data from a high-volume, resource-limited, patient-care setting, reflecting the type of data actually seen in practice.

In conclusion, we found that the choice of LTF definition greatly affected study results and inferences. The choice of definition can also significantly impact programmatic details such as projections for medication and staff requirements, as well as strategies adopted to improve retention. Although our focus has been on HIV, LTF is important for all chronic diseases; careful consideration of the various components of LTF definitions is warranted when evaluating clinical outcomes or studying chronic disease management. Our recommendations for LTF definition are to base the definition on the study/programmatic outcome of interest, available encounter data, and the cohort visit schedule. Except in the context of collaborative networks, we do not advocate a universal standard as the definition of LTF should depend on the intended application and the cohort(s) of study.

Appendix Table 1.

Summary of Impact and General Recommendations for Different Components of the Definition of Loss to Follow-up, Mozambican Cohort, 2006–2011

Component of LTF Definition	Outcome
	Lost to Follow-up	Soft Endpoint (e.g., AIDS-defining Event)	Mortality
Length of time (e.g., 60, 180, or 365 days)	Impact: Substantial and monotonic	Impact: Lesser and variable	Impact: Lesser and variable
	Recommendation: Depends on frequency of visits and desired strictness of LTF definition	Recommendation: Depends on frequency of visits (perhaps 2–3 times the typical length of time between visits where the endpoint is measured)	Recommendation: Depends on frequency of visits (perhaps 2–3 times the typical length of time between visits)
Qualifying visit (e.g., clinic, pharmacy, or any)	Impact: Substantial and monotonic	Impact: Moderate and variable	Impact: Moderate and variable
	Recommendation: Depends on goals	Recommendation: Visits where endpoint is measured (e.g., clinic visits for AIDS-defining event)	Recommendation: Any encounter
Prospective or retrospective	Impact: Substantial and monotonic	Impact: Lesser and variable	Impact: Lesser and variable
	Recommendation: Depends on goals, although typically prospective	Recommendation: Prospective	Recommendation: Retrospective
Last or missed visit	Impact: Minor and monotonic	Impact: Minor and variable	Impact: Minor and variable
	Recommendation: Depends on goals, although typically missed visit	Recommendation: Time from last visit	Recommendation: Time from last visit

Abbreviations: AIDS, acquired immunodeficiency syndrome; LTF, lost to follow-up.