From Access to Engagement: Measuring Retention in Outpatient HIV Clinical Care

Abstract

Engagement in HIV care is increasingly recognized as a crucial step in maximizing individual patient outcomes. The recently updated HIV Medicine Association primary HIV care guidelines include a new recommendation highlighting the importance of extending adherence beyond antiretroviral medications to include adherence to clinical care. Beyond individual health, emphasis on a “test and treat” approach to HIV prevention highlights the public health importance of engagement in clinical care as an essential intermediary between the putative benefits of universal HIV testing (“test”) followed by ubiquitous antiretroviral treatment (“treat”). One challenge to administrators, researchers and clinicians who want to systematically evaluate HIV clinical engagement is deciding on how to measure retention in care. Measuring retention is complex as this process includes multiple clinic visits (repeated measures) occurring longitudinally over time. This article provides a synthesis of five commonly used measures of retention in HIV care, highlighting their methodological and conceptual strengths and limitations, and suggesting situations where certain measures may be preferred over others. The five measures are missed visits, appointment adherence, visit constancy, gaps in care, and the Human Resources and Services Administration HIV/AIDS Bureau (HRSA HAB) performance measure for retention in HIV care. As has been noted for antiretroviral medication adherence, there is no gold standard to measure retention in care, and consideration of the advantages and limitations of each measure, particularly in the context of the desired application, should guide selection of a retention measure.

Introduction

Access to care has long been recognized as a vital factor in promoting and sustaining health and as a contributor to health care disparities.^1,2 This global construct has typically been studied through population-based samples and large administrative databases focusing upon broad questions related to having a usual source of care and dichotomous assessment of medical services receipt at specific clinical venues (e.g., primary care) over a specified historic time interval (e.g., within the past 12 months). Such studies of the domestic HIV epidemic have resulted in estimates that between one third to two thirds of persons with known HIV infection are not in regular outpatient care.^3,4

The term engagement in care embodies the distinct but interrelated processes of linkage to care and retention in care. In distinction to the more global construct of access to care, engagement in care measures outpatient medical service utilization with more granularity and on a longitudinal basis. Delayed linkage and poor retention in outpatient HIV care have been associated with delayed receipt of antiretroviral medications, higher rates of viral load failure, and increased morbidity and mortality.^5–17 Beyond the effects on individual health, engagement in HIV care has important consequences for the public health as it plays a pivotal role in HIV transmission and secondary HIV infections. Studies have demonstrated reduced risk transmission behavior among HIV-infected patients engaged in clinical care,^18,19 and viral load suppression, which is achieved more commonly among those with better retention,^6,9 also reduces risk of transmission.^20–22

Retention in care is being recognized as a crucial step in maximizing patient outcomes. The updated HIV Medicine Association guidelines for primary care of HIV infection state that, “emphasis should be placed on the importance of adherence to care rather than focusing solely on adherence to medications.”²³ Similarly, quality indicators from Human Resources and Services Administration HIV/AIDS Bureau (HRSA HAB), the National Quality Forum (www.qualityforum.org/Measures_List.aspx), and HIVQUAL (www.hivguidelines.org/) all include retention in HIV care as a marker of quality care. The recent emphasis on a “test and treat” approach to HIV prevention highlights the importance of engagement in care as an essential intermediary between the putative benefits of universal HIV testing (“test”) followed by ubiquitous antiretroviral treatment (“treat”). In a recent commentary, leadership from the National Institutes of Health recognized the success of a “test and treat” prevention strategy is contingent upon the use of scientifically proven “efficient and effective means of entering and retaining patients in care.”²¹

This growing and well-deserved recognition that engagement in care is vital in optimizing patient outcomes is a result of emerging research on the topic, and has sparked additional investigation. In contrast to adherence to HIV medications and the more global construct of access to care, there has been a relative paucity of research on engagement in outpatient HIV care. One challenge to researchers and clinicians who want to study engagement is deciding on how to measure retention in care. Measurement of linkage is relatively straightforward as this processes either occurs or does not following HIV diagnosis, thus allowing for dichotomous measurement. In contrast, measuring retention is more complex as this process includes multiple clinic visits (repeated measures) occurring longitudinally over time. Recent studies, the majority of which have been retrospective and observational in design, have used a number of differing approaches to measure retention in care.^5–17 To date, there has not been a published summary of these measures to serve as a guide and provide a common nomenclature for HIV retention measures. A recent review of retention in care noted, “a clear framework for how patient retention is defined and how it should be measured remains outstanding.”²⁴ Our goal is to provide a synopsis of five measures commonly used to study retention in outpatient HIV care, highlighting the advantages, limitations, and potential applications of each measure.

Methods

We reviewed the available published literature and summarized findings of the most commonly used measures to ascertain retention in outpatient HIV medical care. We selected articles using the identified measures of retention to provide example applications of each measure. In this article, we describe the calculation, advantages, and limitations of five commonly used measures of retention in HIV care.

Measures of retention in care

Before defining and describing the calculation of each retention measure, some basic principles regarding a standardized approach to determining qualifying medical visits for computational inclusion are proposed. First, because survival with HIV infection is so dependent on access to antiretroviral therapy (ART), “care” has typically been conceptualized as a scheduled visit with a health care provider (whether physician, advanced practice nurse, or physician's assistant) who can prescribe or manage ART. As a result, studies of engagement in care have focused on primary HIV care visits and excluded medical subspecialty appointments. Notably, although visits with ART prescribers are usually used to measure retention in HIV care, the vital importance of retention for those not on ART cannot be overstated.^7,10,12 Studies have typically included only scheduled outpatient medical appointments while excluding sick call or urgent care visits. In addition, while visits with social services, pharmacists, and nursing personnel are integral to team-based HIV management, those providers cannot directly prescribe or manage therapy and cannot provide longitudinal care independently, so have not be included in these calculations. Finally, appointments for which a patient called in advance to cancel and/or reschedule should be excluded analytically, in contrast to “no show” visits that represent missed appointments with no prior notification to the clinic.²⁵ Similarly, appointments cancelled by the clinic and/or provider should be excluded when calculating measures of retention.

Some of these conventions have application across the five retention measures, whereas others are applicable only to a subset of measures, which is highlighted in the discussion of advantages and limitations. Ultimately, a clear description of the measures and methods employed are essential for studies of engagement in care, and the following section provides suggested nomenclature for the most commonly used measures of retention in care as a proposed common language as the field moves forward. We refer to clinic visit patterns of four fictitious patients during a 12-month observation period to demonstrate calculation of the five measures of retention in care (Fig. 1 and Table 1). Of note, in locations with mandatory reporting, HIV biomarkers (CD4 count and plasma HIV viral load) have been used to measure linkage and retention in HIV care, acting as a surrogate for a completed visit.²⁶ While there are clear advantages to this approach from a public health perspective, a limitation is that biomarkers may be obtained in nonprimary HIV care settings (e.g., emergency department), thereby erroneously assigning some patients as retained in care.

FIG. 1.

Demonstration of calculation of retention measures using four hypothetical patients. Checked boxes indicate primary HIV care clinic visits attended by a patient, and boxes with an “X” indicate “no show” visits. The entire period is 12 months, with vertical marks dividing this into four 3-month periods. The accompanying Table 1 displays the 5 retention measures for the 4 hypothetical patients. The calculation of each measure is described in the methods section.

Table 1.

Retention Measures for Four Example Patients Calculated According to Clinic Visit Attendance during the 12-Month Observation Period

	Missed visits (dichotomous and count measure of “no show” visits)	Appointment adherence (number of completed visits divided by scheduled visits)	Visit constancy (number of 3-month periods with ≥1 completed visit)	Gap in care (6-month time period between completed visits)	HRSA HAB Performance Measure (≥2 completed visits during a 12-month period separated by ≥3 months)
Patient A	Yes; 1	80%	100%	No	Yes
Patient B	Yes; 4	33%	50%	Yes	Yes
Patient C	No; 0	100%	75%	No	Yes
Patient D	Yes; 1	67%	25%	Yes	No

HRSA HAB, Human Resources and Services Administration HIV/AIDS Bureau.

Missed visits

This measure captures the number of missed visits (“no show”) during an observation period of interest. The result is simply a count of missed visits, regardless of how many visits have been scheduled. This measure has been one of the most widely used retention measures in the literature and has been applied as both a dichotomous and as a count measure.^5,10,11 We can see that patients A, B, and D have missed visits (1, 4, and 1, respectively), whereas patient C has no missed visits.

Appointment adherence

This measure is a proportion that captures the number of completed visits in the numerator and the number of total scheduled (completed plus “no show”) visits in the denominator during an observation period of interest.¹⁷ Studies have also modeled appointment non-adherence, sometimes referred to as the missed visit rate or proportion, by placing the number of missed visits (“no show”) in the numerator rather than the number of completed visits.^8,9,16 Appointment adherence in patients A, B, C, and D is calculated as 80% (4/5), 33% (2/6), 100% (3/3), and 66% (2/3), respectively.

Visit constancy

This measure evaluates the proportion of time intervals with at least 1 completed clinic visit during an observation period of interest. We propose the term “visit constancy” for this measure, which heretofore has not had a distinct name. Typically, because treatment guidelines recommend laboratory assessments and visits for patients every 3–6 months, time intervals have ranged between 3 and 6 months.^6,12,27 From the example in Fig. 1, a 12-month observation period is broken down into four 3-month time intervals. Patient A attended clinic visits in all four intervals and had 100% appointment constancy, whereas patients B, C, and D had completed visits in 2, 3, and 1 time interval, respectively, representing 50%, 75%, and 25% constancy. Of note, patient C had no missed visits during the 12-month observation period, but did not have a completed visit in the third interval.

Gaps in care

This measure calculates the time interval between completed clinic visits. The length of the time interval or whether the time interval exceeds a pre-determine threshold, typically ranging between 4 and 12 months, can be used as the measure.^13–15 From the example in Figure 1, we see that patients B and D had a gap of over 6 months between completed visits, whereas patients A and C did not have a gap in care.

HRSA HAB medical visits performance measure

The HRSA HAB measure captures whether a patient had 2 or more completed clinic visits separated by 3 or more months in time during a 12-month observation period. The National Quality Center standards of HIV care includes a measure of 2 or more completed clinic visits separated by 60 days in time and occurring in two 6-month periods during a 12-month observation period. These are hybrid measures incorporating elements of appointment constancy and gaps in care. From Fig. 1 we see that patients A, B, and C all had 2 visits during the 12-month observation period that were separated by at least 3 months but patient D did not.

Discussion

A summary of advantages, limitations and application of the five measures of retention in care is presented in Table 2. The selection of a retention measure is based upon a number of factors including the purpose for measuring retention in care, the type of clinic visit data that are available, clinic scheduling practices, and computational issues. Clinicians providing clinical care and administrators and investigators conducting research will have different purposes for measuring retention, and hence some measures may suit their needs better than others.

Table 2.

Advantages, Limitations and Application of Commonly Used Measures of Retention in Care

Measure	Advantages	Limitations	Applications
Missed visits	Easy to measure	Dichotomous measure is crude	Preliminary research
	Dichotomous and count applications possible	Does not incorporate a denominator or a variable number of scheduled visits	Short-term tracking
	Face validity	Handling cancelled visits analytically	Monitoring individual patient behavior for clinicians
	Widely used	Impact of automatic rescheduling on number of missed visits
		Handling lost to follow-up
Appointment adherence	Easy to measure	Impact of variable number of visits in denominator	Research on exposure / response
	More granular measure	Handling of cancelled visits analytically	Long-term tracking periods
	Face validity	Impact of automatic rescheduling on number of missed visits in the numerator	Panel of patients behavior for clinicians and administrators
	Similarity to ARV medication adherence measures	Handling lost to follow-up
Visit constancy	Face validity/treatment recommendations	Computationally more challenging	Research on exposure/response
	Better accounts for loss to follow-up than missed visits or adherence measures	Appropriate length of visit interval may differ between patients based on disease severity (range, 3–6 months)	Long-term tracking period
	Only need to capture completed clinic visits to calculate	More granular than missed visits but less detail than appointment adherence	Particularly relevant for new patients establishing care and those starting ART
	Automatic rescheduling has no impact
Gaps in care	Easy to measure	Fairly crude, dichotomous measure	Link to research on out-of-care and reengagement
	Surrogate for loss to follow-up	Appropriate length of gap (range, 4–12 months)	Administrative tracking
	Only need to capture completed clinic visits to calculate	Less amenable for use as a time-varying covariate	Monitoring individual patient behavior for clinicians
	Automatic rescheduling has no impact	Patients lost to follow-up will have an undefined gap
HRSA HAB Clinic Visit Performance Measure	Face validity/treatment ecommendations	Computationally more challenging	Administrative tracking and reporting
	Overcomes limitation of appropriate interval length observed with constancy	Less granular than other appointment adherence and constancy	Clinic level quality assurance
	Only need to capture completed clinic visits to calculate		Local, regional and national planning for public health officials, managed care organizations, and third-party payers
	Automatic rescheduling has no impact

ARV, antiretroviral; ART, antiretroviral therapy; HRSA HAB, Human Resources and Services Administration HIV/AIDS Bureau.

In the following sections, salient features that may play a role in the selection of a retention measure are highlighted.

The clinician's perspective

For clinicians, the missed visits and gap in care measure are typically readily available at the point-of-care, and may serve as intuitive measures of retention. Many providers likely already review the medical record and note when the last visit was, or how many times the patient has missed visits recently. They then incorporate these measures in their routine patient care activities, for example, as a means to identify patients in need of case management or social work evaluation of potentially unmet needs (e.g., transportation, mental health services). Patient's patterns of missed visits are likely taken into account by providers in determining the time interval for a subsequent clinic visit, with shorter intervals used for those with more missed visits. Importantly, studies have shown the clinically relevant missed visit and gap measures of retention in care are associated with patient outcomes including clinical events and mortality.^{5,7,10–12,14,15}

The administrator's perspective

Appointment adherence and the HRSA HAB clinic visit performance measures (as well as similar quality measures) may be preferable for administrators. Clinic-wide appointment adherence may allow for longitudinal tracking of retention in care and also for setting an overall clinic benchmark (e.g., >80%). It may also allow administrators to assess variability in appointment adherence between clinics, units within the clinics, or individual provider–patient panels. The HRSA HAB clinic visit performance measure may be used by clinic administrators to assess performance relative to recommended standards and the performance of other clinics, although a benchmark for this measure has not yet been determined by HRSA (http://hab.hrsa.gov/special/performance/measureMedVisits.htm). Furthermore, public health officials, managed care organizations and/or health care payers (e.g., HRSA, CMS) may use these measures to inform local, regional and/or national planning and resource allocation.

The researcher's perspective

The missed visits measure has been widely used for research and clearly has a role, particularly for preliminary research and short-term observation periods. However, appointment adherence and visit constancy may be preferable for research purposes, particularly for longer observation periods. These measures allow for more granular measurement of retention in care, facilitating assessment of exposure/response relationships between retention and outcomes. Notably, the missed visit and appointment adherence approaches have a limitation in patients who are lost to follow-up. These patients, who are clearly poorly retained in care, may be misclassified as engaged if they never had scheduled follow-up visits to miss. As a final research consideration, the missed visits, appointment adherence and visit constancy measures are all easily employed for use as a time-varying measure analytically, an advantage over the gaps in care and hybrid indicators, like the HRSA HAB measure, which are less amenable to use in this fashion.

In addition to the varying professional perspectives and considerations in utilizing retention in care measures, logistical issues are also germane to the selection of a retention measure. These issues are most relevant to administrators and researchers, although some have clear implications for clinicians as well.

Requisite clinic visit data

In contrast to the missed visits and appointment adherence measures, visit constancy, gaps in care, and the HRSA HAB clinic visit performance measure are calculated based on completed visits only. By not incorporating missed clinic visits in calculating these latter three measures, it is not necessary to differentiate which visits a patient failed to attend and did not notify the clinic (“no show”), from those that a patient called to cancel, and those cancelled by the clinic. These measures therefore require less complex data capture and manipulation.

As mentioned earlier, only “no show” visits have typically been included in the missed visit and appointment adherence measures.²⁵ However, there clearly are issues related to the timing of cancelled visits worth noting. For example, a visit cancelled by a patient the week before the appointment is useful to the clinic because it is possible to use that slot for another patient. A visit cancelled 1 hour before a scheduled visit is less useful to the clinic. These issues are not a concern with the visit constancy, gaps in care and the HRSA HAB measures. However, not capturing missed visits in these latter measures also comes with a limitation. While they measure visit attendance at specified intervals serving as a marker of appropriate care, the impact of missed visits beyond these benchmarks of attendance is not taken into account.

Impact of clinic scheduling practices

Clinic scheduling practices must also be considered when selecting a retention measure. Clinics with automatic rescheduling of missed visits may lead to artificially poor missed visit and appointment adherence results, particularly if patient locator information is inaccurate and patients are not aware of rescheduled visits. From Fig. 1, this might have been the case for patient B who missed 3 consecutive clinic visits. However, clinic scheduling practices are not as much of an issue for the appointment constancy, gaps in care and HRSA HAB measure that are based only on completed visits. Accordingly, these measures may be particularly useful when comparing retention across settings in which variable scheduling practices are used.

Other issues related to clinic scheduling practices include how to handle walk-in or urgent care visits in retention measures, particularly when these visits occur with the primary provider but may focus on an acute medical problem rather than primary HIV care. Considerable variability exists among HIV clinics in the availability and approach to handling unscheduled or acute care visits. These local practices have implications for calculating and interpreting retention measures and need to be considered when choosing a measure. Finally, for open access approaches to clinic scheduling, the measures based upon completed visits only (constancy, gaps, and HRSA HAB performance measure) are likely most appropriate, as missed visits may not be as relevant and may be artificially decreased compared to traditional appointment scheduling practices.

Computational issues

The five measures present differing computational complexity. If data about both completed and missed visits are available, the missed visits and appointment adherence measures are the simplest to calculate, as the timing of visits is not necessarily a factor. In contrast, the visit constancy, gaps in care and HRSA HAB clinic visit performance measure that are calculated solely on completed visits require the timing of visits be taken into account.

The gaps and HRSA HAB measures are less computationally complex than the visit constancy measure, as the former two measures only require calculation of the time interval between completed visits. In contrast, measuring constancy requires the observation period of interest be divided into intervals of interest (e.g., 3–6 months) with subsequent determination of the number of intervals with a completed visit for each patient. An additional factor for consideration is whether time intervals are determined based on calendar time, and therefore using the same dates for all patients (e.g., January 1 to March 31, April 1 to June 30, etc.), or if each patient's interval dates are unique (although of the same duration) and linked to a common anchoring point across patients (e.g., date of ART initiation). This latter approach, although more desirable methodologically in many instances, presents more computational and programmatic complexity.

Sensitivity in capturing disease severity and stage of treatment

A final consideration in evaluating and selecting a measure of retention in care includes the sensitivity of each measure in capturing individual patient disease severity and stage of treatment. For example, more frequent medical visits may be indicated for patients newly establishing care, for those initiating or changing antiretroviral regimens, and for patients with more advanced HIV disease. The HRSA HAB performance measure is insensitive in capturing retention as it relates to issues of disease severity and stage, and rather represents a minimum standard of retention in care. Similarly, the gaps measure does not effectively account for disease severity. In contrast, the missed visit and appointment adherence measures may have greater sensitivity in accounting for disease severity as the frequency of scheduled visits reflects the clinician's assessment of the requisite number of visits and visit interval for a given patient during an observation period of interest. Accordingly, implicit in these measures is an assumption that visit frequency reflects, in part, the disease severity and stage of treatment for each individual patient being studied. The constancy measure has the capacity to account for severity and stage of treatment through variation of the time interval (e.g., range 3–6 months) during which the assessment of completed visits is evaluated. For example, for patients initiating clinical care or antiretroviral therapy, a 3-month interval may be most appropriate, whereas a 6-month interval may be indicated for patients on a stable antiretroviral therapy regimen with long-standing HIV viral load suppression and a high CD4 count.

Future directions

A relative paucity of research has focused on the individual and population health implications of differential engagement in HIV care. With growing emphasis on “test and treat” approaches to HIV prevention,²¹ recently referred to as testing, linkage to care plus (TLC+),²⁸ there is a clear need for additional engagement in HIV care research. As HIV care is increasingly team-based with nonprescribing health care professionals taking on expanded roles in direct patient care, future research should evaluate the impact of such visits and their inclusion in HIV retention measures on patient outcomes. Furthermore, with increased emphasis on the patient-centered medical home, the importance engagement in care has taken on increased importance not only for HIV care,²⁹ but for long-term disease management more broadly. Accordingly, the principles and considerations for measuring retention in HIV care described herein may have more widespread application in the future.

Conclusions

In recent years, recognition of the contribution of engagement in care to individual and population health of patients with HIV infection has resulted in the need for a framework defining how retention is measured.²⁴ This article provides a synthesis of the most commonly used measures of retention in HIV care, highlights their methodological and conceptual strengths and limitations, and suggests situations in which certain measures may be preferred over others. As has been noted for antiretroviral medication adherence, there is no gold standard to measure retention in care,³⁰ and consideration of the advantages and limitations of each measure, particularly in the context of the desired application, should guide selection of a retention measure.

Acknowledgments

These data were presented, in part, at a workshop at the 4th International Conference on HIV Treatment Adherence, Miami, Florida, April 5–7, 2009.

This study was supported by grant K23MH082641 (M.J.M.) and grant R34MH074360 (TPG) from the National Institute of Mental Health. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the Department of Veterans Affairs.

Author Disclosure Statement

No competing financial interests exist.