Measuring Engagement in HIV Care: Measurement Invariance in Three Racial/Ethnic Patient Groups

Abstract

Objective:

The objective of the study was to evaluate a novel measure of HIV care engagement in a large sample of non-Latino White, Latino, and African American patients. The Index of Engagement in HIV care (the Index) measures the degree to which a patient feels engaged/disengaged from HIV care. However, its measurement invariance, or the degree to which observed scores can be meaningfully compared across racial/ethnic groups, has not been established.

Methods:

The 10-item Index is a self-report measure initially validated in the Center for AIDS Research Network of Integrated Systems cohort study. Using Center for AIDS Research Network of Integrated Systems survey data, Index scores were linked to patients’ electronic medical records, which included viral load (VL) and appointment attendance data. We conducted measurement invariance analyses to test the Index’s performance in the 3 racial/ethnic groups and its cross-sectional association with VL and retention in HIV care (2 primary outcomes).

Results:

A total of 3,127 patients completed the Index, which showed good reliability across the 3 groups (alphas >.84). Confirmatory factor analysis model fit statistics showed that the Index demonstrated configural, metric, and scalar invariance, supporting the conclusion that the Index is a single factor construct. Lastly, lower Index scores associated with a concurrent detectable VL and poor retention in HIV care for all 3 groups.

Conclusion:

Having demonstrated invariance, the Index scores can be used to compare engagement levels across non-Latino Whites, Latinos, and African Americans in HIV care settings. Improving HIV care retention requires tools that can accurately identify people struggling to stay engaged in HIV care, especially racial/ethnic minorities.

A national priority in HIV research is to ensure that people living with HIV (PLWH) stay retained in HIV care over their life span, especially African Americans and Latinos, thereby facilitating achievement and maintenance of viral suppression (Mugavero, 2016). When PLWH are retained in HIV care, they likely to achieve and sustain viral suppression, approach a near-normal life expectancy, have an improved quality of life, and prevent forward transmission of the virus (Fauci, Redfield, Sigounas, Weahkee, & Giroir, 2019; Mugavero, 2016; Samji et al., 2013). It is no surprise then that HIV research over the past 20+ years has focused on solving the problems of retention in HIV care (Higa, Crepaz, & Mullins & the Prevention Research Synthesis Project, 2016), especially for non-Latino White, African Americans, and Latinos—the three groups with greatest prevalence rates of HIV.

One challenge researchers and clinicians have struggled with over the years is how to measure the key dimensions of the HIV treatment and care experience that are associated with retention in HIV care. Recently we have developed a new measure that captures how engaged PLWH feel in their HIV care that shows promise as a diagnostic tool to detect potential disengagement from care—This measure is called the Index of Engagement in HIV care (the Index). We argue that being retained in HIV care is not the same as being engaged in HIV care, despite the two words often being used interchangeably to convey the broad idea of being connected to HIV care (Mugavero, Amico, Horn, & Thompson, 2013). For example, in HIV surveillance and epidemiological studies, retention is operationally defined by the frequency of viral load (VL) testing, which is a correlate of attending/missing HIV care appointments at arbitrary time points (e.g., proportion of PLWH with two or more VL tests done 3 or more months apart in 1 year) (Centers for Disease Control and Prevention [CDC], 2016; Dandachi et al., 2019; Rebeiro et al., 2016). In research studies, retention is often defined by patterns of attending/missing HIV care appointments (e.g., proportion of kept appointments/scheduled appointments), given the power of appointment attendance as a predictor of future retention, viral rebound (going from suppressed to unsuppressed), and mortality (Colasanti et al., 2016; Pence et al., 2019; Zinski et al., 2015). However, although useful, these retention definitions do not provide insight into how PLWH engage and actively participate in their HIV care (Koester et al., 2019). In this paper, retention refers to appointment attendance and engagement refers to scores on the new measure—the Index—that indicate how engaged PLWH feel in HIV care.

Since 2000, HIV retention rates have improved but are still suboptimal, difficult to sustain, and are indicative of historic disparities (CDC, 2018; Office of National AIDS Policy, 2015). For example, national HIV surveillance data showed that in 2012, 57% of non-Latino Whites, 57% of Latinos, and 52% of African Americans were retained in HIV care within the year (defined as ≥2 CD4 cell count or VL tests ≥3 months apart during 1 year) (CDC, 2016). In a separate analysis of the same data, the prevalence of retention fell to below 50% for all groups, especially among African Americans (37.7%), when the outcome shifted to sustained retention from 2012 through the end of 2013, highlighting retention disparities (Dasgupta, Oster, Li, & Hall, 2016). In 2015, upward of 80% of non-Latino Whites, 78% of Latinos, and 72% of African Americans were initially linked to HIV care (defined as at least one VL or CD4+ test <3 months after diagnosis), but only 59% of non-Latino Whites, 58% of Latinos, and 55% of African Americans were retained 12 months later (CDC, 2018). The reasons for retention disparities are variable (e.g., mental disorders, lack of insurance, limited support), but all negatively impact the motivation to stay in care (Mugavero et al., 2009; Wohl et al., 2011).

And yet, on a national level, the prevalence rate of viral suppression each year has trended upward (CDC, 2016; Nance et al., 2018). More recent data from a large-scale national cohort study showed that in 2015, the prevalence of viral suppression reach >86% for most racial/ethnic groups (Nance et al., 2018). Thus, it is critical to detect who is at risk for falling out of care to sustain the steady improvements in viral suppression. Furthermore, given that most HIV transmission events are estimated to occur between people who are not retained in HIV care (61.3%) (Skarbinski et al., 2015), whereas historically most transmission occurred among people who were unaware of their HIV-positive status, it is vital that PLWH are retained and on antiretroviral therapy (ART) to prevent forward transmission and ultimately, reduce HIV disparities.

To facilitate retention, we focus on measuring engagement in HIV care. Engagement focuses on dimensions of the clinical encounters that facilitate or hinder a person from physically showing up to a HIV care appointment and include a patients’ evaluation of the centrality of communication with staff and providers (Wood et al., 2018), satisfaction with their care and personal treatment they receive (Dang, Westbrook, Black, Rodriguez-Barradas, & Giordano, 2013), their motivation to persist with lifelong medical care, especially during stressful life events (Emlet, Tozay, & Raveis, 2011), and satisfaction with their role in the making medical decisions (Clucas et al., 2011). By approaching the challenges of retention as a product of engagement, the focus shifts to optimizing the care experience to facilitate involvement with one’s care, which complements other retention strategies that focus exclusively on appointment attendance (e.g., reminder systems, patient tracking) (Hall et al., 2017; Thompson et al., 2012).

The Index items correspond to a working definition of engagement as “the ongoing interaction of patients, their providers, and care settings that is characterized by a patient’s sense of connection to and active participation in care” (Johnson et al., 2019, p. 2). Logically, lower Index scores may reflect PLWH who are at risk for disengaging from HIV care because it measures their level of connection to and participation in HIV care. The utility of a tool like the Index is that its scores may be used to prompt support for patients with lower levels of engagement who may be at elevated risk for poor retention in HIV care.

However, measuring complex processes, such as engagement, requires a rigorous development and evaluation plan. To accomplish such a plan, the Index items were developed by using a broad consensus-building Delphi methodology with 66 experts and 12 patient focus groups and then finalized by conducting 25 cognitive interviews (Johnson et al., 2017). The initial validation was published with data coming from a large sample (N = 3,296) within the Center for AIDS Research Network of Integrated Clinic Systems (CNICS) cohort study (Johnson et al., 2019). Results from the initial validation paper showed support of a single factor structure and overall strong psychometric properties. Overall, lower Index scores were associated with greater odds for a detectable VL, poorer retention outcomes, and worse self-reported antiretroviral therapy adherence. Furthermore, forthcoming results from recent prospective analyses show that higher Index scores were predictive of viral suppression and perfect HIV care appointment attendance 1 year after the Index was administered (Christopoulos et al., 2019a).

However, to date, all Index scores have been reported in aggregate with non-Latino White patients contributing the most observations (44% of the sample). Given the persistent and historical disparities in retention in HIV care and viral suppression (Crepaz, Dong, Wang, Hernandez, & Hall, 2018; Dasgupta et al., 2016; Mugavero et al., 2009) and the unique barriers facing Latino and African American populations (e.g., large proportions being uninsured/underinsured and historical medical mistrust) (Eaton et al., 2015; Guendelman & Wagner, 2000), it is plausible that each group may ascribe a different meaning to the HIV care engagement construct. However, before any differences can be inferred, it is necessary to first evaluate whether the Index performs equally in Latinos and African Americans—two groups with the highest prevalence rate of HIV. Differences on the Index, as with all self-report measures, can be meaningfully interpreted only once measurement invariance has been established. Measurement invariance is the concept that care engagement, as measured by the Index, has the same meaning for members of different groups.

The drivers of many biomedical health outcomes and disparities, including retention in HIV care and viral suppression, are psychosocial in nature but unobservable (i.e., internalized HIV stigma, mental illness, and perceived engagement) (Christopoulos et al., 2019b; Rooks-Peck et al., 2018). This requires use of self-report measures to infer that health outcomes, especially disparities, are attributable to some unobservable psychosocial factor (Gregorich, 2006). But when measurement invariance has not been established, meaningful comparisons across groups cannot be made because Classical Test Theory states that all measures are imperfect (DeVellis, 2006). Imperfect measures can lead to biased scores when the underlying construct does not mean the same thing to all groups or when certain groups respond to the items in a way that is systematically different from one another for unrelated reasons, rendering observed-level scores (i.e., raw scores) useless for comparison (Borsboom, 2006; Schmitt & Kuljanin, 2008). This bias has been observed in a commonly used HIV knowledge test, which found that certain test questions did not perform equally for different populations and thus, does not represent knowledge uniformly (Burke, Fleming, & Guest, 2014). A rigorous scientific view is that Index comparisons can be made only if the measure is invariant across groups (Borsboom, 2006; Gregorich, 2006; Meredith & Teresi, 2006).

The Present Study

We evaluated the measurement invariance of the Index using data from the initial validation study but stratified by racial/ethnic groupings: non-Latino Whites, African Americans, and Latinos. Our main objective was to evaluate its psychometric properties in each racial/ethnic group. Our secondary objective was to test for associations between the engagement construct and two major outcomes—VL suppression and retention in HIV care—in each group.

Method

Participants

Data were collected and analyzed from participants in a nationwide multisite cohort study titled the CNICS. CNICS comprises eight academic-affiliated HIV care clinics across the United States (Kitahata et al., 2008). Data came from clinics affiliated with the: University of Alabama–Birmingham; University of Washington; University of California, San Diego; Fenway Health, a community health center with Harvard University; University of North Carolina–Chapel Hill; Johns Hopkins University; and the University of California, San Francisco.

CNICS is an observational cohort study with a centralized data repository of longitudinal information from electronic medical records (EMRs) and other institutional systems from more than 35,000 patients living with HIV. Self-report survey data (i.e., psychosocial measures) are collected from participants systematically and submitted to the centralized data repository (University of Washington), which manages and codes/recodes all data along with standard clinical data, such as VL (Kitahata et al., 2008). All data in this article when the authors received it contained no personal health information. This allowed the authors to self-certify the study be exempt from review by the University of California, San Francisco’s Institutional Review Board.

Participants enrolled in CNICS were: (a) living with HIV and at least 18 years of age and (b) had at least two prior HIV primary care visits in 1 year at a CNICS site (Kitahata et al., 2008). For the current study, the new measure (the Index) was approved by the CNICS leadership group to be entered into the English survey protocol. The Index was given to patients from April 2016 to March 2017 prior to their HIV primary care appointment on a touch-screen tablet (or computer) connected to an Internet network with proper encryption. Survey data from each patient was aggregated and sent to the data repository. We requested data from CNICS on the following variables (see Table 1).

Table 1.

Participant Characteristics

	Number (%) or mean (SD)
Variables	Non-Latino white patients (N = 1,471)	African-American patients (N = 1,342)	Latino patients (N = 314)
Age	49.2 (10.7)	48.0 (11.7)	44.0 (11.0)
Sites
UCSD	264 (18)	78 (5.8)	126 (40.1)
UAB	254 (17.3)	335 (24.9)	3 (1.0)
UW	314 (21.4)	108 (8.0)	59 (18.8)
UNC	179 (12.2)	263 (19.6)	11 (3.5)
Fenway	232 (15.8)	29 (2.2)	53 (16.9)
JH	72 (4.9)	462 (43.3)	5 (1.6)
UCSF	156 (10.6)	69 (5.1)	57 (18.2)
Gender identity
Cis-male	1308 (88.9)	873 (65.0)	280 (89.2)
Cis-female	144 (9.8)	431 (32.0)	22 (7.0)
Gender minority	19 (1.3)	40 (3.0)	12 (3.8)
Sexual orientationa
Sexual minority	1203 (82.3)	572 (43.2)	262 (84.8)
Heterosexual	258 (17.7)	753 (56.8)	47 (15.2)
Clinical outcomes
CD4 cell count	619.1 (335.4)	605.5 (338.3)	556.1 (294.5)
Viral suppression	1111 (91.5)	936 (88.2)	226 (87.3)
Years in CNICS	8.2 (5.8)	8.7 (5.7)	6.2 (4.5)

Note. UCSD = University of California, San Diego; UAB = University of Alabama at Birmingham; UW = University of Washington; UNC = University of North Carolina, Chapel Hill; Fenway = Fenway Institute; JH = Johns Hopkins University; UCSF = University of California, San Francisco; CNICS = Center for AIDS Research Network of Integrated Clinic Systems.

^aSexual orientation data were not present in 10 non-Latino White, 17 African-American, and five Latino cases.

Measures

Index of engagement in HIV care.

The Index has 10 items and the response scale ranges from 1 to 5, with response option wording being specific to the item (e.g., 1 = not at all, to 5 = a great deal, or 1 = not at all open, to 5 = extremely open) (see Table 2). It previously demonstrated a single-factor structure, with higher scores indicating better engagement (Johnson et al., 2019) and was internally consistent (α = .88).

Table 2.

Response Frequencies, Mean Scores, and Reliability for the Index of Engagement in HIV Care

Variables	Response frequencies (%)
1. How much do you trust your HIV care provider?	Not at all	A little	A moderate amount	A lot	A great deal
Non-Latino Whites	.22	1.33	3.70	19.47	75.28
Latinos	1.06	1.42	1.77	23.76	71.99
African Americans	.71	1.34	3.62	21.07	73.27
2. How much does your HIV care provider respect what you have to say?	Not at all	A little	A moderate amount	A lot	A great deal
Non-Latino Whites	.22	1.40	4.94	23.95	69.49
Latinos	1.38	1.38	3.81	25.95	67.47
African Americans	1.34	1.34	4.10	22.65	70.56
3. How much does your HIV care provider really understand you as a person?	Not at all	A little	A moderate amount	A lot	A great deal
Non-Latino Whites	.74	2.97	11.95	31.25	53.08
Latinos	2.07	2.07	8.97	28.28	58.62
African Americans	1.19	2.93	8.39	26.82	60.68
4. How much does the clinic help you meet your most important health needs?	Not at all	A little	A moderate amount	A lot	A great deal
Non-Latino Whites	.37	1.61	6.37	27.82	63.84
Latinos	1.04	1.74	5.21	28.82	63.19
African Americans	1.42	2.05	5.59	24.31	66.64
5. How open do you feel you can be with your HIV care provider?	No at all	Slightly open	Moderately open	Very open	Extremely open
Non-Latino Whites	.30	.67	4.97	24.24	69.83
Latinos	1.41	1.41	7.07	26.86	63.25
African Americans	1.49	1.57	5.82	27.91	63.21
6. How well does your HIV care provider explain things in a way that is easy to understand?	Not at all	A little	A moderate amount	A lot	A great deal
Non-Latino Whites	—	.81	2.87	25.06	71.26
Latinos	1.03	1.03	4.12	25.09	68.73
African Americans	.71	1.18	3.71	25.20	69.19
7. How comfortable do you feel asking questions during your HIV care appointments?	No at all	Slightly comfortable	Moderately comfortable	Very comfortable	Extremely comfortable
Non-Latino Whites	.44	1.03	4.62	25.83	68.09
Latinos	1.03	2.06	6.53	25.09	65.29
African Americans	1.57	1.89	4.40	27.12	65.02
8. How well do you follow through on your HIV care when things in your life get tough?	Not at all	A little	A moderate amount	A lot	A great deal
Non-Latino Whites	2.29	4.21	14.04	30.30	49.15
Latinos	1.76	3.87	15.49	33.10	45.77
African Americans	2.45	3.95	11.60	34.25	47.75
9. How much of a role do you have in making decisions about your HIV care?	No role at all	A small role	A medium-sized role	A big role	A very big role
Non-Latino Whites	1.75	2.40	6.55	26.66	62.64
Latinos	7.51	1.71	5.80	25.60	59.39
African Americans	3.74	3.19	6.54	26.07	60.47
10. How often do you leave your HIV care appointment feeling like you got really good care?	Never	Sometimes	About half the time	Most of the time	Always
Non-Latino Whites	1.83	2.85	1.98	26.85	66.50
Latinos	4.12	3.09	4.47	23.71	64.60
African Americans	3.22	3.92	2.51	19.29	71.06

Note. Cronbach’s alpha for non-Latino Whites was .84, and .87 for Latinos and African Americans. Total Index scores (sum of all items divided by 10, range of 0–4) was 3.54 (.51) for non-Latino Whites, 3.56 (.47) for Latinos, and 3.53 (.53) for African Americans.

Viral load and retention.

HIV-1 VL assay result closest in time to the date in which the Index was administered was used. Because VL is not always assessed at each visit, a 90-day VL window was set both before or after the Index administration (Johnson et al., 2019). VL values of >200 copies/mL were coded as 1 (detectable/not suppressed) and <200 copies/mL as 0 (undetectable/suppressed), which is standard clinical practice. Poor retention was operationalized as an EMR record showing two or more missed visits in the prior 12 months. Retention was coded as 0 (not more than two missed visits) and 1 (two or more missed visits) (Mugavero et al., 2012).

Measures for criterion-related validity.

We hypothesized the Index to be associated with the following and most commonly used clinical variables: (a) past 2-week self-reported depressive symptoms scored continuously on the Patient Health Questionnaire-9 (Kroenke, Spitzer, & Williams, 2001); (b) past 30-day stimulant use from the Alcohol, Smoking, and Substance Involvement Screening Test, coded as 0 (no use), 1 (past use), or 2 (current use) (WHO ASSIST Working Group, 2002); and (c) past 30-day alcohol use scored continuously on the abbreviated three-item Alcohol Use Disorders Identification Test-C (Bush, Kivlahan, McDonell, Fihn, & Bradley, 1998), with higher scores indicating greater risks for hazardous drinking. All measures cited above have good psychometric properties.

Covariates.

The following variables were also collected and analyzed: (a) gender identity (0 = cis-male, 1 = cis-female, 2 = any gender minority), (b) sexual orientation (0 = any sexual minority, 1 = heterosexual), (c) CD4 cell counts, (d) years in CNICS, and (e) age. Gender identity was calculated by cross-tabulating EMR data on sex at birth and current sex, and the yes/no response to a question about transgender identity. Sexual orientation was a single item asking how they identify with responses were coded as “heterosexual” or “sexual minority”.

Racial/ethnic categories.

The multiple-groups confirmatory factor analysis and latent-variable regression were tested in three racial/ethnic categories coded as 0 (non-Latino White), 1 (Latino/Hispanic), and 2 (African American).

Analytic Strategy

All analyses were conducted within each of the three groups simultaneously. Initial descriptive statistics were used to characterize the sample and variables. On the Index, a single-factor structure was supported in an initial validation study when aggregating data across all patients in the sample using a random split-half sampling method to conduct exploratory and confirmatory factor analysis (Johnson et al., 2019).

To test for measurement invariance, a nested hierarchy of hypotheses was tested (Gregorich, 2006). A buildup approach was used in that a baseline model was tested against a series of more constrained models using the latent variable modeling program Mplus version 8 (Mplus (1998–2017)). Each model tested a higher (i.e., more restrictive) level of invariance: (a) configural—For each group, do the items associate with the single factor? (b) metric (or pattern)—are factor loadings equal across the three groups? (this supports the assumption that the single factor has the same meaning across groups); and (c) scalar (or strong)—are factor loadings and item intercepts (means) equal across groups? Scalar invariance allows for raw scores to be meaningfully interpreted as true differences on the construct (Gregorich, 2006). When scalar invariance is achieved, there is no concern that there are systematic (e.g., cultural) differences in how one group understands and responds to each item in ways that are unrelated to the single factor. The weighted lease squares means and variance estimator was used to perform the invariance analyses treating the item responses as ordinal.

For each model fitted and for invariance comparisons between the models, we used the χ² test of exact fit. Because the χ² test is sensitive to trivial model-data fit departures, especially with large sample sizes and this sensitivity extends to invariance comparisons among multiple groups (Cheung & Rensvold, 2002), we also report the following approximate fit statistics: Bentler’s Comparative Fit Index (CFI; value of ≥.95), the standardized root mean square residual (SRMR; value of ≤.08), and the root mean square error of approximation (RMSEA; value of ≤.06) (Hu & Bentler, 1999). We also used a generalized linear model to compare mean differences for the observed Index factor between racial/ethnic groups using the LSMEANS option in SAS PROC GLM with a Tukey adjustment.

For reliability, Cronbach’s alpha was estimated using all 10 items. To estimate criterion-related validity—the extent to which observed scores on the Index associates with related variables—bivariate correlations were estimated with measures of age, gender, time in CNICS, alcohol use, stimulant use, depressive symptoms, VL, and time in CNICS. This is because these were the most relevant measures that could be extracted from the medical records across all sites.

Lastly, for criterion validity—the extent to which the Index associates with a standard measure of retention—we conducted a multigroup latent variable logistic regression to assess whether the Index associated with VL and retention. For each group, regression coefficients were estimated for the Index on VL as well as the Index on retention. To test for equality of regression coefficients, we used a feature in Mplus called a difference of parameter test. This test indicates whether the size of each regression coefficient for each group was different from one another (e.g., is the association of the Index and VL for Whites stronger than that for African Americans?). The objective was to test whether the Index associates with VL and retention in each group and, based on the invariance results, identify if the size of the associations was different. To avoid listwise deletion of cases with partial data on covariates, the variances of the covariates were specified, and thus, covariates were included in the models as random variables. All covariates were selected based on prior literature showing their association with VL or retention.

Results

In the total sample (N = 3,127), Latinos were slightly younger (M = 44.0 years, SD = 11, n = 314) than non-Latino Whites (M = 49.2, SD = 10.7, n = 1,471) and African Americans (M = 48.1, SD = 11.7, n = 1,342). A total of 65.2% of African Americans were cis-gender male, and 88.9% and 89.2% of non-Latino Whites and Latinos were cisgender male, respectively. For African Americans, 43.2% were sexual minorities and 56.8% were heterosexuals. For non-Latino Whites and Latinos, 82.3% and 84.8% identified as sexual minorities, respectively, versus heterosexuals. VL suppression was high in all groups (range = 87.3–91.5%; see Table 1).

Internal consistency reliability for each group was good. For non-Latino Whites, Cronbach’s alpha was .84. For African Americans and Latinos, Cronbach’s alpha was .87 (see Table 2). For non-Latino Whites, higher Index mean scores correlated negatively with stimulant use scores and depression scores (rs = −.13, −.25, ps < .001) and positively with age and time in CNICS (rs = .06, .10, ps < .05). The Index did not correlate with alcohol use scores or other covariates. For African Americans, higher index mean scores correlated negatively with stimulant use scores and depression scores (rs = −.11, −.20, ps < .001) but no other covariates. For Latinos, Index scores correlated negatively only with depression scores, r = −.13, p < .05.

Multiple-groups confirmatory factor analysis for measurement invariance was evaluated using the χ² absolute fit difference test (see Table 3). The difference in the configural versus metric model was not statistically significant, χ²(18) = 21.81, p = .24, supporting a one-factor structure that has the same meaning across the three groups. Next, the configural model was tested against the scalar model and was statistically significant, χ²(50) = 94.12, p < .001, as was the test of the metric model versus the scalar model, χ²(32) = 87.88, p < .001. Following guidance from Cheung et al. (2002), because the χ² statistic gets inflated with large sample sizes (a feature of our study), evaluation of invariance should use approximate fit statistics because they are less sensitive to sample size issues when determining model fit.

Table 3.

Multiple Groups Confirmatory Factor Analysis

	Model fit statistics
Variables	χ2 (df)	CFI	SRMR	RMSEA (90% CI)
Least to most constrained models
Configural, equal factor structure	460.10 (105), p < .001	.990	.025	.057 (0.052, 0.062)
Metric, equal factor loadings	387.21 (123), p < .001	.993	.025	.045 (0.040, 0.051)
Scalar, equal item intercepts	441.75 (155), p < .001	.992	.027	.042 (0.038, 0.047)
χ2 model tests
Configural vs. metric	21.81 (18), p = .240	—	—	—
Configural vs. scalar	94.12 (50), p < .001	—	—	—
Metric vs. scalar	87.87 (32), p < .001	—	—	—

Note. CFI = Comparative Fit Index; SRMR = standardized root mean square residual; RMSEA = root mean square error of approximation; CI = confidence interval. We used the criteria set forth in Hu & Bentler’s (1999) extensive simulation study that one of the following two criteria be met to establish satisfactory global model fit: (a) CFI ≥ .95 and SRMR ≤ .08 or (b) RMSEA ≤ .06 and SRMR ≤ .08.

Invariance was supported using approximate fit statistics at each level of invariance (see Table 3). Each model fit the data well (i.e., least to most constrained) for all racial/ethnic groups. The fit statistics were similar across the three models and indicated excellent fit. First, the configural model fit was excellent (i.e., single item factor and item clustering on one factor): χ²(105) = 460.10, p < .001; CFI = .990, SRMR = .025, RMSEA = .057 (90% CI = .052, .062). Second, the metric model fit was slightly better than the configural model (i.e., each item has equal factor loadings across each group): χ²(123) = 387.21, p < .001; CFI = .993, SRMR = .025, RMSEA = .045 (90% CI = .040, .051). Lastly, the scalar model fit was also excellent (i.e., constraining all item intercepts to equality to draw inferences about observed mean scores): χ²(155) = 441.75, p < .001; CFI = .992, SRMR = .027, RMSEA = .042 (90% CI = .038, .047).

Index mean scores were nearly identical in each group and not statistically significantly different from one another in a Tukey-Kramer test for multiple comparisons. Index scores range from 0 to 4 and was 3.54 for non-Latino Whites (SD = .51), 3.53 for African Americans (SD = .53), and 3.56 for Latinos (SD = .47). Latent variable regressions were conducted for VL and poor retention in HIV care, adjusting for CD4 cell count, age, time in CNICS, gender identity, and sexual orientation. In non-Latino Whites, a higher Index mean was associated with a 55% decrease in the odds for a detectable VL (odds ratio [OR] = .45, p < .001) and 55% decrease in the odds for poor retention (OR = .45, p < .001). In African Americans, it was a 49% reduction in the odds for a detectable VL (OR = .51, p < .001) and 30% reduction in the odds for poor retention (OR = .70, p < .004). In Latinos, it was a 49% reduction in the odds for a detectable VL (OR = .51, p < .02) and 39% reduction in the odds for poor retention (OR = .61, p < .03).

To test whether the size of the regression coefficients of the Index on the two outcomes above across each group was different in size from one another, a difference of parameters test was conducted. For VL suppression, there was no statistically significant difference in the size of the Index coefficients between non-Latino Whites and African Americans (−.13) or Latinos (−.14) or between African Americans and Latinos (0.01), with p values ranging from .65 to .98. There was also no difference by the size of the Index coefficients for retention between non-Latino Whites and African Americans (−.45) or Latinos (−.32) or between African Americans and Latinos (0.13), with p values ranging from .05 to .67.

Discussion

The Index of Engagement in HIV care demonstrated good psychometric properties and equivalence in a large, geographically diverse English-speaking sample of non-Latino White, Latino, and African American patients in HIV care. The 10-item Index measures the underlying construct of perceived engagement in HIV care, a construct that is equally meaningful to the three racial/ethnic groups. The strength of evidence for the Index measure suggests it can now be used in the broader context with diverse HIV patient groups.

The psychometric properties of the Index provide researchers assurance that it is an accurate estimator of engagement in their non-Latino White, Latino, and African American patient populations. This is mainly driven by the achievement of scalar invariance in the racial/ethnic categories, which was largely reflective of the broader HIV epidemic in the United States. This allows for comparison of raw Index scores that are void of systemic response bias. Regarding the χ² results for invariance, we expected it to be statistically significant because of its dependence on sample size, exacerbating minor nonpractical differences between the constrained and less constrained models. In preparation for this, we followed the guidance from Cheung et al. (2002) on use of approximate fit statistics to evaluate invariance in a multiple group confirmatory factor analysis approach with large sample sizes.

Lastly, the Index was internally consistent (reliable) and demonstrated good validity. This good evaluation is because not all criterion-related validity estimates were statistically significant for all three groups. However, this does not affect the evaluation of invariance because it is independent. Furthermore, in our test of criterion validity (i.e., how well does the Index associates with VL and retention?), it performed very well. Overall, lower Index scores were sensitive enough to associate with both a concurrent detectable VL and prior history of poor retention, despite high rates of viral suppression in the cohort (>87%). Furthermore, the effect sizes of the Index on VL and retention were not different in size, suggesting that the Index has an equally strong association with HIV care outcomes irrespective of racial/ethnic categories.

The Index is a tool for researchers, clinic staff, and providers to understand their study or patient populations’ perceived level of engagement in HIV care and can complement other recent indicators of retention based on appointment attendance patterns (Kay et al., 2019). A strength of the Index is that it both associates with viral suppression and retention and provides insights into the dimensions of the clinical encounter that facilitate or hinder engagement.

Whereas the 10 items capture one underlying construct of perceived engagement, the index items do capture facilitators and barriers to engagement that are what have driven disparities in retention outcomes. For example, for Latinos, they may report lower engagement scores if they view their role in making medical decisions as limited. This could be due to the fact that Latinos have historically been the most underinsured group in the United States, may be affected by a limited-English proficiency, or that recent migrant subgroups may have less familiarity with the U.S. health care notion of agency in shared decision making (Guendelman et al., 2000; Harari, Davis, & Heisler, 2008; Shah & Carrasquillo, 2006). Of course, the Index needs to be validated in Spanish to further explore health disparities because Spanish speakers make up a large portion of the United States. HIV epidemic. For disparities among African Americans, the historical nature of stigma and discrimination in medical care may lead certain patients to report less trust in providers or may have a limited ability to be open with their provider, an interpersonal dynamic that is well documented (Earnshaw, Bogart, Dovidio, & Williams, 2015; Eaton et al., 2015; Gaston & Alleyne-Green, 2013). Lastly, the Index also measures a patient’s resilience, that is, persistence to stick with their HIV care during stressful periods in their life. Lower engagement scores could signal to clinic staff that a patient views retention as a personal challenge. Thus, staff can provide more support and intensive follow-up to monitor the patient’s risk for poor attendance (Koester et al., 2019).

A strength of the Index is that it can be quickly administered and implemented in a variety of clinical settings without the need to extract EMR data (i.e., histories of appointment attendance, VL test data). The content of items can be used to inform how clinic staff and providers focus their efforts to promote better engagement, which in turn may increase the likelihood of their patients remaining in care and achieving viral suppression. As it stands, the Index provides a global measure of engagement. But even in a setting with high rates of viral suppression, was sensitive enough to associate with the viral nonsuppression and poor retention.

There are limitations in this study that must be considered. First, this English-speaking sample was part of a clinical cohort with an average length of 6 – 8 years in care at university-based HIV specialty clinics. These sample features may limit generalizability because all questions about engagement in HIV care were likely to be responded to more positively, yet the performance of the Index measure is independent of these sample features. Future research could test how the Index generalizes to other particularly vulnerable groups, such as youth and young adults, transgender women, heterosexual African American women, and populations facing multiple comorbidities and structural barriers to care. Second, many of the Index items capture the dimensions surrounding patients’ interactions with providers, given the requirement of lifelong HIV care. It is unclear how the items would generalize to care models in which patients receive care from multiple, rotating providers (or residents and fellows), rather than one designated provider. Third, VL measurements were set within a window of 90 days, given that although less common, participants may not always have completed the Index measure and a VL test on the same day. Lastly, we used a cross-sectional invariance design, which limits the ability to infer predictive invariance.

The Index of Engagement in HIV Care measures patients’ perceived levels of engagement and performs equally well among the three groups with the highest prevalence rate of HIV in the United States. As a clinical tool, it can detect patients at risk for disengaging from HIV care and thus poor retention. As a research tool, it can be used to investigate the potential causes and consequences of disparities.