Psychometric evaluation of a patient-reported item bank for healthcare engagement

Abstract

Purpose:

Healthcare engagement is a core measurement target for efforts to improve healthcare systems. This construct is broadly defined as the extent to which healthcare services represent collaborative partnerships with patients. Previous qualitative work operationalized healthcare engagement as generalized self-efficacy in four related subdomains: self-management, collaborative communication, health information use, and healthcare navigation. Building on this work, our objective was to establish a healthcare engagement instrument that is sufficiently unidimensional to yield a single score.

Method:

We conducted cognitive interviews followed by a nation-wide mail survey of US Veteran Administration (VA) healthcare users. Data was collected on 49 candidate healthcare engagement items, as well as measures of self-efficacy for managing symptoms, provider communication, and perceived access. Items were subjected to exploratory bifactor, statistical learning, and IRT analyses.

Results.

Cognitive interviews were completed by 56 patients and 9,552 VA healthcare users with chronic conditions completed the mail survey. Participants were mostly white and male, but with sizable minority participation. Psychometric analyses and content considerations reduced the item pool to 23 items, which demonstrated a strong general factor (OmegaH of .89). IRT analyses revealed a high level of reliability across the trait range, and little DIF across groups. Most health information use items were removed during analyses, suggesting a more independent role for this domain.

Conclusion.

We provide quantitative evidence for a relatively unidimensional measure of healthcare engagement. Although developed with VA healthcare users, the measure is intended for general use. Future work includes short-form development and validation with other patient groups.

Introduction

Healthcare engagement is one of the core measurement targets for efforts to improve the performance of health systems [1]. This construct is broadly defined as the extent to which healthcare services represent collaborative partnerships between patients and providers and systems that deliver care [2, 3]. Engagement is distinct from other patient-focused constructs like patient experience or satisfaction, which address how consumers feel about their care rather than what they do to interact with their care. There are explicit financial incentives for healthcare engagement in publicly funded health care, through legislation such as the Medicare Access and CHIP Reauthorization Act (MACRA). The ongoing transformation in healthcare – emphasizing organizational accountability to promote population health – has galvanized efforts to promote patient engagement through patient-centered care. Patient-centered care aligns services with patient knowledge, skills, social circumstances and preferences for their care, and therefore depends on patient participation in processes such as goal setting, shared decision-making, and self-management. These participatory behaviors optimize the benefit one receives from healthcare services, enhancing the efficiency and effectiveness of care [4]. To date, however, much research on strategies to enhance patient engagement infers, rather than measures, healthcare engagement. A reliable and valid patient-reported measure of healthcare engagement would provide actionable data to evaluate health systems and help providers tailor care.

Like many psychological constructs, healthcare engagement is multifaceted and complex, with a range of approaches to operationalize the construct. Candidate measures have emerged, but require several subscales to cover the construct [5], or focus on patient traits, such as psychological readiness, motivation, or skills as a proxy for engagement [6, 7]. Efforts to operationalize the reciprocal nature of patient engagement have identified salient participatory behaviors. The engagement behavior framework [8] established a comprehensive inventory of patient participatory healthcare behaviors across the continuum of preparation, interaction, and follow-up. Subsequent research investigating barriers and facilitators to patient engagement suggest reliability across heterogeneous populations and settings in the behaviors patients describe as most important for benefiting from health care [9–15].

Our theoretical model for the present study organizes these behaviors into four inter-related domains [16]. We conceptualize patient engagement as the generalized self-efficacy across: (1) self-management, including health promotion, illness management, and treatment plan adherence; (2) collaborative communication, including behaviors such as asking questions or sharing feedback to establish shared goals and partnerships; (3) health information use, or behaviors required to seek, evaluate, and apply health information; and (4) healthcare navigation, consisting of behaviors related to maneuvering systems to obtain needed care. Most theoretical models maintain that engagement behavior is influenced by policy, organizational, and provider contexts [2, 17], but these elements do not map easily to a person-level measurement model. However, self-efficacy judgements reflect individual behavioral competencies within these organizational or relational contexts, because self-efficacy is a context-dependent self-assessment of behavioral capability. In other words, better engagement occurs where providers or systems make engagement behaviors easier to perform. The application of self-efficacy theory lends predictive utility and a framework that adapts well to measurement efforts [18]. In addition, self-efficacy can generalize across behaviors that require similar competencies, or that occur within shared contexts [19]. In this way, cumulative self-efficacy judgements for engagement behaviors can place individuals along a single continuum that reflects the propensity to engage with care.

The goal of the current study was to establish a patient-reported outcome measure that reflects a patient’s propensity to engage with health care. We aimed to establish an item bank on healthcare engagement that is sufficiently unidimensional to yield a single score, is free from differential item functioning based on race, gender, or chronic condition, and maximizes parsimony and precision across the continuum of the latent construct.

Methods

Item pool development

We developed a Healthcare Engagement item bank using methods established by the patient-reported outcomes measurement information system (PROMIS) [20, 21]. We previously validated the healthcare engagement construct using a literature review, expert reviews, and qualitative concept elicitation interviews with Veterans [16]. Though this work was conducted in the US Veterans Affairs (VA) healthcare system and our primary goal was to develop a measure that resonated with Veteran experiences, a secondary objective was to ensure that the construct would generalize beyond the VA healthcare system.

The item pool for this study (49 items) was generated by mapping items from publicly available measures of similar constructs and writing new items to achieve construct coverage. All items were written or edited to a 6^th grade reading level. Cognitive interviews with 56 patients included think aloud responses, prompts for comprehension, retrieval, judgement or other difficulties responding to items, and relevance [22]. The a priori standard for item inclusion was three consecutive interviews establishing comprehension, relevance, and no difficulties. Minor wording changes were based on consensus of study interviewers and the principal investigator, major wording changes were addressed in dedicated item review meetings of the study staff, investigators, and consulting primary care or mental health providers.

Participants and Procedures

We conducted a nation-wide mail survey of 9552 veteran users of 136 Veteran Administration (VA) medical centers in the continental US (response rate 38%). Our sampling priorities for this initial quantitative validation were: a) variability across the full range of the construct; and b) representation of diverse clinical samples from target populations for the measure’s use.[23] The sampling frame was identified from administrative data in the VA Corporate Data Warehouse [24] We sampled from the VA primary care population, who are predominantly male, and older, with poorer health status, and less racially diverse as compared to other health care populations [25]. Inclusion criteria were: Veteran, aged 18–80, White or Black race (Asian, Pacific Islander, and Native Americans are each ≤ 1% of users and accuracy in public sector administrative data is poor [26]), a valid mailing address in the medical record, at least one past-year primary care visit to the assigned primary care panel, and a diagnosis of one of four prevalent chronic conditions (diabetes, hypertension, depression, PTSD), as indicated by one inpatient stay or two outpatient visits on different dates of service. Exclusion criteria were active diagnoses of dementia or psychotic disorders, or 90-day risk for hospitalization or death in the top 5% as indicated by the VA’s Care Assessment Needs score [27]. Stratified sampling was used to ensure sufficient cell size for subgroup comparisons and DIF analyses. Sampling was stratified by chronic condition cohort, and within cohorts to oversample Women, Black Veterans and Hispanic Veterans. Cohorts were also balanced on health status, as indicated by risk adjustment scores [28]. Study information sheets were mailed with the surveys in lieu of consent forms; return of a completed survey was considered implied consent. The IRB at Stanford University School of Medicine approved the study. Data were collected October 2018 – January 2019.

Measures

Candidate Healthcare Engagement items.

We tested 49 candidate items representing each of the four domains of the engagement construct [16]: self-management (14 items), collaborative communication (12 items), health information use (12 items), and healthcare navigation (11 items). The measure asks respondents to evaluate the truth of each statement based on the “current time” and based on the respondent’s care from any doctor or provider. Candidate items were provisionally scored on a 5-point Likert-type scale, with response options of “Not true”, “A little true”, “Somewhat true”, “Mostly true,” and “Very true.”

Concurrent validity indicators.

The PROMIS Self-Efficacy for Managing Chronic Conditions - Managing Symptoms Short Form 4a [29] is a 4-item instrument that addresses managing symptoms during daily activities, interference with relationships, managing symptoms in public, and managing symptoms with a provider. T-scores were obtained via the Heath Measures scoring service and Cronbach’s alpha was .88 for this study. The Consumer Assessment of Healthcare Providers and Systems, Provider Communication Measure (CAHPS) [30] is a 4-item measure comprised of patient ratings of how well a provider explains things, listens carefully, shows respect, and spends sufficient time with the patient. Top-box scoring methods were used (proportion of items with top scores), and Cronbach’s alpha was .92 for this measure. Perceived access to care was measured using a single item scored on a 5-point Likert scale [31] that captures the patient’s global assessment of how well the care they received met their needs, representing the inverse of unmet need for care.

Data Analysis

Our analysis strategy was to pare down the item pool to reflect the latent construct of healthcare engagement. Though our theoretical model was comprised of 4 inter-related domains of engagement behaviors, engagement was hypothesized to stem from generalized self-efficacy across these behaviors [16], making the common variance across domains our primary interest, rather than comprehensive coverage of each domain that would result in separate scales or subscales. We assumed an effect-indicator model from the beginning of the analysis [32] and therefore the analysis strategy also targeted removal of smaller, extraneous factors in an effort to identify a sufficiently unidimensional set of items that could form a single scale. A different set of theoretical assumptions, such as creating an inventory of knowledge and skills, could be consistent with a causal-indicator model, where the focus of item selection would be to maximize the unique content represented by each subdomain [33]. While there are merits of each approach, we felt the latter would have contributed to proliferation of similar constructs in the field, overemphasized each subdomain’s independence, increased respondent burden, and would have had relatively limited utility in health care systems due to the need to reliably assess and interpret multiple domains and scores.

Preliminary analyses examined classical item-level statistics, polychoric correlations, and hierarchical cluster analysis [34] to identify poorly functioning items. Next, we conducted iterative exploratory bifactor analyses using the psych package in R. We started by selecting 4 subfactors corresponding to the 4 hypothesized domains of the construct, but considered solutions with 3 subfactors as well. (For a detailed illustration of how to apply bifactor modelling to patient experience data, see Reise, Morizot, & Hays, 2007 [35]) We evaluated dimensionality based on general factor saturation and computed Omega-Hierarchical (Omega-H), where values > 0.70 suggest sufficient unidimensionality for IRT modeling [36]. In addition, we calculated explained common variance (ECV), the relative strength of a general factor compared to group factors, targeting ECV values > 0.60 as a tentative benchmark [36].

As a final step, we used a statistical learning theory (SLT) approach to optimize the parsimony of the item pool, minimize expected prediction error and incorporate criterion validity into the item selection process [37]. SLT methods have good potential to inform the development of latent trait scales. The measurement of complex constructs, such as patient engagement, requires investigators to balance the theoretical and conceptual basis of the construct with empirical results in making judgements in the presence of structural ambiguity [38]. Well-documented pitfalls are associated with too great an emphasis on optimizing inter-item correlations, such as the proliferation of reliable but narrow scales or subscales that under-represent a construct [39].

In our context, the SLT procedure helped us to identify items that could be deleted because they were not sufficiently related to criterion variables. We derived a cross-validated set of items that correlated with the validity indicators for each of the three domains of the model using the BISCUIT (Best Items Scale that is Cross Validated, Unit-weighted, Informative and Transparent), which has been shown to produce parsimonious models with comparable levels of predictive accuracy, even at high levels of missing data when compared to other statistical learning techniques [40]. The tuning parameter was selected to exert a moderate degree of influence on item selection, identifying the best 20 items for each criterion, with 1000 bootstrap replications. All items that appeared in optimal scales for all 3 validity indicators in > 1 replication were retained in the item pool.

To illustrate the structure of our final set of items prior to IRT analyses, we ran a confirmatory bifactor model corresponding to the final structure determined by exploratory factor analyses. For this final model, we report Omega-H, ECV, Comparative Fit Index (CFI), the Tucker Lewis Index (TLI), and the root mean square error of approximation (RMSEA). Next, we estimated IRT parameters using the graded response model (GRM) [41], as well as the corresponding IRT-based bifactor model with three subfactors [42]. We did this with the understanding that unidimensional parameters might still be distorted by subfactors evident in a bifactor model. Therefore, we applied a procedure to convert conditional general factor slopes into marginal slopes, using formulas 10 and 11 provided in a didactic article by Toland et al. [43], citing detailed work on these procedures [44]. Once in marginal form, the slopes and location parameters from the bifactor model can be fairly compared to unidimensional models, and may be scored “as if” they were from unidimensional models.

Differential item functioning (DIF) analysis compared race (Black/African-American vs. White), age (65 and above vs below 65), mental health vs physical health cohorts, and financial strain. We applied logistic ordinal regression with IRT scoring, commonly used in the development of PROMIS instruments [45]. We used the chi-squared likelihood-ratio statistic as the initial DIF detection criteria (alpha < 0.01), and a cut-off of McFadden pseudo R²Δ ≥ 0.02 in model comparisons to determine substantial DIF, a reasonable threshold used in the development of self-reported health outcomes [46].

All analyses were conducted in R, using the psych package [47] and lavaan [48] for factor analyses, mirt [49]for IRT analyses and lordif [45] for DIF analyses.

Results

Participant Characteristics

In total, 9,569 individuals responded to the survey. Participants who returned the survey, did not opt out, and completed at least 50% of the healthcare engagement item pool were considered complete responders and included in the study (N = 9,552). Non-responders included Veterans who opted out of the survey (N = 1,022), returned a blank questionnaire (N = 15), and did not return anything (N = 14,584). Ineligible Veterans included those who were deceased (N = 9) or whose address was undeliverable (N = 304). The response rate was approximately 38%.

Table 1 shows the demographic and clinical characteristics of the sample used for item analysis. Participants were mostly white and male, but with sizable minority participation (i.e., 27% Black, 15% Hispanic/Latinx).

Table 1.

Demographic and clinical variables for Veteran sample (N = 9552)

Variable	N	%
Age
< 50	1548	16
50–64	2863	30
65–74	4432	46
≥75	709	7
Female	2134	22
Race/Ethnicity
Hispanic / Latinx	1442	15
Non-Hispanic Black or African American	2584	27
Non-Hispanic White	5440	57
Other/Unknown	85	<1
Married / cohabitating	5958	62
Education
High school / GED	2369	25
Some college or technical school	4070	43
College graduate or higher	3045	32
Financial Strain
Comfortable	2579	27
Income can provide for basic needs	4507	47
Difficult/Very difficult to get by on present income	2350	25
Self-Rated Health Status
Poor	647	7
Fair	3091	32
Good	3609	38
Very Good	1590	17
Excellent	242	3
Any past-year mental health diagnosis	6668	70

Note: Race/Ethnicity, Education, Financial Strain, Self-Rated Health Status categories add to less than 9552 due to missing data.

Initial item analysis

Thirteen pairs of items in the polychoric correlation matrix were highly correlated (r > .70). After removing 8 items, these pairs disappeared (several items appeared in multiple pairs). We note that these pairs of items were expected because we tested several items with alternate wording. Next, we examined how individual items (or small groups of items) might separate from larger sets of items with a hierarchical item cluster plot [34]. This plot showed final clustering of a pair of items with narrow content (social support) and a single item that had poor wording and the lowest cluster loading (i.e., corrected item-total correlation). We deleted these 3 items, and an additional item with redundant wording, leaving 39 in the item pool at this stage.

Exploratory bifactor analysis

We removed a total of nine items in this step, based on empirical and conceptual considerations. The bifactor models with four subfactors identified 3 items with low general factor loadings (< .50) or high specific factor loadings (>.50). These items focused on individual health behavior maintenance. In addition, we noticed that a single specific factor was driven by 3 items from the “health information use” domain, also showing high specific factor loadings and low general factor loadings, suggesting a potential for multidimensionality. They focused on independent health information use (e.g., “It is easy to find information on my own that helps me manage my symptoms”). All other items in this domain showed lower loadings on the “health information use” factor (< .35). We opted to remove these 3 items, and model 3 subfactors representing the remaining subdomains. This decision to shift from 4 to 3 subfactors was based not only on these empirical considerations, but on the overarching conceptual assumption that the construct of engagement is essentially collaborative, and that the multidimensionality of items addressing independent skills was inconsistent with this assumption.

In the subsequent 3-factor bifactor model, we identified 3 more items with general factor loadings of .50 or less. These items addressed resilient self-efficacy for independent self-management behaviors (e.g., “I attend all of my appointments, even when life gets busy or stressful”) and were removed based on similar empirical considerations and conceptual assumptions. The resulting 30 item set showed a high proportion of general factor variance relative to specific factor variance (item-level proportions ranging from .58 to .83), with overall indices of OmegaH as .79 and ECV .69. These values suggest sufficient levels of unidimensionality and that the items mostly reflect a single construct [36].

Criterion Validity

Finally, we derived a cross-validated set of items that correlated with the validity indicators for each of the three domains of the model using BISCUIT [40]. As noted above, we ran models separately for each of the three criterion variables: CAHPS communication, PROMIS self-management, and perceived access (as proxy for healthcare navigation, since few measures exist), and selected all items retained in all three models. Average item-criterion correlations for these items ranged from .29 to .61 (SD = .01). Three items were not retained in any model (two items from the communication subfactor and one item from the self-management subfactor), and an additional 4 items were retained in some, but not all, models (two from the communication subfactor, and two from the self-management subfactor), yielding a final item pool of 23 items. This final item set correlated with criterion variables at .62, .64, and .65 for communication, self-management, and perceived access, respectively. Table 2 shows the full item text of the final 23 items, as well the frequency of endorsement for each response option.

Table 2.

Final item text for the Healthcare Engagement Measure and item frequencies

		Frequency of Endorsement (%)
Item Key #	Item Text	Not at all true	A little bit true	Somewhat true	Mostly true	Very true
1	If I think my treatment plan needs to change, I have no problem bringing it up with my provider.	4	6	14	29	48
2	When I need more information, I ask, even when my provider is in a rush.	4	6	14	32	44
3	I make sure I understand all of my test results.	2	4	12	31	51
4	I know I can express my doubts, even when my provider might disagree.	4	7	17	32	40
5	If I didn’t think a treatment was working, I would tell my provider.	1	3	6	23	67
6	It is easy to find the health care resources I need (such as classes, support groups).	13	16	25	25	21
7	I know I can think through the pros and cons when I need to make a choice about my health.	2	6	17	34	40
8	I know I can get the information I need about the pros and cons of treatments.	3	8	19	35	34
9	I always know who to contact when I have a health issue.	4	8	15	32	42
10	It is easy for me to refill medications on time.	3	5	10	28	53
11	I know I can get the health care services I need, even if I must arrange it myself.	8	8	16	31	37
12	I know I can get a provider to deal with my main health concerns.	6	9	15	30	41
13	I can make sure my concerns are fully addressed before I leave appointments.	3	7	16	36	38
14	I know I can find a way to get in touch with my provider or care team when I need to.	6	8	14	27	45
15	When I need information about my care, like test results, I can get it easily.	6	9	17	30	39
16	I have a provider who I can trust to act in my best interests.	5	6	12	25	53
17	I can get the care I need without getting discouraged.	1	4	13	42	39
18	I know I can always follow my doctor’s instructions.	1	4	15	33	47
19	Learning more about my health issues helps me manage them better.	6	13	24	35	22
20	Even if I am tired or in pain, I know I can stick to my treatment plan.	5	12	26	32	25
21	I have clear goals to improve my health.	2	8	20	34	35
22	Monitoring how well my treatments are working helps me get the most out of my care.	3	9	22	40	26
23	I know I can get myself to keep doing the things that keep me healthy, even when life gets challenging.	4	6	14	29	48

Note. Missing frequencies ranged between 1 and 3 % (mean was 1.5% across items). When used in practice, the first and second categories should be collapsed. Specifically, item scores assigned to each categories should be as follows 1 = Not at all true, 1 = A little bit true, 2 = Somewhat true, 3 = Mostly true, and 4 = Very true.

Bifactor model of final 23 item set

A bifactor EFA model of the final item set showed that the highest subfactor loadings largely corresponded to our a priori classification. Exceptions were two items that focused on making sure health concerns are addressed (#12 and #13); these loaded on the navigation factor instead of communication. In addition, three items originally conceptualized as health information (#3, #7, #8) were distributed across the three subfactors.

To provide detail on the structure of our selected items, we ran a final confirmatory bifactor model on a subset of complete data (N = 8426), specifying ordered categorical data, and using the EFA result to assign items to factors. This model showed relatively high general factor indices, with OmegaH of .89 and ECV of .79. Fit statistics were: CFI = 0.996, TLI = 0.995, RMSEA = 0.044 [90% CI: 0.045,0.046]. Table 3 shows these loadings and the item-level ECV. As Table 3 shows, item #8 had the highest general factor loading (.81) and could be said to represent the construct (“I know I can get the information I need about the pros and cons of treatments”).

Table 3.

Confirmatory bifactor model of the final 23-item set and item-level explained common variance (I-ECV)

Item Key #	Truncated item text	General Factor	Navigation Factor	Self-Management Factor	Communication Factor	Item-ECV
14	get in touch w provider	0.66	0.53	0	0	0.61
12	provider main concerns	0.71	0.46	0	0	0.70
15	get info on care easily	0.68	0.41	0	0	0.73
16	provider best interests	0.67	0.41	0	0	0.73
11	services I need	0.58	0.28	0	0	0.81
13	concerns fully addressed	0.76	0.28	0	0	0.88
17	not discouraged	0.77	0.28	0	0	0.88
9	know who to contact	0.70	0.26	0	0	0.88
10	easy to refill	0.55	0.25	0	0	0.83
6	easy to find resources	0.62	0.24	0	0	0.87
8	info on treatment	0.81	0.12	0	0	0.98
23	things to keep healthy	0.60	0	0.48	0	0.61
21	goals to improve health	0.57	0	0.45	0	0.62
19	learning about health	0.61	0	0.37	0	0.73
20	stick to treatment plan	0.59	0	0.36	0	0.73
22	monitoring treatment	0.65	0	0.34	0	0.78
18	follow dr. instructions	0.61	0	0.20	0	0.91
7	think through pros/cons	0.67	0	0.20	0	0.92
5	tell my provider	0.73	0	0	0.39	0.78
1	bringing up plan change	0.69	0	0	0.38	0.77
4	express doubts	0.71	0	0	0.36	0.80
2	ask for more information	0.73	0	0	0.33	0.83
3	understand results	0.76	0	0	0.18	0.95

Item response theory (IRT) model

We first ran a nominal response IRT model to examine category functioning within the items. Results indicated the lowest two categories (“Not at all true” and “A little bit true”) did not reliably distinguish engagement levels among respondents. Consequently, we collapsed these two categories in subsequent models such that 3 thresholds are estimated for 4 categories of item data (for additional detail, see Reise et al., under review).

We proceeded with estimating item parameters for our 23 item set with both a unidimensional and bifactor model. Because slopes from the two models cannot be fairly compared [43], we converted the bifactor conditional slopes into marginal slopes. The resulting marginal slopes were highly correlated with the unidimensional slopes (r = .86), but slightly lower on average, showing a mean slope value of 1.67 (range 1.67 to 2.61) compared to 1.83 (range 1.30 to 2.54) for the pure unidimensional solution. Given this slight difference, we decided to use the converted bifactor parameters, which have more accurate (higher) standard errors [43]. The mean location value of the final item parameter set (averaged across the 3 values for each item) was −0.76 (range −1.57 to 0.14). The lowest location value was −2.68 and the highest was 1.25. Figure 1 shows the average location parameter across the items for the final item parameter set. This plot illustrates that item #5 (“If I didn’t think a treatment was working, I would tell my provider”) was easiest for participants to endorse, but item #6 (“It is easy to find the health care resources I need (such as classes, support groups)”) was the hardest. In other words, any given participant was more likely to rate #5 as true compared to #6.

Figure 1. Plot of Healthcare Engagement Measure items, ordered by average location parameter.

Minimum and maximum values of the lines represent the lowest and highest value of the location parameters associated with each item. Dot size is proportional to the discrimination value. Truncated item text is shown.

Figure 2 shows the marginal reliability curve across the latent trait, customarily denoted by theta, which has a population mean of 0 and population standard deviation of 1. For the x-axis of Figure 2, we converted theta to T-scores (theta *10 + 50). This shows that the estimated reliability is .90 or higher ranging from T-scores of 22 to 65. Figure 3 shows the distribution of IRT-based scores for the current sample, using our estimated item parameters and Bayes expected a posteriori (EAP) scoring [50]. The scores mostly form a bell-like shape, with the exception of a small subset of persons with very high levels of healthcare engagement.

Figure 2.

Reliability estimates for the final 23-item set across the trait range, based on the Graded Response Model.

Figure 3.

Histogram of Bayes expected a posteriori (EAP)-based T-scores for the final 23-item set across the trait range

DIF analysis

None of the DIF analysis results exceeded (or approached) the cut-off of McFadden pseudo R²Δ ≥ 0.02 for any of the items, suggesting that the item parameters are valid for the different demographic and clinical groups examined.

Discussion

In an effort to address the lack of pragmatic, predictive, and publicly available measures that can guide patient engagement efforts, we developed an item bank to measure patient engagement in healthcare, building on prior conceptual and qualitative work [16]. Our measure scales patients on their propensity to engage with healthcare by measuring self-efficacy for engagement behaviors. Although the items are sufficiently unidimensional to represent engagement as a single score, three subdomain concepts were still evident: Healthcare Navigation, Collaborative Communication, and Self-Management. Theoretically, engagement scale scores can be thought of as the generalized self-efficacy across these three domains of healthcare behaviors. Indeed, patients who can effectively navigate their health system, establish a collaborative relationship with their provider characterized by effective communication, and self-manage chronic conditions receive optimal benefits from their healthcare services. The final Healthcare Engagement Measure items demonstrated excellent reliability across the range of trait scores, including the lower range, where discriminating engagement levels is likely to be most relevant. We found no evidence of substantial DIF across age, race, gender, financial strain, or the presence of a mental health condition. The absence of item bias in these domains is encouraging in the development of an instrument that could be used with diverse populations, including in public sector health care settings, and with a range of health conditions.

Our results reaffirmed that collaborative interaction is at the core of healthcare engagement. Exploratory bifactor modeling identified several items addressing independent skills or attributes in health information use and self-management domains, which demonstrated high loadings on the specific factor, but poor general factor loadings. While these items measured their narrower domains well, they were not strong indicators of the latent engagement construct. This appears to be especially the case for the health information domain, of which only 3 items were ultimately retained. These retained health information use items showed strong loadings on the general factor, while also having interpretable (albeit very small) loadings on subfactors. For example, understanding test results (#3) might involve communication with the provider, and finding information about one’s treatment (#8) also reflects healthcare navigation.

The absorption of health information into the other three subfactors is consistent with results from our qualitative work [16]: individuals with low health literacy demonstrated high engagement in the presence of collaborative communication. Being able to reliably turn to a provider for support can engender proxy efficacy, or a sense of efficacy that is socially mediated, gained from relationships with others who possess helpful expertise or agency. We do not discount the value that individual health literacy has to promote self-efficacy for engagement behaviors, but posit that individual strengths and vulnerabilities may be indirect indicators of engagement to the extent that these skills facilitate collaborative interaction with providers and systems of care. Our results are also consistent with efforts to promote engagement by system redesign to reduce the health literacy demands on patients [51]. Interventions such as the use of infographics, teach-back, decision aids, or team-based medical homes can enhance self-efficacy for engagement behaviors by decreasing task demands and making it easier for patients to participate in their care.

We used innovative methods to incorporate validity indicators into item selection in an effort to identify the most parsimonious, yet predictive, set of measurement items using SLT. We implemented analyses that optimized out-of-sample predictive validity as an exploratory check on potential measurement bias stemming from judgement calls made in item selection and factor structure. Based on these results, we were able to improve parsimony by reducing the item bank from 30 to 23 items, make small but meaningful improvements in unidimensionality indices, and enhance confidence in predictive validity.

Our results should be interpreted in light of several considerations. First, this initial calibration was conducted with users of VA healthcare. The Veterans Health Administration is the largest integrated healthcare system in the US, which, like other public sector settings and systems, provides care to lower income patients with high rates of mental health and chronic conditions [25]. While subsequent evaluation with users from other healthcare settings will lend more confidence to generalizability, especially in more racially diverse settings, initial work within VA healthcare system has the potential to support applicability to vulnerable populations. Application to vulnerable populations is essential for a measure of health care engagement, as settings that serve greater proportions of low SES, or minority patients, deliver less patient-centered care and fare worse under alternative payment models, both within and outside VA system.[52–54] Second, a scale of 23 items is too long to be practical for many clinical and research purposes. Though the calibrated item bank lends itself to administration via computerized adaptive testing, the next step in our research is to develop a short form that can be more easily administered and interpreted at the point of care. Validation of the short form against indicators of engagement and healthcare quality measures will also be informative.

Efforts to measure healthcare engagement are particularly relevant for the increasing number of health care systems whose financial performance rests on effective population health management, or promoting and maintaining the health of a defined group of patients [55]. A reliable and valid patient-reported measure of healthcare engagement could be used in risk stratification models to enhance population health management through tailored care, to derive methods for equitable financial reimbursement, or as an outcome measure for organizational efforts to promote healthcare engagement. The precision, predictive validity, and lack of DIF with respect to race, ethnicity, or socioeconomic status for this item bank suggests considerable promise for these applications.