Measuring Nonspecific Factors in Treatment: Item Banks that Assess the Healthcare Experience and Attitudes from the Patient's Perspective

Abstract

Purpose

Nonspecific factors that accompany healthcare treatments, such as patients’ attitudes and expectations, are important parts of the experience of care and can influence outcomes. However, no precise, concise, and generalizable instruments to measure these factors exist. We report on the development and calibration of new item banks, titled the Healing Encounters and Attitudes Lists (HEAL), that assess nonspecific factors across a broad range of treatments and conditions.

Methods

The instrument development methodology of the Patient-Reported Outcomes Measurement Information System (PROMIS^®) was used. Patient focus groups and clinician interviews informed our HEAL conceptual model. Literature searches of 8 databases yielded over 500 instruments and resulted in an initial item pool of several thousand items. After qualitative item analysis, including cognitive interviewing, 296 items were included in field testing. The calibration sample included 1657 respondents, 1400 obtained through an internet panel and 257 from conventional and integrative medicine clinics. Following exploratory and confirmatory factor analyses, the HEAL item banks were calibrated using item response theory (IRT).

Results

The final HEAL item banks were Patient-Provider Connection (57 items), Healthcare Environment (25 items), Treatment Expectancy (27 items), Positive Outlook (27 items), and Spirituality (26 items). Short forms were also developed from each item bank. A 6-item short form, Attitudes toward Complementary and Alternative Medicine (CAM) was also created.

Conclusions

HEAL item banks provided substantial information across a broad range of each construct. HEAL item banks showed initial evidence of predictive and concurrent validity, suggesting they are suitable for measuring nonspecific factors in treatment.

Introduction

Two key Triple Aim healthcare goals are enhancing the patient experience of care and improving outcomes [1]. Therefore, it is essential to assess patient perspectives of their healthcare environment and their personally relevant outcomes. Although understanding the specific effects of treatments is vital and the focus of most biomedical research, nonspecific or contextual features of healthcare are also important as they contribute to the care experience and often influence outcomes [2]. The nonspecific, contextual features of healthcare treatments are broad-ranging and multifaceted, and may include interactive processes such as rituals and conditioning [3, 4], as well as patients’ attitudes and their views of healthcare relationships and the treatment environment.

Trust in the healthcare provider, comfort with the clinical environment, and optimism regarding treatment are important contextual aspects of the patient experience of care. Such contextual factors may be independent of specific actions of a medication or procedure, yet there is growing recognition that they can influence treatment outcomes. A meta-analysis of 13 studies found significant effects of the patient-provider relationship on outcomes [5]. Trials that experimentally manipulate provider warmth show enhanced treatment effects even with inert treatments [6, 7]. Expectations regarding treatments are particularly influential in symptom improvement [6, 8-11]. For example, back pain patients with the highest treatment expectations exhibited an adjusted odds ratio of 5.3 for improvement [12]. Additionally, when given pills clearly marked ‘placebo’, and told of potential mind-body benefits, 59% of irritable bowel syndrome patients reported adequate improvement [13]. Likewise, patients’ own characteristics shape their experience of care. Taking an active role can contribute to therapeutic outcomes [14], as can optimism [15, 16], and spirituality [17]. A review of the biological mechanisms of nonspecific factors supports their influence on outcomes, often greater than the specific treatment [18]. Thus, enhancing nonspecific effects may be the most direct method for enhancing the impact of all therapies.

Efforts to understand nonspecific factors are increasing, particularly through experiments designed to elucidate placebo effects. However, there has been little effort to measure nonspecific, contextual factors, particularly from the patient's perspective. Instruments that measure these factors would have widespread value as they would allow researchers and clinicians to understand these dimensions as well as enhance them in the treatment setting.

This study reports on the development and initial validation of a set of patient-report measures, the Healing Encounters and Attitudes Lists (HEAL) that assess several nonspecific factors in healthcare and healing. We used instrument development methodology of the Patient-Reported Outcomes Measurement Information System (PROMIS) [19, 20], which includes patient and clinician input, iterative steps of item revision, and modern psychometric methods (see Online Resource). HEAL instruments are designed to be administered as computerized adaptive tests (CATs), in which a small number of items can provide precise information. However, brief static (6- to 7-item) forms were also created using this methodology. The University of Pittsburgh IRB approved this study (PRO13090474).

Methods

Overview

HEAL item banks were developed using the rigorous instrument development methodology of PROMIS [19, 21-23]. Several iterative steps were involved: formulation of a conceptual model; development of item banks, and psychometric analyses including both classical test theory (CTT) and item response theory (IRT).

Formulation of the conceptual model

Multiple sources of information informed the model of contextual factors that contribute to health outcomes. We included the viewpoints of clinical experts and patients as well as existing scientific literature. Through interviews with over 20 healthcare providers, six patient focus groups that explored patients’ ideas about contributors to positive and negative treatment outcomes, and literature reviews of placebo mechanisms and patient characteristics that moderate outcomes, we developed a preliminary conceptual model of HEAL (Figure 1). The model included within-person characteristics and interpersonal relationship perceptions, and consisted of 7 initial domains: positive / negative attitudes, spirituality, locus of control, treatment expectancy, health and wellness attitudes, perceptions of the patient-provider relationship, and patient perceptions of the healthcare environment.

Figure 1.

Initial Conceptual Model of HEAL Domains

Development of item banks

Item bank development included 1) comprehensive literature searches to find existing questionnaires, 2) categorization, review and refinement of existing items, and writing new items, and 3) cognitive interviews with patients to establish clarity (see Figure 2).

Figure 2.

Overview of HEAL Item Banks Development

Comprehensive literature searches

Searches of PubMed, EMBASE, CINAHL, AMED, ICL, MANTIS, HaPI, and PsycINFO databases were conducted using a search methodology developed by the Pittsburgh PROMIS research site [24]. The searches generated 14,864 abstracts, which were individually reviewed. Articles documenting development and psychometric validation of instruments were examined, yielding 535 unique questionnaires with over 16,000 items potentially related to the domains in the HEAL model.

Categorization and review of items

The initial pool of existing questionnaire items was coded into the conceptual categories of the HEAL model. The codings were based on independent ratings by teams of 2-4 healthcare clinician-researchers (RG, CG, NM, MJS) and instrument development experts (PP, KJ, ND, JC, CM), with disagreements resolved through discussion. Using the qualitative item review procedures of PROMIS [19], the pool of items was reduced by removing redundant items and those that were too narrow in focus (e.g., disease- or treatment-specific). Further refinement included removing or rewriting poorly written items (see [25] for examples) and writing new items. The retained items were standardized for first-person subject, verb tense (primarily present tense), simple vocabulary, and 5-point response scales reflecting frequency (Never to Almost always) or intensity (Not at all to Very much). From the initial pool of items, a total of 359 were retained for further review by patients in the cognitive interview stage.

Cognitive interviews

Forty-two patients participated in cognitive interviews, during which patients ‘think aloud’ while reviewing one item at a time with a trained interviewer. Each of the 359 items was reviewed by at least 6 patients representing a broad range of ages, diverse races and reading levels, and both genders. Patients provided feedback on the item clarity, vocabulary, and appropriateness of the response scale. Based upon participant feedback, 63 items (17.5%) were rewritten or removed. All revised items were subsequently reviewed in further cognitive interviews. The item pool contained 296 items following the cognitive interview phase.

Sampling

We field tested the items on two samples of patients who received conventional or integrative medicine treatments for a medical or mental health condition (details in Online Resource). One sample of 1,400 persons was provided through the internet survey company, YouGov.com. Patients who reported receiving treatments within the past year were eligible. The second sample included 257 patients at the University of Pittsburgh Medical Center (UPMC) who had recently started a new integrative medicine (n = 127) or conventional medicine (n = 130) treatment. This clinical sample repeated the computerized assessment 6 weeks later.

Measures

The 296 HEAL items retained following cognitive interviews represented several domains relevant to interpersonal and contextual aspects of treatment as well as potentially important patient attitudes.

To reduce burden for the internet sample to 30 minutes or less, only HEAL items and demographics were included. Also, we administered subsets of HEAL items to respondents in a blocked design, ensuring that each item was paired with every other item a comparable number of times without requiring that participants respond to all items. In the clinical sample, in order to explore concurrent and discriminant validity, the PROMIS 29 health status profile and several well-known ‘legacy’ questionnaires were also completed (see Online Resource). At the 6-week follow-up, clinical participants also completed the single-item Clinical Global Impression (CGI) scale, which asks for comparison of current symptoms to earlier symptoms [26] to assess predictive validity of the HEAL measures.

Classical test theory analyses

The initial CTT analyses involved descriptive statistics and factor analysis. Although our conceptual model included 7 domains deemed relevant to patients’ experience of healthcare and healing, we made no assumptions regarding the most appropriate factor structure for the HEAL items. Our goal was to identify the most robust latent constructs with sufficient unidimensionality to proceed with IRT analyses. We also wanted to capture a variety of clinical indicators to ensure content validity. Thus, we intended to strike a balance between unidimensionality and adequacy of content.

The entire sample, both internet and clinical, was randomly divided into two subsamples. The first subsample was used for exploratory factor analysis (EFA, n = 799), and the second was used for subsequent confirmatory factor analysis (CFA, n = 858) [27]. Factor loadings, scree plots, and eigenvalues were evaluated.

IRT analyses

Item response theory (IRT) refers to a set of psychometric methods for developing and scoring tests based upon the idea that one's response to each item reflects one's level on the underlying domain of interest. The latent trait (for example, treatment expectancy) is scaled along a dimension called theta (θ). IRT models include discrimination parameters (i.e., how well the item distinguishes among individuals higher vs. lower on the θ scale) and location/ threshold parameters (i.e., the value of θ at which an individual has the highest probability of choosing the particular response to the item). Thus, IRT provides psychometric information about each item separately, as well as information for the overall test (see Online Resource).

Items remaining in the pools after CTT were calibrated with the two-parameter graded response model (GRM) using MULTILOG 7.03 [28]. Each item's fit to the IRT model was examined using the SAS macro IRTFIT program [29]. Misfitting items (χ² p<.01) were considered for exclusion. Several additional outcomes from IRT analyses, such as differential item functioning (DIF) and indicators of local independence were used to identify items for possible exclusion (see Online Resource).

Content-expert review

To ensure the relevance of the remaining items, content experts (CG, PP) re-examined items from the clinical perspective to eliminate items with questionable psychometric properties or clinically overlapping content. Conversely, items with important clinical implications were considered for retention in the pool even if they did not meet some conventional psychometric guidelines.

Preliminary validity evidence

In order to evaluate concurrent and discriminant validity, we correlated θ scores on HEAL domains with corresponding legacy measures and the PROMIS health status measures. As an initial estimate of predictive validity, we correlated HEAL θ scores with participants’ perceptions of improvement on the CGI. The clinical sample was the primary source for validity information, given the more extensive data we had on presenting complaints and treatment history.

Results

Factor analyses

In the EFA, we found a 5-factor structure fit well to the data. Correlations among the 5 factors ranged from 0.17 to 0.49, indicating that they were related but sufficiently distinct to deserve separate status. The items in the original HEAL domains of Patient-Provider Connection and Healthcare Environment loaded on the first factor. The items in the original HEAL domains of Optimistic Attitudes and Locus of Control loaded on factor 2. Spirituality items, Health and Wellness Attitudes, and Treatment Expectancy items loaded primarily on factors 3, 4, and 5 respectively. Nine items with cross loadings (.40 loading on more than one factor) and 39 items with single factor loadings less than .40 were dropped. Notably, 17 of the original 34 items in the Health and Wellness Attitudes item bank had low single factor loadings (less than .40) and were among the items dropped. A second round of EFA was performed to confirm the 5-factor structure and to re-assess the magnitude of the factor loadings. No further items were eliminated after the second EFA.

Single-factor CFAs were performed for each of the five factors, based on items retained after EFAs. Iterations of CFA were performed until all retained items had loadings greater than .50. A total of 250 items were retained at this stage for subsequent IRT analyses. To ensure construct validity, all items were reviewed again for content relevance. Based on this review, factor 1, which had 94 items, was separated into 1a, Patient Provider Connection (PPC) and 1b, Healthcare Environment (HCE). CFA was rerun for PPC and HCE, and all factor loadings were larger than .50.

IRT calibrations

The item banks corresponding to the 6 factors from CTT were calibrated separately using the two-parameter GRM. Item-parameter estimates for the final items for each item bank are shown in the Online Resource in Tables 1 – 6.

Item information functions (IIF), differential item functioning, and local independence

Item information curves, which reflect the overall performance of individual items, were examined. Items with limited information (i.e., with peaks on the IIF < 1.0) were removed. Following IIF-based refinement, 190 items remained (see Table 1). No items showed differential functioning based upon gender, age, or education. Therefore, no items were removed due to DIF. Five locally dependent items were removed.

Table 1.

Items Retained following CTT and IRT Analyses

		CFA		IFF		Local Independence		IRTFIT
Item Bank	Domain	Factor	Retained	Removed	Retained	Removed	Retained	Removed	Retained
PPC	Patient-Provider Connection	1a	61	2	59	0	59	2	57
HCE	Healthcare Environment	1b	33	6	27	0	27	2	25
TE	Treatment Expectancy	5	42	1	41	3	38	11	27
PO	Positive Outlook	2	64	37	27	0	27	0	27
SP	Spirituality	3	33	4	29	2	27	1	26
HWA	Health and Wellness Attitudes	4	17	10	7	0	7	1	6 (Attitudes toward CAM)
			250	60	190	5	185	17	168

IRT model fit

We examined each item's fit to the IRT model. The SAS macro IRTFIT identified 17 misfitting items, and these were removed (see Table 1).

The domain Health and Wellness Attitudes was originally broadly defined, and included social and family factors, attitudes toward diet and exercise, fear of illness, and views of conventional and complementary medicine. The multidimensionality of this domain rendered 17 of the original 34 items inappropriate for fitting to IRT models. A further 11 items in this domain were dropped due to low item information (10 items) and IRT misfit (1 item), leaving too few items for CAT administration of this item bank. A set of 6 items concerning views of complementary medicine were the only survivors, and they were retained as a short, static measure entitled Attitudes toward CAM.

Preliminary validity evidence

In the clinical sample, we examined concurrent validity between the 6 HEAL θ scores and legacy instruments measuring similar constructs. HEAL Patient-Provider Connection and Healthcare Environment correlated .38 and .39 (p< 0.01) with a measure of outpatient clinical care (ACES [30]), indicating similarity but not complete overlap. HEAL Treatment Expectancy was associated with Credibility .71 (p< 0.01) and Expectancy .58 (p<0.01) factors of the Credibility Expectancy Questionnaire [31]. HEAL Positive Outlook was inversely associated with PROMIS29 depression (−.71, p<0.01) and anxiety (−.54, p<0.01). Likewise, Spirituality, Positive Outlook, and Attitudes toward CAM were significantly correlated with corresponding legacy measures (see Online Resource), with correlations ranging from .81 to .30 (all p's<0.01). Treatment-related item banks PPC and HCE were unrelated to PROMIS health status measures, providing preliminary support for discriminant validity.

To explore predictive validity, baseline HEAL scores were compared with follow-up CGI ratings in the clinical sample. Correlations ranged from .36 (p<0.01) for HEAL Treatment Expectancy to .13, (p<0.05) for HEAL Spirituality, indicating that HEAL scores account for some variability in patients’ perceived improvement across a broad range of treatments.

Selection of items for short forms

To administer the HEAL where computerized adaptive testing (CAT) is not available, static short forms can be a useful alternative. To develop short forms, we evaluated the items in the HEAL banks based on their psychometric properties: their discrimination parameters, the percentage of time the item would have been selected in a simulated CAT [32] based on the observed data from the calibration samples, the expected information under the standard normal distribution (mean 0, SD 1), and the expected information under an extended distribution (mean 0, SD 1.5) [33]. For 5 of the HEAL banks, we selected items for short forms based upon the convergence of the psychometric criteria together with review of the clinical importance and content balance of the items. Clinical experts (PP, CG) performed these content reviews and choose items for short forms that had both excellent psychometric properties and represented the clinical breadth of the domains. The other HEAL item bank, originally conceptualized as Attitudes toward Health and Wellness, included only 6 items following CTT and IRT analyses which is an inadequate number for CAT. Therefore, this bank is available only as a short form: Attitudes toward Complementary /Alternative Medicine (CAM). The short forms are provided in Table 2.

Table 2.

Calibrated HEAL Short Form items

Patient-Provider Connection (PPC)	Slope	Location thresholds
Think of the HCP (Healthcare Provider) who provides your current/ongoing treatment...	(Discrimination)	Not at all vs. a little bit	A little bit vs. somewhat	Somewhat vs. quite a bit	Quite a bit vs. very much
I am satisfied with my healthcare provider.	4.29	−1.41	−0.88	−0.09	0.71
I trust my healthcare provider.	3.93	−1.66	−0.95	−0.19	0.74
My healthcare provider pays attention to my individual needs.	3.80	−1.36	−0.85	0.08	0.89
My healthcare provider gives me enough information.*	3.57	−1.82	−0.98	0.00	0.95
My healthcare provider respects me.	3.55	−1.81	−1.10	−0.23	0.71
I feel my healthcare provider understands me.	3.52	−1.22	−0.68	0.19	1.00
My healthcare provider gives me support and encouragement.	3.51	−1.41	−0.80	0.07	0.95

Healthcare Environment (HCE)	Slope	Location threshold
Think of the place where you receive your current/ongoing treatment...	(Discrimination)	Not at all vs. a little bit	A little bit vs. somewhat	Somewhat vs. quite a bit	Quite a bit vs. very much
The staff was respectful.	4.19	−2.75	−1.92	−1.17	−0.29
The staff was friendly.	3.91	−2.60	−1.75	−1.13	−0.21
The staff was helpful.	3.73	−2.56	−1.91	−1.05	−0.14
My care was well organized.	3.09	−2.54	−1.91	−0.98	−0.04
The healthcare provider's office respected my privacy.	3.08	−2.51	−2.00	−1.25	−0.28
The waiting area was comfortable.	2.35	−2.70	−1.98	−0.83	0.15

Treatment Expectancy (TE)	Slope	Location threshold
	(Discrimination)	Not at all vs. a little bit	A little bit vs. somewhat	Somewhat vs. quite a bit	Quite a bit vs. very much
I am confident in this treatment.	4.64	−1.56	−0.90	−0.03	0.86
This treatment will be successful.	4.34	−1.71	−0.87	0.06	0.99
I feel good about this treatment.	4.25	−1.60	−0.98	−0.15	0.74
I expect good outcomes from this treatment.	4.23	−1.68	−0.86	−0.02	0.85
This treatment is right for me.	4.02	−1.68	−1.06	−0.14	0.80
I value this treatment.	3.47	−1.67	−1.15	−0.25	0.67

Positive Outlook (PO)	Slope	Location threshold
	(Discrimination)	Not at all vs. a little bit	A little bit vs. somewhat	Somewhat vs. quite a bit	Quite a bit vs. very much
I feel positive about my life.	4.25	−1.95	−1.18	−0.30	0.68
I am hopeful about my future.	3.03	−2.06	−1.15	−0.30	0.59
My future looks good.	2.95	−1.92	−1.19	−0.11	0.85
I am satisfied with my life.	2.82	−1.81	−1.12	−0.13	1.01
I feel confident about myself.	2.56	−2.11	−1.16	−0.21	0.91
I feel I can cope with my problems.	2.09	−2.72	−1.63	−0.46	0.98

Spirituality (Sp)	Slope	Location threshold
	(Discrimination)	Not at all vs. a little bit	A little bit vs. somewhat	Somewhat vs. quite a bit	Quite a bit vs. very much
Spiritual beliefs give meaning to my life.	4.29	−1.04	−0.42	0.14	0.73
Spiritual beliefs give me hope.	4.29	−1.13	−0.49	0.09	0.64
I find comfort in my faith.	4.15	−0.98	−0.42	0.10	0.70
My spirituality gives me inner strength.	3.89	−1.22	−0.46	0.10	0.76
Prayer is a meaningful part of my life.	3.87	−0.78	−0.26	0.23	0.71
I feel supported by a higher power.	3.73	−1.03	−0.47	0.06	0.63

Attitudes toward CAM	Slope	Location threshold
CAM (Complementary and Alternative Medicine) is a non-conventional, holistic, or natural approach to healthcare. Common CAM treatments may include acupuncture, massage therapy, meditation, or herbal remedies.	(Discrimination)	Not at all vs. a little bit	A little bit vs. somewhat	Somewhat vs. quite a bit	Quite a bit vs. very much
CAM is effective.	3.18	−2.20	−0.83	0.39	1.27
I prefer CAM over conventional medicine.	2.80	−0.49	0.15	0.91	1.55
It is important to be open to CAM.	2.65	−2.06	−1.00	−0.12	0.65
CAM can be used to treat serious illness.	2.43	−1.51	−0.43	0.60	1.35
CAM can prevent health problems.	2.27	−1.99	−0.78	0.46	1.39
I prefer natural remedies.	2.15	−1.38	−0.33	0.57	1.31

^*Response categories are: Never, Rarely, Sometimes, Often, Almost Always

The short forms’ internal consistency alpha coefficients were 0.92 for Healthcare Environment and Positive Outlook, 0.96 for Patient-Provider Connection and Treatment Expectancy, and 0.97 for Spirituality. The correlations between the theta scores derived from the short forms and their corresponding full item banks were high, ranging from 0.93 to 0.97, indicating that the short forms scores are highly consistent with those of the full banks.

Discussion

Nonspecific or contextual aspects of healthcare treatments are complex and extensive, and contribute to patients’ experience of care. Nonspecific factors may moderate and mediate treatment outcomes, and thus, despite the challenges, are important to assess in medical research. Barriers to such assessments include patient and clinician burden and a lack of precise measures that are relevant across a broad range of treatments and health conditions. Many existing measures of patient engagement and other relevant factors are lengthy, specific to a particular setting, or are not informative across the full range of the construct. We developed the HEAL item banks to overcome these barriers and provide researchers and clinicians with precise tools for assessing contextual factors that influence treatment outcomes and patients’ care experiences.

The HEAL banks of Patient-Provider Connection (57 items), Healthcare Environment (25 items), Treatment Expectancy (27 items), Positive Outlook (27 items), and Spirituality (26 items) demonstrated adequate unidimensionality, high item and test information, and initial construct validity. The items of our original model domains of Locus of Control and Self-Efficacy were not found to be separate factors distinct from Positive Outlook, and thus these items are contained within the Positive Outlook item bank. The domain Attitudes toward Health and Wellness originally included many items describing health attitudes ranging from views of exercise to opinions regarding the role of advanced technology in healthcare. Due to this diversity, upon psychometric testing very few of the items in this domain formed a unidimensional set of items. This set of items concerned Attitudes toward CAM, and were redefined as a short 6-item form. We were able to derive short forms for the other five HEAL banks that provide brief, efficient measures of nonspecific moderating and mediating factors that may contribute to healing outcomes. Short forms may be especially useful in clinic to assess the contribution of nonspecific factors to patients’ experiences and outcomes.

HEAL CATs and associated short forms fill an important gap in the measurement of nonspecific factors that may affect treatment outcomes. The HEAL are unique in that they provide precise information on potentially influential interpersonal and attitudinal factors that cut broadly across medical conditions and treatments. Because they are administered as CATs or short forms, using HEAL does not unduly burden patients. Investigators and clinicians can choose to use whichever HEAL instruments are most relevant and appropriate for their purposes. In research, the use of HEAL CATs may be a step toward dismantling the ‘black box’ of the placebo effect. In clinical applications, HEAL may inform clinicians regarding factors that can enhance health outcomes and improve the experience of care [34].

Future directions

One goal is to provide further validation of the HEAL item banks among patients who are initiating a treatment that involves ongoing meetings with a healthcare provider. Our initial validation work in the present sample provides useful clues to predictive validity of the HEAL item banks, yet is limited by the fact that treatments were of differing lengths and intensity, and clinical details were limited. We are currently validating the HEAL CATs and Short Forms in patients with chronic back or neck pain receiving new treatments. A related goal is to determine, through both qualitative and quantitative methods, the clinical utility of the HEAL for improving patient-provider engagement and clinical outcomes in an expanded group of patients. Future work to disseminate and implement HEAL as well as other PROMIS measures will involve their systematic inclusion in large behavioral and pharmacologic RCTs.