Confirmatory Factor Analysis of the Pain Care Quality Surveys (PainCQ^©)

Abstract

Objective

To examine the reliability and validity and to decrease the battery of items in the Pain Care Quality (PainCQ^©) Surveys.

Data Sources/Study Setting

Patient-reported data were collected prospectively from 337 hospitalized adult patients with pain on medical/surgical oncology units in four hospitals in three states.

Study Design

This methodological study used a cross-sectional survey design. Each consenting patient completed two PainCQ^© Surveys, the Brief Pain Inventory-Short Form, and demographic questions. Clinical data were extracted from the medical record.

Data Collection/Extraction Methods

All data were double entered into a Microsoft Access database, cleaned, and then extracted into SPSS, AMOS, and Mplus for analysis.

Principal Findings

Confirmatory factor analysis using Structural Equation Modeling supported the initial factor structure. Modification indices guided decisions that resulted in a superior, parsimonious model for the PainCQ-Interdisciplinary Care Survey (six items, two subscales) and the PainCQ-Nursing Care Survey (14 items, three subscales). Cronbach's alpha coefficients all exceeded .80.

Conclusions

Cumulative evidence supports the reliability and validity of the companion PainCQ^© Surveys in hospitalized patients with pain in the oncology setting. The tools may be relevant in both clinical research and quality improvement. Future research is recommended in other populations, settings, and with more diverse groups.

Measuring and improving the quality of health care has become imperative for health care services. Performance measurement, in which health care processes, outcomes, and patient perceptions are quantified, has achieved new significance with national financial repercussions such as pay for reporting and performance. Valid and reliable performance tools are critical in establishing baseline measures and informing subsequent quality improvement efforts (Naylor 2007). Within this context, health services researchers must intensify efforts to develop and rigorously test performance measures that can be applied both locally and nationally for quality improvement and ultimately public reporting.

The Robert Wood Johnson Foundation recognized this need and prioritized performance measure development as part of the Interdisciplinary Nursing Quality Research Initiative (INQRI 2011). The intent of this report is to summarize progress on an INQRI-funded project to develop a patient-centered quality measure focused on pain management. Specifically, we will report findings from a confirmatory factor analysis of the Pain Care Quality (PainCQ^©) Surveys. A major goal of the analysis was to reduce the battery of items to produce more parsimonious and valid measures of patients' experiences of the management of their pain.

Standards for widespread adoption of quality measures are very rigorous. In addition to standard evaluation criteria such as reliability and validity, evidence that the measure can discriminate across clinical settings and populations is necessary. The National Quality Forum (NQF) includes four criteria to assess a quality measure: importance, scientific acceptability, usability and relevance, and feasibility of data collection (NQF 2010). This manuscript adds to the evidence to support the scientific acceptability of the PainCQ^© Surveys, their validity, reliability, and feasibility as quality measures in clinical oncology units. This work is preliminary to assessing the measures' ability to discriminate across diverse settings.

To date, assessment of patient-centered care has been primarily global (e.g., overall satisfaction with care or pain management), not specific to particular providers (e.g., nurses, physicians, or other care providers), and temporally framed within the entire hospital stay. For example, one question from the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) national survey asks patients whether hospital staff did everything they could to help with pain (CMS 2009). Such questions lack specific details that could be used to strategically improve pain care quality within the organization.

Development of the Pain Care Quality Surveys (PainCQ ^©)

The PainCQ^© Surveys were designed to target specific aspects of the patient experience with pain care quality in the context of a specific provider during a specific encounter or shift of care (Beck et al. 2010a). Expert and patient feedback in the formative stages of instrument development suggested that it was also important to consider the care provided by the patient's team during his/her hospital experience. Thus, the PainCQ^© Surveys are two companion tools that measure the quality of nursing and interdisciplinary care related to pain management as perceived by individuals hospitalized with varying types of pain. The reports of testing to date have focused on hospitalized medical and surgical patients in the oncology setting.

The PainCQ^© Surveys were developed using a mixed methods approach by combining qualitative and quantitative methods that have been reported in detail (Beck et al. 2010a). Items were generated from qualitative data and systematically evaluated. Cognitive interviews with 39 patients revealed that some items were relevant to nursing care during a specific time frame, whereas others were more relevant to care provided by a patient's interdisciplinary team during the hospital stay. Thus, the referent (nurse vs. team) and the timeframe (last shift vs hospital stay) for each survey differ. These findings supported an approach that included two tools: an Interdisciplinary Care Survey (PainCQ-I) and a Nursing Care Survey (PainCQ-N). This model was further supported in a series of exploratory factor analyses (EFA)—one with each survey—using principal axis factoring with an oblimin rotation. Data were obtained from 109 hospitalized patients from three oncology settings (Beck et al. 2010c).

PainCQ-I

The initial 11-item PainCQ-I generated from the EFA resulted in two subscales, Partnership with the Health Care Team and Comprehensive Interdisciplinary Pain Care; each exceeded the internal consistency reliability estimates (Cronbach's alpha = .70) recommended for new tools (.85 and .76, respectively) (Nunnally and Bernstein 1994; Beck et al. 2010c). The first subscale, Partnership with the Health Care Team (PainCQ-I Partnership), contained six items that relate to collaboration among health care team members and with the patient. Partnership occurs when health care team members believe the patient and involve that person in pain management (Beck et al. 2010b). The second subscale, Comprehensive Interdisciplinary Pain Care (PainCQ-I Comprehensive), was composed of five items that reflected care focusing on the whole person and how pain influences patients' relationship and activities. The items reflect pain management that is more comprehensive in scope such as nonpharmacological approaches, impact on family and friends, and fear of addiction (Beck et al. 2010c).

PainCQ-N

The 22-item PainCQ-N contained three subscales; the coefficient alphas ranged from .77 (Comprehensive Nursing Pain Care) to .95 (Being Treated Right), again indicating sufficient internal consistency for all three subscales (Beck et al. 2010c). The Being Treated Right (PainCQ-N Being Treated Right) subscale represents care provided by concerned nurses who are listening, anticipating problems, responding promptly, and believing the individual's reports of pain. The patient feels that there is a plan and medications available. Comprehensive Nursing Pain Care (PainCQ-N Comprehensive) reflects pain management provided by nurses that includes patient education about side effect management and the use of approaches such as positioning, distraction, relaxation and breathing, and/or massage. Efficacy of Pain Management (PainCQ-N Efficacy of Pain Management) indicates that patient comfort results from medications that work effectively and quickly (Beck et al. 2010c).

Need for a Confirmatory Factor Analysis of the PainCQ^© Surveys

In scale development, it cannot be assumed that all of the items that define a given construct have been delineated with a single evaluation of the psychometric properties of a generated scale. Additional confirmatory analytic studies are needed to assess the extent to which the scale structure holds up in additional populations. Although the exploratory factor analyses of the PainCQ^© Surveys yielded five promising subscales for evaluating effective interdisciplinary and nursing care management of patient pain, we were concerned about the large number of items (k = 15) and the very high coefficient alpha (.95) in the PainCQ-N Being Treated Right and the total number of items in the two surveys. This evidence suggested that some redundant items could be removed without affecting scale reliability. Such a reduction would increase the feasibility of its use.

The primary purpose of this current research, therefore, was to undertake a confirmatory factor analysis (CFA) using Structural Equation Modeling (SEM) to determine the stability of the structure and reliability of the PainCQ^© Surveys in a second larger national sample of an additional 337 hospitalized patients. Two additional goals of this study were to further reduce the number of redundant items in the PainCQ-I and PainCQ-N subscales and to examine the relationships between the PainCQ^© subscales and pain reports.

Methods

Participants

This methodological study used a cross-sectional survey design. Participants (n = 337) were recruited from hospitals in three geographically diverse settings with medical or surgical oncology units: Huntsman Cancer Hospital in Salt Lake City, Utah (n = 198; two inpatient units of 25 beds each, one medical and one surgical); St. Vincent Healthcare in Billings, Montana (n = 31; one 32-bed medical oncology unit [40–50 percent oncology] and one 16-bed surgical unit [35 percent oncology]); Norris Cotton Cancer Center (NCCC) at Dartmouth Hitchcock Medical Center in Lebanon, New Hampshire (n = 103; one 29-bed medical hematology/oncology unit); and the Department of Veterans Affairs Medical Center, White River Junction, Vermont (n = 5 oncology patients from one 24-bed medical/surgical unit). The use of similar types of units in diverse geographic sites allowed for accrual in a timely fashion and increased the representativeness of the sample.

An oncology population was purposively selected in this study to provide some homogeneity in terms of diagnosis but heterogeneity regarding the types of pain experienced. We anticipated that the sample of patients would experience various sources of pain: tumor, surgery, diagnostic procedures, or treatment-related side effects (e.g., oral mucositis) as well as neuropathic pain and chronic pain related to comorbid conditions. Such heterogeneity is desirable in a psychometric study, enhancing the generalizability of the findings. The University of Utah Institutional Review Board (IRB) and the IRB at each site approved this study.

Sample inclusion criteria included adult inpatients (≥18 years of age) on one of the designated units with an expected length of stay of more than 24 hours; diagnosis of cancer, surgery for cancer, a suspected cancer diagnosis, or a hematological disorder; and a positive response to screening regarding the presence of pain. We excluded patients with overt psychiatric disorders or who were cognitively or physically unable to participate.

We targeted patient recruitment to occur within 2 hours of completion of a nursing care shift. The initial screening was completed by the inpatient charge nurse or unit manager who indicated whether the patient was in pain and able to participate. Patients who met the eligibility criteria were invited to participate by trained research staff that explained the study purposes and obtained a written informed consent and authorization to obtain medical record information. As a component of the consent process, the patient was asked to summarize in his/her own words the purpose of the study to determine whether the patient understood the consent process and was cognitively able to participate.

Methods of Data Collection

Each patient completed both of the PainCQ^© Surveys described above (33 items total) and the Brief Pain Inventory-Short Form (BPI-SF) at the end of a nursing care shift. The BPI-SF is a widely used measure of time in pain, pain intensity, interference with function, and pain relief. There is ample evidence of reliability and validity of the BPI-SF (Cleeland 1989; Serlin et al. 1995). A short demographic and clinical data form was also completed by research staff. If the patient had difficulty with reading, or completing the survey, research staff assisted, usually by reading the questions to the patient.

Approach to Data Analysis

Each stage of the analysis was highly iterative, nonlinear, and involved multiple statistical approaches; results were evaluated by the team for the best theoretical and statistical fit. With each change (e.g., item deletion or modeling of interitem correlations), a new model was generated and evaluated. The types of approaches that guided our decision making are summarized below.

We began with an a priori framework based on the two generated surveys with five subscales (PainCQ-I and PainCQ-N) obtained from the previous exploratory factor analyses with 109 patients (Beck et al. 2010c). Next, data from a second sample (n = 337) were submitted to a series of confirmatory factor analyses using the SEM programs, AMOS, version 16 (Arbuckle 2007) and Mplus, version 6 (Muthén and Muthén 2010). Because multiple patients could be seen by the same nurse (n = 112 nurses, average number of patients/nurse: 2.9, range = 1–14), we represented both ‘nurse’ and ‘hospital’ as separate clustering factors in a two-level analysis and examined the extent to which such clusters had an effect on item outcomes. The intraclass correlations (ICC) were very small for both Hospital and Nurse, indicating minimal higher level clustering. Treating the PainCQ-N items as continuous variables, Nurse-clustered ICCs ranged from .013 to .072 (median = .033), whereas Hospital-clustered ICCs ranged from .000 to .008 (median = .002). Given that these cluster effects were negligible, we decided to continue with a standard SEM analysis.

Of 411 potential patients, 400 were eligible and 337 (84.25 percent) consented to participate. Missing data for the SEM analysis were replaced by the participant's mean for the subscale to which the item had been assigned if the participant had completed at least 50 percent of the subscale items. If more than 50 percent of the items were missing from any subscale, the participant's data were excluded from the SEM analysis. A maximum of 4.4 percent (n = 15) of the participants were excluded because of missing data. Examination of the missing data indicated random patterns; the items maintained similar means, variances, and covariances before and after imputation. A final sample of 327 was used for the PainCQ-I and 322 was used for the PainCQ-N.

An examination of the distributions of the PainCQ^© items revealed that 86.3 percent (19/22) of the PainCQ-N likert scale items and 63.6 percent (7/11) of the PainCQ-I likert scale items were significantly negatively skewed (Fisher skewness coefficients <−2.0). As a result, it could be assumed that these discrete categorical likert-style items did not strictly meet the data requirements for maximum likelihood estimation (MLE) that assume multivariate normality. Analyses using conventional MLE for continuous data were therefore compared with independently conducted weighted least squares (the WLSMV estimator of Mplus) analyses based on a theoretically correct response model for ordered categorical data. As both approaches resulted in highly similar solutions for the factor structures, for simplicity purposes the MLE solution will be reported here.

Several conditions needed to be satisfied in order for an item to be retained in the generated model. The path coefficients between an item and its predicted subscale on the PainCQ-I or PainCQ-N Survey identified from the Beck et al. (2010c) study needed to be statistically significant (p < .05). In addition, the items could not be highly correlated (r > .80) with other items in the survey. Modification indices generated from the structural parameters presented in the SEM analyses were used as guidelines to identify additional statistically significant and theoretically meaningful paths that were obtained, but not hypothesized, in the Beck et al. (2010c) study. These modification indices are typically used in SEM to provide suggestions for model modifications that are likely to result in a better fit of the model and a more desirable reduced chi-square value. The researcher can then determine the extent to which such modifications to the model make theoretical sense. Each modification also required reexamining the model to evaluate the effects of the change on the model fit.

The normed chi-square (χ²) goodness-of-fit test (χ²/df), root mean square error of approximation (RMSEA), and various incremental (normed fit index [NFI] and comparative fit index [CFI]), predictive (expected cross-validation index [ECVI], and Akaike information criteria [AIC]) and absolute (goodness-of-fit index [GFI]) fit indices assessed the quality of the model fit to the data (Clayton and Pett 2011). A minimum standard of a normed chi-square value between 2 and 5, values of at least 0.90 for the incremental and absolute fit indices and .08 for the RMSEA, was set (Browne and Cudeck 1992). The hypothesized values for both the ECVI and AIC needed to be smaller than both the saturated and independence models (Clayton and Pett 2011).

In addition, each of the items on the PainCQ-I and PainCQ-N subscales was examined for its redundancy with other items in the given subscale using Kaiser's (Kaiser 1958) simplicity criterion for item parsimony (Mulaik 1972). That is, each item was ranked according to the variance of its squared loadings obtained from an EFA analysis (Varimax rotation with Kaiser normalization) of the full model. Items with fairly high loadings on one subscale and near zero loadings elsewhere will have higher simplicity values compared with items that have high loadings on multiple subscales. To increase scale internal consistencies with minimal scale redundancy, theoretically and clinically defensible items were retained in order of their simplicity rankings and clinical significance. This approach facilitated our ability to achieve scale parsimony generated from a real-world clinical setting rather than merely maximizing fit.

To further evaluate the PainCQ^©'s construct validity, we assessed the correlations between the PainCQ-I and PainCQ-N and several BPI-SF items (time in pain, time in severe pain, pain intensity, interference, and relief). To have greater precision, we combined the samples from the EFA (n = 109) and CFA (n = 337) studies for a total N of 446 patients in this analysis. Given that we did not impute missing values on the BPI-SF, sample sizes vary slightly due to pairwise deletion.

Results

Participants

Table 1 outlines the demographic characteristics of the participants. Participants ranged in age from 19 to 97 years (M = 54.0, SD = 15.5); 25.8 percent were 65 or older; 56.4 percent were female, and 92.3 percent were non-Hispanic white. The majority was married (65.2 percent); 65.9 percent had some education posthigh school.

Table 1.

Demographic Characteristics of Hospitalized Patients with Pain (n = 337)

Characteristic	n	%
Gender
Female	190	56.4
Male	147	43.6
Race/Ethnicity*
Non-Hispanic white	311	92.3
Hispanic white	14	4.2
Native American/Alaskan native	11	3.3
African American	4	1.2
Asian	4	1.2
Hawaiian/Pacific Islander	1	0.3
Other	5	1.5
Marital status
Single	40	11.9
Separated or divorced	44	13.1
Widowed	33	9.8
Married/partnered	220	65.2
Education
8th grade or less	5	1.5
Some high school	17	5.2
High school	92	27.3
Tech school graduate	26	7.7
Some college	91	27.0
College graduate	64	19.0
Post grade/professional	41	12.2
	M (SD)	Range
Age in years	54.0 (15.5)	19–97

Note.

^*Percentages may not sum to 100 because more than one option can be chosen.

Clinical Characteristics of the Sample

The most common reasons for hospitalization were supportive care and management of complications (46.3 percent) and surgery (33.8 percent). Most patients (68.8 percent) reported at least one comorbidity; the most common were hypertension (23.7 percent) and arthritis (17.2 percent). The day of hospitalization at the time of survey completion ranged from 1 to 28 (Median = Day 3). A palliative care or pain specialty team provided care to 14.5 percent of participants.

Thirty percent of the respondents reported being in pain frequently or constantly during the last shift; 12 percent reported being in severe pain frequently or constantly. The most predominant causes of pain were surgery (40.7 percent), cancer (26.9 percent), and treatment complications (10.8 percent).

Results of the SEM Analyses

The results of the SEM analyses (AMOS, MLE) are presented in Figures 1 and 2. All items loaded significantly on the subscales specified initially by Beck et al. (2010c). An examination of the modification indices indicated that only two additions to the resulting model were theoretically meaningful: the correlation between error terms for two items on the PainCQ-N Survey (Figure 2) and the indication that one of the items that had originally loaded on the Being Treated Right subscale on the PainCQ-N Survey was also significantly associated with the Comprehensive Nursing Pain Care subscale.

Figure 1.

The Initial PainCQ-I (k = 11 Items Including the Shaded Items) and the Reduced PainCQ-I(k = 6 Items, with Shaded Items Removed)

Figure 2.

The Initial PainCQ-N (k = 22 Items, Including the Shaded Items) and the Reduced PainCQ-N(k = 14 Items, Shaded Items Removed)

PainCQ-I

Because high interitem correlations resulted in item redundancy, the original 11-item PainCQ-I Survey was reduced to six items with two subscales each containing three items (Figure 1, Table 2). The content of the two subscales was consistent with the initial model. All of the items loaded significantly (p < .05) on their respective factors with standardized regression coefficients ranging from .67 to .83 (Table 2).

Table 2.

Confirmatory Factor Analysis of the PainCQ-I: Standardized Regression Weights, Intersubscale Correlations, and Cronbach's Alpha Coefficients (n = 327)

		Standardized Regression Weights
Item Description	M (SD)*	PS	COMP
PainCQ-I partnership (PS)
• My health care team worked together to manage my pain	5.25 (1.14)	.78
• My health care team responded to changes in my pain	5.05 (1.35)	.77
• My health care team involved me in decisions about controlling my pain	4.90 (1.50)	.74
PainCQ-I comprehensive (COMP)
• My health care team explained that taking pain medication may increase my activity level	3.60 (1.93)		.83
• My health care team explained that patients will not become addicted to pain medication over time	3.66 (2.04)		.81
• My health care team asked about how my pain affected my relationship with others	3.10 (1.89)		.67

	Intersubscale Correlations (Cronbach's Alpha)
PainCQ-I	k	PS	COMP
PS	3	(.80)	—
COMP	3	.46	(.81)

Note. Blank cells indicate that parameters in the CFA were constrained to 0.

^*Range: 1 = strongly disagree to 6 = strongly agree.

The Cronbach's alpha coefficients for the Partnership and Comprehensive subscales were 0.80 and 0.81, respectively. These results suggest that, despite item reduction, the 6-item PainCQ-I subscales remained internally consistent for this sample of 337 hospitalized cancer patients. The correlation between the scores for the two subscales was .46 (Table 2).

PainCQ-N

Figure 2 and Table 3 present the results of the SEM analyses of the PainCQ-N. As with the PainCQ-I, the number of items in the PainCQ-N was reduced because of high interitem correlations and subsequent redundancy. The 15-item Being Treated Right subscale was reduced to seven items.

Table 3.

Confirmatory Factor Analysis of the PainCQ-N: Standardized Regression Weights, Intersubscale Correlations, and Cronbach's Alpha Coefficients (n = 322)

		Standardized Regression Weights
Item Description	M (SD)*	TR	COMP	EPM
PainCQ-N being treated right (TR)
• My nurse did a good job helping to control my pain	5.47 (1.01)	.84
• My nurse believed my reports about my pain	5.52 (1.05)	.83
• My requests for better pain relief were handled quickly	5.18 (1.32)	.83
• I felt comfortable talking to my nurse about my pain	5.54 (1.03)	.81
• I had pain medication available when I needed it	5.45 (1.15)	.80
• My nurse answered questions about my pain promptly	5.17 (1.30)	.75
• My nurse asked me about my pain	5.52 (1.07)	.74
PainCQ-N comprehensive (COMP)
• My nurse taught me that it is important to prevent the pain by taking the medication sooner rather than later	4.70 (1.69)		.75
• My nurse discussed side effects of the pain medications with me	3.98 (1.90)		.64
• Approaches, in addition to medications, worked well to control my pain. Examples are positioning my body, thinking about other things, deep breathing exercises, relaxation, and massage	3.79 (1.93)		.62
• In addition to medications, my nurse suggested approaches to help manage my pain. Examples are positioning my body, thinking about other things, deep breathing exercises, relaxation, and massage	3.59 (1.97)		.60
PainCQ-N efficacy of pain management (EPM)
• The pain medication kept me comfortable	4.97 (1.33)			.90
• The pain medications worked well to control my pain	5.00 (1.30)			.90
• The pain medication worked quickly to ease my pain	4.86 (1.37)			.88

	Intersubscale Correlations (Cronbach's Alpha)
PainCQ-N	k	TR	COMP	EPM
TR	7	(.92)	—	—
COMP	4	.53	(.80)	—
EPM	3	.57	.41	(.92)

Note. Blank cells indicate that parameters in the CFA were constrained to 0.

^*Range: 1 = strongly disagree to 6=strongly agree.

As indicated earlier, the modification indices indicated that one of the items that originally loaded on the Being Treated Right subscale also significantly double loaded >.40 on the Comprehensive Nursing Pain Care subscale. It was also one of the two items with a low simplicity value (1.67). Because the content of this item (My nurse taught me that it is important to prevent the pain by taking the medication sooner rather than later) fit better with the items in the Comprehensive subscale, it was assigned to that subscale. This resulted in a 4-item Comprehensive subscale. The third subscale, Efficacy of Pain Management, was also reduced from 4 to 3 items.

The Cronbach's alpha coefficients for the three subscales of the PainCQ-N ranged from .80 to .92 (Table 3). Interscale correlations ranged between .41 and .57 (Table 3). The overall Pearson product moment correlations among the five PainCQ-I and PainCQ-N subscales were low to moderate, ranging between .32 (PainCQ-I Comprehensive and PainCQ-N Being Treated Right, and PainCQ-I Comprehensive and PainCQ-N Efficacy of Pain Management) and .60 (PainCQ-I Partnership and PainCQ-N Being Treated Right). A similar range of correlations for these subscales was obtained with the Spearman rho correlation (.30–.67).

Goodness-of-Fit Statistics for the PainCQ^© Surveys

Table 4 compares the initial and final goodness-of-fit statistics for the PainCQ-I and PainCQ-N Surveys. For both the PainCQ-I and PainCQ-N, the final model was superior to the initial model that contained more items. Despite their statistical significance (owing to a large sample size) the χ² statistics were reduced from the initial to the final model. The more revealing normed χ² statistics indicated a satisfactory fit of the model with values of 2.89 and 2.52 (desired range of values: 2–5). For both the PainCQ-I and PainCQ-N, all of the incremental and absolute fit indices (.93–.98) were above the desired level (.90) and the RMSEA coefficients were within the desired confidence intervals (.04–.11 and .06–.08, respectively).

Table 4.

Comparison of the Fit Measures for the Initial and Final Solutions for the PainCQ^© Surveys: PainCQ-I and PainCQ-N

	PainCQ-I		PainCQ-N
Solution	Initial (k = 11)	Final (k = 6)	Initial (k = 22)	Final (k = 14)
n	327	327	322	322
χ2 goodness of fit	150.91***	23.09***	943.56***	184.23***
df	43	8	206	73
Normed χ2 (χ2/df)	3.51	2.89	4.58	2.52
RMSEA (90% confidence interval)	.09 (0.7–.10)	.08 (.04–.11)	.11 (.10–.11)	.07 (.06–.08)
SRMR	.06	.03	.08	.04
Incremental fit indices
NFI	.92	.97	.83	.94
CFI	.94	.98	.87	.96
Predictive fit indices
ECVI
Hypothesized	.60	.15	3.23	.77
Saturated	.41	.13	1.58	.65
Independence	5.63	2.38	17.84	9.78
AIC
Hypothesized	196.91	49.09	1,037.56	248.23
Saturated	132.00	42.00	506.00	210.00
Independence	1,835.93	774.30	5,726.63	3,139.83
Absolute fit indices
GFI	.92	.98	.79	.93

Note. *p < .05, **p < .01, ***p < .001.

AIC, akaike information criterion; CFI, comparative t index; ECVI, expected cross-validation index; GFI, goodness-of-fit index; NFI, normed t index; RMSEA, root mean square area of approximation; SRMR, standardized root mean square residual.

Construct Validity

Statistically significant but low correlations were found between PainCQ-I Partnership subscale scores and pain outcomes: time in pain, time in severe pain, pain intensity, pain interference with function, and pain relief as measured using the BPI-SF) (Table 5). PainCQ-I Comprehensive subscale scores were not correlated with the pain outcome measures except for pain relief.

Table 5.

Correlations of the PainCQ-I and PainCQ-N Subscales with BPI-SF Pain Outcomes (n = 443)

	Time in Pain	Time in Severe Pain	Pain Intensity	Pain Interference	Pain Relief
BPI-SF	n = 441	n = 441	n = 441	n = 440	n = 436
Mean (SD)	2.78 (1.39)	1.69 (1.42)	3.86 (2.00)	4.10 (2.67)	7.33 (2.55)
Range	0–5	0–5	0–10	0–10	0–10
PainCQ-I
PainCQ-I partnership	−.13*	−.15**	−.13*	−.24***	.23***
PainCQ-I comprehensive	−.10	.01	.01	−.07	.14**
R2	.017	.038	.028	.072	.051
PainCQ-N
PainCQ-N being treated right	−.24***	−.26***	−.26***	−.27***	.33***
PainCQ-N comprehensive	−.23***	−.12*	−.12*	−.12*	.25*
PainCQ-N efficacy of pain management	−.43***	−.39***	−.41***	−.44***	.55***
R2	.189	.180	.195	.223	.305
Total R2	.201	.186	.210	.238	.314

Note. *p < .05, **p < .01, ***p < .001.

For this analysis, two samples (n = 109 and n = 337) were combined; sample size varied due to missing values.

Statistically significant moderate correlations were found between all PainCQ-N subscale scores and the pain outcome measures (Table 5). When compared with PainCQ-N Being Treated Right and PainCQ-N Comprehensive subscale scores, PainCQ-N Efficacy of Pain Management subscale scores were the most highly correlated with pain outcomes (r = −.44 [pain interference] to r = +.55 [pain relief], p < .001). The PainCQ^© Surveys as a whole explained 19–31 percent of the variance in pain outcome scores. Although these findings support the construct validity for all of the PainCQ-N and PainCQ-I Partnership subscales, the construct validity for the PainCQ-I Comprehensive subscale was not consistently supported.

Discussion

The findings of this study add to the evidence supporting the reliability and validity of patient-centered measures of the quality of nursing and interdisciplinary care related to pain management among individuals hospitalized on medical and surgical oncology units. The results of our analyses suggest that both the PainCQ-I and PainCQ-N Surveys with five subscales are potentially effective measurement tools. The support for the relationship of the PainCQ-N subscales and the PainCQ-I Partnership subscale with pain outcomes was stronger than with the items in the PainCQ-I Comprehensive subscale. However, as patients rated their comprehensive care lower, these items may inform the need for future improvement efforts in this area. Additional testing is warranted.

The iterative process of confirmatory factor analysis and item analysis indicated that the final model fit better than the saturated or independence models. Our conclusion, therefore, was that the final models for the PainCQ-I and PainCQ-N were the best solution for the analyzed data. These models confirmed the theoretical constructs that have been developed from both qualitative interviews and exploratory factor analysis (Beck et al. 2010a,b,c). Moreover, we were able to reduce the number of items from 33 to 20, thus creating a more parsimonious set of items for each construct. This reduction will improve the feasibility and reduce the mean time to completion in future studies. The reliability of the shorter subscales was supported by Cronbach's alpha coefficients of greater than .80 for each of the subscales, exceeding recommended standards (Nunnally and Bernstein 1994).

Study Limitations

Although the oncology setting provides a good prototype for pain in hospitalized patients, it will be essential to test the PainCQ^© Surveys in other populations of hospitalized patients. Pain is a highly prevalent and distressing symptom in other patient populations and across all health care settings. Therefore, it will be important to evaluate pain care quality in other types of patients as well (e.g., chronically ill, pediatric, and critically ill populations). The tool may also be adapted for testing in other settings such as ambulatory care, home care, or long-term care. Our team is currently extending this research to compare our findings across more heterogeneous clinical settings and populations. Such multilevel research with larger samples will hopefully provide evidence of the stability of the factor structure across populations and multiple settings, and will enable us to evaluate the extent to which pain care quality varies by setting, such as a nursing unit or hospital (Donaldson 2005). In addition, future research will need to focus on more racially and ethnically diverse patient populations and evaluate the tool in other languages. Such evidence would also support potential adoption of the PainCQ^© as a national quality measure.

Implications

We believe that the systematic work to develop and test the PainCQ^© Surveys has led to accumulating evidence to support the reliability and validity of these measures of pain care quality in hospitalized oncology patients. Patients in pain, including the elderly, can complete the surveys and make judgments about their care. The shorter versions of the tools could be evaluated for use in quality improvement (QI) efforts within the oncology acute care setting. It is recommended that each of the five subscales be scored separately by computing a mean of the relevant items. This approach is consistent with recommendations by DeVellis (2006) when a factor structure exists and would more clearly guide areas for improvement. Although in a few cases the correlations across the PainCQ-I and PainCQ-N subscales were slightly higher than the within factor correlations, the impact of the referent group (the nurse or interdisciplinary team) would indicate a different focus for improvement. Thus, combining the subscales would reduce their clinical relevance.

The low number of patients per nurse in this study did not support aggregation to the level of the nurse and additional research is warranted. QI projects in which a greater number of patients per nurse are evaluated would add to an evaluation of the use of surveys in this manner. An evaluation at the nursing unit level is also recommended. In clinical research, the tools could be used to test the efficacy of specific interventions designed to reduce the intensity of pain experienced by patients, including psycho-educational, communication, pharmacologic strategies, and other integrative approaches.

Within a quality improvement perspective, these patient-experience assessment tools may directly link perceptions of patient-centered care and pain outcomes to the effectiveness of day-to-day practices within hospital units. Unlike existing patient response tools, the PainCQ^© Surveys solicit detailed information surrounding the management of pain in the hospital settings. Potentially, organizations can use these data to “drill-down” to elements in pain care quality that require the most improvement. Additional investigation of the tools' use at either the unit or hospital level is recommended. For example, a unit or organization may have scored well on the subscale; PainCQ-N Being Treated Right, but PainCQ-I Comprehensive and PainCQ-N Comprehensive scores may be low. A targeted intervention that addresses the Comprehensive items could be employed to improve pain care quality within the individual nursing unit or organization.

As a part of quality improvement efforts, the tools could be used to test the efficacy of specific and tailored interventions designed to reduce the intensity of pain experienced by patients. Interventions could include educational or organizational strategies to improve care processes. In addition, changes in hospital nurse staffing could be tested as an example of a systems-based intervention that could lead to improved care quality. Such uses would build evidence related to the tools' sensitivity to change.

The value of the PainCQ^© Surveys as quality measures for purposes of benchmarking and ultimately public reporting is yet to be determined. In this study, the impact of the hospital was negligible, partly because relatively homogeneous sites were selected. Extending the evaluation to a greater number of diverse hospitals and patient populations beyond oncology is necessary. Evaluation at the nursing unit level also is recommended. The low impact of the nurse was surprising. This finding may be explained by the low number of patients per nurse, which may have been inadequate to reflect differences among nurses. It is also possible that the tool is not sensitive enough to detect such differences or that the differences are negligible. Perhaps there is inadequate exposure to an individual nurse or the nursing quality is fairly homogeneous for one patient. Additional evaluation is warranted.

Overall, measurement is imperative to initiate quality processes that will improve the nation's health. This report provides an example of a systematic and iterative approach to developing a patient experience measure focused on a significant and costly clinical problem pain. We are completing a project to replicate this analysis in a sample of medical and surgical hospitalized veterans. A recently funded initiative will translate representative items from the PainCQ^© Surveys into pain quality indicators in over 300 U.S. hospitals that participate in the National Database of Nursing Quality Indicators. We are aggregating data to the nursing unit level in this project. Findings from these projects will provide additional evidence to evaluate the use of such measures in quality efforts.

SUPPORTING INFORMATION

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.