Resident Vignettes for Assessing Care Quality in Nursing Homes

Abstract

Objectives

Validated process measures that correlate with patient outcomes are needed for research and quality improvement.

Design

Cross-sectional analysis within a cluster-randomized fall prevention study.

Setting

Nursing homes in North Carolina, n=16.

Participants

Nursing home staff (n=541) and residents with 1 or more falls in 6 months (n=597).

Measurements

Fall-prevention process measures in 4 categories derived from Assessing Care of Vulnerable Elders (ACOVE) quality indicators were measured in two ways: 1) chart abstraction; and 2) staff responses to clinical vignettes of hypothetical residents at risk for falls. Recurrent fall rates (falls/resident/year) were measured. The proportion of the total variation in falls rates explained by the scores for each method (chart abstraction or vignette) was calculated using multi-level adjusted models.

Results

Chart and vignette measures of comorbidity management were moderately correlated (Pearson correlation coefficient 0.43) while other process measure categories had low or negative correlation between the two methods (psychoactive medication reduction 0.13, environmental modification -0.42, and exercise/rehabilitation -0.08). Measures of environmental modification and comorbidity management explained a moderate amount of the total variation in recurrent fall fates, vignettes (7-10% variation explained) were superior to chart abstraction (2-6% variation explained). Vignette responses from unlicensed staff (Nurse Aides and Rehabilitation Aides) explained more variance than Registered Nurses, Licensed Practical Nurses, or other Licensed Staff in these categories. Process measures for psychoactive medication reduction and exercise/rehabilitation did not explain any of the variation in fall outcomes. Overall, vignette process measures explained 3.9% and chart abstraction measures explained 0% of the variation in fall outcomes.

Conclusion

Clinical vignettes completed by nursing home staff had greater association with resident recurrent fall rates than traditional chart abstraction process measures.

Background

While resident-level outcome measures are the gold standard for research and quality improvement in nursing homes (NH), the large sample size requirements and need for case-mix adjustment frequently make them impractical. Therefore, measures describing the care process (called process measures or quality indicators) are commonly employed.¹ Process measures describe how well care delivery matches “best practice” as defined by clinical trials or expert opinion, and are typically collected by chart abstraction. Specific process measures for a variety of common conditions are recommended in practice guidelines from the American Medical Directors Association and Assessing Care of Vulnerable Elders (ACOVE), and many are collected in the Minimum Data Set (MDS).^2,3

There are several challenges in using process measures to assess care quality. First, there must be an accurate and efficient means of collecting them. While chart abstraction is appropriate for single, consistently documented measures (e.g., receipt of influenza vaccine, completion of advanced directives), it is much less feasible for complex conditions with multiple process measures (e.g., falls, pressure-ulcer prevention, heart failure management). Furthermore, chart abstraction measures rely on the accuracy and completeness of the medical record; prior studies have demonstrated low and inconsistent documentation in NHs.⁴ Finally, trained staff and assessment of inter-rater reliability are required for valid chart abstraction, adding to measurement burden.

Another problem is that process measures collected by chart abstraction do not always correlate with the resident outcomes they are meant to improve.⁵ For example, process measures for pressure ulcer prevention have low correlation with actual rates of pressure ulcers in NHs.⁶ More often, the correlation between the process measures and outcome of interest is not known. Therefore, additional measurement options that accurately and efficiently describe the quality of care delivered, and are demonstrated to correlate with resident outcomes are urgently needed.

One such option may be the use of clinical vignettes.⁷ In vignette measures, providers are presented with one or more patient scenarios and are asked to describe how they would manage them. Multiple process measures can be assessed in a single scenario and data can be collected efficiently with either paper tools or computer applications. Because staff respond to the same patient scenarios, it is unnecessary to adjust for patient-level differences in risk. Studies in primary care suggest that clinical vignettes completed by physicians have better correlation with care quality, as measured by standardized patients, than process measures obtained by chart abstraction.^7-9

Vignettes have been used in NHs to describe variations in provider decision-making¹⁰, staff knowledge and attitudes¹¹, and resident preferences¹². However, to our knowledge no studies have examined the use of clinical vignettes in assessing care quality. We evaluated the feasibility and validity of using clinical vignettes to measure fall process measures in 16 NHs. We examined the correlation between process measures collected via vignettes compared to chart abstraction, their association with resident fall rates, and their sensitivity to change following a fall reduction quality improvement initiative.

Methods

Design

Observational study leveraging data from a cluster-randomized fall prevention trial.¹³ For this analysis, staff in 16 NHs completed fall prevention vignettes before and immediately after a 3-month fall prevention quality improvement program.¹⁴ Chart abstraction process measures were also collected before and after the program. Resident fall rates in the 6 months following the quality improvement program were measured. Half of the NHs were randomized to an additional staff education intervention as part of the parent trial (CONNECT, designed to enhance information flow and problem solving), but CONNECT had no impact on chart abstraction process measures or resident fall rates.¹⁵ Therefore, all 16 facilities are included in this analysis, but participation in CONNECT is indicated in all statistical models. This study was approved by the Duke University Institutional Review Board.

Setting

NHs in North Carolina participating in the parent study had at least 90 beds and accepted Medicare and Medicaid.

Staff Participants

All staff in nursing (including Registered Nurses, Licensed Practical Nurses, and Nurse Aides), rehabilitation, and social work with direct resident care roles were invited to participate. Staff had to speak English and provide individual signed informed consent (n=541). Participants self-reported their role, training level, years in the facility, age, gender and race/ethnicity at baseline.

Resident Participants

Residents were 65 years of age or older, experienced at least 1 fall in the 6 months following the falls quality improvement intervention, and remained in the facility for at least 30 days after the fall event. These criteria allowed adequate time for staff to document fall prevention plans in high-risk residents. If more than 50 residents in the facility fell during this period, a random sample of 50 unique residents was selected. A waiver of informed consent was obtained for the resident sample (n=597).

Chart Abstraction Measures

Fall prevention process measures recommended by the Assessing the Care of Vulnerable Elderly (ACOVE) group and the American Medical Directors Association clinical practice guideline were abstracted from the medical record for the 30 days following the resident's first fall. Measures included orthostatic blood pressure measurement or intervention; sensory impairment evaluation or intervention; footwear change; gait, exercise, or assistive device intervention; toileting schedule implemented; environmental modification completed; at least 1 psychoactive medication reduced; vitamin D supplements prescribed. Medical records were abstracted by trained nurses employed by the state Quality Improvement Organization; a random sample of 10% of records were abstracted by a second nurse and at least 90% agreement maintained.

Vignette Measures

The vignette measure was designed by the authors based on prior vignette studies in other settings,^7-9 and the content of staff education materials included in the Agency for Healthcare Research and Quality's Falls Management Program.¹⁴ Brief resident “stories” were constructed by varying demographic characteristics (gender, race, older/younger age, long-term care/rehabilitation stay) and the presence or absence of fall-related risk factors (prior falls, dementia, multiple fall-related medical conditions, ambulatory status and assistive device use). Age, race, and gender were depicted by photo, while risk factor descriptions were written at an 8^th grade reading level. After reading the resident story, staff indicated how likely it was that each of 12 fall prevention activities would be completed by anyone in their facility for that hypothetical resident by circling a number between 0 (never) and 10 (always). These 12 activities were the same as the process measures collected via chart abstraction as described above. A sample vignette is found in figure 1. Vignettes were refined in a convenience sample of nursing home staff prior to deployment, and validation of the vignette's factorial structure has been previously reported.¹⁶ Each vignette requires less than 5 minutes to complete.

Figure 1.

Sample clinical vignette completed by nursing home staff. Resident characteristics were randomly generated from a matrix of dimensions. Gender, race, and older/younger age were depicted by a photo. The presence or absence of fall risk factors (prior falls, dementia, multiple chronic conditions, type of stay, ambulatory status and assistive device use) were described in a brief story. Each staff member completed up to 4 vignettes with different combinations of resident characteristics.

Staff were asked to complete 2 clinical vignettes before and immediately after the falls quality improvement intervention, for a maximum of 4 vignettes over 3 months (n=1615 total vignettes). The vignettes stories were generated using a factorial design, such that 576 possible combinations of the demographic characteristics and fall-related risk factors described above were available. Each staff participant received 4 different vignettes randomly selected from this pool using a random number generator. The randomized selection of vignettes ensured balanced resident characteristics in the vignettes across facilities, while also allowing us to examine the impact of vignette resident characteristics on fall prevention, as previously reported.¹⁶

Fall Rates

Median recurrent fall rates (falls/resident/year) among residents with at least 1 fall in 6 months were calculated. Falls were assessed by chart abstraction supplemented with review of facility fall logs. Resident follow-up days were calculated from the index fall date to the end of the 6-month study window, censoring for death, discharge, and hospital stays.

Analysis

To reduce multiple statistical comparisons, the 12 fall prevention activities in the vignette and chart abstraction measures were grouped into 4 categories based on the Assessing the Care of Vulnerable Elders (ACOVE) quality indicator groups.¹⁷ These categories were: (1) environmental modification (low bed or floor mat, footwear, lighting, alarms, bathroom equipment, room clutter, room location), (2) psychoactive medication reduction, (3) exercise/ rehabilitation (therapy referral, assistive device change, regular exercise, increased activities), and (4) comorbidity management (orthostatic hypotension measurement/management, vision assessment, toileting schedule, vitamin D). Cronbach's alpha, principal components analysis, and scree plots were used to confirm the four-factor structure of the vignette survey.

The total variance in fall rates within- and between-facilities was calculated accounting for clustering at the facility level using multilevel modeling. Variance was adjusted for resident-level risk factors (age, gender, race, prior falls, cognitive impairment, Parkinsonism, neuropathy, low vision, stroke, ambulatory status, assistive device use), staff-level factors (number of years at the facility, race, gender), and facility-level factors (staff turnover rate, staffing levels, intervention status, facility bed-size, RN Staffing Rating). First, a multilevel null model was constructed without the quality indicator scores for either chart abstraction or vignettes. Then, multilevel comparison models were constructed with mean facility scores for each quality indicator category as measured by chart abstraction or vignettes. Vignette responses were adjusted for vignette patient characteristics (age category, race, gender, falls history, assistive device use, dementia, and multiple co-morbidities). The proportion of the total variance in falls rates explained by the scores for chart abstraction or vignettes was calculated for each category, and overall for each measurement method as 100%×(total variance in null model - total variance in comparison model)/(total variance in null model). Missing data on individual items were imputed from the average of that staff member's responses to the other items on the fall intervention category, if available, or censored if not. There was no adjustment for multiple comparisons.

Results

Characteristics of NHs, staff respondents to vignettes, and residents with falls who underwent chart abstraction are listed in Table 1. All NHs were for profit, with a mixture of chain and free-standing facilities, and a mixture of urban and rural setting. Consistent with staffing levels in the NHs, a majority of respondents were unlicensed staff (Nurse Aides and Rehabilitation Aides) (47.1%) with LPNs, RNs, and other licensed staff (rehabilitation and social work) representing 14.0%, 11.4% and 7.4% of the sample, respectively. Residents with falls were predominantly female (61.3%), white (71.4%), receiving long-term care (58.2%), and had mean age of 81.9 years.

Table 1. Characteristics of participating nursing homes, staff, and residents who fell.

Nursing homes	N=16
Bed size (mean)	127
Non-chain (%)	5 (31%)
Urban location (%)	8 (50%)
Staff participants	N= 541
Age (%)
18–35 years	159 (29.4%)
36–55 years	254 (47.0%)
56 years and older	78 (14.4%)
Unknown	50 (9.2%)
Female (%)	443 (81.9%)
Race (%)
White	191 (35.3%)
Black or African American	268 (49.5%)
Others	40 (7.4%)
Unknown	42 (7.8%)
College or Associate Degree (%)	179 (33.1%)
Role (%)
Unlicensed	319 (47.1%)
LPN	95 (14.0%)
RN	77 (11.4%)
Other licensed staff	50 (7.4%)
Residents who Fell	N= 597
Mean age, years (SD)	81.9
Female (%)	366 (61.3%)
Race (%)
Caucasian	426 (71.4%)
Black	163 (27.3%)
Other	8 (1.3%)
Receiving Long-term Care	347 (58.2%)
Falls in 6 months prior to index (%)	282 (47.2%)
Cognitive Impairment (%)	384 (64.3%)
Parkinsonism (%)	50 (8.4%)
Neuropathy (%)	71 (11.9%)
Vision Impairment (%)	156 (26.1%)
Ambulatory Status (%)
Independent	454 (76.0%)
Dependent	143 (24.0%)
Stroke (%)	214 (35.8%)

For the vignette process measures, 712 of 1320 eligible clinical staff consented to participate in the parent study, and 541 of consented participants completed at least 1 vignette for overall participation rates of 41.3%% of licensed and 41.7% of unlicensed staff. Of those who completed at least 1 vignette, 56.9% completed all 4 vignettes. Participation rates varied across facilities from 29.3% to 70.2%. As we reported previously, the vignette measures were strongly influenced by staff respondent role, with RNs tending to report a higher likelihood that activities would be completed for all 4 intervention categories than did LPNs, unlicensed staff (Nurse Aides and Rehabilitation Aides), and especially other licensed staff (Rehabilitation Staff and Social Work).¹⁶

The correlation between process measures as assessed by chart abstraction and vignettes within each ACOVE quality indicator category is shown in Table 2. Chart and vignette measures of comorbidity management were moderately correlated (Pearson correlation coefficient r= 0.43) and measures of psychoactive medication reduction were slightly correlated (r= 0.13). In contrast, chart and vignette measures of environmental modification (r= -0.42) and exercise/rehabilitation interventions (r= -0.08) were negatively correlated. None of the correlations were statistically significant due to the small NH sample size that limited the power of statistical testing.

Table 2. Correlation between chart abstraction and vignette process measures within facilities (N=16).

Process Measure Category	Pearson Correlation Coefficient	P value
Environmental Modification	-0.42	0.10
Exercise/Rehabilitation	-0.08	0.76
Comorbidity Management	0.43	0.10
Psychoactive Medication Reduction	0.13	0.62

The median recurrent fall rate among residents who fell at least once was 2.2 falls/resident/year, and ranged from a minimum of 0 to a maximum of 18.3 falls/resident/year across facilities. The proportion of total variance explained at both the resident and facility levels by each of the quality indicators as measured by chart abstraction or vignette is shown in figure 2. Process indicators for environmental modification and comorbidity management explained a moderate amount of the total variation in recurrent fall rates; in both cases vignette measures (7-10% variation explained) were superior to chart abstraction measures (2-6% variation explained). Process measures for psychoactive medication reduction and exercise/rehabilitation interventions did not explain any of the variation in fall outcomes, and indeed were negatively associated with them, regardless of whether they were measured by vignette or chart abstraction. All vignette process measures combined explained 3.9% of the variation in fall outcomes, while chart abstraction measures combined explained 0% of the variation.

Figure 2.

Proportion of total outcome variation (variation of the outcome at resident and facility level) explained by process measures as measured by chart abstraction and vignette.

Because the job category of respondents was strongly associated with their vignette responses in a prior analysis of this dataset¹⁶, we repeated the analysis using the mean vignette responses from each of 4 job categories; unlicensed staff (Nurse Aides and Therapy Aides), RNs, LPNs, and other licensed staff (Rehabilitation and SW). Responses from unlicensed staff explained substantially more of the variance for the 2 categories in which vignettes were associated with fall rates. For environmental modification, responses from unlicensed staff explained 8.3% of the variance whereas responses from all other job types were negatively associated with fall rates. For comorbidity management, responses from unlicensed staff explained 8.9% of the variance, responses from LPNs explained 5.3% of the variance, and responses from RNs and other licensed staff were negatively correlated with fall rates.

Discussion

Selecting valid process measures that correlate with the resident outcomes that they are meant to affect is a key step in NH research and quality improvement. To our knowledge, this is the first evaluation of the use of clinical vignettes completed by NH staff as a facility-level measure of care quality. We found that 2 of 4 fall prevention process measures assessed by clinical vignettes, specifically environmental modification and comorbidity management, explained 7-10% of the variation in resident- and facility-level recurrent fall rates. Given the multifactorial pathogenesis of falls, this magnitude of variance explained by a process measure is fairly substantial. For example, a multicomponent model including measured patient reaction time, postural sway, quadriceps strength, vision, and cognition explains 18% of the variance in recurrent falls.¹⁸ Notably, vignettes were superior to chart abstraction for all process measures.

Our findings suggest that clinical vignettes offer a promising alternative to chart abstraction for measuring NH quality. Vignettes were acceptable to staff and were simple to administer and analyze, whereas chart abstraction was time-consuming and required substantial resources for abstractor training, validation, and database development. Vignettes also use standardized patient examples, eliminating the need to adjust for differences in resident-level risk factors in chart abstraction measures. Therefore, vignettes may be especially cost-effective for complex conditions requiring multiple process measures and case-mix adjustment. It is intriguing, but perhaps not surprising, that the responses of unlicensed staff (and to a lesser extent, LPNs) were more highly associated with falls than the responses of RNs and other professional staff. Nurse Aides provide 80% of the hands-on care in NHs¹⁹ and would be directly involved in implementing many of the fall prevention interventions measured by the vignettes (e.g., toileting schedules, monitoring orthostatic blood pressure). We and others have reported a disconnect between written care plan for multifactorial problems such as falls (created by the professional staff) and actual resident care (implemented at the bedside by the unlicensed staff and LPNs).^20,21 Future studies should explore if it is possible, or even desirable, to limit the distribution of vignette surveys to the staff most engaged in providing the care. Our findings also provide critical support for directly incorporating the knowledge of direct-care staff in process measures of quality, even when such knowledge differs from clinical managers.

While promising, further work is needed before vignettes can be used routinely in NH research or quality improvement programs. First, not all vignette process measures were associated with facility fall rates, including assessments of important components of evidence-based fall prevention programs (psychoactive medication reduction, exercise and rehabilitation interventions). This may be because not all fall reduction activities are observed by all staff; for example, psychoactive medication reduction may occur without the knowledge of unlicensed staff, and may be better answered by facility pharmacists or medical providers. Additional refinement of the language, the number of items, and the optimal respondents in these categories is needed. Second, while we showed a cross-sectional association between vignette process measures and falls, it remains unknown whether improvements in process measures are associated with improvements in fall rates. Because fall rates did not change in our facilities after implementation of the Agency for Health Care Research and Quality Falls Management Program, we were unable to test this important question. Third, we found that staff role is a more powerful driver of responses on clinical vignettes than the facility where staff member is employed. Therefore, it is crucial to measure and control for staff role in any vignette measure. Finally, it would be useful to determine whether vignette measures can distinguish differences in care quality between different nursing units within the same facility so that quality improvement efforts can be targeted to those with lower performance.

Our findings also highlight the need to establish the relationship between process measures and the resident outcome they are meant to reflect. Many clinical practice guidelines, quality improvement programs, and regulatory documents (e.g., MDS) recommend specific process measures that have face validity based on findings from prior clinical trials or cohort studies.²² Frequently, these are developed using methodologies summarizing expert opinion.²³ However, differences in the operationalization of interventions or measurement of their implementation across sites can result in process measures that are not associated, or even negatively associated with the true outcomes of interest.⁶ Therefore, these process measures require validation before they are widely adopted by regulators or administrators. Further, our results highlight that not all process measures for a given clinical condition have same strength of association with the outcome, and measurement efforts should be prioritized to those most strongly associated.

The study findings should be interpreted in light of its strengths and weaknesses. We had a large staff sample size with over 40% participation of eligible clinical staff. We had a rigorous chart abstraction process with over 90% interrater agreement for process measures and falls. However, our NH sample was small (n=16) and from one geographic region, limiting generalizability. We had no participation from medical or pharmacy team providers, who play a key role in fall prevention interventions. The lack of benefit of the falls quality improvement program on fall rates prevents us from testing the sensitivity of the vignette measure to change.

We conclude that clinical vignettes completed by NH staff were more feasible and had greater association with resident recurrent fall rates than traditional chart abstraction process measures. Additional research is needed to refine the fall clinical vignettes, and determine whether this approach is useful for assessing the quality of care in other geriatric syndromes.