Show Back: Developing and Testing of a Simulation Based Assessment Method for Identifying Problems in Self-Management of Medications in Older Adults

Introduction

Older adults are at significant risk of adverse drug events (ADEs) after hospital discharge.^1,2 Current methods of assessing for medication management proficiency such as the teach back approach are suboptimal, missing impairments in vision, literacy, attention, and dexterity which are common in older adults. By simulating medication management, an older adult could demonstrate proficiency or lack thereof in areas where errors commonly occur.

Home health nurses are well-positioned to identify management problems. Currently these nurses do not systematically assess medication management proficiency across domains. A systematic method of observing and appraising medication management across domains would enable nurses to identify and correct errors.

In this study, we developed and tested a comprehensive simulation which assesses older adult medication self-management proficiency.

Methods

Development of Simulation

We convened an advisory panel of nurses from the Visiting Nurse Association (VNA) Care Network of Massachusetts to develop the simulation. We identified several domains where older adults make medication errors and eventually selected five domains for testing: medication identification, medication purpose explanation, organization, administration, and timing of doses. Through discussions, we also decided to rate proficiency with the simple yes or no score for each domain, to limit our simulation to ten medications, to assess only non-pill medications for organization, and to limit evaluation of administration to injectable and inhaled medications or pills requiring cutting.

Setting / Population for Evaluation of Simulation

We enrolled English-speaking clients of the VNA aged 65+. We excluded patients with significant cognitive impairment.

Simulation Reliability Testing

A VNA nurse and a physician tested the assessment method during home visits. After several revisions, we codified a manual with rules for conducting the simulation (Appendix A). We then assessed ten patients using the manual. From these visits, we were able to measure interrater agreement and time required for the simulation.

To calculate interrater reliability adjusted for correlation by patient, we followed the procedure of Yang et al.³ We characterized kappa values following Landis and Koch.⁴

We performed all calculations in Statistical Analysis Software © version 9.4.⁵

Results

The average age of our patients was 76 ± 7.1 years. We assessed 81 medications (Table 1). The mean time required to conduct each simulation was 22 ± 14 minutes. Eight of nine patients found our assessment helpful or very helpful. One patient did not answer our question about satisfaction.

Table 1.

Characteristics of Medications Assessed

Medication Route	n (% out of 81)
Inhaled	4 (4.9)
Injection	3 (3.7)
Oral/Pill	74 (91.4)
Medication Class	n (% out of 81)
Vitamin/mineral/supplement	28 (34.6)
Anticoagulant	7 (8.6)
Antihyperlipidemic	7 (8.6)
Antihypertensive	7 (8.6)
Opioid	6 (7.4)
Antidepressant	4 (4.9)
NSAID	4 (4.9)
Antibiotic	3 (3.7)
Bronchodilator	3 (3.7)
Gastrointestinal	3 (3.7)
Antidiabetic	2 (2.5)
Other	7 (8.6)

In cases in which there was no disagreement between raters, we found high proficiency across all five domains (83%–100%). Six patients had at least one medication in which both raters scored not proficient for at least one domain. Most cases of being not proficient stemmed from patients being unable to explain reason for taking a medication (n = 13 of 81 medications). Inter-rater agreement for explanation and organization was near-perfect, (k= 0.837 95% CI 0.627–1.046 and 0.840 95% CI 0.442–1.229, respectively). For timing, agreement was moderate (k= 0.702 95% CI 0.409–0.997); for identification it was fair (k = 0.220 95% CI −0.142–0.584). We had few injectable and inhaled medications, limiting our ability to assess administration although agreement was moderate in those cases (k=0.633 95% CI 0.232–1.034).

Discussion

We successfully developed a comprehensive, standardized simulation to assess medication self-management proficiency in older adults by home nurses. Although similarities with other instruments are limited, our medication assessment compares favorably with the Medication Discrepancy Tool (MDT), which identifies medication discrepancies based on patient and systemic factors for older adults recently discharged from hospital to home.⁶ In their testing, the developers identified moderate agreement (k=0.56) across three different types of clinicians. We investigated a different source for errors, i.e. lack of self-management proficiency. We achieved similar agreement as the MDT although we did not pool across domains. We believe assessing proficiency discretely for five domains captures a broader set of vulnerabilities for the development of an ADE compared with MDT. We plan to study our instrument in a larger sample to confirm our hypothesis.

Limitations

We found only fair agreement for identification of medications. We plan to improve training to bolster agreement of our simulation for future use. We also did not calculate a summary score to determine proficiency across all domains. We plan to review the role of a summary statistic for predicting ADEs.

Conclusion

We successfully developed a comprehensive, standardized simulation to assess medication self-management proficiency in older adults. The simulation was reliable, time efficient, and satisfactory to patients. In future work, we plan to test this assessment method for its ability to predict ADEs and related readmissions.