Usability Testing of a Sick-Day Protocol in CKD

Assessing the performance at self-management of patients with CKD is essential for effective strategies to safeguard care. We evaluated the use of a sick-day protocol with structured usability testing by patients with CKD. The Sick-Day Protocol is a self-management protocol developed by National Health Service Highland in Scotland to prevent AKI, and it is made available through the Scottish Patient Safety Program on a “Medicine Sick Day Rules” card (1).

The Sick-Day Protocol card describes a gastrointestinal or fever-related and volume-depleting illness; then, it directs patients to withhold specified medications and resume them when well (after 24–48 hours of eating and drinking normally). Five medication classes and examples are listed, including angiotensin-converting enzyme inhibitors, angiotensin receptor blockers, diuretics, nonsteroidal anti-inflammatory drugs, and metformin.

We recruited volunteers with stages 3–5 CKD from the Safe Kidney Care program (2). All participants independently scheduled and completed testing. The protocol was deemed exempt by the University of Maryland Baltimore Institutional Review Board. Usability testing sessions were conducted in a private room by a moderator and recorder. Participants were educated about the purpose of the Sick-Day Protocol and qualifying illnesses. The moderator presented the card to all participants and requested each to summarize it in their own words. The study team developed four scenarios representing commonly encountered medical problems. All were reviewed and adjudicated by two physicians (C.J.D. and J.C.F.) for clinical validity and population suitability. Participants were presented the scenarios, including one for which the Sick-Day Protocol would be appropriately activated (index) and one with nonqualifying illnesses, including nephrolithiasis, congestive heart failure, and a “minor” gastrointestinal illness. One half of the sample had scenarios presented in a fixed sequence, starting with the index. The remainder had the scenarios presented randomly. The moderator reviewed the mock patient’s medications and comorbidities in each scenario and then, provided mock medication bottles. Participants were asked about the appropriateness of the Sick-Day Protocol, if any of the mock medications should be withheld in each scenario, and if so, which ones. Completion of each task was classified as a success or error. After completing the testing, the coordinator offered participants feedback on their performance.

Twenty participants, enrolled between April 13, 2017 and June 8, 2017, were predominately men and over the age of 65 years old, with 70% reporting diabetes and 45% seen in the emergency department or hospital during the past 6 months. Most reported at least graduating high school, and over one half had household incomes ≤$50,000. All participants reported managing their own medications. When asked to evaluate the index scenario, a majority (18 of 20) correctly identified it as qualifying for the Sick-Day Protocol and recognized the need to hold medications during the illness. Correct classification of the remaining scenarios as ineligible for the protocol ranged from seven to 12 participants per scenario. The proportion of correct scenario classifications did not differ on the basis of order of scenario presentation.

Participants were also asked to specify medications to hold during the index scenario as well as with the nonqualifying “minor” illness if, in fact, it did qualify. Reference to the protocol card was encouraged for decision support. Table 1 shows the significant proportion of medications that were incorrectly identified to be withheld or overlooked for withholding. Although a majority correctly identified the index scenario as qualifying for the protocol, 95% of participants made errors with appropriate medication selection. When asked to identify medications for withholding in the minor gastrointestinal illness, responses showed a slight decrease in errors. Only one participant completed the assigned tasks error free.

Table 1.

Performance with selection of medications to be withheld during a sick day applied to scenario 1 (index scenario) and scenario 3 (minor gastrointestinal illness)

Participants	Scenario 1: “Show Me Which Medications Mr. Smith Should Temporarily Stop Taking”					Scenario 3: “Show Me Which Medications Mrs. Johnson Would Consider Stopping If Her Illness Qualified for the Sick-Day Protocol”
	Medicines Not to Be Stopped		Medicines to Be Stopped			Medicines Not to Be Stopped			Medicines to Be Stopped
	Tylenol with Codeine	Terazosin	Lisinopril	Metformin	Spironoa	Omeprazole	Multivitamin	Calcium	Losartan	Furosemide
1	Error	Error		Error		Error	Error	Error
2	Error	Error		Error		Error			Error
3	Error	Error		Error	Error				Error
4	Error	Error		Error	Error
5	Error	Error		Error	Error				Error
6	Error	Error		Error
7	Error	Error
8	Error	Error							Error	Error
9	Error	Error	Error	Error	Error		Error	Error	Error	Error
10	Error	Error	Error	Error		Error	Error	Error
11	Error	Error	Error	Error		Error		Error	Error	Error
12	Error	Error	Error	Error		Error
13	Error	Error
14	Error	Error		Error
15	Error	Error	Error		Error	Error	Error	Error	Error	Error
16	Error	Error	Error	Error	Error	Error			Error	Error
17
18	Error	Error				Error
19	Error	Error	Error	Error	Error			Error
20	Error	Error
Incorrect actions (errors)/total	12/20	11/20	6/20	10/20	6/20	8/20	4/20	6/20	8/20	5/20

^aSpironolactone.

Usability testing of the Sick-Day Protocol revealed a notable error rate in protocol use and highlighted a striking rate of failure in identifying the correct medications to be withheld with such a protocol. Although the efficacy of the Sick-Day Protocol remains to be determined, its utility expressed on a portable index card for personal use was disappointing and raises safety concerns, including its activation for improperly classified illnesses, such as heart failure, where holding medications might be deleterious (3).

The usability testing has limitations related to the qualitative nature of the study. The small sample size limited hypothesis testing but is consistent with the sample size recommended with usability testing (4). Conducting scenario evaluations directly after Sick-Day Protocol education may have inflated performance rates, because performance could deteriorate when participants are removed from the initial education. The results provide insight into performance with simulated health conditions, but they do not capture participants’ performance with their own medications or social and environmental factors that might influence protocol use.

For effective self-management, patients with CKD need to understand an illness with few overt signs of exacerbation. This study illustrates that this may be challenging for many individuals. Health information technology may enhance medical self-management, but usability testing directed at assessing the human factor needs of the target population is critical for successful implementation of chronic care protocols intended for patients.

Disclosures

Between 2014 and 2017, T.B. was a member of National Health Service England’s Think Kidneys Programme Board. Since 2017, T.B. has been a Royal College of General Practitioners’ Clinical Champion for AKI. R.M.D., C.J.D., L.-A.W., B.M.S., M.V.-C., W.J.F., and J.C.F. have nothing to disclose.