Table 2.

Representative comments from physician testers, about each of the CDS interventions, through successive iterations of usability testing.

Post-testing feedback	Round 1	Round 2	Round 3
Pain Assessment Alerts	“Too complex; I assumed it was an error and ignored it.”	“If I click the box “Will order pain meds now”, it would only be helpful if it took me to where I can make the order.”	“Great, straightforward. I would want to do something about a pain score of 10.”
Pain HPI Keycept	“I didn’t fully understand this intervention. “It’s a little much.”	“I actually liked this one. It was a reminder to do something about a pain score of 10.“	Those are really good because they’re in your face, if someone had forgotten to order pain meds, then it would be helpful.
One-Click Pain Score Update Button	N/A (this intervention was added in Round 2)	(Providers offered feedback that the idea of a one-click button for updating the pain score would be helpful)	“Before, you would click on the vitals and there would be 800 different boxes to check so it always takes a long time to find the pain assessment. So I liked that it is very easy to see the button and it takes you only to the pain scale” quickly.
Elderly Analgesia Order Set	“Good. Has all options laid out, including small doses. ‘Easy’ to order them all. Reminds to start low, slow.”	“Best practice pearls in the order set. I think this is a great reminder, especially for more junior people.”	“I like that the order set gives suggestions of pain control in the elderly.”
Discharge Pain Prompt	“A little different – comes in a different place than the BPA. Thought it was an error. Forced me to re-evaluate the pain score.”	“VERY annoying. Because it forces me to document in a specific way. I did document in the text. I felt it didn’t benefit the patient in any way.”	[Repeat-user]: “This obviously has worked for me because I am already programmed to update patients’ pain score before they go home.” [New user]: “I didn’t like it but it was good for me.”