Table 2.

Categories of usage problems identified, illustrative instances and references to the papers they are retrieved from. Users’ comments are in italic font.

Usage problems	Illustrative instances from the 111 items	References
Behavioral issues
Increased workload due to the alerting function	“If I have to consider every DDI, then I am busy with it, all day, and that is not my job.” [32]	[22; 24; 27; 29; 31; 32; 37; 38; 40; 41]
Users do not use the system at all	“Two subjects did not use the decision support feature” [25]	[25; 28; 29]
Users voluntarily ignore the alerts	“Five nurses and two providers were observed to skip all or some of the reminders” [37]	[10; 22; 26; 28; 29; 32; 34-37; 40; 41]
Users use the system ineffectively	“The physician reported that specific features of the system (…) were hindering the use” [41]	[24; 29; 37; 41]
Users use workarounds	“Provider arbitrarily selected a date to satisfy the reminder” [37]	[25; 37; 38]
Users blindly follow the advice even if they do not understand it	“MD clicks through [the alert]” [accepts the advice without understanding the alert] [26]	[26]
Users are lost/stuck: they do not know how to go on	“Physicians were lost” [27]	[27; 37]
Cognitive issues
Information involuntarily missed: they cannot access or find it	“Not having noticed the DDI alert that appeared as a second DDI alert” [32]	[10; 24; 29; 30; 32; 35]
Increased memory load while using the alerting system: users must rely on their memory	“Some prescribers relied solely on their memory of the patient profile” [29]	[29]
Users experience difficulties in understanding the alert	“Had difficulty identifying the patient’s risk factors for the interaction” [30]	[26; 27; 29; 30; 33; 41]
Users experience difficulties in identifying alert’s components (including icons, features or specific data)	“They misidentified the alert as a general guideline reminder and did not notice the dose calculations embedded in text.” [25]	[25; 36; 37]
Users misinterpret alerts’ components (including icons, features or specific data)	“A user thought that the appearance of the ‘stamp’ window implied that the patient had a chronic pain problem or diagnosis” [36]	[36;39]
Users misinterpret alerts’ content	“Misinterpretation was rife, as shown by the high numbers of wrong or inapplicable rules and reasoning.” [32]	[23;32]
Users are interrupted by alerts while making their decision or interviewing the patient	“There were several cases where inadequate alert design (…) disrupted their workflow.” [22]	[22;24;29;34]
Emotional issues
Annoyance/irritation	“Repetitive alerts are both annoying and unnecessary.” [29]	[24; 26; 28; 29; 32; 41]
Frustration	“Physicians became frustrated” [27]	[22; 24; 27; 28]
Ugly experience	“Reading them is ugly” [22]	[22]
Stress, pressure	“Place prescribers under pressure” [29]	[29]
Cynicism	“This lack of information led to prescriber cynicism.” [22]	[22]
Attitudinal issues
Users question the behavior of the system: how the system is working, how it responds to users’ actions	“Did it accept my changes?” [26]	[24-26; 29]
Users question the triggering and sorting model of alerts	“I am not confident it’s checking all the interactions that I want it to check.” [29]	[22; 26; 29; 38]
Users question the usefulness of the alerting system	“The alerts were most likely to be helpful if they [were] presented when the users were entering orders or were otherwise at the point of making a decision about the issue in question or closely related issues.” [24]	[24; 33; 36]
Users question the validity of alerts	“That’s not true to my knowledge. The patient doesn’t like to take it; I doubt he’s taking it [from a non-VA source]. I will talk to the patient about it.” [31]	[22; 29; 31; 33; 34]
Users experience alert fatigue/desensitization	“Some doctors recognized that they had become desensitized to the alerts.”[40]	[22; 29; 31-34; 37; 40]
Users have negative feelings towards the system	“Justification requirement often viewed as time burden” [29]	[24; 29; 35; 36]