Table 3.

Examples of post-deployment user feedback and related adapted integrated clinical prediction rule 1 tool modifications per usability theme.

Usability issue/driver	User feedback examples	Modification
Lack of training	Providers that did not receive original academic detailing never tried tool.	Additional academic detailing
Alert fatigue/lack of sensitivity	Lack of specificity in symptom (e.g., cough) yields lack of specificity with regard to firing of tool.	None identified
Workflow barriers	Triggered too early in the process, especially if patient had multiple issues (unaware they could retrieve tool).	Methods to return to tools covered in repeat academic detailing
Added burden	Tool is “one more thing” to do during visits.	Click counts examined, further reductions deemed not possible
Lack of buy-in/tool not useful	Providers familiar with criteria so stopped using/tool did not change how care was provided. After using it providers are comfortable with the criteria so do not need to use it.	None identified