Table 2.

User feedback and developer response. Example qualitative themes elicited during semi-structured interviews which generated actionable insights for improving Disease Manager.

Construct	Representative quote(s)	Action taken by developers
Trust: Is the user confident in the validity of information and recommendations?	“It said last attempt to quit was January, but then here it says former smoker.”	Active smoking cessation medications added to Smoking Cessation section to emphasize difference between patients who are trying to quit and patients no longer using tobacco.
Trust: Is the user confident in the validity of information and recommendations?	“I suppose he probably needs Prevnar 13, but those guidelines are actively changing, so. And Prevnar is going away maybe.”	Pneumonia vaccine section modified to give user choice in which vaccine to order given that guidelines are under revision.
Action/feedback: How do users identify the results of their actions taken within the interface?	“And then, I didn't see that orange thing that popped up the first time, and then I did see it, so then I realized that I had to do something differently, and that's why I have two orders for albuterol.”	More obvious messaging added to where users naturally look after they select treatments to reflect that orders have been pended to a “shopping cart” queue.
Mental model mapping: How does the user’s conception of the information space match the display?	“I'm trying to decide…what dyspnea means. And I thought maybe it was dyspnea on exam versus dyspnea in the history.” “I would have liked some description of what they meant by certain things, like dyspnea, for example. What I’m used to in other interfaces is you hover over to say, ‘By dyspnea, we mean this.’ Because that’s kind of subjective, so I wasn’t quite sure what study they were referring to or whose definition of dyspnea are you using?”	User input options for “dyspnea status” changed from binary categories to symptom descriptions based on GOLD guidelines in line with providers’ conceptualization of shortness of breath.
Affordance: How intuitive is the display in supporting the user’s information needs, navigation, and desired actions?	“The thing with [the usual EHR] is there’s multiple clicks you have to make in different areas… there's always a struggle to find it… it takes me forever to find it and everything.”	User-identified areas of duplicative clicks addressed to minimize redundant actions.

COPD: chronic obstructive pulmonary disease; EHR: electronic health record; GOLD: Global Initiative for Chronic Obstructive Lung Disease.