Table 4.

Example feedback from oncologists about the interface, by round and usability heuristic domain.

Usability domain	Rounds 2 and 4 (n = 8 participants)	Round 5 (n = 5 participants)
Match between the system and the real world (Definition: The system should use words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order)
Positive comment	“As oncologists we sometimes have rose-colored glasses on and like to overestimate the benefits of second, third, fourth line treatment. And so, I think that this can maybe ground you and bring you back to reality like, hey, look it’s probably not such a good idea. Let’s think about alternatives.” (#7) “I think this would be useful for family members.” (#3) “…helpful in…patients who…see cancer as like a battle that they have to fight…often unwilling to stop treatment despite all evidence that treatment might harm them rather than help them.” (#7)	“The tool [is] communicating the uncertainty…in a way that I’m able to interpret that and maybe communicate to the patient.” (#9) “It will give me more confidence that there’s a good survival and we can be aggressive.” (#10) “It would [help] me be [more] quantitative about what I should expect for the patient and help the patient understand that.” (#11) “It would be particularly useful…for patients who are wanting to push on despite me feeling that they’ve maybe used up their good options and would be better served with hospice care.” (#11) “Low makes sense…I can understand likely as well. So, low to me, I think it’s perfect for this point, it’s less than 50 percent.” (#9)
Issue	“Often…what I’m trying to convey is your survival chance with and without treatment. There’s no without treatment on this graph.” (#2)	“Tool needs to be used with someone to explain it for patients” (#13) “Often times we are getting new patients…who have been formally treated in community…[When] we think of their first treatment at Huntsman, it’s probably their second or third… The system [should be] able to adjust for that fact.” (#9)
Trust and transparency (Definition: Trust in the system should be supported by transparency and disclosure of relevant information)
Positive comment	“If it was a validated tool…I would have a high trust in it.” (#5) “I wholeheartedly agree with the recommended actions.” (#7)	“They look valid…The error bars and the confidence intervals…makes it clearer that some of the data are fuzzy.” (#10)
Issue	“EHR data isn’t always accurate…the machine learning [algorithm] can be coded inaccurately…a quality control step to make sure that the input into this model is as accurate as possible would be important.” (#5)	“I just need the literature behind how you developed the tool.” (#12) “If they’ve had three or four lines of therapy somewhere else and then they get the fifth line of therapy here even though it’s the first one here they’re going to have a very different prognosis than somebody who has been treated here the whole time.” (#11)
Aesthetic and minimalist design (Definition: Every extra unit of information in a dialogue competes with relevant units of information and diminishes their relative visibility.)
Positive comment	“That [holding hands] icon is nice. …[It] obviously represent[s] …we’re coming to a decision together…That’s a nice way of…displaying that language in graphic form.” (#7)	“I like the tool…It captures key information…This graphic [showing likelihood of surviving beyond six months] helps hit that quickly.” (#9)
Issue	“Alot of people have trouble with graphs…particularly a graph that’s got three lines and any two of which have error bars.” (#2) “I would do it with a single curve that…best fit the patient.” (#1) (NOTE: issue resolved)	“I feel like it doesn’t…necessarily communicate to someone of a low education status.” (#9)
Recognition rather than recall (Definition: Minimize the user’s memory load by making objects, actions, and options visible. Users should not have to remember information from one part of the process to another. Instructions for use of the system should be visible or easily retrievable)
Positive comment	“This figure makes sense.” (#4)	“If this pops up as a recommended action, I can see that as being a useful communication tool.” (#9)
Issue	“Trends are really important…Albumin is probably the most important… I look at weight…knowing what it’s trended…is really important” (#2)	“…the ECOG, the performance status is missing, which is something that you need to decide for further treatments.” (#12) “What are their goals of treatment?” (#12)
Consistency and standards (Definition: Users should not have to wonder whether different words, situations, or actions mean the same thing. Standards and conventions in product design should be followed.
Issue	“When you say ‘Next steps,’ I’m expecting very concrete information. That doesn’t provide any concrete information.” (#5) (NOTE: issue resolved)	“…on the X axis the number of days, instead of six months, so they are different metrics… Someone might have to do mental math to convert 180 days to six months…use months to keep it simple.” (#9)
Help and documentation (Definition: While it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large)
Issue	“I don’t know [what] this Frequently Asked Questions will…show the provider…Maybe you can just put up a note what index date means or you can put something…the provider…can search.” (#4) (NOTE: issue resolved)	“This tool needs to be used with someone to explain it for the patient…especially if…you show a very low survival.” (#13)

Text in [] provides additional information to enhance quote clarity. (#x), interview number. We do not include positive comments about “Consistency and standards” and “Help and documentation” because this category was uncommon for both positive comments and issues, and positive statements are not expected when the system functions as expected.