Table 4.

Quoted feedback of participants on various usability domains and overall qualitative interpretation.

Themes	Sub-Themes	Feedback Quote	Overall Assessment
Ease of understanding	Web application presentation neatness, and simplicity	“Seems pretty clear. The left is giving the predicted grade based on the Gleason score. The right shows the ‘work’ behind the scenes to demonstrate how it go to that score”—User 4   “One the left side, have you ever considered using a vertical bar to represent different percentage of tissue, including benign one and cancers of different ISUP grades? My rationale is that the pathologists derive these mental percentage from a linear biopsy, your linear representation may make the mental transition more seamless and less frictional.”—User 3	Users generally found the web application’s presentation neat and clear. They appreciated the left side’s predicted grade based and the right side’s representation of the model’s decision-making process (refer Figure 2). Some users suggested using different charts to represent different percentages of tissue, including benign and cancerous tissue of various ISUP grades. This could potentially improve the mental transition when interpreting the results. There was a recommendation to include specific requirements for the images to be uploaded, this aimed to ensure data quality and consistency. In summary, the feedback highlighted the importance of clear visualization, potential enhancements to the presentation format, and the need for specific image upload requirements to improve the overall usability and effectiveness of the web application.
	AI result visualization charts and their self-explainable quality	“Should there be specifications for the image to be uploaded such as the magnification used for the slide scan, the model of the scanner, that it must be an H&E slide, that the slide must be deidentified and contain a date of the scan, etc.”—User 1
Ease of usage	Clarity with instructions and navigation	“The data is easy to interpret on the right. I can see the breakdown of how the AI predicted the various scores and percent probability. There is a disconnect on the left because if it is using the data on the right for prediction, then it should simply state that the predicted score is 4 + 3 = 7 with Grade group of 3 and not 3.8”—User 4   “Easy to Navigate”—User 5	Users found the application relatively easy to use and navigate. To use this feedback to improve our application, we can ensure that the application maintains consistency in presenting data and that users can easily understand how predictions are made. We will also continue to gather user feedback and conduct usability testing to identify any other areas where improvements can be made. This will help in refining the user experience and making the application even more user-friendly. By implementing these actions, we can enhance the clarity and usability of our application, making it more effective and user-friendly for our target audience.
Accessibility	Positive feedback upon completion of web session	“It’s good. A little more detail on the Gleason score graphic would be useful. Caters to a wide healthcare audience.”—User 4   “Make the right column info optional: the user has to click a button for it to appear. I am not exactly certain on the details of the information in that column. If you explain to me further, I might have some ideas as how to make it user friendly.”—User 3	User feedback suggests that our application has been generally well received by healthcare professionals, with positive remarks about its quality and suitability for a broad audience. However, users have expressed a desire for more detailed information in the Gleason score graphic, indicating room for improvement in enhancing the clarity of predictions. To address this, we are considering making the right column information optional, requiring user interaction to access it unless we can transmit our intended results better to a non-technical audience. By noting down additional information aspects valuable for diagnosis, we will try our best to include suggested information by making changes to tabs as well as removing non-useful information based on received feedback. These insights from users will guide our ongoing efforts to refine and optimize our application, ultimately delivering a more robust and user-friendly tool for healthcare practitioners.
	Difficulties faced during interaction	“Information on family history of prostate cancer not present, useful for diagnosis”—User 2   “Not very comprehensive (don’t know about prior biopsies or PSA level)”—User 5
Tab-specific value proposition	Biopsy Tab	“Would be better to see the biopsy in a separate image viewer or management system.”—User 4   “Looking at the biopsy is very helpful”—User 5	Key takeaways from user feedback indicate a preference for a dedicated image viewer or management system for biopsy viewing with higher image resolutions. Users find the biopsy examination to be particularly valuable. On the Demographics Tab, there is a consensus that more comprehensive demographic data, apart from age, should be included to assess potential risk effectively. However, users emphasize excluding unrelated risk factors. Regarding the NLP Summary Tab, it is suggested that its current content does not significantly contribute to prostate diagnosis and grading, indicating room for improvement in its relevance. Lastly, users express a strong preference for viewing all information together in the Detailed Summary Tab, emphasizing the need for a streamlined and efficient presentation. To enhance our application, we plan to explore dedicated biopsy viewing options, incorporate more pertinent demographic data while excluding irrelevant factors, refine/disregard the NLP Summary Tab for better relevance, and optimize the presentation of data in the Detailed Summary Tab.
	Demographics Tab	“Insufficient demographics shown to assess potential risk”—User 1   “Other than age, other risk factors should not be here.”—User 3
	NLP Summary Tab	“Not very helpful for prostate diagnosis/grading”—User 5   “OK to know but does not really lend benefit for the current study”—User 3
	Detailed Summary Tab	“Better to see all information together (Faster)”—User 5   “Yes, I’d prefer to see all this information together Yes, it doesn’t seem overwhelming.”—User 2
Personalized suggestions regarding interface	Optional suggestions provided by each user	“Helpful tool”—User 1   “No strong suggestions for improvement at the moment”—User 2   “It provides too much information. There are only 3 things crucial to the pathologists: whether there is cancer, if yes where are they on the biopsy, and what are the grades.”—User 3   “Minor point but the light blue background does not seem to fit. It takes attention away from the visuals and should either be completely light or darkened without the background images.”—User 4   “Nice design/visually appealing”—User 5	Feedback on the personalized suggestions for the interface is generally positive from each user, with users finding the tool helpful and visually appealing. However, there is a suggestion to streamline the information presented, focusing on key elements crucial to pathologists. Additionally, a minor design adjustment is recommended to improve the background’s visual coherence. To enhance our application based on this feedback, we will work on optimizing the information presented for relevance and clarity, while also addressing the design element to create a more visually cohesive experience for users.