Table 4.

Summary of evaluating explanation effectiveness.

	Participant	Methods	Measurement/Instruments	Limitation/Future work
Alahmadi, A. et al., 2021 [29]	7 cancer patients, 2 female nurses, 1 male physician	Two focus group discussions	Understandability, Usefulness, Reliability : Guidance for Reporting Involvement of Patients and Public (GRIPP) reporting checklists	The focus groups included a limited number of participants mentioning the need for a more diversified clinical population. Evaluating explanation effectiveness is essential to demonstrate the technique's applicability in clinical practice.
Born, J. et al., 2021 [30]	2 physicians	A user study : a questionnaire and comments	Scoring of −3 (the heatmap is only distracting) to 3 (the heatmap is very helpful for diagnosis), The average ratio of correctly explained patterns	Explanation parts of the incorrectly highlighted visible patterns were detected.
Neves, I. et al., 2021 [31]	1 expert cardiologist, 1 graduate medical student, 1 resident	A user study : an online questionnaire in random order of 20 ECGs and free-text comments	Performance accuracy, Task completion time, Usefulness levels: a 5-point scale, Typicalness levels: a 5-point scale	There was a lack of agreement on evaluating the quality of explanations and usefulness of model outputs.
Sabol, P. et al., 2020 [32]	14 pathologists	A clinical trial : a questionnaire	Objectivity, Details, Reliability, Quality : average score	The broad experiment to include other pathologists from varied domains was necessary.
Tan, W. et al., 2021 [33]	2 chief physicians, 3 associate chief physicians, 1 attending physician, 1 resident	An experiment for the diagnostic performance assisted by the LNN model comparing to otosclerosis-LNN, otologists, and XAI-assisted otologists	Average accuracy, Sensitivity, Specificity	During the experiment, otologists often combine clinical diagnoses utilizing diverse patient information, including CT scans, clinical complaints, medical records, and audiological examinations.
Derathé, A. et al., 2021 [34]	6 experienced digestive surgeons	A survey : a questionnaire and comments	Level of agreement with the statement for each surgeon : a 5-level Likert scale	Due to the ambiguity of the survey questions, the respondents provided responses that were inconsistent with the question's intent.