Table 3.

Classification failure mode analysis.

Case		Image	Model output and analysis
Out-of-frame examination
	Error type: false negative Model classification: “no doctor” True classification: “physical examination”		Model output: the patient is sitting on the chair. There is no doctor in the image. Failure reasoning: the camera fails to capture the full examination that takes place around the legs of the patient.
Patient view occlusion
	Error type: false negative Model classification: “consulting” True classification: “physical examination”		Model output: the patient is sitting on the chair. The doctor is consulting the patient. Failure reasoning: the medical student is obstructing the view of the patient, while the doctor is palpating the hands of the patient.
Verbal instruction examinations
	Error type: false negative Model classification: “consulting” True classification: “physical examination”		Model output: the doctor appears to be seated and observing the patient, possibly talking with the patient. Failure reasoning: without the audio, it is very difficult to differentiate a patient walking to the medical chair and a patient being tested on their gait.
Preparation activity
	Error type: false positive Model classification: “physical examination” True classification: “preparation”		Model output: the doctor is standing up next to the patient, appearing to examine or interact with the patient’s shoulders, indicating a physical examination is taking place. Failure reasoning: while the doctor is lowering the chair for the patient, the model believes that the medical student is assessing the shoulders of the patient due to the student’s hand position on the patient.
Bird’s eye view
	Error type: false positive Model classification: “physical examination” True classification: “consulting”		Model output: the doctor is crouched in front of the patient, appearing to examine or interact with the patient’s lower body, indicating a physical examination is taking place. Failure reasoning: due to the camera being positioned higher than the eye level, if the medical student is close to the patient while taking notes with their hands hovering over the patient’s legs while they explain something to the patient, the model thinks that the student is assessing the patient’s legs from their hand position on the patient’s legs.