Surgical Instant Replay: A National Video-Based Performance Assessment Toolbox

What is the innovation?

Taking inspiration from sports broadcasting, the application of a surgical ‘instant replay’ is an innovative approach for advancing post-operative performance review and sharing of best practices. Recent advances in audiovisual technology for professional-grade capture, editing, storage, annotation and dissemination of operative video has opened up new opportunities for surgeons and surgical educators to enhance technical and non-technical (e.g. situation awareness, decision-making, leadership) performance. In one application, a video camera aimed at the operating table and lavalier microphone under the operating surgeon’s gown were linked to a datastore in the surgical hallway, and the resulting video segments presented to the team post-operatively. When reviewing exposure, technique, and errors from film, the participating teams were highly critical of their own performance, concluding that “Irrelevant motions by surgeon and assistant within the operative field delay the completion of the operation, may injure the tissues, occur frequently, and are usually unnoticed at the time of the operation” (p144)(1). This futuristic vision of surgical performance enhancement was actually conducted in the late 1960s by surgeons from Temple University (Philadelphia, Pennsylvania). Despite the ubiquity of video for minimally invasive and robotic surgery, sharing of recorded operations at surgical professional conferences, and demonstrated feasibility of surgical video for assessment(2), there remains substantial variability in the use of operative video for review and improvement across surgical specialties(3). Opportunities for widespread implementation of video to improve technical and non-technical performance should be prioritized to enhance patient safety.

What are the key advantages over existing approaches?

Public reporting of institutional (and in some instances individual surgeon) performance is here to stay, so the importance of scientific strategies to assess performance cannot be overemphasized. Surgical video can increase the reliability and scalability of performance assessment, improve team engagement, and reduce the limitations and biases of human memory in assessment. We can learn from other industries in their approach; professionals in civil aviation, space exploration, railway, maritime, offshore oil and gas production are routinely trained and evaluated on their technical and non-technical skills, in order to demonstrate competency relative to regulated standards. For example, the Federal Aviation Authority (FAA) and Civil Aviation Authority (CAA) require regular non-technical skills evaluations of pilot behavior in the cockpit by expert peer pilot assessors, using validated behavior observation tools (e.g., NOTECHS(4)). These assessments aim to: (a) ensure that pilots are competent to fly; (b) improve performance in non-normal circumstances; and (c) test pilots’ ability to prevent high risk environmental threats leading to poor outcomes. Other industries, especially sports, use in-game video analysis (specific to individual players, the team, coaches and medical staff) for performance review, coaching, and progression. Analogous to what could be possible in surgery; the US Tennis Association has innovated artificial intelligence systems for video annotation of training and tournament performance, with the specific aims of developing data-driven tactical analysis to support success during identified ‘pressure points’(5). However, there are limited applications for data-driven “coaching to mastery” within medicine. While the normalization of surgeon performance assessment has not yet occurred, the required hardware and systems are now at an advanced level of technological readiness (see table).

Table:

Blueprint for a National Video-Based Surgical Assessment Toolbox, with technology readiness level assessed via NASA criteria. This could be used to select components for a specific purpose, identify innovation needs in specific areas, and track progress towards the goal of widespread video assessment in surgery to enhance performance.

Toolbox category	Video assessment component	Technology innovation status	NASA Technology Readiness Level (1–9)
Hardware and Technology	Video capture - operative field view	Routinely captured for open and minimally invasive (robotic, laparoscopic) procedures	TRL9
	Secure video hosting	Several commercial and academic HIPAA-compliant solutions exist for secure viewing and assessment of surgical video	TRL9
	Video capture - operative team view	Available in commercial applications, and less routinely captured than operative field view. Mainly quality improvement/ research	TRL7
	Audio capture	Gathered via individual or shotgun mics; usually to support non-technical skills assessment alongside team video	TRL7
Assessment process and type	Automated video segmentation	Feasibility demonstrated (e.g. gastric surgery)	TRL6–7
	Assessments - crowdsource	Feasibility of public vs surgeons for technical skills demonstrated	TRL6–7
	Competence standards	‘Cut scores’ for human assessment of technical and non-technical skill proficiency demonstrated in general surgery	TRL6–7
	Cognitive load monitoring	Feasibility of sensors to measure LF/HF ratio in cardiac surgery teams demonstrated	TRL4–7
	Assessments - peer	Feasibility demonstrated in multiple studies	TRL4–5
	Artificial intelligence assessment - surgeon’s technical skill	Several groups developing automated identification of instruments, hands, progression, bleeding, team motion	TRL2–4
	Artificial intelligence assessment - surgeon/ team non-technical skills	Emerging evidence on communication skills and cognitive load. Proof of concept for team motion and surgical sabermetrics	TRL1–3
Certification process	Surgeon certification process	Currently solely incorporates a fund of knowledge and judgment tests. Feasibility of video as adjunct to be established	TRL1–2

Technology Readiness Levels (TRL): TRL1: Basic principles observed; TRL2: Conceptual; TRL3: Proof of concept; TRL4: Laboratory studies; TRL5: Validity testing; TRL6: Prototype demonstration; TRL7: System demonstration in Operating Room (OR); TRL8: System complete and working in OR, TRL9: Successfully implemented and proven. (Criteria adapted from National Aeronautics and Space Administration (NASA): https://www.nasa.gov/directorates/heo/scan/engineering/technology/technology_readiness_level)

How will this affect clinical care?

Widespread surgical video will normalize performance review, facilitate self-assessment and sharing of best practices, and enhance patient safety. The primary aim of the 10 U.S. surgical certifying Boards is to advance the quality and safety of care by establishing knowledge, education, and training standards. There is now sufficient evidence for extending surgical certification standards to include operative technical and non-technical skills performance review in addition to the existing rigorous examination process of clinical knowledge and surgical judgment. Surgical video could be an invaluable tool for formative assessment and the curation of an evidence basis demonstrating competence in support of initial and maintenance of certification processes and decisions.

Is there evidence supporting the benefits of the innovation?

There is substantial evidence of surgical video applications for research (e.g. evaluation of training, technical and non-technical skills(6), identifying risk, associating measured skills with patient outcomes(7)), and formative assessment (e.g. coaching, mentoring, simulation debriefing, resident feedback). Beyond “watching the game in the stands”, it is imperative to partner with engineers and human factors scientists to advance the field by building tools to improve surgical video assessment and early warning indicators of performance variability.

What are the barriers to implementing this innovation more broadly?

We share a vision for surgical video to be embedded in every procedure to support safe surgery while maintaining patient confidentiality. Some surgeons may be resistant to change, having spent their professional life working towards and exceeding the set standards without video, while others are interested in optimizing self-assessment. Healthcare systems in the U.S. are unique among other industries exposed to major hazards, and surgeons represent one of the few professional groups where civil and criminal liability for poor performance is a reality. This cultural barrier to disclosure must be addressed by positioning video as a surgeon-led innovation to share best practices, learn from adversity, demonstrate competence, and most importantly protect patients. If video has a role in enhancing surgeon certification, it is important that the process does not erode the psychologically safe space for non-punitive formative assessment and learning from error.

In what time frame will this innovation likely be applied routinely?

It is anticipated that surgical video will be widely used for formative assessment of technical and non-technical skills within five years. Summative assessment may follow soon thereafter as surgeons and regulatory representatives (e.g., risk management, legal counsel) recognize that the benefits of video assessment outweigh its potential risks. The routine application of embedded surgical video in every procedure unlocks exciting future directions for safe and highly reliable surgery, including real-time clinical guidance, artificial intelligence-based video coaching, and baseball-style sabermetrics to support individual and team performance. By implementing the ‘surgical instant replay’ for education and assessment at scale, we have the opportunity to protect patients by continually raising performance standards in surgery.