Evaluation of colonoscopy technical skill levels by use of an objective kinematic-based system

Abstract

Background

Colonoscopy requires training and experience to ensure accuracy and safety. Currently, no objective, validated process exists to determine when an endoscopist has attained technical competence. Kinematics data describing movements of laparoscopic instruments have been used in surgical skill assessment to define expert surgical technique. We have developed a novel system to record kinematics data during colonoscopy and quantitatively assess colonoscopist performance.

Objective

To use kinematic analysis of colonoscopy to quantitatively assess endoscopic technical performance.

Design

Prospective cohort study.

Setting

Tertiary-care academic medical center.

Population

This study involved physicians who perform colonoscopy.

Intervention

Application of a kinematics data collection system to colonoscopy evaluation.

Main Outcome Measurements

Kinematics data, validated task load assessment instrument, and technical difficulty visual analog scale.

Results

All 13 participants completed the colonoscopy to the terminal ileum on the standard colon model. Attending physicians reached the terminal ileum quicker than fellows (median time, 150.19 seconds vs 299.86 seconds; p < .01) with reduced path lengths for all 4 sensors, decreased flex (1.75 m vs 3.14 m; P = .03), smaller tip angulation, reduced absolute roll, and lower curvature of the endoscope. With performance of attending physicians serving as the expert reference standard, the mean kinematic score increased by 19.89 for each decrease in postgraduate year (P < .01). Overall, fellows experienced greater mental, physical, and temporal demand than did attending physicians.

Limitation

Small cohort size.

Conclusion

Kinematic data and score calculation appear useful in the evaluation of colonoscopy technical skill levels. The kinematic score appears to consistently vary by year of training. Because this assessment is nonsubjective, it may be an improvement over current methods for determination of competence. Ongoing studies are establishing benchmarks and characteristic profiles of skill groups based on kinematics data.

Colonoscopy is widely used to diagnose and treat a variety of colon diseases from colon cancer to inflammatory bowel disease. It is currently recommended that everyone in the United States undergo at least 1 colonoscopy at age 50, with many patients requiring earlier intervention and repeat procedures.

Colonoscopy is an invasive procedure for which there is currently no reliable or validated process to determine when an endoscopist has attained technical competence.¹ Medical societies have guidelines that define the quantity of colonoscopies that a trainee must perform in order to be considered technically competent; however, the actual range is wide, because some endoscopists may be proficient at 100 procedures, whereas others may require several hundred.^2,3 Additionally, there is no continual assessment or recertification of endoscopists to make certain that their endoscopic skills are maintained at an acceptable standard. This is simply because there is no standard measure available.

A growing body of evidence supports the fact that simulator training is useful in enhancing surgical skill.^4–6 Several simulators that rely on various methods for skill assessment have been developed.^7,8 Aggarwal et al⁹ have demonstrated that significant differences exist between novice and experienced surgeons for procedure time, total path length, and number of movements. Others have shown that novices have the tendency to manipulate a surgical instrument with greater velocity and jerk when compared with the smooth motion of experts.^10,11 This analysis of motion, or kinematics, has been used to define a model of surgical expertise and to establish an objective metric for evaluation of performance in laparoscopic surgery.^11–13

Colonoscopy has long been neglected in the expanding field of kinematic analysis because the methods and metrics derived for laparoscopy could not be directly applied to flexible instrument manipulation. This is largely because of the limitations of prior technologies and software systems. Recently, our group has developed a novel system for acquisition of kinematics data that was initially applied to the assessment of EUS performance and has now been adapted to colonoscopy.¹⁴ The system allows for precise description of colonoscope movement as it traverses the colon. Because this is the first system that allows the collection of kinematics data in colonoscopy, it is currently unknown what parameters will be important in defining a metric and whether kinematic variables can allow for the differentiation between novices and experts.

We aimed to use the novel kinematic system to define the significant variables for performing a mechanically successful colonoscopy and to use kinematic measurement to differentiate between novices and experts in this initial feasibility pilot study.

METHODS

Thirteen physicians, including 4 attending physicians (with >2000 total career colonoscopies) and 9 fellows (3 first-year, 3 second-year, and 3 third-year, each with <500 total career colonoscopies), were randomly selected from the Division of Gastroenterology at our university hospital to perform a single colonoscopy on a standardized, nonhuman, colon model (CM-1; Olympus Inc, Tokyo, Japan), with a variable-stiffness pediatric colonoscope (PCF-Q180AL; Olympus). The CM-1 is fabricated from thin latex that has basic internal and external likeness to the human colon (length, shape, configuration) and is secured to a wood base (Fig. 1). Additionally, CM-1 has elastic properties and mesenteric attachments that allow for “looping” and “loop” reduction. Kinematics data collection started at anus insertion and ended when the terminal ileum was visualized on direct forward view. After the colonoscopy, each endoscopist completed the National Aeronautics and Space Administration (NASA) Task Load Index questionnaire (a reliable and validated instrument that assesses 6 parameters: mental demand, physical demand, temporal demand, performance, effort, and frustration in relation to a task; NASA Task Load Index v1.0, NASA Ames Research Center, Moffett Field, CA, USA) and assigned the colonoscopy a level of difficulty score based on a visual analog scale.

Figure 1.

Standardized latex colon model (CM-1; Olympus Inc, Tokyo, Japan).

The system to collect kinematics data consisted of a flat-plate electromagnetic transmitter that was placed under the colon model and 4 small (1.8 mm), 6 degree-of-freedom, electromagnetic sensors (Ascension Technologies, Milton, Vt) that were attached to the colonoscope along the same x-axis plane at 0 cm (sensor [S] 1), 10 cm (S2), 30 cm (S3), and 55 cm (S4) from the endoscope tip (Fig. 2). A Dell Precision T3400 MiniTower Q6600 with 4 GB RAM, a 2.40-GHz processor, and an nVidia Quadro FX1700 graphics card (Dell Inc, Round Rock, Tex) running MATLAB R2009a Simulink v4.1 (The MathWorks Inc, Natick, Mass) were used to manage and store the kinematics data. Total system cost was less than $20,000 U.S. dollars. Computed parameters included time, path length, flex, velocity, acceleration, jerk, tip angulation, angular velocity, rotation, and curvature (Appendix, available online at www.giejournal.org).

Figure 2.

A, Colonoscope with attached electromagnetic sensors (white arrows). B, Detailed view of colonoscope tip with attached electromagnetic sensor (white dashed arrow). S, sensor.

With performance of the attending physician group used as the criterion standard reference, a distance score was calculated for each fellow by computing the difference (D_ij) between a specific fellow variable (V_ij) and the attending physicians’ mean value for that variable (μ_j): (D_ij = V_ij − μ_j). This difference (D_ij) was then used to compute an individual score (Z_ij) by the following formula: Z_ij = [(D_ij − D_μj) / (σD_j)], where D_μj is the mean difference for all fellows for variable j, and σD_j is the standard deviation of the mean difference for all fellows for variable j. The distance score for fellow i was then calculated as the square of the summation of all variables [distance score_i = ∑(Z_ij)²]. A score of zero indicates perfect performance, whereas scores greater than zero indicate lower performance.

All participants were blinded to their individual kinematics data and distance scores. In addition, participants were unaware of other endoscopists’ data or comparisons being made between and within groups. The colonoscopies were performed in clinical isolation with a system engineer present for system set-up and maintenance. All data were de-identified and securely stored on an encrypted workstation.

Medians and quartiles were calculated for the attending physician and fellow groups. The Wilcoxon test was used for univariate comparison of continuous variables. Linear regression models were used for tests of trend. SAS v 9.2 (SAS Institute Inc, Cary, NC) was used for all statistical analysis, and significance was set at P < .05. Approval was obtained from the hospital institutional review board before initiation of data collection.

Take-home Message.

• Kinematics data and score calculation appear to be useful in the evaluation of colonoscopy technical skill level. Because kinematic assessment is quantitative and nonsubjective, it may be an improvement over current methods of technical skill assessment for determination of competence.

RESULTS

Four attending physicians and 9 fellows were enrolled in the research study. Baseline characteristics are identified in Table 1. All participants completed the colonoscopy on the standard colon model. No participant had used the colon model or the kinematics data collection system prior to this research study. There were no incomplete data sets or missing data. Individual attending physician kinematic parameters are presented in Table 2. Compared to fellows, attending physicians had reduced time to terminal ileum visualization (150.19 seconds vs 299.86 seconds, P < .01), lower path lengths for all 4 sensors, reduced flex (1.75 m vs 3.14 m, P = .03), smaller tip angulation, reduced absolute roll, and lower curvature of the endoscope (Table 3). The mean distance score increased by 19.89 for each decrease in postgraduate year (P < .01) (Table 4). Additionally, for each unit increase in time, the distance score increased by 0.07 (P = .01). There was variance between the individual attending physician kinematic parameters. Overall, fellows experienced greater mental, physical, and temporal demand at completing the colonoscopy than did the attending physicians. The fellows also rated the task as causing increased levels of frustration, requiring more effort, and being of higher technical difficulty when compared with the ratings of attending physicians (Table 5). Individual attending physician task load index and technical difficulty visual analog scale scores are presented in Table 6.

TABLE 1.

Characteristics of endoscopists

	Attending physicians (n=4)	Fellows (n=9)	P value
Male, no. (%)	3 (75)	8 (89)	.62
Total no. colonoscopies performed in career, median (Q25, Q75)	>2000	210 (70, 317)	< .01
FY3 (n = 3), median (Q25, Q75)		329 (317, 338)	.02
FY2 (n = 3), median (Q25, Q75)		212 (201, 300)	.02
FY1 (n = 3), median (Q25, Q75)		63 (57, 70)	.02

FY, Fellowship year; Q25, lower quartile (25% quantile); Q75, upper quartile (75% quantile).

TABLE 2.

Select kinematic variable categories by individual attending physician^*

Category	Attending physician (n = 4)				Variance
	1	2	3	4
Time (s)	149	243	151	82.5	4377.4
Path length (m)
Sensor 1	3.1	5.2	3.0	2.1	1.7
Sensor 2	3.3	4	2.6	2.0	0.7
Sensor 3	4	3.6	3.5	2.0	0.8
Sensor 4	4.1	4.7	3.3	2.0	1.4
Flex (m)	2.7	1.0	2.5	0.7	1.1
Maximum acceleration (mm/s2)
Sensor 3	8.5	10.8	6.6	4.7	6.8
Absolute Tang (degrees)
Angle y	526,872.412	863,139.3261	530,596.2133	286,251.1218	56,173,112,949
Angle z	7108.367565	11,862.14848	4086.38586	2671.90698	16,528,063.5
Absolute roll (degrees)
Sensor 1	1273.23316	2498.726022	1237.619708	1017.250157	450195.9
Sensor 2	1827.170491	1958.002769	1682.274866	1163.587323	121217.7
Sensor 3	1961.509438	1857.437374	1577.160617	1052.985291	165,369.1
Sensor 4	1221.033545	1837.883285	1236.006966	1250.697179	90738.8
Maximum curvature (mm−1)
Section 1	0.28	37.13	2.34	0.51	326.5
Section 2	0.39	0.20	3.20	0.43	2.1
Section 3	0.48	0.62	0.19	0.04	0.1
Maximum jerk (mm/s3)
Sensor 1	441	1179	455	575	122,069.9
Sensor 2	675	1073	686	591	46,529.5
Sensor 3	716	1104	664	407	82,885.2
Sensor 4	95.3	78.5	78.1	78.6	71.7

T_Ang, Tip angulation.

^*Remainder of variable categories available upon request.

TABLE 3.

Select kinematic variable categories among attending physicians and fellows, median (Q25, Q75).^*

Category	Attending physicians (n = 4)	Fellows (n = 9)	P value
Time (s)	150.19 (115.68, 197.48)	299.86 (284.23, 692.33)	< .01
Path length (m)
Sensor 1	3.05 (2.55, 4.13)	6.88 (5.36, 8.86)	.01
Sensor 2	2.93 (2.27, 3.63)	6.53 (4.38, 8.34)	< .01
Sensor 3	3.55 (2.77, 3.77)	8.10 (5.86, 9.38)	< .01
Sensor 4	3.73 (2.68, 4.40)	8.17 (6.39, 9.77)	.03
Flex (m)	1.75 (0.83, 2.61)	3.14 (2.94, 3.36)	.03
Maximum acceleration (mm/s2)
Sensor 3	7.55 (5.63, 9.63)	10.77 (10.23, 11.30)	.04
Absolute Tang (degrees)
Angle y	528,734 (406,562; 696,868)	1,178,470 (1,162,487; 2,717,401)	< .01
Angle z	1,294,429 (873,046; 1,809,529)	2,746,110 (2,513,621; 7,195,719)	< .01
Absolute roll (degrees)
Sensor 1	1255.43 (1127.43, 1885.98)	4876.23 (3383.50, 6128.04)	.01
Sensor 2	1754.72 (1422.93, 1892.59)	2075.14 (1813.31, 2868.29)	.12
Sensor 3	1717.30 (1315.07, 1909.47)	3778.88 (2884.45, 5340.38)	< .01
Sensor 4	1243.35 (1228.52, 1544.29)	2871.76 (1885.32, 3286.02)	.04
Maximum curvature (mm−1)
Section 1	1.42 (0.40, 19.73)	679.86 (16.65, 1303.93)	.06
Section 2	0.41 (0.29, 1.81)	3.98 (3.49, 6.49)	.03
Section 3	0.34 (0.12, 0.55)	2.10 (1.58, 6.02)	.04
Mean curvature (mm−1)
Section 1	0.07 (0.06, 0.50)	1.33 (1.26, 2.11)	.03
Section 2	0.04 (0.02, 0.06)	0.21 (0.09, 0.22)	.02
Section 3	0.01 (0.009, 0.02)	0.14 (0.02, 0.14)	.04

T_Ang, Tip angulation; Q25, lower quartile (25% quantile); Q75, upper quartile (75% quantile).

^*Remainder of variable categories available upon request.

TABLE 4.

Distance score among fellows^*

Fellow (n=9)	No. colonoscopies performed in career	Individual distance score	Distance score, mean (±SD)
Third-year			46.88 (±9.23)
1	329	37.10
2	338	55.44
3	317	48.10
Second-year			51.48 (±11.4)
1	201	38.72
2	212	60.61
3	300	55.10
First-year			61.65 (±24.2)
1	57	63.32
2	70	36.66
3	63	84.96

^*β₀ = 19.89, P < .01.

TABLE 5.

NASA Task Load Index and technical difficulty VAS scores^*

Category	Attending physicians (n = 4)	Fellows (n = 9)	P value
Mental	12.5 (7.5, 17.5)	35 (25, 50)	< .01
Physical	12.5 (7.5, 15)	50 (30, 65)	.02
Temporal	10 (10, 12.5)	45 (15, 50)	.04
Performance	10 (5, 30)	45 (20, 50)	.08
Effort	10 (5, 27.5)	60 (55, 70)	< .01
Frustration	7.5 (3, 12.5)	50 (50, 65)	.02
Technical difficulty	6.5 (3.5, 11)	50 (34, 60)	< .01

NASA, National Aeronautics and Space Administration; VAS, visual analog scale.

^*Median (Q25, Q75), Q25, lower quartile (25% quantile); Q75, upper quartile (75% quantile).

TABLE 6.

NASA Task Load Index and technical difficulty VAS scores by individual attending physician

Category	Attending physician (n = 4)				Variance
	1	2	3	4
Mental	20	5	15	10	42
Physical	15	5	15	10	23
Temporal	10	10	15	10	6
Performance	45	15	5	5	358
Effort	40	5	15	5	273
Frustration	15	5	10	1	37
Technical difficulty	15	7	6	1	34

NASA, National Aeronautics and Space Administration; VAS, visual analog scale.

DISCUSSION

In the current study, kinematics data were successfully recorded from the position sensors on the flexible endoscope for each endoscopist. A single colonoscopy was chosen in order to minimize any learning curve or practice bias that would occur if multiple colonoscopies were performed by each endoscopist on the model. There was a significant difference between attending physicians and fellows for key kinematic parameters that correspond to common clinical scenarios. Clinically, attending physicians typically have reduced insertion time for reaching the terminal ileum when compared with fellows. That same finding was upheld in our study population. In addition, fellows had more curvature and flex when compared with the attending physician group. Curvature and flex may, in theory, be indicators of looping and therefore may translate into measures of patient comfort.

When kinematics data were used to define a model of expertise based on technical skill level, attending physicians were chosen as the expert group, and their score was standardized to zero. This score served as the comparison mark for assessment of fellow technical skill level. A linear relationship existed for the overall kinematic score whereby as fellows’ experience increased, their scores decreased (ie, came closer to zero). This indicates that as the experience level of endoscopists improves (first-year, second-year, third-year), their overall kinematic scores approach that of the defined expert group (attending physicians). The variance among attending physicians for kinematic parameters indicates that kinematics data collection is likely a sensitive measure of skill and not simply a measure of year of training. Hence, kinematics may provide a quantitative method for the assessment of endoscopist technical skill level, independent of the quantity of colonoscopies an endoscopist has performed.

Additionally, fellows found the colonoscopy to be more difficult and frustrating, requiring greater effort and mental demand when compared with attending physicians. This corresponds with the differences in overall kinematic scores and with what is commonly seen in clinical practice. As endoscopist technical skill level improves, the level of task demand, frustration, and perception of task technical difficulty decreases.

The kinematics data collection system did not encumber the endoscopists and was simple to assemble immediately before the procedure. The CM-1 model provided a physical simulation for the endoscopists and was used to keep colon conditions constant among all users to avoid human factor variables such as anatomy, bowel preparation, manual external pressure, patient age/colon health, and sedation. Therefore, each endoscopist was able to perform a colonoscopy on the exact same colon under the exact same operating conditions, thereby reducing potential confounding variables or bias. This may have resulted in reduced times to the terminal ileum because of optimal colon conditions when compared with those of human patients who have real-life variables to account for and at times suboptimal endoscopy factors.

Potential limitations of this study include selection of the expert group, because there may be other colonoscopists who have improved technical skill and therefore may change the expert reference model. If the reference model were to change, we anticipate that the same relationship will hold for overall kinematic scores among trainees. Additionally, because this study was conducted on a colon model, direct clinical correlation could not be proven. With the research study examining kinematic variables that mark endoscope mechanical manipulation ability, it may not be necessary for the model itself to reflect a true human colon. Because detectable kinematic differences and scores can be calculated in an objective manner between groups, it likely does not matter what model one chooses—the colon model is, for all intents and purposes, simply a basic “maze.” The study reflects that there are differing technical skills with endoscope mechanical manipulation that can be measured and that with improved mechanical ability, one has lower scores/greater technical skill to navigate the maze. Validation studies are underway in human patients to determine whether our kinematic parameter assessment of endoscopist technical skill level is upheld in clinical practice.

In summary, kinematic data collection and analysis appear to be useful in the objective evaluation of colonoscopy technical skill levels. Because this assessment is quantitative and nonsubjective, it may be an improvement over the current methods of skill assessment for determination of competence. Kinematic parameter assessment has the potential to serve as a useful tool in training programs and may be useful for endoscopist recertification—something that is currently lacking in endoscopy. Ongoing studies are focused on establishing benchmarks and characteristic profiles of skill groups based on kinematics data. These profiles may then be used as reference points for the determination of competence based on comparative kinematic gestures between endoscopists.

Abbreviation

S

sensor

i

fellow number

j

variable number

D

difference

V

variable

μ

mean

Z

individual score

σ

standard deviation

APPENDIX

Definition of measured kinematic parameters

Time = Time measured from S (sensor) 1 in seconds (s)

Path length (PL) = Total distance that each individual sensor travels in the x, y, z coordinate system measured in meters (m)

Flex (F) = Calculated as the difference in path length between S4 and S2 measured in meters; PL_S4 − PL_S2

Velocity (v) = The change in distance divided by the change in time for each individual sensor as it travels in the x, y, z coordinate system measured in millimeters per second (mm/s)

Acceleration (a) = The second derivative of the change in distance divided by the change in time for each individual sensor as it travels in the x, y, z coordinate system measured in millimeters per second squared (mm/s²)

Jerk (j) = The third derivative of the change in distance divided by the change in time for each individual sensor as it travels in the x, y, z coordinate system measured in millimeters per second cubed (mm/s³)

Tip angulation (T_Ang) = The difference in degrees between the y and z coordinate angle for S1 fixed on S2

Angular velocity (v_Ang) = The change in tip angulation divided by the change in time for S1 fixed on S2 measured in radians per second (radian/s); 1 radian/s ~ 57.29578 degrees/s

Rotation (Rol) = Total operational turn for each individual sensor around the x-axis measured in degrees

Curvature (Curv) = The geometric shape of the endoscope in the x, y, z coordinate system measured as the magnitude of the derivative of the tangent vector (to the endoscope shape) with respect to path length based on endoscope section (mm⁻¹); where section 1 is between S1 and S2, section 2 is between S2 and S3, and section 3 is between S3 and S4