Abstract
Background
An increasing quantity of research in the domain of telemedicine show a growing popularity and acceptance of care over distance systems among both clinicians and patients. We focus on telementoring solutions, developed for providing remote guidance to less experienced surgeons. Telestration is often regarded as an extra functionality of some telementoring systems. However, we advocate that telestration must be viewed as a core feature of telementoring due to its advantages.
Objective
To analyze and define concepts, parameters, and measurement procedures to evaluate the impact of using telestration while telementoring.
Methods
A systematic review of research dealing with telestration during remote guidance sessions was performed by querying three major online research databases (MEDLINE, Association of Computing Machinery, and Institute of Electrical and Electronics Engineers) using a predefined set of keywords (“laparoscopy”, “annotate”, “telestrate”, “telestration”, “annotation”, “minimally invasive”, and “MIS”).
Results
The keyword-based search identified 117 papers. Following the guidelines for performing a systematic review, only 8 publications were considered relevant for the final study. Moreover, a gap in research defining the impacts of telestration during telementoring was identified. To fill this niche, a framework for analyzing, reporting, and measuring the impacts of telestration was proposed.
Conclusions
The presented framework lays the basics for the structured analysis and reporting of telestration applied to telementoring systems. It is the first step toward building an evidence knowledge base documenting the advantages of live video content annotation and supporting the presented connections between the concepts.
Keywords: telementoring, telestration, annotation, impact, benefits, theoretical models, evaluation framework
Introduction
The shift from open to laparoscopic surgery presents a fertile ground for an expansion of telemedicine (a set of medical practices without direct physician-patient interaction that are often realized via interactive audio-video communication [1]). Advances in information and communication technologies have resulted in the development of relatively low cost and high reliability devices as a medium for telemedical solutions. The demand for telementoring systems in laparoscopic surgery, comprising the real-time interactive teaching of techniques by an expert surgeon to a student at a distant site [1], in laparoscopic surgery was partially motivated by the decreasing ratio of general practice surgeons to population, a problem that is predicted to increase in the future [2]. Studies have shown the benefits of using telementoring, especially in the field of laparoscopic surgery [3,4]. Time and cost reductions as well as a more efficient surgical education are only a few of the many, mostly evident advantages of this developing technology [3].
Telementoring has been discussed extensively in the literature, with multiple solutions demonstrated for laparoscopic surgery [5,6]. Its place in surgery has been described by Doarn [7] as “a natural fit”. Furthermore, videoconferencing has gained increasing popularity in all fields of medicine, especially in follow-up and out-patient treatment [8-10].
In spite of the recent advances in mobile devices, the current body of literature tends to limit the telementoring approach to stationary platforms. However, the Mobile Medical Mentor project team advocates using mobile devices for telementoring due to additional benefits compared to the stationary ones [11]. The fact that mentors rarely stay at one location and are highly mobile should be sufficient impetus for further exploration of mobile platforms for telementoring. Having a telementoring device within reach should increase the availability of experts. Therefore, the main challenge is to ensure that the required telementoring infrastructure is available on a mobile medium [11].
Telestration is defined as a technique for enabling the drawing of freehand markups (annotations) over image or video (Figure 1). Although it is mostly used in commenting on sports and weather forecasts, it often attracts the attention of medical personnel [12,13]. In the domain of telemedicine, telestration is normally considered to be an additional accessory function of telementoring systems. As the overall impacts and benefits of telestration are not clear, this paper aimed to analyze the available studies reporting the use of telestration to form a systematic assessment of the reported outcomes of using telestration in minimally invasive surgery (MIS).
Figure 1.
Surgical telestration.
The paper is structured as follows: after a short introduction to the developing domain of telemedicine, the motivation for using telementoring systems is presented. The telestration feature is analyzed in greater detail.
The Methods Section defines the procedures that were followed to summarize the current body of knowledge in using telestration and guidelines to develop the evaluation framework. The Results Section represents the achievements of a systematic review procedure and highlights the identified gap in the available research. The following section discusses the identified niche in research, introducing an evaluation framework for analyzing, measuring, and reporting the impacts of using telestration during telementoring sessions. Evaluation biases and confounders were analyzed in the Discussion Section.
Methods
Overview
This section is divided into two sub-sections, the first of which describes the search process for relevant research. This search was performed to summarize the available body of knowledge in evaluating the impacts of telestration applied to telementoring systems. The second part studies the development of an evaluation framework in more detail by introducing the methodology for developing theoretical models.
Systematic Literature Review
To form a solid basis for this research, a systematic literature review of telementoring applications that include a telestration feature in MIS was performed. The review was carried out following the guidelines defined by Kitchenham [14]. The aim of reviewing the available body of literature in the field was motivated by the need for a summarized and structured evidence knowledge base. Moreover, we intended to identify the gaps in current research and develop guidelines for further investigation of the domain [14,15].
The search for related studies was limited by publication date (1992-2012) due to technological progress. Search results (published before 1992) were considered too old to be relevant. Three major online research databases (MEDLINE, Association of Computing Machinery [ACM], and Institute of Electrical and Electronics Engineers [IEEE]) were selected as data sources for the review. The search was performed on the 13th of November, 2012. The selected databases were queried using the same set of keywords, containing combinations of the following terms: “laparoscopy”, “annotate”, “telestrate”, “telestration”, “annotation”, “minimally invasive”, and “MIS”. Keywords were selected based on earlier research in the domain [3,11,12], in accordance with the corporate decision of the co-authors. The results of the search were analyzed by one reviewer to determine the eligibility of the studies. We admit the possible weakness of including only one reviewer in the study selection process; however, due to the narrow focus and straightforward evaluation of the papers, this bias should be minimal.
Developing the Evaluation Framework
The methodology presented by Sjøberg et al [16] was employed to develop the theoretical fundamentals for evaluating the impacts of the technology. The initial focus was to define the basic elements of the theory-constructs. We aimed to use concrete and easily measurable concepts instead of abstractions. This approach makes the framework less universal; however, our purpose was to develop a specialized evaluation methodology for a narrow field telestration system. Relationships between the concepts were defined to highlight the existing dependencies. The framework was supplemented by explanations and a description of the scope. Constructs, relationships, and explanations are the main components for developing theoretical models [16].
Results
Systematic Literature Review
The keyword-based search resulted in 117 papers in total, as shown in Figure 2. We aimed to identify and analyze papers reporting the use of telestration feature. The main focus was on the analyses of the impacts of telestration to the overall procedure toward research-based proof of the benefits for telementoring. Nine instances were discarded as duplicates. Title analysis resulted in 88 exclusions due to a focus on other disciplines. An additional 8 records were discarded after analyzing the abstracts of the papers due to a lack of relevance to the current research topic. Only 12 (10%) studies were considered eligible for the final stage-full text analysis, which revealed that only 8 papers were relevant for the current study: [13,17-23] (4 papers were excluded due to lack of focus on telestration).
Figure 2.
Systematic review flow diagram.
The review was performed to assess the impacts and benefits of telestration as applied to telementoring systems and then analyze these data in a systematic manner. The study revealed a gap of knowledge in assessing the outcomes of telestration. All of the selected studies reported the use of telestration; however, no analysis or assessment of its impacts was identified. The only outcome was a significant improvement of the mentoring session due to the ability to annotate graphic content, reported in 3 papers [18,20,21]. No support for the claims was provided, making them sound subjective. The remainder of the papers declared the feature to be an integral part of the telementoring system; although, no additional details were made available.
The purpose of the review was to assess the potential benefits of telestration in telementoring based on reported experiences. However, we were unable to achieve our goal due to the lack of existing research. The shortage of publications motivated the need to propose an evaluation framework for analyzing the outcomes of telestration in the scope of telementoring.
Definition of Concepts and Theoretical Models
An extra layer of interaction and complexity is added by providing the ability to annotate the graphic content used in a telementoring session. On the one hand, telestration simplifies pointing actions, eliminating the need for discussion to define the exact location in the visual representation; while on the other hand, it introduces extra complexity into the workflow model of the procedure. This complexity includes controlling of the telementoring system and annotations as well as extra hardware in the operating room. In this Section, we aim to define the potential benefits of telestration and discuss the impact of telestration on the outcome of the overall procedure.
To define the influence of telestration while telementoring, the outcomes were divided into Educational and Clinical Impact groups, highlighting the direct influence of the education process on potentially improved patient outcome (see Figure 3). To lay the basis for determining the impact of this feature, we focus on the case of laparoscopic surgery as an initial point for a video conferencing-based telementoring and surgical education. The laparoscopic procedure was selected because of its high visual component and the fact that the procedure is already transmitted to a monitor, which allows the easy adoption of the mentoring approach. As the local surgeon, hereafter referred to as mentee, is observing the progress of the procedure on the monitor, sharing this representation with a remote mentor is highly feasible. However, representation sharing and verbal interaction between endpoints (a common telementoring approach) is not sufficient in some cases. From an educational perspective (Figure 4) telestration is a feature that adds interactivity to the learning process, which should result in increasing quality of education through more comprehensive instructional material. An increased quality of education (defined quantitatively as increasing scores in the rating scale) should be enough impetus to suggest the possibility of a shorter duration of surgical training. This change could potentially lead to a rising number of highly experienced medical personnel and lower education costs. However, a lack of research leaves the assumptions poorly supported until the validation of the framework on real-life cases.
Figure 3.
Impacts of telestration.
Figure 4.
Educational outcomes of telestration.
From a clinical perspective, the overall goal is the improved patient outcome, defined quantitatively as the ratio and severity level of complications, as shown in Figure 5. We assume that the use of the telestration feature in mentoring should result in increased accuracy of the surgical decisions, relation A in Figure 5, and overall mentoring process as the exact location of interest can be defined visually instead of more ambiguous verbal description. Secondly, the duration required to indicate the exact location in the operative field should decrease, resulting in a decreased duration of the overall procedure, relation C in Figure 5. We assume that following more accurate directions of the mentor should also decrease the duration of the procedure, relation D in Figure 5, help avoid clinical errors and shorten the recovery period, contributing to the overall goal–improved patient outcome, relation E in Figure 5. Moreover, shorter recovery and hospitalization is directly related to lower costs of treatment. The concept of “surgical education” represents the connection of educational and clinical outcomes, depicted in Figure 3, while relation B in Figure 5 summarizes the impacts of telestration with respect to improving surgical education as discussed above. F is one of the fundamental relations in the model dealing with the direct impact of improved education on the final well-being of the patient.
Figure 5.
Clinical outcomes of telestration.
Another important aspect that should not be omitted is time consumption. From the perspective of the mentor, time savings contribute to a reduction of the workload as well as increased availability of the expert. The mentee should also experience decreased time consumption, resulting in shorter duration of the overall procedure. These issues pose the following question: given that the resources available in the hospital are constant, is it possible to treat more patients during the same period of time by decreasing the duration of the procedure? Moreover, the costs of the treatment should also diminish.
To summarize the Section, we assume that telestration is a feature that improves telementoring techniques. However, due to lack of studies on the possible impacts on the overall mentoring process, our claims remain poorly supported. Only the most obvious relations were discussed due to the simplicity of the models. Other dependencies may exist.
Impact Measurement
To prove the hypotheses formulated in Section Definition of Concepts and Theoretical Models, a measurement system needs to be established. The purpose of such system is to assess if the introduction of a particular feature resulted in expected outcomes. This system calls for a comparison of measurements before and after the procedure workflow was supplemented by the new technology.
Evaluating the educational outcomes, shown in Figure 4, is straightforward, as the “decreased duration” and “increasing scores in the rating scale” concepts are easy to measure. The same strategy applies to relations C and D (“decreased knife time”) as well as E and F (“reduced ratio and severity of complications”), which refer to the clinical outcomes of telestration in Figure 5. The established Clavien-Dindo classification of surgical complications is proposed as a metric for assessing the grade of postoperative complications [24]. Relation B in Figure 5 represents the link between telestration and improved educational outcomes, depicted in Figure 3. The main challenge is to prove the dependency between using telestration and improved accuracy of mentoring (relation A, Figure 5). Although this dependency may appear self-evident, due to the increased accuracy of pointing actions, a reliable measure is complicated. Therefore, we propose a combined measure for determining the existence and benefits of this relationship. This measure assesses the following: (1) the number of guidance misinterpretations (possibly leading to errors), (2) the number of requests to clarify the advice, (3) the total time spent on guidance during the procedure, and (4) the number of mentoring interruptions to the flow of the procedure. The combined score is used to determine the accuracy of the mentoring accompanied by telestration. The measures are summarized in Table 1.
Table 1.
Measurement system.
| Parameter | Relationship in Figure 5 | What is measured? | Measure unit |
| Knife time | C | Duration: starting point–first incision, end point–end of the procedure. | Minutes, seconds |
| Ratio and severity level of complications |
E, F | Number of complications / overall number of performed procedures and severity of the complications, if any. |
Ratio coefficient and a grade in the Clavien-Dindo classification scheme for surgical complications [24] |
| Score in rating scale (surgical education) | B | Scores in the predefined scale to measure the progress of education. | Rating score |
| Duration (surgical education) | B | Duration required to reach the same level in the rating scale. | Hours |
| Accuracy of mentoring | A | 1) Number of guidance misinterpretations (possibly leading to errors), 2) Number of requests to clarify the guidance, 3) Total time spent on guidance during the procedure, and 4) Number of interruptions of the flow of the procedure for mentoring. |
Combined score |
To evaluate our hypotheses, a comparison of measures before and after introducing the technology to the operating room must be performed. ALFA Toolkit [25] is one of the tools for analyzing the captured videos, including assessing the durations and other parameters. Results from similar research prove the feasibility of using the tool for the analysis of multi-channel videos in order to measure the impacts of the technology on the workflow of the procedure [26-28]. Three video sources (the output from the laparoscopic camera, the telementoring video and an overview of the operating room, which captures the actions of the surgical team as well as the laparoscopic and telementoring monitors) accompanied by an audio record will be used to evaluate the changes in the procedure workflow. The analysis requires a manual video coding to determine the start and end points of a particular event. This step may become ambiguous; therefore, it should be performed by multiple independent coders. Average value should be used in a case of coders’ disagreement, kappa coefficient should be reported. The following parameters should be encoded and measured using combined videos of the procedure:
Start and end point of the procedure (knife time)
Number of guidance misinterpretations
Number of requests to clarify the guidance
Start and end point of every telementoring interaction between mentor and mentee
Initiator of each interaction
Duration and timing-based measures are automatically collected by ALFA Toolkit during the coding process, while the numerical data (number of misinterpretations and clarification requests, initiator of the interaction) is recorded by the coders in an MS Excel sheet, which is combined with the values exported from ALFA toolkit for further analysis. Average values together with dispersion coefficients are used to represent the final results.
Discussion
Principal Findings
Research biases were analyzed in detail by Hartman et al [29] to increase the awareness of methods of mitigating their undesirable impacts. Although a number of biases were studied, in this phase of the project, the most attention is given to measurement-related biases.
An accurate measure of the parameters defined in Table 1 is a challenge. However, to achieve accurate results in each experiment, the requirements for the analyzed procedure and stakeholders should be formalized to keep them as standard as possible. Unfortunately, the observed procedures are never the same, even if a strictly predefined workflow is followed. Moreover, the human factor shapes the workflow according to the preferences and previous experiences of the surgeon in charge. Complete control of the mentioned factors may be impossible; however, to increase the accuracy, only very similar procedures are to be studied and represented in the final results. The similarity of the cases is based on preoperative data (medical imaging, observations, etc.). Moreover, it is assumed that the remote mentoring approach is also highly influenced by the social connection between mentor and mentee, their previous experiences in telementoring, and their current attitudes. An accurate determination of the levels of experience of mentor and mentee is also a subjective parameter influencing the final results of the study. Randomization of the observed procedures and surgical team may be a way to achieve more consistent results in this case.
From the point of view of the measurement process itself, ambiguity is also inevitable. To avoid biases in duration and score estimation based on video representation, only the consensus decision of the project team should be considered correct.
The presented framework addresses the theoretical side of the project. It was developed as a methodology section for the on-going research. Logical relations between the concepts are represented in order to understand the influence of the technology on procedural workflow and define it in a measurable manner. The research still suffers from the lack of evidence and validation on real life test cases.
To conclude the section, we admit the potential weaknesses of the proposed research caused by the mentioned biases. The list of the biases influencing the final outcome of the experiment is not exhaustive; however, we aimed to mention only the most obvious cases.
Conclusions
The paper aimed to summarize the experience of using telestration during telementoring sessions to highlight the impacts of this feature on the mentoring process and workflow of the procedure. However, an absence of research reflecting the use of telestration was identified. Therefore, a framework for analyzing the impact of live video content annotation was proposed. To keep the models simple and adaptable, easy-to-measure concepts were employed and only the most obvious dependencies were discussed. To support our claims and the presented models, an impact measurement procedure was defined.
The presented framework and impact measurement procedures form a methodology for the further analysis and reporting of research on telestration and telementoring systems. Having a more formalized method should increase the quality and quantity of publications in this field, providing an evidence-based knowledge base supporting the development of the telementoring domain and the introduction of new technologies and features with the aim of improving patient outcomes.
Acknowledgments
This research was funded by a Helse Nord grant (ID 5614/HST1025-11) to the Norwegian Centre of Integrated Care and Telemedicine (NST), University Hospital of North Norway. The study would not have been possible without the support of the Mobile Medical Mentor (M3) project team at NST.
Abbreviations
- ACM
Association of Computing Machinery
- IEEE
Institute of Electrical and Electronics Engineers
- MIS
minimally invasive surgery
Footnotes
Conflicts of Interest: None declared.
References
- 1.Guidelines for the surgical practice of telemedicine. Society of American Gastrointestinal Endoscopic Surgeons. Surg Endosc. 2000 Oct;14(10):975–9. doi: 10.1007/s004640000290. [DOI] [PubMed] [Google Scholar]
- 2.Williams TE, Ellison EC. Population analysis predicts a future critical shortage of general surgeons. Surgery. 2008 Oct;144(4):548–54; discussion 554. doi: 10.1016/j.surg.2008.05.019. [DOI] [PubMed] [Google Scholar]
- 3.Augestad KM, Chomutare T, Bellika JG, Budrionis A, Lindsetmo RO, Delaney CP. Clinical and Educational Benefits of Surgical Telementoring. In: Patel HRH, Joseph JV, editors. HRH Patel and Joseph JV. Simulation Training in Laparoscopy and Robotic Surgery. London: Springer London; 2012. pp. 75–89. [Google Scholar]
- 4.Augestad KM, Lindsetmo RO. Overcoming distance: video-conferencing as a clinical and educational tool among surgeons. World J Surg. 2009 Jul;33(7):1356–65. doi: 10.1007/s00268-009-0036-0. http://europepmc.org/abstract/MED/19384459. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Rothenberg SS, Yoder S, Kay S, Ponsky T. Initial experience with surgical telementoring in pediatric laparoscopic surgery using remote presence technology. J Laparoendosc Adv Surg Tech A. 2009 Apr;19 Suppl 1:S219–22. doi: 10.1089/lap.2008.0133. [DOI] [PubMed] [Google Scholar]
- 6.Damore LJ, Johnson JA, Dixon RS, Iverson MA, Ellison E, Melvin W. Transmission of live laparoscopic surgery over the Internet2. The American Journal of Surgery. 1999 Nov;178(5):415–417. doi: 10.1016/S0002-9610(99)00203-2. [DOI] [PubMed] [Google Scholar]
- 7.Doarn CR. Telemedicine in tomorrow's operating room: a natural fit. Semin Laparosc Surg. 2003 Sep;10(3):121–6. doi: 10.1177/107155170301000305. [DOI] [PubMed] [Google Scholar]
- 8.Merrell RC, Doarn CR. Is it time for a telemedicine breakthrough? Telemed J E Health. 2008 Aug;14(6):505–6. doi: 10.1089/tmj.2008.8499. [DOI] [PubMed] [Google Scholar]
- 9.Lim EC, Seet RC. In-house medical education: redefining tele-education. Teach Learn Med. 2008;20(2):193–5. doi: 10.1080/10401330801991931. [DOI] [PubMed] [Google Scholar]
- 10.Steventon A, Bardsley M, Billings J, Dixon J, Doll H, Hirani S, Cartwright M, Rixon L, Knapp M, Henderson C, Rogers A, Fitzpatrick R, Hendy J, Newman S, Whole System Demonstrator Evaluation Team Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial. BMJ. 2012 Jun;344:e3874. doi: 10.1136/bmj.e3874. http://www.bmj.com/cgi/pmidlookup?view=long&pmid=22723612. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Budrionis A, Augestad KM, Bellika JG. Telestration in Mobile Telementoring. Proceedings of The Fifth International Conference on eHealth, Telemedicine, and Social Medicine (eTELEMED 2013); The Fifth International Conference on eHealth, Telemedicine, and Social Medicine (eTELEMED 2013); 2013; Nice, France. 2013. Feb 24, pp. 307–309. http://www.thinkmind.org/index.php?view=article&articleid=etelemed_2013_16_40_40278. [Google Scholar]
- 12.Budrionis A, Augestad KM, Patel HRH, Bellika JG. Towards Requirements for Telementoring Software. Proceedings of Scandinavian Conference on Health Informatics (SHI2012); Scandinavian Conference on Health Informatics (SHI2012); 2012 October 2-3; Linköping, Sweden. 2012. pp. 27–33. http://www.ep.liu.se/ecp/070/005/ecp1270005.pdf. [Google Scholar]
- 13.Zhou Q, Liu D. Interactive visual content sharing and telestration: A novel network multimedia service. Proceedings of 14th International Conference on Intelligence in Next Generation Networks (ICIN); 14th International Conference on Intelligence in Next Generation Networks (ICIN); 2010 October 11-14; Berlin, Germany. 2010. http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=5640915&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F5623872%2F5640879%2F05640915.pdf%3Farnumber%3D5640915. [DOI] [Google Scholar]
- 14.Kitchenham B. Keele University Technical Report TR/SE-0401 ISSN:1353-7776, National ICT Australia Ltd NICTA Technical Report 0400011T.1. Keele, UK: Keele University; 2004. [2013-07-16]. Procedures for Performing Systematic Reviews http://tests-zingarelli.googlecode.com/svn-history/r336/trunk/2-Disciplinas/MetodPesquisa/kitchenham_2004.pdf. [Google Scholar]
- 15.Kitchenham B, Brereton OP, Budgen D, Turner M, Bailey J, Linkman S. Systematic literature reviews in software engineering – A systematic literature review. Information and Software Technology. 2009;51:7–15. doi: 10.1016/j.infsof.2008.09.009. [DOI] [Google Scholar]
- 16.Sjøberg DIK, Dybå T, Anda BCD, Hannay JE. Building Theories in Software Engineering. In: Shull F, Singer J, Sjøberg DIK, editors. Guide to Advanced Empirical Software Engineering. London, UK: Springer-Verlag London; 2008. pp. 312–336. [Google Scholar]
- 17.Tan GY, Goel RK, Kaouk JH, Tewari AK. Technological advances in robotic-assisted laparoscopic surgery. Urol Clin North Am. 2009 May;36(2):237–49, ix. doi: 10.1016/j.ucl.2009.02.010. [DOI] [PubMed] [Google Scholar]
- 18.Agarwal R, Levinson AW, Allaf M, Makarov D, Nason A, Su LM. The RoboConsultant: telementoring and remote presence in the operating room during minimally invasive urologic surgeries using a novel mobile robotic interface. Urology. 2007 Nov;70(5):970–4. doi: 10.1016/j.urology.2007.09.053. [DOI] [PubMed] [Google Scholar]
- 19.Rafiq A, Moore JA, Zhao X, Doarn CR, Merrell RC. Digital Video Capture and Synchronous Consultation in Open Surgery. Annals of Surgery. 2004;239(4):567–573. doi: 10.1097/01.sla.0000118749.24645.45. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Schulam PG, Docimo SG, Saleh W, Breitenbach C, Moore RG, Kavoussi L. Telesurgical mentoring. Initial clinical experience. Surg Endosc. 1997 Oct;11(10):1001–5. doi: 10.1007/s004649900511. [DOI] [PubMed] [Google Scholar]
- 21.Ali MR, Loggins JP, Fuller WD, Miller BE, Hasser CJ, Yellowlees P, Vidovszky TJ, Rasmussen JJ, Pierce J. 3-D telestration: a teaching tool for robotic surgery. J Laparoendosc Adv Surg Tech A. 2008 Feb;18(1):107–12. doi: 10.1089/lap.2007.0051. [DOI] [PubMed] [Google Scholar]
- 22.Visentini-Scarzanella M, Merrifield R, Stoyanov D, Yang GZ. Tracking of irregular graphical structures for tissue deformation recovery in minimally invasive surgery. Med Image Comput Comput Assist Interv. 2010;13(Pt 3):261–8. doi: 10.1007/978-3-642-15711-0_33. [DOI] [PubMed] [Google Scholar]
- 23.Bauer JJ, Lee BR, Stoianovici D, Bishoff JT, Janetschek G, Bunyaratavej P, Kamolpronwijit W, Ratchanon S, O'Kelley S, Cadeddu JA, Micali S, Micali F, Man KLi, Goh P, Png D, Kavoussi LR. Remote telesurgical mentoring: feasibility and efficacy. Proceedings of the 33rd Annual Hawaii International Conference on System Sciences; 33rd Annual Hawaii International Conference on System Sciences; 2000 January 4-7; Hawaii, US. 2000. http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=926802. [DOI] [Google Scholar]
- 24.Dindo D, Demartines N, Clavien PA. Classification of surgical complications: a new proposal with evaluation in a cohort of 6336 patients and results of a survey. Ann Surg. 2004 Aug;240(2):205–13. doi: 10.1097/01.sla.0000133083.54934.ae. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.de Lusignan S, Kumarapeli P, Chan T, Pflug B, van Vlymen J, Jones B, Freeman GK. The ALFA (Activity Log Files Aggregation) toolkit: a method for precise observation of the consultation. J Med Internet Res. 2008;10(4):e27. doi: 10.2196/jmir.1080. http://www.jmir.org/2008/4/e27/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Pflug B, Kumarapeli P, van Vlymen J, Ammenwerth E, de Lusignan S. Measuring the impact of the computer on the consultation: an open source application to combine multiple observational outputs. Inform Health Soc Care. 2010 Jan;35(1):10–24. doi: 10.3109/17538150903358693. [DOI] [PubMed] [Google Scholar]
- 27.Gibbings-Isaac D, Iqbal M, Tahir MA, Kumarapeli P, de Lusignan S. The pattern of silent time in the clinical consultation: an observational multichannel video study. Fam Pract. 2012 Oct;29(5):616–21. doi: 10.1093/fampra/cms001. [DOI] [PubMed] [Google Scholar]
- 28.De Lusignan S, Kumarapeli P, Debar S, Kushniruk AW, Pearce C. Using an open source observational tool to measure the influence of the doctor's consulting style and the computer system on the outcomes of the clinical consultation. Stud Health Technol Inform. 2009;150:1017–21. [PubMed] [Google Scholar]
- 29.Hartman JM, Forsen JW, Wallace MS, Neely JG. Tutorials in clinical research: part IV: recognizing and controlling bias. Laryngoscope. 2002 Jan;112(1):23–31. doi: 10.1097/00005537-200201000-00005. [DOI] [PubMed] [Google Scholar]