Abstract
Background
Learning how to conduct a medical interview and perform a physical examination is fundamental to the practice of medicine; however, when this project began, the methods used to teach these skills to medical students at the University of Toronto (U of T) had not changed significantly since the early 1990s despite increasing outpatient care, shorter hospital stays, and heavy preceptor workloads. In response, a Web-based clinical skills resource was developed for the first-year undergraduate medical course—The Art and Science of Clinical Medicine I (ASCM I).
Objectives
This paper examines our experiences with the development of the ASCM I website and details the challenges and motivators inherent in the production of a Web-based, multimedia medical education tool at a large Canadian medical school.
Methods
Interviews and a focus group were conducted with the development team to discover the factors that positively and negatively affected the development process.
Results
Motivating factors included team attributes such as strong leadership and judicious use of medical students and faculty volunteers as developers. Other motivators included a growing lack of instructional equivalency across diverse clinical teaching sites and financial and resource support by the Faculty of Medicine. Barriers to development included an administrative environment that did not yet fully incorporate information technology into its teaching vision and framework, a lack of academic incentive for faculty participation, and inadequate technical support, space, and equipment.
Conclusions
The success of electronic educational resources such as the ASCM I website has caused a significant cultural shift within the Faculty of Medicine, resulting in the provision of more space, resources, and support for IT endeavours in the undergraduate medical curriculum.
Keywords: Undergraduate medical education, Internet, clinical skills, medical history taking, teaching methods, training techniques, qualitative research, focus groups, interviews
Introduction
The life so short, the craft so long to learn...
– Hippocrates
No one has time to sit down and teach people how to do a history anymore, but it is the most important thing you have to learn.
– Physician contributor to the ASCM I website
Learning how to conduct a medical interview and perform a physical examination is fundamental to the practice of medicine. Documentation of the history and physical is critical for communication between medical personnel and for the long-term recording of patient data. The methods used to teach these skills to students at the University of Toronto (U of T) had not changed significantly since the early 1990s despite changes in the practice of medicine that included increasing outpatient care, shorter hospital stays, and heavy preceptor workloads. As a result, the traditional teaching and learning of clinical skills appeared to be increasingly inadequate in today's health care environment [1,2], a concern that is now being recognized and addressed nationally by the Association of Faculties of Medicine of Canada (formerly known as the Association of Canadian Medical Colleges).
The Art and Science of Clinical Medicine (ASCM I)
In the medical program at the U of T, clinical skills training begins in the second week of first year with the course The Art and Science of Clinical Medicine I (ASCM I). This course is taught at six affiliated teaching hospital sites by approximately 90 volunteer physician tutors, who teach groups of five to seven students. Like many other medical schools, the U of T teaches a standardized clinical curriculum [3-5] and is faced with many challenges. For example, the clinical learning experience varies between hospital sites and small groups, and student course evaluations have repeatedly identified a need for standardized teaching methods and expectations. Similarly, tutors have requested faculty development to aid their understanding of the level of expertise necessary in the clinical encounter in order for a first-year student to meet the course objectives.
Origin of the ASCM I Website
To address these concerns, a medical student proposed the development of a computer-based tool to meet the demand for instructional equivalency across clinical sites, to standardize the expectations of both students and tutors, to assist the teaching of clinical skills to undergraduate medical students, and to supplement the course content. In response, a research and development team was formed to create the ASCM I website, an interactive, multimedia online resource for both student education and faculty development in clinical skills training [6]. Such online resources are playing an increasingly important role in the delivery of medical education programs [7-15].
A Web-based platform was chosen to supplement the hands-on clinical curriculum due to the following benefits of the Internet: (1) increasingly wide availability to students and tutors at home, school, and hospital; (2) on-demand and round-the-clock access; (3) increasing popularity; (4) expanding interactive and multimedia capabilities; (5) ability to accommodate many learning styles; and (6) ease of website updates and expansions. A lack of physical examination sites suitable for our local and Canadian context was also a factor in choosing a Web-based platform. The course director and three medical students developed the password-protected website over two years. Formative and summative evaluations of the site by a medical educator and member of the Wilson Centre for Research in Education have been ongoing. Summer student scholarships and Faculty of Medicine funding supported basic budget requirements.
Focus of This Study
In this paper we share our experience of the development of the ASCM I website in order to detail some of the challenges and motivators inherent in the production of a Web-based, multimedia medical education tool at a large Canadian medical school. The experiences, practical knowledge, understanding, and expectations that individuals (eg, faculty and students) brought to the information technology (IT) development process clearly influenced the final product. Similarly, the institutional context (eg, established roles, norms, and mandated practices of the medical school) and the broader social context (eg, political and economic) in which the electronic resource was embedded also strongly shaped the development process. Therefore, this study examines the factors influencing the development, implementation, and evaluation of e-based resources in our setting. Ultimately, it aims to generate both qualitative and quantitative data for explanatory theory building [16]. By discussing individual, structural, and political issues that impinged upon the development and implementation of the ASCM I website, we also hope to assist others who are contemplating or developing similar projects in their own settings.
Methods
Website Description
The website was developed in two phases: the history home page was developed in 2000, and the physical examination home page was developed in 2001. The complete site has been in operation since September 2001 [17,18]. Selected portions of the website may be accessed at http://ascm.med.utoronto.ca/examples/.
The history section (Figure 1) consists of (1) a video interview (26 minutes) conducted by a first-year medical student with a standardized patient, (2) eight interactive modules outlining key components of medical history taking, (3) case report assignment modules to improve case report writing skills, and (4) activity modules to improve verbal and nonverbal communication skills and to increase student comfort with patient visitation on the wards.
Figure 1.
The ASCM I history home page
The physical examination section (Figure 2) contains 10 modules for the physical skills (excluding the neurological exam, which has its own separate website) taught in ASCM I. Each module includes a video demonstration performed by a faculty member, accompanied by explanatory graphics and text. The modules include demonstrations of draping and positioning, inspection, palpation, percussion, auscultation, and special maneuvres as they apply to the specific examinations. Subsections of the modules can be viewed separately or together as a continuous piece. Graphics and notes specific to the subsections of the examinations provide information on anatomy, physiology, and the mechanics of the examination. Several modules include unrehearsed examinations performed by a first-year student that are accompanied by faculty feedback in order to address common challenges students face when learning particular examinations and to illustrate for physician tutors how to give immediate formative feedback to students.
Figure 2.
The ASCM I physical examination home page
Focus Group
Following completion of the website prototype, a focus group of 10 participants (physicians, standardized patients, medical students, and production staff involved in the creation of the site) was held in July 2002 to discuss the motivators, challenges, and barriers to the development of the ASCM I site. A funnel technique of questioning was used to identify broad question areas first and then to progressively discuss specific domains of interest. Initial probes fell into four main domains: technical, content, and contextual issues and future directions for site development. The focus group discussion was audio taped and transcribed.
Interviews
Semi-structured interviews (N = 5) were conducted with the remaining members of the development team and university staff (eg, course coordinators, clinical faculty, and administrative staff) in order to garner their insights on the website development and maintenance. As in the focus group, participants were asked to identify the difficulties and challenges they faced, along with their successes and lessons learned.
Data arising from the interviews and focus group were analyzed using the methods previously described by Miles and Huberman [19] and Krueger [20]. Each transcript was read recursively by at least two readers to code the data and create thematic categories representing trends in the perceptions, attitudes, and experiences of the participants. Patterns were then jointly identified. The jointly coded data underwent a process of increasingly finer categorization until all trends and variations were accounted for and cross-referenced. Finally, the completed academic manuscript was presented to participants for review to ensure the accuracy and comprehensiveness of the data interpretation (ie, interpretive validity). This study received ethical approval from the Research Ethics Board of the University of Toronto.
Results
All participants involved in the development of the ASCM I website shared an interest in and a commitment to innovation in the undergraduate medical curriculum, as will as a willingness to contribute a great deal of their own personal time to realize that goal. They discussed their motivations for joining the development team, the importance of team leadership for project completion, the impact of a lack of resources (eg, space, finances, and personnel), and the need for institutional support for IT. These themes are discussed below and are summarized in the Table 1.
Table 1.
Motivators and barriers experienced by the ASCM I development team
| Motivator | Description/Characteristics |
| Curriculum gap |
|
| Team leadership |
|
| Medical students |
|
| Faculty |
|
| Teamwork |
|
| Financial resources |
|
| Space |
|
| Barrier | |
| Administrative structure of institution |
|
| Development mandate |
|
| Faculty |
|
| Support staff |
|
| Equipment |
|
Motivations for Joining the Development Team
In the focus group, team members explained their motivations to contribute. Medical students selected from the ASCM I course to construct the site the following summer were intrigued by the project's informatics focus (“Well, isn't this different and neat!”) and were motivated to join by altruism. They “wanted to improve the course” and felt that “the university was lagging behind in terms of using Internet computing.” As newly graduated members of the course's target audience, the medical students could readily identify difficult curriculum topics from a student's perspective and focus the teaching content of the site accordingly. The students showed tremendous enthusiasm, creativity, and dedication to the project [3,21]. According to other team members, the students “worked marvelously well together,” had complementary skills, and appeared to “live” in the computer facilities where they studied at night while developing the site and solving user problems and programming glitches. Together they “created something that is absolutely amazing.”
All physician contributors were already involved in the ASCM I course as tutors or hospital coordinators, had a personal interest in medical education, and volunteered considerable personal time and energy to the project, often as a personal favor to the course director. One clinician explained, “[The course director] motivated me very strongly, but I also have an interest in undergraduate education and looking at different ways of teaching the musculoskeletal exam. This project seemed like a perfect fit.” Similarly, the support staff on the project all had strong track records in multimedia and medical education along with a willingness to contribute extensive personal time.
Together, these players formed an enthusiastic and committed team that worked well together. Focus group participants said, “The team work here was unbelievable!” and they recalled late night work sessions with pizza and beer to keep creativity flowing. Mutual support of team members was critical when it seemed that the fledgling project would collapse and was crucial to its ultimate success. As one student put it, “[Success] is when you learn to work as a team. It is when you know that if you start to slip, someone else is going to back you up or say, ‘No, don't give up yet!'”
Such volunteer-driven resource development requires a very dedicated and motivated team of faculty, staff, and students who recognize the importance of the project. Unfortunately, the majority of clinicians do not have protected time for teaching and have difficulty finding time for educational work [4]. As one physician explained, “My teaching is unsupported, ie, I do it by generating my income with the other 75% of my time.” Similarly, Berge and Muilenberg [22] identified lack of faculty time and compensation as the most frequently identified barrier to the development of distance education strategies. However, another clinician asserted, “You make time for this. You make the time!” Unfortunately, in general, there appears to be little incentive for clinical faculty to contribute to or create innovative projects [22]. In spite of personal interest, clinician respondents in the focus group felt unable to spearhead resource development projects “because of the time and energy it would require.” One clinician noted, “The University has to realize that anybody you could get has other commitments.” The U of T partially addresses this concern by making “creative professional activity” and teaching platforms viable routes for faculty promotion.
Team Leadership
Interviewed participants discussed the importance of the course director as an e-learning innovator and project champion, thereby highlighting his role as a promoter of organizational change. The team leader had a vision of the project and the passion to see it through. As one interviewed administrator noted, “The only way things get developed in the faculty is that the key teachers say ‘I want to do something,' and ways are found to do it. It certainly doesn't come from the top down.” The course director had “a good reputation as a teacher,” “had never been involved in anything like this before,” “really got excited about it,” and was then able to move the project forward to completion “on time, on budget.” This project arose from the grassroots rather than from a university-mandated approach to curriculum development.
An ability to deal with and adapt to uncertainty and a lack of resources (including protected time) was an important leadership quality discussed in the focus group. This innovative project was very vulnerable to disruptions in its early days (eg, loss of a key team member). The team leader created a unified team that capitalized on the personal strengths of the individual members. Important project management skills included the ability to spearhead the development process, to network and partner with members of other medical programs such as the Standardized Patient Program and the Wilson Centre for Research in Education, and to bring together, support, and motivate a diverse and talented development team [4,23]. The course director provided the project's cohesive leadership by working on multiple administrative levels to secure support for the project. He expended energy and time recruiting and supporting team members, advocating for project funding, and facilitating product evaluation. He observed that an ongoing motivator for him was the encouragement and financial support of the Associate Dean of Undergraduate Medicine and the Director of Teaching Labs and Educational Computing.
Lack of Resources
The project was initially hindered by a lack of resources as the Faculty did not yet have the necessary infrastructure or IT support. As one respondent noted, “In these kinds of projects we are always trying to find a little help here and a little help there. So they are put together from lots of little bits and fragments.” The entire website was produced for less than $25000 Canadian (primarily salary support for students). Others in the focus group stated bluntly, “We were all putting this together with scotch tape,” and were “doing things on a shoestring budget and [with] Band-Aids.” Developers faced outdated and limited computer and audiovisual equipment and minimal support staff. Students had to teach themselves audiovisual skills and were frequently “bogged down in” technical issues, resulting in “frustrations” and much wasted time and effort. Technical tasks such as film editing initially took weeks to learn and accomplish, compared with one afternoon the following year after the necessary equipment was purchased. One student recalled, “There was no one there to show me the ropes. I was just learning as I went. . . . There was no personnel support.” Another observed, “Everyone was just kind of thrown together, and we made do with what we had.”
Innovative work on a low budget without technical support means making mistakes. According to one focus group physician, the visuals of the student interview with a standardized patient “are pretty poor” due to inexperience with the camera, “but the content was fantastic.” The necessary technical skills gradually developed within the team, and by the second summer, new students could rely on pre-existing expertise by “standing on the shoulders” of the previous students. Experience taught the team valuable (and, in hindsight, simple) lessons about the usefulness and cost-effectiveness of pre-production meetings in which the different needs and styles of clinicians and the production staff were anticipated and reconciled. Simple but effective tricks to simplify editing and improve video quality were passed down. In particular, practical tips for the filming (eg, camera setup, lighting, and sound) and selection of audio and video codecs were most helpful.
Site Maintenance
Once the website was completed and was online (hosted on a faculty server), a lack of university IT infrastructure to support this new resource became apparent. The course director recalled,
“We found out in the middle of September, when the first glitch occurred, that there was no one...to look after this. We spent all these hundreds of hours doing this, and now it was ready to go, but I didn't know how to fix the little problems—I am the course director!... There was nobody to look after the damn thing!” As a temporary solution, one of the students came in regularly to back up and maintain the database. Furthermore, the students did “all the tech support” for users, including staffing an electronic user help line. An instruction package and online instructions were provided to all students currently enrolled in ASCM I; however, the course administrative staff also devoted significant time to dealing with password concerns and questions from both students and faculty.
Institutional Support for IT
Interviewees noted that, since 1999, support for IT innovation increased steadily among key stakeholders (eg, upper administration, academic deans), and there was increased accountability for technological innovation that falls within academic portfolios. Greater numbers of people are taking responsibility for and promoting innovation. Interview participants recognized a growing shift in thinking within the university hierarchy, from very little evidence of support from administration for online education to attitudes of “This is what is coming and there is a definite move in that direction,” and “Hey, you can get a lot of research funds by doing something like this.” One person noted, “They realized subsequently that, in fact, good research came out of it.” This evolution in thinking was termed “big C change” by one interview respondent, signaling the beginning of the acceptance of IT as a teaching and learning tool by the university hierarchy. Interviewees also noted, however, that the funding necessary to maintain already developed projects and to provide student computer infrastructure was still seriously insufficient. As one focus group participant observed, “Another frustration with developing this resource was there was really no precedent that U of T will look at another project and say, well, they went through this. These were their pitfalls, so we should avoid making the same mistakes.”
Discussion
Effective development of a multimedia education tool requires a multidisciplinary team with diverse skills and creative talents. The effective use of technical non-experts, such as students and clinicians, for the development and implementation of IT products requires readily available technical support personnel (see also [5]). Ideally, collaboration of IT experts, content experts, and developers allows the content to be conceptualized by content specialists, enacted and captured by a production crew, and placed and maintained in an electronic framework by technical staff.
This study used qualitative methods (a focus group and individual interviews) to investigate the complex social factors involved in the development of the ASCM I website. Limitations to these methods can include recall bias by the participants and a potential social desirability bias in the focus group, as participants may not have wished to sound negative about the website or the development process in front of their colleagues. However, private individual interviews resulted in similar discussions of the motivators and challenges experienced during site development. All participants appeared very open, thoughtful, and concerned about the future of university-based electronic resource development. Future research can assess the transferability of our findings to other situations and development teams.
Institutional Context
The administrative and support structure of most educational institutions were created well before the computer and Internet revolutions, and established medical schools appear less likely to accommodate innovation within their organizations' structure [22,25]. Therefore, although computers have permeated educational administration, in many schools they are still ancillary in education itself [26], and few medical schools have developed a strategic approach to the use of technology in medical education [27]. As was experienced here, the lack of acceptance and integration of computer-based education in the mindset of an institution hinders both the development and the use of computer-based educational tools. For institutions that do use distance education for mission critical goals, other barriers include a lack of technical expertise and support, and concerns regarding resource evaluation and effectiveness. In the absence of a unified academic plan on IT and medical education, independent projects and resources may be poorly developed, uncoordinated, and not well integrated into the curricula [5]. An effective, IT-supported medical curriculum requires an accompanying organizational change strategy to adopt and develop technology suitable for that context and to prepare and support e-learners [24]. For example, an institution needs to develop an “institutional memory” of IT trailblazing by its faculty and students so that they are not forced to continually reinvent the wheel.
Fortunately, since the start of this project in 2000, the Faculty of Medicine has taken important steps to improve the IT development climate. In January 2002, the Medical Alumni Association sponsored an Education Innovations Lab to provide space and hardware for faculty, staff, and students to develop new IT applications for educational purposes. One full-time and one part-time staff member have been hired to assist in the design and maintenance of the educational tools created. Two internal sources of funding are now available for the development of IT courseware: The Dean's Excellence Fund in Medical Education and the Information Technology Courseware Development Fund. We believe that the value of the electronic educational projects developed to date has triggered these improvements and will continue to prompt changes necessary for IT to be more fully utilized in the future.
A medical faculty committed to e-education requires an environment and policies that are conducive to IT development and use. For example, the ASCM I website remains password protected in the absence of a broader university policy or official disclaimer to protect participants from liability claims. In our experience, fundamental development support for in-house creation of multimedia resources necessitates provision of physical space, technical support, hardware, and software and licensing. The budget of a single medical course cannot support even one of these requirements. Moreover, Internet-based educational tools require long-term basal funding because of the need for evaluation, technical support for users, and site housing and maintenance. In order that projects such as this one can be incorporated into the Faculty funding scheme, changes in administrative policy and structure are required. Research and dissemination of effective IT development practice and theory are integral to the continued growth of the field. In order to promote such dissemination, the Faculty of Medicine sponsors a yearly Educational Achievement Day to network and share new developments.
National Context
As we were implementing the first electronic iteration of the ASCM I website to increase standardization of clinical skills teaching across our own curriculum, a national discussion of undergraduate clinical skills teaching in Canada was being initiated by the Association of Faculties of Medicine of Canada (AFMC). In 2002, the Canadian National Clinical Skills Working Group was formed to standardize the curriculum content and evaluation methods across Canadian medical schools and to specify the clinical skills Canadian medical students should master by graduation. Electronic and Web-based teaching and learning tools such as the ASCM I website are ideally suited for promoting nationally recognized clinically useful and evidence-based clinical skills, and for addressing many of the objectives put forth at the national AFMC clinical skills meeting. A similar website for all Canadian medical schools would be an ideal forum for the presentation of standardized history taking, physical examinations, and technical skills, and it could positively affect the practices of both students and teachers. The ASCM I website can be considered a successful pilot project for the feasibility of a Canada-wide, online resource for clinical skills training.
Conclusions
This study investigated the climate surrounding the creation of an Internet-based learning tool for medical education at a major Canadian medical school. Capitalizing on strong leadership and the skills of a multidisciplinary team of collaborators, we developed an effective and widely used resource for students. The enthusiasm, creativity, sense of ownership and altruism, and content knowledge of the students and faculty involved with this grassroots project were key to its development (see also [28]). We found that, when information technology has not yet been incorporated into the fundamental educational structure of an academic institution, the resource development process can be arduous and can result in challenges around funding, personnel, and resource allocation for electronic curriculum development. At the heart of the difficulties experienced at our institution was lack of a centralized policy on the use of information technology in medical education. This barrier hampers financial support for educational information technology as part of the medical school's core curriculum. For the effective development of future electronic resources, the barriers documented in this paper must be addressed, and the key motivators capitalized on and enhanced.
Future Directions
Systematic evaluation of innovative electronic teaching and learning initiatives is crucial in order to ensure continued excellence and user-centered program development [16,29]. We believe that our website is a positive educational resource, and a future publication will document our formative and summative evaluation strategies. In ASCM I, our strategy of ongoing formative evaluation has allowed continued resource development to be driven by user (and other stakeholder) feedback on the site's strengths and weaknesses [18,30]. Continued formative and summative evaluation of the ASCM I learning tool is proving key to its expansion to address the requirements of the second-year ASCM course, which teaches more complicated history taking and examination skills.
Acknowledgments
We gratefully acknowledge the support of a 2002 University of Toronto, Information Technology Course Development Fund grant for $12800 (Can) from the Provost and the Department of Anesthesia to Joyce Nyhof-Young and John Bradley. Funds to support three medical students were also received from the Faculty of Medicine's Summer Research Scholarship Program. The tireless creative work of Hance Clarke, Errol Colak, and the rest of the development team is also gratefully acknowledged, as is the mentorship and support provided by Dr. Lawrence Spero, Director of the Teaching Laboratories and Educational Computing in the Faculty of Medicine.
Abbreviations
- ASCM I
The Art and Science of Clinical Medicine, year one
- IT
information technology
- U of T
University of Toronto
Footnotes
None declared.
References
- 1.Mangione S, Nieman L Z. Cardiac auscultatory skills of internal medicine and family practice trainees. A comparison of diagnostic proficiency. JAMA. 1997 Sep 3;278(9):717–22. doi: 10.1001/jama.278.9.717. [DOI] [PubMed] [Google Scholar]
- 2.Mangione S, Nieman L Z, Gracely E, Kaye D. The teaching and practice of cardiac auscultation during internal medicine and cardiology training. A nationwide survey. Ann Intern Med. 1993 Jul 1;119(1):47–54. doi: 10.7326/0003-4819-119-1-199307010-00009. http://www.annals.org/cgi/pmidlookup?view=long&pmid=8498764. [DOI] [PubMed] [Google Scholar]
- 3.Leong Shou Ling, Baldwin Constance D, Adelman Alan M. Integrating Web-based computer cases into a required clerkship: development and evaluation. Acad Med. 2003 Mar;78(3):295–301. doi: 10.1097/00001888-200303000-00012. [DOI] [PubMed] [Google Scholar]
- 4.Petrusa E R, Issenberg S B, Mayer J W, Felner J M, Brown D D, Waugh R A, Kondos G T, Gessner I H, Mcgaghie W C. Implementation of a four-year multimedia computer curriculum in cardiology at six medical schools. Acad Med. 1999 Feb;74(2):123–9. doi: 10.1097/00001888-199902000-00011. [DOI] [PubMed] [Google Scholar]
- 5.Fleiszer David M, Posel Nancy H. Development of an undergraduate medical curriculum: the McGill experience. Acad Med. 2003 Mar;78(3):265–9. doi: 10.1097/00001888-200303000-00005. [DOI] [PubMed] [Google Scholar]
- 6.Bradley J, Nyhof-Young J, Clarke H, Colak E, Macdonald R. Multimedia in Health Sciences Education: Proceedings of the Slice of Life 2001 & Computers in Healthcare Education Symposium. Berlin, Germany: Logos Verlag; 2001. Mrs. Fisher tells her story: an interactive Web-based learning tool to assist medical students with history taking and case report writing skills; pp. 43–46. [Google Scholar]
- 7.Walsh JB, Goepfert A, Berkow R, Phelan S. Designing Web sites to achieve instructional equivalency across clinical sites. Association of American Medical Colleges Annual Meeting; November 2-7, 2001; Washington DC. [Google Scholar]
- 8.Israel JL, Quintana J, McArdle P, Swartz MH. Meeting new patients: an interactive video evaluation tool. Association of American Medical Colleges Annual Meeting; November 2-7, 2001; Washington DC. [Google Scholar]
- 9.Kreger C. An interactive multimedia Web-based program to teach physical examination skills. Association of American Medical Colleges Annual Meeting; November 2-7, 2001; Washington DC. [Google Scholar]
- 10.Wittenberg L, Milligan J, Macnab AJ. Development and evaluation of an educational Website: The UBC pediatric clinical skills Website. ACMC-ACAHO-CAME Annual Meeting; April 30, 2001; Toronto, Ontario, Canada. [Google Scholar]
- 11.University of Toronto Department of Radiation Oncology. Virtual experience in radiation oncology. [2005 Apr 19]. http://www.bluelemonmedia.com/vero/
- 12.Mosbaugh K, Stapleton L. Teaching the fundamentals of physical diagnostic techniques on the Web; Slice of Life Conference; June 18-22, 2002; Toronto, Ontario, Canada. [2005 Apr 19]. http://www.slice.gsm.com/2002/Thurs_demo/Stapleton-Phys%20DX.html. [Google Scholar]
- 13.Cain TJ, Fish CA, Finneran CN, et al. Enhancing physical examination skill instruction with digital learning objects: using technology to build the basics. Demonstration at: Slice of Life Conference; June 24ñ28,; Slice of Life Conference; June 24-28, 2003; Philadelphia, PA, USA. [2005 Apr 21]. http://www.slice.utah.edu/2003/Friday_demo/Cain,_demo.htm. [Google Scholar]
- 14.Cain TJ. An interactive Web-based primer for teaching physical examination skills: an integration of Flash, SMIL, and Cold Fusion; Slice of Life Conference; June 18-22, 2002; Toronto, Ontario, Canada. [2005 Apr 21]. http://www.slice.utah.edu/2002/Sat_audit/Cain.html. [Google Scholar]
- 15.Hohol MJ. The neurological examinationóa Web-based multi-modal educational tool; Slice of Life Conference; June 18-22, 2002; Toronto, Ontario, Canada. [2005 Apr 21]. http://www.slice.utah.edu/2002/Thurs_demo/Hohol.html. [Google Scholar]
- 16.Somekh B. The role of evaluation in ensuring excellence in communications and information technology initiatives. Education, Communication & Information. 2001;1(1):75–101. doi: 10.1080/14636310120048065. [DOI] [Google Scholar]
- 17.Nyhof-Young J, Bradley J, Clarke H, Colak E, Macdonald R. Mrs. Fisher tells her story: an interactive web-based learning tool to assist medical students with history taking and case report writing skills; Slice of Life Conference; July 31 - August 4, 2001; Munich, Germany. [2005 Apr 19]. http://www.slice.utah.edu/2001/Fri_program/Nyhof-Young_1.html. [Google Scholar]
- 18.Bradley J, Clarke H, Colak E, Simmons C, Nyhof-Young J. An Interactive web-based learning tool to assist medical students with history taking, case report writing, and physical examination skills; Slice of Life Conference; June 18-22, 2002; Toronto, Ontario, Canada. [2005 Apr 19]. http://www.slice.utah.edu/2002/Fri_audit/Bradley.html. [Google Scholar]
- 19.Miles MB, Huberman AM. Qualitative Data Analysis: An expanded Source Book. Thousand Oaks, CA: Sage; 1994. [Google Scholar]
- 20.Krueger Richard A. Developing Questions for Focus Groups (Focus Group Kit) Thousand Oaks, CA: SAGE Publications; 1997. Jul 24, [Google Scholar]
- 21.Riza L, Connors M. The use of medical students in developing Web-based curriculum applications. Acad Med. 2001 May;76(5):551–2. doi: 10.1097/00001888-200105000-00096. [DOI] [PubMed] [Google Scholar]
- 22.Berge ZL, Muilenburg LY. Obstacles faced at various stages of capability regarding distance education in institutions of higher learning. Tech Trends. 2001;46(4):40–45. [Google Scholar]
- 23.Mackenzie J D, Greenes R A. The World Wide Web: redefining medical education. JAMA. 1997 Dec 3;278(21):1785–6. doi: 10.1001/jama.278.21.1785. [DOI] [PubMed] [Google Scholar]
- 24.Kurchner-Hawkins R. Preparing and supporting e-learners: the organizational change imperative. In: Piskurich GM, editor. Preparing Learners for e-Learning. San Francisco, Calif: Pfeiffer; 2003. [Google Scholar]
- 25.Bloom S W. Structure and ideology in medical education: an analysis of resistance to change. J Health Soc Behav. 1988 Dec;29(4):294–306. [PubMed] [Google Scholar]
- 26.Moehr J R, Grant A. Medical informatics and medical education in Canada in the 21st century. Clin Invest Med. 2000 Aug;23(4):275–80. [PubMed] [Google Scholar]
- 27.Moberg T F, Whitcomb M E. Educational technology to facilitate medical students' learning: background paper 2 of the medical school objectives project. Acad Med. 1999 Oct;74(10):1146–50. doi: 10.1097/00001888-199910000-00020. [DOI] [PubMed] [Google Scholar]
- 28.Gupta G, Park J, Nyhof-Young J, et al. Student teachers: medical students developing content for their own Website supported, self assessment, question database; Slice of Life Conference; June 28 - July 3, 2004; Amsterdam and Leiden, The Netherlands. [2005 Apr 19]. http://www.slice.utah.edu/2004/thursday_posters/Gupta,%20poster.htm. [Google Scholar]
- 29.Worthen Blaine R, Sanders JR, Fitzpatrick JL. Program Evaluation. Alternative Approaches and Practical Guidelines. 2. White Plains, NY: Longman; 1997. [Google Scholar]
- 30.Colak E, Clarke H, Simmons C, Spero L, Nyhof-Young J, Bradley J. Evaluation of phase one of an on-line educational tool to assist medical students with history taking and case report writing skills. Slice of Life Conference; June 18-22, 2002; Toronto, Ontario, Canada. [Google Scholar]