Using Contextual Design to Identify Potential Innovations for Problem Based Learning

Abstract

We report on the use of Contextual Design (CD) to develop models of the information management, resource integration, and collaborative processes of medical students in problem-based learning groups. CD is a modified ethnographic technique designed to provide a detailed understanding of the user’s needs. Although the technique has been used in non-healthcare related fields, there is limited published data on the application of CD within healthcare settings. In this pilot study, we evaluated the feasibility of the CD methodology for this domain, developed an initial set of CD models, and formulated a series of design ideas based on the data. The study helps to clarify the effectiveness and feasibility of CD as well as the limitations for using this method in health-related domains.

INTRODUCTION

Contextual Design (CD)¹ is a highly structured method for collecting, interpreting, and aggregating qualitative data about work processes. The methodology is designed specifically for the purpose of creating software that addresses user’s needs. Since its original description by Beyer and Holtzblatt, CD has mainly been used by large software corporations. It has only rarely been applied towards the development of medical information systems ^2,3. In this pilot study, we apply CD to the study of educational work processes in Problem-Based Learning (PBL)-a ubiquitous medical educational setting with a unique set of information needs and limited software designed to support this process.

RESEARCH GOALS

This pilot study was performed to (1) determine the feasibility, advantages and disadvantages of adapting CD for use in a medical and educational domain, (2) provide an initial model of information seeking and information flow for students engaged in PBL and (3) generate an initial set of design ideas for a collaborative learning environment for PBL.

BACKGROUND

Problem-based learning

PBL was originally developed at McMaster University by Barrows and colleagues^4–6, and describes a form of case-based learning in which domain knowledge and problem solving skills are acquired as students work through a specific clinical problem. In PBL, students work in small groups with the guidance of a faculty preceptor. The students are first presented with a clinical problem that the y do not possess adequate knowledge to solve. The group relies on their current knowledge to discuss the case and identify learning issues. The group then adjourns to pursue independent study on the selected topics and later reconvenes to discuss their research and apply this new knowledge to solving the case. Students engaged in PBL are typically active information-seekers, utilizing multiple types of sources (printed material, online books and websites) and institutional resources (libraries, e-mail) while they integrate information for a specific purpose. Although the PBL method is now incorporated in some form in a majority of medical school curricula, there have been few attempts to design software that supports this collaborative learning process ^7,8.

Contextual Design

Knowledge of the end-user is a prerequisite to designing successful software. Often, needs-assessment is limited to demographic data and end-user opinions. However, detailed data about actual work processes can provide critical information for developing relevant and useable information systems. CD incorporates traditional ethnographic approaches into a multi-part process that supports data-driven design. Unlike traditional ethnography, extensive training in CD is not required for proficiency. In contrast to the extensive narrative of ethnography, the intermediate deliverables of CD are a set of highly formalized paper models of the work process that can be understood at a glance.

The initial aspects of CD involve three steps: data collection, modeling and consolidation. Data collection encompasses observation, interviewing and collection of artifacts. Contextual Inquiry (CI) interviews take place while anticipated users are performing their work. The process consists of watching the individual perform their work and periodically interrupting to discuss and clarify some aspect of work just performed. Typically, prewritten questions are not used. In order to keep the amount of data collected manageable and ensure the data is pertinent, the CD team chooses a focus prior to initiating the interviews. Similar to the focus in ethnography, the CI focus is a small set of themes that determines what aspects of the end-user’s work the interviews will emphasize. Ideally, 10 –20 CI interviews a re performed, with individuals who represent different roles within the work environment.

Modeling is the first step in the data analysis process. During a given interpretation session, the CD team works together to develop a set of five paper models as they work through notes, transcripts and/or videotape from a single CI interview. These paper models are formalized diagrams that depict a global picture of the work process. The five models are the flow, sequence, cultural, artifact and physical models.

The flow model (see figure 2) documents the communication and coordination involved in the work. Individuals and well-defined groups are represented by circles. Information sources and sinks are drawn as rectangles. Labeled arrows between individuals and groups show the directionality and content of information flow. Physical objects passed between parties are also represented and annotated with their purpose. The sequence model depicts the steps used to achieve the individual’s work. Sequences within the model are annotated with the intent of the sequence as well as the trigger that initiated the sequence. The sequence model reveals the individual’s strategy and intent. The artifact model consists of annotated photocopies or drawings of objects used in the work process, and could include paper documents, or screen captures of software used. The model reveals the structure, usage and intent of an object manipulated in the sequence model or passed in the flow model. The cultural model provides a representation of the cultural constraints placed on the worker. Individuals, groups or entire organizations are depicted as overlapping circles connected by appropriately labeled arrows representing the influence of one on the other. Constraints may be due to policy, personal values, organizational culture or other influences. The physical model shows the structure of the work environment as it impacts the individual’s work. Breakdowns in communication, coordination and operability of physical artifacts that interfere with the individual’s work are marked on any of the five models by red lightening bolts. In addition to these models, a running list of observations and design ideas is generated during the interpretation session.

Figure 2.

Flow Model of a single subject.

Consolidation is then used to aggregate data from multiple CI’s, resulting in five consolidated models. In addition, an affinity diagram is produced that distills and organizes the information in the observations and design ideas list, generated during individual interpretation sessions. Affinity diagramming is a categorization method, where numerous ideas are sorted into categories based on the natural relationship (affinity) between the ideas. The team thus organizes all ideas into themes, providing a bridge from CI data to design process.

METHODS

Defining focus

Three foci were selected for this pilot study: (1) student information management strategies, (2) integration of references and resources, and (3) collaboration.

Participants and case materials

For this pilot study we followed one PBL group of seven students and a preceptor, working on a single PBL case over a two-week interval (three classroom meetings). During this interval, we observed all classroom interactions (referred to as “group meetings”), and conducted two types of CI interviews with each student (Figure 1). In addition, we conducted interviews with the preceptor, course director and assistant course director, and a second faculty preceptor. Medical students were paid for their participation. The study design and use of human subjects were approved by the University of Connecticut Health Center Institutional Review Board.

Figure 1.

Timeline of work and CI interviews

Data Collection

We collected observational and interview data, as well as work process artifacts during group meetings and as students worked individually to prepare for PBL sessions. All CI videotapes or audiotapes were transcribed verbatim. A representative subset of CI interviews and group meetings were used in subsequent modeling.

CI of Individual student work

We anticipated that much of the “unseen work process” in PBL was occurring as students individually researched their cases prior to the PBL session. To capture these “unseen work process”, we interviewed students as they worked in the library, researching learning issues identified in the group, and accumulating supporting resources, and screen captured any interactions with the Health Library Information system, using Fantasia Studio ™ continuous screen-capture software. Transcribed text and Camtasia Studio ™ video were not generated for one of the seven individual work interviews due to technical difficulties.

Observation of Group meetings

Digital video of 3 group meetings was recorded. To limit obtrusiveness, the video equipment was placed in a far corner of the room and run without user intervention during the meeting. We retained copies of individual student concept maps, as well as other materials used by students during the case. Videotape was used to capture and preserve all use of the whiteboard for group concept maps and other collaborative purposes.

CI of Group meetings

CI requires that interviewers ask questions about work processes as they observe work being performed. In the case of PBL, a significant part of the work was performed during group interactions. Given the intensity of these interactions, it was impossible to conduct CI interviews during PBL sessions. Therefore, we modified the CI format–observing, videotaping and transcribing t he group processes, but then meeting with individual students soon after the group (1–4 days) to conduct CI as we played selected video clips from the previous group meeting to provide surrogate contextual references. The video clips were selected by the interviewer (MB) prior to the interview and were chosen to illustrate specific events relevant to the focus. Students were asked to interpret various events and group processes, and CI was conducted along the lines of student responses.

Faculty interviews

A semi-structured, standardized, interview format was used to interview the group preceptor, a second preceptor for a group that we did not follow, the course director and the associate course director. Pre-written questions followed the predefined foci.

Data analysis/modeling

Due to the volume of data generated, we analyzed only a subset of data for the purposes of this pilot study: (a) the second of three group meetings, (b) individual work interviews of 3 students, and (c) post group interviews of 2 students and the preceptor. The modeling team consisted of three of the four authors of this paper (VM, RC, and MB). RC and VM had prior experience performing CI and CD. Typically CD teams involve a larger number of individuals in the interpretation process.

For all data, we read verbatim transcripts aloud and in many cases watched relevant videotape. We progressed, usually a few transcript lines at a time, through each transcript-interpreting, discussing and reaching consensus about (a) whether to include the data in one of the five models, (b) which model to include it in, and (c) how to include it. Flow, sequence and cultural models for a given individual were generated in parallel. Camtasia Studio™ screen capture videos of the student searches were also used for the individual student modeling sessions and served as the main reference for developing the sequence models. Artifact models were made for two instructive examples. Physical models were not drawn due to time constraints. Individual transcript and line numbers were added for each item used in the models to point back to the original data from which it came. Maps were drawn in pencil on paper, and later converted to Microsoft® Visio® diagrams.

Affinity diagram

After all models were complete, we produced an affinity diagram to solidify issues, using the breakdowns, insights, design ideas, sequence events, cultural influences and individual roles identified during construction of the previous models. Each item was written on a post-it note. A first draft affinity diagram was produced according to the procedure described in Beyer and Holtzblatt ¹.

RESULTS

The complete corpus consists of 18 (14 student and 4 faculty) interviews resulting in 9994 total lines of transcribed text. The group meetings generated 196 total minutes of video. The observations of individual work generated 383 minutes of Camtasia Studio™ video. The pilot study used data from six of the 18 interviews and the second group meeting to generate 18 models, a summary affinity diagram and a list of insights, questions and design ideas. Interpretation sessions consumed approximately twelve and half hours in total. These 6 sessions represent interpretation of 3979 lines of transcription by three individuals, or 104.6 lines of transcription per person-hour. The models revealed 30 unique breakdowns. A total of 62 insights, 9 questions and 48 design ideas were recorded during the associated interpretation sessions.

The design ideas were typically generated after identifying a particular breakdown in the work process. For example, the students expressed numerous problems with using the whiteboard to generate group concept maps. The inability to efficiently edit complex maps was identified as one such obstacle. This breakdown led to a design idea for a collaborative concept-mapping tool using wireless technology and editable physical node objects, placed and manipulated on a digital whiteboard. Additional findings are outlined in Table 1.

Table 1.

Examples of Contextual Inquiry Findings

Information revealed by Contextual Inquiry1	Contextual Inquiry Foci for Study	Example Finding
Structure	Information Management	Consensus about starting point for concept map is critical for developing searching goals
	Knowledge Integration	Students don’t have access to references during discussion
	Collaboration	Although students were taught to use whiteboard during classroom time, the whiteboard is not always used
Breakdowns	Information Management	Students sometimes don’t have access to case or learning issues when searching for information
	Knowledge Integration	Whiteboard seen as place to record final knowledge about case, but not the working/in progress representation of the case
	Collaboration	Incorrect information is discussed during group meetings
Low-Level Details	Information Management	Cutting and pasting relevant text from resources loses the context and future access to the reference
	Knowledge Integration	Concept map often not returned to student
	Collaboration	Students run out of room for concept map on 8.5 x11 piece of paper

Figure 2 illustrates a flow model developed during an interpretation session of the transcripts from a single student. The model demonstrates that the student perceives their responsibility to include “gathering information” and “building concepts maps”. The diagram reflects the fact that t he student retrieves information almost exclusively from electronic resources. However, information gathering from electronic sources is hampered by difficulties reading from typical computer screens, requiring that all resources be printed. Nonetheless, although information is printed, it is not referred to during group meetings resulting in inaccurate statements based on remembered “facts”. In addition, all printed materials are eventually discarded. A design idea emergent from the model was to use a library-based collaborative information portal for storing and managing resources which would be accessible by students from tablet-based personal computers during group meetings as well as during their information gathering sessions. This design idea would not have been conceived without the detailed end-user data provided by CI.

The affinity diagram contained 349 individual elements. These consist of breakdowns, insights, design ideas, sequence events, cultural influences, and individual roles identified during construction of the models. The affinity established 54 different categories and the relationships between them, and served as an aggregation across models of significant design constraints that must be reflected in the software design process. As a result of this process we generated 48 design ideas which included broad design strategies for collaborative software including the library-based collaborative information portal for storing and managing resources, a wireless “gameboard” for creating concept maps with physical objects, and a PDA system for accessing/manipulating electronic resources among users.

DISCUSSION

The pilot study demonstrated the feasibility of this method, including adaptations that we made for this domain. Additional work is needed to model the remaining data, and to perform CIs with other PBL groups in order to develop consolidated models that express general emergent themes. An important advantage of the technique was that the interviewer required only minimal training and no prior practical experience in CI. The combination of a highly structured data analysis process, and the interpretation of the data by a team of individuals with prior CI experience, makes CI a practical alternative to traditional ethnographic study, as it can be used by software design groups without an ethnographer. Although the use of t his method is feasible, and provided extremely valuable data to the design team, there were some significant limitations:

Labor and Time Intensiveness

The acquisition and analysis of this data was time-consuming. Large amounts of raw data were generated and required analysis by multiple individuals working as a team. The time and labor costs of the method must be weighed against the usefulness of the data collected. CD may be best suited for domains where (a) it is unclear how to best use technology and there are few examples to guide design, (b) other kinds of studies (such as information needs assessments) are limited, and/or (c) the potential for failure without user acceptance is high. PBL is an example of a domain that fits these criteria.

Application in Healthcare Environments

Healthcare work is often not interruptible and thus not easily amenable to standard CI methods. Beyer and Holtzblatt¹ briefly discuss the use of video recordings as surrogate contextual references during CI interviews conducted after the work has been performed. Although cumbersome, we found this approach worked quite well and see no obvious reason such an approach would not work in clinical settings including hospital emergency rooms, operating rooms and outpatient medical clinics.

Access to Patient Information

However, another important aspect researchers must consider is the impact of new HIPAA legislation on data collection of this kind. Access to work processes in Medicine means access to patient identifiers. Audio-taping of CI interviews may only be possible where no patient identifiers are recorded (for example in educational domains) or where patient’s can be consented prior to incidental exposure of the study staff. These requirements may or may not extend to studies using CI with the purpose of Quality Assurance and improvement of supporting technologies and infrastructures.

CONCLUSIONS

Initial aspects of Contextual Design were successfully adapted to study a medical education domain and yielded important insights t hat will be integrated into the design of user-centered software for collaborative medical learning. The authors identified several potential limitations of this technique. The technique could be used in many other domains requiring user-centered design, for example development of new health information portals, laboratory information systems, or electronic medical record systems.