Skip to main content
PMC home page
AMIA Annual Symposium Proceedings logoLink to AMIA Annual Symposium Proceedings
. 2012 Nov 3;2012:1168–1175.

A Mixed Methods Approach for Measuring the Impact of Delivery-Centric Interventions on Clinician Workflow

Rhonda G Cady 1, Stanley M Finkelstein 1
PMCID: PMC3540554  PMID: 23304393

Abstract

Health interventions vary widely. Pharmaceuticals, medical devices and wellness promotion are defined as ‘outcome-centric.’ They are implemented by clinicians for the use and benefit of consumers, and intervention effectiveness is measured by a change in health outcome. Electronic health records, computerized physician order entry systems and telehealth technologies are defined as ‘delivery-centric.’ They are implemented by organizations for use by clinicians to manage and facilitate consumer health, and the impact of these interventions on clinician workflow has become increasingly important. The methodological framework introduced in this paper uses a two-phase sequential mixed methods design that qualitatively explores clinician workflow before and after implementation of a delivery-centric intervention, and uses this information to quantitatively measure changes to workflow activities. The mixed methods protocol provides a standardized approach for understanding and determining the impact of delivery-centric interventions on clinician workflow.

Introduction

Health interventions range from wellness promotion to electronic health records. While the primary motivation behind the development of these interventions is improved health outcomes, the person implementing and using the health intervention varies. Items such as pharmaceuticals, medical devices and wellness promotion are implemented by clinicians for the use and benefit of consumers, and intervention effectiveness is measured by a change in health outcome. These interventions are defined as ‘outcome-centric.’ Electronic health records, computerized physician order entry systems and telehealth technologies are implemented by organizations for use by clinicians to manage and facilitate consumer health. These interventions are defined as ‘delivery-centric’ because clinician interaction with these interventions during health care delivery is frequent and often mandatory, and the impact on clinician workflow has become increasingly important.1

Qualitative ethnography and quantitative time-motion study have been used separately to analyze the impact of delivery-centric interventions on clinician workflow.2,3 The primary purpose of this paper is to describe a novel mixed methods protocol that combines these methodologies to measure the impact of a delivery-centric intervention on clinician workflow. The aims of the workflow analysis are achieved using a two phase sequential mixed methods protocol that explores the workflow of clinicians before and after the implementation of a delivery-centric intervention, and uses this information to measure the efficiency and describe the consequences of adding the intervention to the workflow. Results from a workflow analysis of pediatric triage nurses are highlighted to demonstrate the usefulness of the mixed methods protocol.

Background

The gold-standard for health intervention research is the randomized controlled trial (RCT),4 but applying the methodology to delivery-centric interventions can be difficult. Randomization is rarely possible due to business unit/organizational implementation schedules, and statistical methods based on a randomized, normally distributed population are no longer valid. While RCTs are good at measuring pre-determined outcomes of delivery-centric intervention performance, “they are not well suited to answering questions concerning whether systems will be used or how they will be used.”5 More importantly, positive RCT results from narrowly-defined boundaries may not generalize to complex delivery systems,6 while negative RCT results provide little context for understanding why the intervention lacked a significant effect.7

Newly released Best Practices for Mixed Methods Research in the Health Sciences8 from the Office of Behavioral and Social Sciences Research of the National Institutes of Health indicates the growing need for and acceptance of methodological diversity. The primary strength of a mixed methods design comes from incorporating qualitative and quantitative components that offset the weaknesses of one methodology with the strengths from the other.9 A quantitative strand may be considered weak in understanding the context of the research question whereas a qualitative strand is inherently influenced by investigator interpretation and bias, and lacks generalizable results. Furthermore, the ability to corroborate findings across the strands strengthens the validity of results,10 and provides a rigorous answer to research questions not suited to a single methodology.8

Complex adaptive systems (CAS) rely on interrelationships between individuals, use distributed rather than centralized control, and display unpredictable behavior.11 Health care organizations have been portrayed as CAS,12 and embedding delivery-centric interventions within this system can lead to unpredictable and unintended consequences.13 The socio-technical viewpoint takes a collaborative and cooperative perspective of work that is prone to exceptions, and evaluation focuses on the degree to which technology supports a person in accomplishing their work.14 Since health care is a complex adaptive system dependent on collaborative and cooperative work, workflow analysis must employ a socio-technical viewpoint to capture the collaboration-dependent variables. The Interactive SocioTechnical Analysis (ITSA) Framework provides a basis for evaluating the impact of a delivery-centric intervention on these variables.15

The theory of distributed cognition provides a mechanism for understanding the people, activities, and artifacts of a collaborative work process.16 A key component of distributed cognition is identifying the interactions between artifacts and people. Artifacts are “an artificial device designed to maintain, display or operate upon information in order to serve a representational function.”17 Examples include paper and electronic records, e-mail, voice mail, telephone, fax and video. Artifacts do not directly change a person’s capabilities, but they do “change the nature of the activity performed”17 and can increase or decrease a person’s overall performance. In addition, artifacts can distribute actions across time, across people, and/or change the actions required to perform the activity. The primary methodology for studying distributed cognitive systems is a distributed cognitive activity analysis utilizing cognitive ethnography.18 Since technology is an artifact, incorporating distributed cognitive theory into a mixed methods workflow analysis that espouses a socio-technical viewpoint, provides the foundation for evaluating the impact of a delivery-centric intervention on clinician workflow.

Methods

A workflow analysis of triage nurses working in a large urban General Pediatrics Clinic is conducted using a novel mixed methods protocol that qualitatively explores nurse workflow before and after implementation of a delivery-centric intervention and uses this information to quantitatively measure changes to workflow (Fig 1). Qualitative data is collected via observation, semi-structured interview, and documentary analysis and analyzed within the theory of distributed cognition to identify the activity and interaction themes of the workflow. Qualitative themes are developed into a time-motion workflow activity list that is used during the quantitative strand of the mixed methods design. The time-motion study measures the efficiency of the before and after workflows. Efficiency is operationalized as the timely delivery of services and a process measure of time provides a direct calculation that allows comparison of the workflows. The consequences of the delivery-centric intervention are operationalized as changes to workflow that occur after implementation. Qualitative and quantitative results are merged and integrated8 within the Interactive SocioTechnical Analysis (ITSA) Framework15 to provide an interpretive analysis of the positive and negative consequences of the delivery-centric intervention on clinician workflow.

Figure 1:

Figure 1:

Open in a new tab

Research Methodology Protocol

The first component of the mixed methods workflow analysis is a cognitive ethnography that strives to understand the impact of a delivery-centric intervention of clinician workflow (Fig 2). Data collection during the ethnographic component focuses on workflow before and after implementation of the delivery-centric intervention. The qualitative analysis relies on two basic strategies: “asking questions and making comparisons.”19 Asking questions is accomplished with a process analysis that focuses on the “who, what, when, where and how”19 of workflow, resulting in process maps that describe ‘before and after’ workflows. Comparisons are made using a directed content analysis grounded in the theory of distributed cognition,16 with the purpose of extending the distributed cognition framework20 to the clinical workflow. The overall rigor of the ethnographic component of the workflow analysis is enhanced by the mixed methods protocol, where mixing qualitative and quantitative methodologies offsets the weaknesses of one method with the strengths of the other and allows corroboration of findings across methods.9

Figure 2:

Figure 2:

Open in a new tab

Cognitive Ethnography Methodology

The second component of the mixed methods workflow analysis connects the qualitative and quantitative components. Connecting data sources is an integral component of mixed methods design and in this protocol, they are interactively mixed. The categories and subcategories emerging from the directed content analysis of ethnographic data become the basis for the activities, persons and artifacts of the workflow activity list (before and after implementation). Validity of the workflow activity list is established across three domains: content, concurrent and convergent. The activity list is the data collection tool used during the time-motion study.

The third component of the mixed methods workflow analysis utilizes time-motion study to determine whether a difference in efficiency exists between the ‘before and after’ implementation workflows. Data collection during the time-motion study utilizes the validated workflow activity list developed in the connecting data sources component, to measure workflow activities before and after implementation of the delivery-centric intervention. At the start of each data collection session, the researcher notes the informant being observed and snaps-back a digital stop-clock to zero. As the informant moves from one activity to another, four pieces of data are coded: 1) time from the stop-clock; 2) activity observed; 3) person(s) interacted with; 4) artifact(s) interacted with. The data from the coded workflow observations comprise the sample used in each time-motion study.

Research questions are examined using summary statistics and descriptive data analyses and research hypotheses are tested with traditional and non-parametric statistical analyses. Primary observer reliability testing is conducted to ensure accurate coding of the observation sample. Inter-observer agreement and reliability are determined using the Kappa statistic and intraclass correlations. Between-group comparison using the nonparametric Wilcoxon rank sum tests for a difference in the time required to conduct workflow activities before and after implementation of the delivery-centric intervention. Chi-square analysis tests whether an association exists between the activities and interactions (persons and artifacts) of workflow. Results from the time-motion study are corroborated with ethnographic findings, strengthening the overall rigor of the protocol.

The final step of the mixed methods workflow analysis merges results from the quantitative and qualitative strands for an interpretive analysis of the impact of a delivery-centric intervention on clinician workflow. Questions about the positive and negative consequences of the intervention are analyzed using the five dimensions of the Interactive SocioTechnical Analysis Framework (Fig 3).15 The first dimension, HIT Changes Existing Social System, describes how new technology can alter the “patterns of work, communication, or relationships among clinicians.”15 The second dimension, Technical and Physical Infrastructures mediate HIT Use, describes the consequences of a mismatch between new technology and organizational infrastructure.15 The third dimension, Social System Mediates HIT Use, describes how the actual use of new technology differs from the intended use.15 The fourth dimension, HIT-In-Use Changes Social System, describes how the actual use of new technology leads to changes in the organizations social system.15 The final dimension, HIT – Social System Interactions Engender HIT Redesign, capture the unchecked consequences when technology use becomes mandatory.15

Figure 3:

Figure 3:

Open in a new tab

Merging Qualitative and Quantitative Strands of Workflow Analysis

Results

A workflow analysis was conducted in the triage office of a large urban General Pediatrics Clinic affiliated with a free-standing, non-profit children’s hospital using the mixed methods protocol The delivery-centric intervention measured by the mixed methods workflow analysis protocol was video triage. The workflow analysis protocol was approved by the Institutional Review Boards of the researcher’s institution and the General Pediatrics Clinic.

Only two registered nurses work in the triage office and written informed consent was obtained from both nurses. While triage nurses are the primary informants, their workflow involves health-related interactions with parents and children, resulting in a secondary group of subjects, parent/child informants. A total of eleven parent/child informants provided written informed consent that allowed observation of nurse-parent-child interactions during video triage and documentary analysis of the child’s EMR.

A cognitive ethnography of triage nurse workflow was conducted before and after the implementation of video triage, a new delivery-centric intervention. Data collected during 32 hours of direct observation identified the activities and interactions of triage conducted by telephone, and triage conducted by video. Video triage changes work processes by adding new activities and interactions to triage workflow. The interactions added by video triage facilitated verbal and visual communication between nurse, parent and child, and allowed additional data collection during assessment. The activities added by video triage, troubleshooting video session and preparing for video session, were the result of institutional network architecture incompatibilities and a new communication (video) platform. The activities and interactions of telephone and video triage were validated by additional observation and became the basis for the time-motion data collection tool.

A time-motion study of the same triage nurses performing telephone and video triage found that the time spent on the activities of video triage was significantly longer (p=0.0002) than the time spent on the activities of telephone triage. This finding suggests that video triage decreases nurse efficiency. Merging qualitative and quantitative findings within the ITSA framework provides context to these results. Triage nurses state video is useful because it is often the first ‘face-to-face’ interaction with both parent and child and more importantly, visual communication increases the depth and breadth of data available for assessment, which increases the nurse’s confidence in triage assessment.

Conclusion

Prior workflow research has employed a mixed methods protocol, with qualitative data collection and analysis informing quantitative data collection, typically in the form of survey or interview questions.21 The primary users of delivery-centric inventions are clinicians who are experts in their respective fields, recognizing patterns or situations, acting from past experience and problem solving when needed.22 It is often difficult for an expert to explain their ‘expertise’ since many elements of the process are performed automatically or sub-consciously.23 This could result in an incomplete description of workflow.

The workflow analysis described in this paper uses a mixed methods protocol to develop a customized workflow activity list developed via cognitive ethnography rather than a pre-existing or expert-generated workflow activity list. Health care is complex, collaborative and unpredictable. The workflow relies on problem solving, information flow and the coordination of multiple activities, people and things. Pre-existing workflow activity lists may not capture this data and expert-generated workflow activity lists may inadvertently omit activities and/or interactions. Content analysis of ethnographic data using categories from the theory of distributed cognition identifies specific workflow activities and interactions. Conducting content analysis within a theoretical framework allows customized workflow activities to be grouped within consistent categories that support comparative meta-analysis.

This research has limitations that could impact the validity of the results. The triage office is staffed by only two nurses. This is not a limitation to the consistency of qualitative strand of the mixed methods protocol since data collection focuses on purposeful observations from key informants.24 However, the same sample of triage nurses is used for quantitative data collection. While this limits the generalizability of the results to other populations, it does not threaten the validity of this research, since the results of a workflow analysis are primarily intended for the population tested. The sequential design of the mixed methods protocol could limit the interpretive analysis if one methodology is emphasized over the other.9 Merging qualitative and quantitative findings within the ITSA framework minimize this limitation.

Proper evaluation of workflow before and after implementation of a delivery-centric intervention requires tools that reflect this type of workflow. A cognitive ethnography framed within the theory of distributed cognition generates the requisite tools: a time-motion workflow activity list and workflow process maps. Because both tools are created using established frameworks (theory of distributed cognition and statistical process control) comparison of the results from workflow analysis utilizing these tools in a consistent manner is possible. Time-motion study quantitatively measures the impact of the delivery-centric intervention on work activity time. Use of the methodology to evaluate other delivery-centric interventions is a necessary step to ensure transferability of the protocol and support comparative meta-analysis. Knowledge generated by meta-analysis of workflow studies utilizing the mixed methods protocol provides evidence as to how and why a delivery-centric intervention impacts clinician workflow and advances the overall efficiency of health care delivery.

References

  • 1.Lapointe L, Mignerat M, Vedel I. The IT productivity paradox in health: A stakeholder’s perspective. Int J Med Inf. 2011;80(2):102–115. doi: 10.1016/j.ijmedinf.2010.11.004. [DOI] [PubMed] [Google Scholar]
  • 2.Peute LW, Aarts J, Bakker PJ, Jaspers MW. Anatomy of a failure: A sociotechnical evaluation of a laboratory physician order entry system implementation. Int J Med Inf. 2010;79(4):e58–70. doi: 10.1016/j.ijmedinf.2009.06.008. [DOI] [PubMed] [Google Scholar]
  • 3.Hollingworth W, Devine EB, Hansen RN, et al. The impact of e-prescribing on prescriber and staff time in ambulatory care clinics: A time motion study. J Am Med Inform Assoc. 2007;14(6):722–730. doi: 10.1197/jamia.M2377. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Meldrum ML. A brief history of the randomized controlled trial. from oranges and lemons to the gold standard. Hematol Oncol Clin North Am. 2000;14(4):745–60. doi: 10.1016/s0889-8588(05)70309-9. [DOI] [PubMed] [Google Scholar]
  • 5.Kaplan B, Brennan PF, Dowling AF, Friedman CP, Peel V. Toward an informatics research agenda: Key people and organizational issues. Journal of the American Medical Informatics Association. 2001;8(3):235–241. doi: 10.1136/jamia.2001.0080235. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Delpierre C, Cuzin L, Fillaux J, Alvarez M, Massip P, Lang T. A systematic review of computer-based patient record systems and quality of care: More randomized clinical trials or a broader approach? International Journal for Quality in Health Care. 2004;16(5):407–416. doi: 10.1093/intqhc/mzh064. [DOI] [PubMed] [Google Scholar]
  • 7.Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: Barriers and challenges. BMJ. 1998;316(7149):1959–1961. doi: 10.1136/bmj.316.7149.1959. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.NIH office of behavioral and social sciences research (OBSSR) - mixed methods research index http://obssr.od.nih.gov/scientific_areas/methodology/mixed_methods_research/index.aspx. Accessed 10/11/2011, 2011
  • 9.Creswell JW. Designing and conducting mixed methods research. Los Angeles: SAGE Publications; 2011. Los Angeles. [Google Scholar]
  • 10.Bryman A. Integrating quantitative and qualitative research: How is it done? Qualitative Research. 2006;6(1):97–113. [Google Scholar]
  • 11.Holland JH. Adaptation in natural and artificial systems : An introductory analysis with applications to biology, control, and artificial intelligence. 1st MIT Press ed ed. Cambridge, Mass: MIT Press; 1992. p. 211. [Google Scholar]
  • 12.Weaver CA, editor. Nursing and informatics for the 21st century : An international look at practice, trends and the future. Chicago: Healthcare Information and Management Systems Society; 2006. [Google Scholar]
  • 13.Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: The nature of patient care information system-related errors. Journal of the American Medical Informatics Association. 2004;11(2):104–112. doi: 10.1197/jamia.M1471. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Protti D. A proposal to use a balanced scorecard to evaluate information for health: An information strategy for the modern NHS (1998–2005) Computers in Biology and Medicine. 2002;32(3):221–236. doi: 10.1016/s0010-4825(02)00017-3. [DOI] [PubMed] [Google Scholar]
  • 15.Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health care an interactive sociotechnical analysis. J Am Med Inform Assoc. 2007;14(5):542–549. doi: 10.1197/jamia.M2384. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Hollan J, Hutchins E, Kirsh D. Distributed cognition: Toward a new foundation for human-computer interaction research. ACM Trans Comput -Hum Interact. 2000;7(2):174–196. [Google Scholar]
  • 17.Carroll JM, editor. Designing interaction : Psychology at the human-computer interface. Cambridge ; New York: Cambridge University Press; 1991. (Cambridge series on human-computer interaction ; 3) [Google Scholar]
  • 18.Ball LJ, Ormerod TC. Putting ethnography to work: The case for a cognitive ethnography of design. International Journal of Human-Computer Studies. 2000;53(1):147–168. [Google Scholar]
  • 19.Corbin JM. Basics of qualitative research : Techniques and procedures for developing grounded theory. 3rd ed. Los Angeles, Calif: Sage Publications, Inc; 1942. 2008. Los Angeles, Calif. [Google Scholar]
  • 20.Hsieh H, Shannon SE. Three approaches to qualitative content analysis. Qualitative Health Research} 2005;15(9):1277–1288. doi: 10.1177/1049732305276687. [DOI] [PubMed] [Google Scholar]
  • 21.Sittig DF, Ash JS, Guappone KP, Campbell EM, Dykstra RH. Assessing the anticipated consequences of computer-based provider order entry at three community hospitals using an open-ended, semi-structured survey instrument. Int J Med Inf. 2008;77(7):440–447. doi: 10.1016/j.ijmedinf.2007.08.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Dreyfus HL, Dreyfus SE. Frictionless forecasting is a fiction. In: Åkerman N, editor. The necessity of friction. Boulder, Colo.: WestviewPress; 1998. Boulder, Colo. [Google Scholar]
  • 23.Bereiter C. Surpassing ourselves : An inquiry into the nature and implications of expertise. Chicago: Open Court; 1993. Chicago. [Google Scholar]
  • 24.Muecke MA. On the evaluation of ethnographies. In: Morse JM, editor. Critical issues in qualitative research methods. Thousand Oaks, Calif: Sage Publications, Inc; 1994. pp. 187–209. [Google Scholar]

Articles from AMIA Annual Symposium Proceedings are provided here courtesy of American Medical Informatics Association

RESOURCES