Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 May 23.
Published in final edited form as: Innov Pharm. 2012;3(3):88. doi: 10.24926/iip.v3i3.270

Using Think Aloud Protocols to Assess E-Prescribing in Community Pharmacies

Olufunmilola K Odukoya 1,✉, Michelle A Chui 2
PMCID: PMC4031678  NIHMSID: NIHMS448429  PMID: 24860689

Abstract

Introduction

Think aloud protocol has rarely been used as a method of data collection in community pharmacies.

Purpose

The aim of the report is to describe how think aloud protocols were used to identify issues that arise when using e-prescribing technology in pharmacies. In this paper, we report on the benefits and challenges of using think aloud protocols in pharmacies to examine the use of e-prescribing systems.

Methods

Sixteen pharmacists and pharmacy technicians were recruited from seven community pharmacies in Wisconsin. Data were collected using direct observation alongside think aloud protocol. Direct observations and think aloud protocols took place between January-February, 2011. Participants were asked to verbalize their thoughts as they process electronic prescriptions.

Results

Participants identify weaknesses in e-prescribing that they had previously not conceived. This created heightened awareness for vigilance when processing e-prescriptions. The main challenge with using think aloud protocols were interruptions in the pharmacy. Some participants found it difficult to remember to continue verbalizing during think aloud sessions.

Conclusion

The use of think aloud protocols as method of data collection is a new way for understanding the issues related to technology use in community pharmacy practice. Think aloud protocol was beneficial in providing objective information on e-prescribing not based on pharmacist’s or technician’s opinion of the technology. This method provided detailed information and also a wide variety of real time challenges with e-prescribing technology in community pharmacies. Using this data collection method can help identify potential patient safety issues when using e-prescribing and suggestions for redesign.

Keywords: Think aloud protocol, Electronic prescribing, Community pharmacy

INTRODUCTION

Healthcare settings are increasingly implementing health information technology (HIT) with different capabilities such as electronic prescribing (e-prescribing) functionalities.1,2 E-prescribing is the direct transmission of prescriptions from prescribers such as physicians and nurse practitioners to community pharmacies.3 E-prescribing systems are being used by clinics and pharmacies to improve the safety of the medication use process.4 Several studies have demonstrated the benefits and challenges of using e-prescribing in hospital settings. One widely reported benefit of using e-prescribing technology is the likelihood of reduction in medication errors associated with illegible handwritten prescriptions.5 On the contrary, e-prescribing use by physicians can result in problematic electronically received prescriptions for community pharmacies.6,7 Common problems with electronically received prescriptions include but are not limited to: - omission of information such as drug direction and drug dosing errors.8

There has been extensive research investigating physician use of e-prescribing systems.9,10 However, fewer studies have reported on community pharmacy personnel interactions with e-prescribing systems to ensure accurate dispensing of electronically received prescriptions (e-prescriptions). In order to understand how healthcare professionals interact with various types of HIT systems, patient safety experts have recommended the adoption of human factor engineering methods.11 Human factor engineering methods are frequently employed to identify causes of problems when using HIT and have been shown to be valuable in developing effective and practical solutions.11–13 Thus far, there is no known study that has employed human factors approaches to e-prescribing systems in community pharmacies. This paper reports on the benefits and challenges of using a human factors technique called think aloud protocol to assess the use of e-prescribing technology in community pharmacies.

Think aloud protocols

Think aloud protocol is a method used by human factors researchers to provide empirical and procedural information about work processes or tasks. This is done by having participants verbalize task performance procedures to yield insight into the cognitive components of the task.14–17 Think aloud protocols have been used to elicit information on the workflow challenges and information needs of pharmacy personnel when processing electronic prescriptions (e-prescriptions).18 One study also applied think aloud protocol analysis to evaluate the textual signals used by pharmacists for detection of adverse drug events.19 Results from the study showed that think aloud protocol were useful in understanding pharmacists’ information needs when processing medication orders.

OBJECTIVE

The aim of this paper is to describe the benefits and challenges of using think aloud protocols to examine use of e-prescribing systems from the perspective of community pharmacists and technicians.

METHOD

Participants

Sixteen pharmacists and pharmacy technicians were recruited from seven community pharmacies in Wisconsin. These pharmacy personnel were invited to participate in this study through the Pharmacy Society of Wisconsin. Pharmacies that used three common e-prescribing systems (PDX, QS/1, and PharmaServ) were recruited.

Data Collection

Data were collected using direct observation alongside think aloud protocol. Direct observations and think aloud protocols took place between January–February, 2011. The duration of observation ranged from two to five hours depending on frequency of receiving e-prescriptions. The researcher also observed general pharmacy workflow practices that pertained to use of e-prescriptions (e-prescribing workflow includes the time point from when an e-prescription is received in the pharmacy to the time the patient receives the medication).

During the think aloud protocol, participants were asked to verbalize their thoughts as they process e-prescriptions and to highlight the signals that help them to ensure safe medication dispensing practices. Each participant was observed processing at least five consecutive e-prescriptions. With every step in the dispensing process, the participant was asked to verbally state what they were thinking about, what information they needed to fulfill each step, what questions they had, and how they would proceed to the next step (a diagram summarizing steps of e-prescribing processing is provided in the results section). The responses of each participant were recorded and transcribed for analysis. Information was also collected by the researcher using field notes to contribute to the responses that were digitally recorded. In addition, specific instructions for the think aloud process are found in the appendices.

RESULTS

Participant Demographics and Pharmacy Characteristics

Participating pharmacies were a variety of owner/corporation models with different dispensing computer systems– three pharmacies with PDX, two pharmacies with PharmaServ, and two pharmacies with QS/1or QS/1NRx dispensing systems. Four of the pharmacies were independent pharmacies while three were chain pharmacies. The volume of e-prescriptions received daily by these pharmacies ranged from 66 to 320 e-prescriptions. Participating pharmacies had been using their pharmacy dispensing system to receive e-prescriptions for a range of six months to five years.

Sixteen participants took part the think aloud protocols and direct observations. Of all the sixteen individuals who participated, fifteen were female, six were pharmacists and ten were pharmacy technicians. All but one participant were Caucasians. There was a wide age range among the participants (25 to 49 years) with a mean age of 39.8 years. (Std. Dev = 7.6). A total of twenty-four new e-prescriptions were processed during think aloud protocols. The vast majority these e-prescriptions were for medications for chronic diseases. None of the e-prescriptions were for control substances because at the time of the study, controlled substances could not be e-prescribed.

Basic steps to conducting think aloud protocol

Step 1 – Explain purpose of using the method to the participant

Rationale: To indicate to participants that the researcher is trying to understand the process by which the technology is used rather than judging their performance

Step 1a – Explain to participants that it may be helpful to assume they are educating the researcher about the process or training the researcher about how to use the technology

Step 2 – The researcher provides a demonstration of thinking aloud while using a cellphone

Rationale: To help the participant apply the thinking aloud of a cell phone to thinking aloud when processing an e-prescription

Step 3 – The researcher put on tape recorder and ensures it is working

Step 4 – Keep reminding participants to think aloud

Rationale: Participants may forget to keep talking and explaining their tasks aloud, since this is not typical when performing the specific task

Step 5 – Repeat think aloud protocol at least five times to provide variability in cases or data obtained

In pharmacies where both pharmacists and technicians processed e-prescriptions, both were involved in the think aloud process. At least five e-prescriptions were processed in each pharmacy to provide a broad spectrum of real time e-prescriptions. Figure 1 depicts the basic steps for e-prescription processing20 which was created from the data obtained from think aloud protocols. The process for handling e-prescriptions was not always fixed but was sometimes dependent on the peculiarity of the e-prescribing system, the pharmacy busyness, and the person processing the e-prescription. Figure 2 is an example of how data from think-aloud was analyzed to understand how e-prescribing is used in community pharmacies.

Figure 1.

Figure 1

Open in a new tab

E-prescribing Processing in Community Pharmacies

Figure 2.

Figure 2

Open in a new tab

Data analysis of think aloud with a pharmacist

Benefits of using think aloud protocols

There were two main benefits to using think aloud protocols as a data collection method.

  1. Identify pharmacy staff information needs – Think aloud protocol was beneficial in providing objective information on e-prescribing not based on pharmacist’s or technician’s opinion of the technology. This method provided detailed information and also a wide variety of real time challenges with e-prescribing technology in community pharmacies. This method also helped to deduce information that pharmacy staff had to memorize to process e-prescriptions. One participant stated that the think aloud protocol was easy and was similar to training someone on processing an e-prescription.

  2. Identify real-time patient safety and workflow usability issues – During think aloud protocol, participants identified weaknesses in e-prescribing., the researcher was also able to observe real time errors and challenges of e-prescribing technology in pharmacies. Some participants recalled past errors as they were thinking aloud. This created heightened awareness for vigilance when processing e-prescriptions. The method also allowed the researcher to observe aspects of e-prescribing technology that pharmacy personnel may not fully comprehend and could lead to workflow issues in the pharmacy.

Challenges with using think aloud protocols

  1. Interruptions – A major challenge while conducting the think aloud protocol in the pharmacies were interruptions in the pharmacies. Such interruptions included the following: interruptions by patients coming into the pharmacy (with a baby crying), interruptions from co-workers needing to use the computer being used to process e-prescriptions, interruptions by intermittent phone calls from physicians or patients, issues related to the dispensing robot, and interruptions by other members of the pharmacy team. Interruptions occurred when participants had to attend to matters in the pharmacy that needed immediate attention. Think aloud sessions were also interrupted when participants needed to get medications out of the dispensing robot or get labels ready for filled medications. These interruptions resulted in the researcher having to frequently stop between think aloud sessions to allow for participants to perform their normal responsibilities in the pharmacy. Interruptions also occurred more frequently in pharmacies with limited staffing. In busy pharmacies background noise was a major challenge. Noise in the pharmacy came in the form of constant ringing of the pharmacy phone, noise from dispensing robots, or conversations in the pharmacies which affected the quality of think aloud recordings.

  2. Remembering to think aloud – Another challenge to efficiently collecting data through this method was that some participants found it difficult to remember to continue verbalizing their thoughts as they processed each e-prescription. The researcher had to keep reminding some participants to verbalize their thoughts out loud. One participant stated that although the think aloud protocol was not difficult it was difficult to think aloud and use the right terms while processing an e-prescription.

  3. Can Slows down pharmacy work - In busy pharmacies, performing the think aloud protocol slowed down the speed of processing the prescriptions. E-prescriptions volume also varied with the time of day and day of the week which sometimes resulted in long delays in obtaining five e-prescriptions for the think aloud process. Think aloud protocol was performed in the early afternoon, while in some pharmacies e-prescription volume was higher at later hours in the day. One participant stated that processing e-prescriptions while conducting the think aloud protocol slowed down pharmacy workflow. For pharmacies whereby pharmacists and technicians worked on different aspects of the e-prescriptions and processed a large number and a wide variety of e-prescriptions at different times, it was hard to follow through with one prescription.

DISCUSSION

The goal of using the think aloud protocol for data collection was to provide objective information on how pharmacists and pharmacy technicians interact with e-prescribing systems. This data method was chosen to identify information needs and reasoning strategies when handling e-prescriptions (effects of e-prescribing on pharmacists and pharmacy technicians work). This method was used to provide extensive verbal reports of pharmacists’ and technicians’ thought sequences and decision-making processes when processing e-prescriptions.

Think aloud has rarely been used in community pharmacy research. A major contribution of this study is that researchers looking to use this method for data collection can benefit from our experience of using this method to examine e-prescribing processes in community pharmacies. We found that using think aloud to understand technology use can help identify potential patient safety issues when using the technology and provide suggestions for modifying the design of the technology to improve safety and efficiency. When using think aloud to assess technology use it is important to select study participants that are familiar with the technology, that is, they use the technology frequently.

The use of this method in examining use of HIT has been shown to enhance the understanding how healthcare professionals interact with these systems.19 There are several benefits but also challenges to using think aloud protocols as a method of data collection in pharmacies. As reported in this, participants were typically faster after performing the think aloud process once. It was useful having more than one personnel in a pharmacy perform the think aloud process with e-prescribing as it showed differences or similarities in how the technology is used. In addition, it might be best to conduct think aloud in the pharmacy at times where there is likely to be minimum interruptions to ensure good quality data is audio recorded. One notable limitation of this study is that the frequency of challenges when using think aloud was not recorded.

CONCLUSION

As a qualitative study, this investigation produced rich, in depth knowledge about pharmacies use e-prescribing systems. However, we recommend that future use of think aloud protocols should involve a combination with other data collection methods to examine the reliability and validity of data obtained using think aloud protocols. The use of think aloud protocols a method of data collection is a new way for understanding the benefits and challenges technology use in community pharmacy practice.

Acknowledgements

The study was funded by a grant from the Community Pharmacy Foundation and by the Clinical and Translational Science Award (CTSA) program, previously through the National Center for Research Resources (NCRR) grant 1UL1RR025011, and now by the National Center for Advancing Translational Sciences (NCATS), grant 9U54TR000021. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH

Appendix A: Instruction Guide for Think Aloud Protocol

  1. Introduce the researcher and the study (title and importance)

  2. State goal of think aloud protocol

    For example - Think aloud protocol will allow the researchers better understand the process involved in filling an e-prescription. This may give insight to some of the difficulties encountered while filling an e-prescription

  3. Explain the think aloud protocol to the participant

    For example - The think aloud protocol will involve you thinking aloud as you process five e-prescriptions. This procedure will be audio taped. The recording will only be heard by the investigators of this study. The tapes will be kept only for the duration of the study. Please do not share patient specific information. I will record your verbalizations and might ask questions as you fill the e-prescriptions. I might occasionally remind you to please keep talking if you lapse into silence. Please explain in sufficient detail every step taken when processing an e-prescription.

I am not testing your ability to process an e-prescription but to describe how e-prescriptions are processed in various pharmacies. Please you can stop the task at any time you become uncomfortable. Please feel free to ask questions at any point in the process. I will not tell you when you have completed the task but you must determine this on your own.

I will practice the think aloud protocol to help you get familiarized with the process.

Do you have any questions about this process?

You can begin the process.

Please keep talking.

Thank you for participating in this process. Please can you provide any feedback on how you found this process? Thank you.

Footnotes

Disclosure

We declare no conflicts of interest or financial interests that the authors or members of their immediate families have in any product or service discussed in the manuscript, including grants (pending or received), employment, gifts, stock holdings or options, honoraria, consultancies, expert testimony, patents, and royalties.

Contributor Information

Olufunmilola K. Odukoya, University of Wisconsin, Department: Social & Administrative Sciences, School of Pharmacy; Postal address: 777 Highland Avenue, Madison Wisconsin 53705, USA; odukoya@wisc.edu, Telephone: 16086985054; Fax: 16082625262.

Michelle A. Chui, University of Wisconsin, Department: Social & Administrative Sciences, School of Pharmacy; Postal address: 777 Highland Avenue, Madison Wisconsin 53705, USA; mchui@pharmacy.wisc.edu, Telephone: 1608-262-0452; Fax: 16082625262.

REFERENCES

  • 1.Hersh W. Health care information technology: Progress and barriers. JAMA. 2004;292(18):2273–2274. doi: 10.1001/jama.292.18.2273. [DOI] [PubMed] [Google Scholar]
  • 2.Siska MH, Tribble DA. Opportunities and challenges related to technology in supporting optimal pharmacy practice models in hospitals and health systems. Am J Health Syst Pharm. 2011;68(12):1116–1126. doi: 10.2146/ajhp110059. [DOI] [PubMed] [Google Scholar]
  • 3.Moniz TT, Seger AC, Keohane CA, Seger DL, Bates DW, Rothschild JM. Addition of electronic prescription transmission to computerized prescriber order entry: Effect on dispensing errors in community pharmacies. Am J Health Syst Pharm. 2011;68(2):158–163. doi: 10.2146/ajhp080298. [DOI] [PubMed] [Google Scholar]
  • 4.Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: A review of the recent literature shows predominantly positive results. Health Aff (Millwood) 2011;30(3):464–471. doi: 10.1377/hlthaff.2011.0178. [DOI] [PubMed] [Google Scholar]
  • 5.Kaushal R, Kern LM, Barron Y, Quaresimo J, Abramson EL. Electronic prescribing improves medication safety in community-based office practices. J Gen Intern Med. 2010;25(6):530–536. doi: 10.1007/s11606-009-1238-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Gilligan AM, Miller K, Mohney A, Montenegro C, Schwarz J, Warholak TL. Analysis of pharmacists’ interventions on electronic versus traditional prescriptions in 2 community pharmacies. Research in Social and Administrative Pharmacy. 2012 doi: 10.1016/j.sapharm.2011.12.005. [DOI] [PubMed] [Google Scholar]
  • 7.Hincapie AL. e-Prescribing/Patient safety analysis utilizing the pharmacy and prescriber e-prescribing experience reporting (PEER) portal. 2011 [Google Scholar]
  • 8.Warholak TL, Rupp MT. Analysis of community chain pharmacists' interventions on electronic prescriptions. J Am Pharm Assoc (2003) 2009;49(1):59–64. doi: 10.1331/JAPhA.2009.08013. [DOI] [PubMed] [Google Scholar]
  • 9.Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. The effect of electronic prescribing on medication errors and adverse drug events: A systematic review. J Am Med Inform Assoc. 2008;15(5):585–600. doi: 10.1197/jamia.M2667. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: A systematic review. Arch Intern Med. 2003;163(12):1409–1416. doi: 10.1001/archinte.163.12.1409. [DOI] [PubMed] [Google Scholar]
  • 11.Gurses AP, Ozok AA, Pronovost PJ. Time to accelerate integration of human factors and ergonomics in patient safety. BMJ Quality & Safety. 2011 doi: 10.1136/bmjqs-2011-000421. [DOI] [PubMed] [Google Scholar]
  • 12.Carayon P. Human factors in patient safety as an innovation. Appl Ergon. 2010;41(5):657–665. doi: 10.1016/j.apergo.2009.12.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Carayon P. Human factors and ergonomics in health care and patient safety. In: Carayon P, editor. Handbook of human factors and ergonomics in healthcare and patient safety. Mahwah, NJ: 2006. pp. 699–700. [Google Scholar]
  • 14.Lundgren-Laine H, Salantera S. Think-aloud technique and protocol analysis in clinical decision-making research. Qual Health Res. 2010;20(4):565–575. doi: 10.1177/1049732309354278. [DOI] [PubMed] [Google Scholar]
  • 15.Khajouei R, Hasman A, Jaspers MW. Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system. Int J Med Inform. 2011;80(5):341–350. doi: 10.1016/j.ijmedinf.2011.02.005. [DOI] [PubMed] [Google Scholar]
  • 16.Fairbanks RJ, Caplan SH, Bishop PA, Marks AM, Shah MN. Usability study of two common defibrillators reveals hazards. Ann Emerg Med. 2007;50(4):424–432. doi: 10.1016/j.annemergmed.2007.03.029. [DOI] [PubMed] [Google Scholar]
  • 17.Wu RC, Orr MS, Chignell M, Straus SE. Usability of a mobile electronic medical record prototype: A verbal protocol analysis. Inform Health Soc Care. 2008;33(2):139–149. doi: 10.1080/17538150802127223. [DOI] [PubMed] [Google Scholar]
  • 18.Odukoya OK, Chui MA. Retail pharmacy staff perceptions of design strengths and weaknesses of electronic prescribing. Journal of the American Medical Informatics Association. 2012 doi: 10.1136/amiajnl-2011-000779. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Phansalkar S, Hoffman JM, Hurdle JF, Patel VL. Understanding pharmacist decision making for adverse drug event (ADE) detection. J Eval Clin Pract. 2009;15(2):266–275. doi: 10.1111/j.1365-2753.2008.00992.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Odukoya OK, Chui MA. Exploring the relationship between processing E-prescriptions and community pharmacy workflow. Journal of American Pharmacists Association, JAPhA. 2012 doi: 10.1331/JAPhA.2012.12066. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES