Abstract
The objective of this study is to assess the initial perception of referring providers and radiologists to a virtual consultation solution (the Virtual Radiology Reading Room, VR3). VR3 is specifically designed to replace the radiology reading room phone and enable the radiologist to continue to work as part of the clinical care team fielding radiology consults without being confined to the same physical space. Surveys of providers’ and radiologists’ initial experience were conducted approximately 6 months after initial deployment. Users were asked about their overall impression, and how well the solution integrates with their workflow as well as how it compares to traditional phone calls to the reading room. Forty of 71 referring providers and 27 of 44 radiologists responded to our survey. VR3 was rated 4.7 out of 5 stars by referrers and 4.1 by radiologists. Seventy percent of referrers and radiologists preferred VR3 to the phone while 4.5% of referrers and 11% of radiologists preferred the phone. Referring providers and radiologists expressed a positive initial perception of the Virtual Radiology Reading Room and prefer it to traditional phone calls to the reading room.
Keywords: Consultation, Reading room, Communication, Virtual, Value, Asynchronous
Introduction
The role of the radiologist as an imaging consultant, always available for consultation with referring providers, is a critical role emphasized by the leading radiology societies [1–3]. Prime among the goals of the American College of Radiology Imaging 3.0™ initiative is to “support radiologists transformation into the diagnostic imaging consultant to all clinicians, both before and after the imaging encounter.”[4]. With the transition to PACS about 3 decades ago, radiology consultations have shifted from being mostly in-person to being conducted primarily through the reading room phone [5]. Phone calls coming to the reading room are a major source of interruptions [6, 7] which has led some institutions to employ reading room assistants to help manage the chaos [6]. We developed a virtual radiology consulting service called the “Virtual Radiology Reading Room” (VR3) that expands the traditional reading room experience beyond the confines of a physical space leveraging modern communication tools such as Microsoft Teams to replace the reading room phone. Referrers submit consults via a simple web app to a specific reading room. In phase 1 of the project that was surveyed in this study, the webapp interface allowed the sender to select a callback method (MS Teams vs phone number), include patient information (optional), include a free-text question (optional), mark the consult urgent if desired (optional), and select a reading room (e.g., Body, Neuro, MSK). The sender’s name and email are included with the consult automatically. We allowed senders to provide a phone number for callback instead of MS Teams on a temporary basis to encourage adoption by providers with the intention of removing that option in future phases. The consults are routed to a group of radiologists covering the specified reading room that are not confined to the same location and may be working in dispersed onsite locations, remotely from home, or at an off-site outpatient center. Via VR3, this group of radiologists can continue working together as a team fielding incoming consults. The radiologists are notified of the consults on MS Teams which is integrated into their workstations. One of the radiologists claims the consult and responds to the consult via MS Teams where they can utilize voice call, text, screen share, and/or video as they see best (or via phone if sender selected phone callback). Once one radiologist claims the consult, the consult is no longer visible to the rest of the team. In phase 1 of the project, there was no integration with PACS, so the radiologist would search the patient manually in PACS if needed. A pilot of phase 1 of this solution was implemented in our institution starting May 2021. The tool was made available to all providers in our hospital, but we incrementally publicized it to specific departments within our hospitals. In this study, we assessed the initial perception of referring providers and radiologists to the Virtual Radiology Reading Room.
Methods
Surveys of providers and radiologists of the VR3 initial experience were conducted. The survey was sent to all VR3 users among providers at 4 months and 8 months after initial deployment with a single reminder within 1 week for non-responders; each user was surveyed only once. The survey was sent to all VR3 users among radiologists at 6 months with a single reminder within 1 week for non-responders. Referring providers were asked about their overall impression, the reliability of VR3, the efficiency of VR3, and how well it integrates with their workflow. They were then asked how VR3 compares to traditional phone calls and how easy it was to access the platform. Questions were answered on a 5-point Likert scale. Additionally, referring providers were asked a series of free-text response questions, including what they would like to see added to the application and the expected turnaround time for “standard” and “urgent” consults. Referring providers were further divided into two subgroups: those that used VR3 at least three times (group A) and those that used the application only once or twice (group B). For group B, we asked additional questions to evaluate the causes of infrequent use. We provided possible reasons for infrequent use and asked for responses to rate each on a 5-point Likert scale [8]. The reasons provided were “No occasion to use it again”, “I forgot about it”, “ I am just more used to the phone”, “VR3 is not easily accessible when I need it most”, “VR3 is cumbersome to use”, “I don’t want to wait for a response”, “Most times I prefer to come to the reading room in person”, and “Other reason”.
Radiologists were given a survey of their initial experience as well. They were also asked on a 5-point Likert scale about their overall impression, ease of use, how well it integrates with their workflow, and how it compares to traditional phone calls. Additional questions were asked to evaluate the value of VR3, including whether VR3 contributes positively to patient care, whether VR3 contributes positively to the perceived role of radiology, and whether VR3 allows the role of the radiologist to be quantified and valued. Radiologists were also asked a series of free-text response questions, including what they would like to see added to the application and the expected turnaround time of “standard” and “urgent” consults.
Results
Seventy-one unique referring providers used VR3 to send approximately 200 consults in the first 8 months of deployment. Of those, 40 referring providers (56%) responded to our survey. Thirty-eight of 40 (95%) referring providers had a favorable initial overall impression and gave VR3 at least 4 out of 5 stars (average rating 4.7). Responding referring providers agreed or strongly agreed that VR3 is reliable (90%), efficient (92.5%), and integrates well with workflow (87.5%). Compared to contacting the reading room via traditional phones, 70% of responding referring providers felt that VR3 is better or much better and only 4.5% felt that the phone was better or much better. Responding referring providers also agreed (defined as providing a score of 4 or 5 on a 5-point scale) that VR3 was easy to use (85%). Figure 1 depicts referring providers’ responses. The top three reasons that occasional users (group B) did not use VR3 more frequently were “I am just more used to using the phone” (55%), “no occasion to use it again” (48%), and “I forgot about it” (45%). The least common reason group B did not use VR3 more frequently was that “VR3 is cumbersome to use” (10%). Expectations for turnaround time were provided as free-text answers and have therefore widely varied from 5 min to 48 h for standard consults with a major peak around < 30 min and a second minor peak around “same business day”. For “urgent” consult, responses ranged from 0 min to “same business day” with a major peak around < 5 min and a second minor peak around 1 h. Figure 2 depicts referring providers’ responses to the expected turnaround time for “standard” and “urgent” consults. Table 1 shows free-text responses from referring providers, including what they want to see added to the application.
Fig. 1.
Referrer responses to VR3 initial perception survey questions on a Likert scale from 1 to 5 with 5 being "strongly agree" or the most positive response
Fig. 2.
Referring providers’ expected turnaround times for "Standard" and "Urgent" consults as depicted by frequency charts
Table 1.
Referring providers free-text responses
| “I used this service twice and found it very helpful…I think the benefit of a tool like this is that we don’t interrupt the radiologists’ workflow and they may have more time to review the images before speaking to us.” |
| “Love the easy access, although I enjoy bringing the team down for in person-radiology discussion, I hope this doesn’t take away from that opportunity.” |
| “On the form, it might be helpful to have a separate “Submit” button to send the request instead of clicking on the location to send the request as the sending function. I have accidentally sent a request before the form was completely filled out because I selected a location.” |
| “The one time I tried it, I waited quite a while but never got a response and ended up calling anyway. However, I’m willing to try again next time I need to discuss scans. Honestly, forgot about this option until recently and will try this prior to calling.” |
| “The one time the radiologist contacted me to review images with me it worked great. The one time I used it to request a consult, nobody responded. This was a while ago and it was cumbersome so much so that I forgot how to access it. If it is more user friendly, I would use it.” |
| “As the need is relatively infrequent, I always seem to forget how to access it precisely when I need it, once I remember it is user friendly.” |
| “Excellent resource!” |
| “Although I’ve only used it twice, I felt that the response time was excellent and often was easier to use than calling down to the reading room.” |
| “I’ve found it incredibly helpful when I am contouring a challenging case (I am in radiation oncology).” |
| “I was surprised how easy it was and how quickly I received a response. Great work!” |
| “I have really enjoyed using this. It is easy to use and the response time has been great.” |
Forty-four unique radiologists used VR3 to respond to approximately 140 consults at the time radiologists’ survey was created at 6 months of deployment. Of those, 27 (61%) radiologists responded to our survey. Twenty-one of 27 (78%) had a favorable initial overall impression and gave VR3 at least 4 out of 5 stars (average rating 4.1). Responding radiologists agreed or strongly agreed that VR3 was easy and intuitive (81%), integrates well with workflow (63%), and is less disruptive than phones (67%). Seventy percent felt that VR3 is better or much better than getting phone calls from ordering providers while 11% felt the phone was better. In terms of the perceived value of VR3, radiologists agreed or strongly agreed that VR3 contributes positively to patient care (81%), contributes positively to the perceived role of radiology (78%), and allows the role of the radiologist as a consultant to be quantified and valued (74%). Figure 3 depicts radiologist responses. For “standard” consult response time, radiologist free-text responses ranged from 5 min to 3 days with a majority expecting a turnaround time between 15 min to an hour (70% — 19/27). For “urgent” consult response time, radiologist free-text responses ranged from “less than 5 min” to 1 h with a majority expecting a turnaround time under 15 min (63% — 17/27). Figure 4 depicts radiologists’ responses to the expected turnaround time for “standard” and “urgent” consults. Table 2 shows free-text responses from radiologists, including what they want to see added to the application.
Fig. 3.
Radiologist responses to VR3 initial perception survey questions on a Likert scale from 1 to 5 with 5 being "strongly agree" or the most positive response
Fig. 4.
Radiologists’ expected turnaround times for "Standard" and "Urgent" consults as depicted by frequency charts
Table 2.
Radiologist free-text responses
| “I believe teams is a great way to collaborate with clinicians, e.g. showing them images as a replacement for coming to the reading room. I think that it would be more helpful while on call, when phones can interrupt workflows.” |
| “I actually found it disruptive to get a random message. Would rather have someone make a call. Also from a learning perspective, residents should be fielding these calls.” |
| “I really do like it much as I hate it when the alarm goes off.” |
| “I think the turnaround time for each of the request has been vague, which puts all request in highest urgency as of now. I think if the clinicians can stratify their requests in terms of urgency, that would be great. Also if clinicians can clarify how they want to be contacted (message through teams, calls through teams, images/screen share on teams, phone call, etc.) would also be helpful.” |
| “Clinicians will learn to use “urgent” in all cases–it would be nice to have some feedback to them, or a query that says, is this really urgent.” |
| “There seems to be a diffusion of responsibility, where people seem reluctant to answer the call in a timely manner. It would be nice to have the consult go directly to the first year resident first, then escalated to the next appropriate person if not claimed — similar to how phone calls are handled.” |
| “It’d be nicer to have consulting team on Teams for easy asynchronous communication. Often the questions are fairly straightforward to answer (i.e. was the lesion there before/ larger or bigger) and could easily be answered with a Teams message.” |
Discussion
Providing consultation to referring providers is one of the key responsibilities of radiologists and is becoming increasingly important in the era of value-based patient care. These consultations have significant patient management implications [9, 10] and constitute a significant portion of the value added by radiology. Currently, the vast majority of these consults are conducted informally, primarily through the radiology reading room phone. The limitations of this approach are well recognized. We have very limited ability to quantify the volume or types of consults, or the time spent by radiologists providing this service. Furthermore, in the vast majority of these consults, the radiologist maintains no record that the consultation ever happened nor is there documentation of what was discussed [9]. Phone interruptions in the reading room are frequent and occur between 2 and 27 times per hour [6, 7, 11]. As shown by Ratwani et al., these interruptions can be highly disruptive, occurring during case reads and lasting well over 2 min, which is likely to result in substantial memory decay [7], meaning greater memory loss for the primary task, and a longer resumption lag. These limitations were further compounded by the fragmentation of the reading rooms that occurred during the COVID-19 pandemic [12]. Post-pandemic, the reality remains that our teams of subspecialty radiologists are increasingly physically separated in different buildings and sites, making the reading room phone an increasingly inadequate solution. Our virtual radiology reading room (VR3) provides a novel alternative to the reading room phone that leverages modern communication tools to improve accessibility, control interruptions, enrich communication, and quantify and track the consults. VR3 represents a paradigm shift for both our referring providers and radiologists, and therefore we conducted these surveys to assess their initial perception of this solution. Referring providers had a very favorable overall impression of VR3 (95%) and the majority preferred VR3 over calling the reading room, with only a small fraction that favored the phone. These results are critical because achieving the full potential of this solution hinges on changing referring providers practice patterns and habits. Radiologists also had a favorable overall impression of VR3 (78%) with a majority (70%) preferring it to the traditional phone call. Understandably, some radiologists showed less enthusiasm for VR3 during the pilot as the full benefit of the solution, particularly with regard to disruption control, cannot be achieved during the pilot phase. Only a small portion of the total volume of consults has been shifted to VR3 during the pilot so the radiologists’ perception of interruptions is unlikely to be changed.
One particular consideration that emerged during the pilot is the varying expectations of turnaround times. We made the decision not to make explicit promises regarding turnaround time during the pilot phase and decided instead to observe the expectations of both referrers and radiologists. Radiologists were advised to address consults during natural interruptions to attention (e.g., between cases) as opposed to during attention-intensive tasks. Since the radiologists are working as a group in fielding these consults, we expected that at least one would reach a natural stopping point in under 15 min. In the survey, we asked both referring providers and radiologists for their expectations for a reasonable turnaround time. While there were some differences in expectation between the radiologist and referring providers, with the radiologists expecting longer turnaround times, we were pleased to find natural overlaps in the expectation that could guide our design of the solution. In aggregate, the answers we received spanned a wide range of values with a cluster of responses in the 15–30 min range and another cluster in the few hours to “same business day” range. For urgent consults, response times were clustered in the 5–15 min range as well as in the range of 30 to 60 min. For reference, during the pilot, the vast majority of VR3 consults (~ 90%) were answered in well under 20 min irrespective of urgency. Based on these answers, we concluded that a service-level agreement defining three levels of consults may address the varying needs of referring providers and provide sufficient flexibility to the radiologists. In our subsequent phase of VR3 following the survey, we introduced 3 levels of consults: same-day (turn around time < 4 h), expedited (turn around time < 30 min), and urgent (turn around time < 10 min).
To our knowledge, there is only one published solution of a virtual consult system used in radiology [13]. Our solution differs from the report in multiple ways. First, our solution is designed to maintain the concept of the reading room defined as a group of radiologists that may include trainees providing coverage as a team. We did not need to create a separate consulting service or dedicate a specific radiologist to field consults. Our solution is designed to integrate into the routine daily radiologists’ workflow effectively replacing the reading room phone and therefore enabling radiologists to continue working as a team fielding consults without being confined to the same physical space. Providers can connect with any “reading room” during business hours and with on-call radiologists after hours using the same web app. Second, our solution emphasizes semi-synchronous communication, which we define as asynchronous communication occurring under preset timing parameters. Consults are submitted asynchronously and are addressed within preset turnaround times depending on their level of urgency. The consultation session that follows can utilize both synchronous (voice, video, or screenshare) and/or asynchrnous (text chat) communication. The goal is to shift interruptions from happening during attention-intensive tasks such as during case reads to time-periods in between case reads, to reduce the potentially harmful effects of disruptions such as reduced efficiency and the potential for errors.
The results of the surveys have shaped our design and development of VR3 in other ways. We have improved the integration with the medical record system and enhanced the visibility of the tool to the referring providers. We have also enhanced the radiologist interface by providing a one-button click to open the patient/study in question in PACS. We are currently piloting the solution in an affiliated community hospital to prove its value beyond academic radiology departments.
The limitations of this study include the subjective nature of the surveys. Additionally, the surveys were conducted as part of a quality improvement initiative where it was more important for us to respond to user suggestions and concerns than it was to maintain anonymity. The lack of anonymization of the survey could have affected the ratings and the opinions expressed.
Conclusion
Referring providers and radiologists expressed a positive initial perception of the Virtual Radiology Reading Room. The majority of referring providers and radiologists feel that VR3 integrates well with their workflow and is superior to traditional phone calls.
Take Home Points
Radiology consultation is increasingly important in the era of value-based patient care.
Radiology consultations are predominantly done informally, untracked, and are major cause of disruption.
The virtual radiology reading room provides a novel alternative to the reading room phone that leverages modern communication tools to improve accessibility, control interruptions, enrich communication, and quantify and track the consults.
Varying expectations of turnaround times can be harmonized into few levels of urgency acceptable to both referring providers and radiologists.
Referring providers and radiologists prefer the virtual radiology reading room solution over phone calls to the reading room.
Summary Sentence
Referring providers and radiologists expressed a positive initial perception to semi-synchronous digital radiology consultations and prefer it to traditional phone calls to the reading room.
Declarations
Conflict of Interest
The authors declare no competing interests.
Footnotes
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.McGinty GB, Allen B, Geis JR, Wald C. IT Infrastructure in the Era of Imaging 3.0. J Am Coll Radiol 2014;11:1197–204. 10.1016/j.jacr.2014.09.005. [DOI] [PubMed]
- 2.Silva E, Breslau J, Barr RM, Liebscher LA, Bohl M, Hoffman T, et al. ACR White Paper on Teleradiology Practice: A Report From the Task Force on Teleradiology Practice. J Am Coll Radiol. 2013;10:575–585. doi: 10.1016/j.jacr.2013.03.018. [DOI] [PubMed] [Google Scholar]
- 3.Gunderman RB, Chou HY. The Future of Radiology Consultation. Radiology. 2016;281:6–9. doi: 10.1148/radiol.2016152781. [DOI] [PubMed] [Google Scholar]
- 4.IMAGING 3.0 n.d.
- 5.Reiner B, Siegel E, Protopapas Z, Hooper F, Ghebrekidan H, Scanlon M. Impact of filmless radiology on frequency of clinician consultations with radiologists. Am J Roentgenol. 1999;173:1169–1172. doi: 10.2214/ajr.173.5.10541082. [DOI] [PubMed] [Google Scholar]
- 6.Smith EA, Schapiro AH, Smith R, O’Brien SE, Smith SN, Eckerle AL, et al. Increasing Median Time between Interruptions in a Busy Reading Room. 2021;41:E47–56. 10.1148/RG.2021200094. [DOI] [PubMed]
- 7.Ratwani RM, Wang E, Fong A, Cooper CJ. CASE STUDIES IN CLINICAL PRACTICE MANAGEMENT A Human Factors Approach to Understanding the Types and Sources of Interruptions in Radiology Reading Rooms. 2016 doi: 10.1016/j.jacr.2016.02.017. [DOI] [PubMed] [Google Scholar]
- 8.Sullivan GM, Anthony R. Artino J. Analyzing and Interpreting Data From Likert-Type Scales. J Grad Med Educ 2013;5:541. 10.4300/JGME-5-4-18. [DOI] [PMC free article] [PubMed]
- 9.Won E, Rosenkrantz AB. JOURNAL CLUB: Informal Consultations Between Radiologists and Referring Physicians, as Identified Through an Electronic Medical Record Search. Am J Roentgenol. 2017;209:965–969. doi: 10.2214/AJR.17.18050. [DOI] [PubMed] [Google Scholar]
- 10.Walker D, MacDonald DB, Dennie C, Afkham A, Liddy C, Keely E. Electronic Consultation Between Primary Care Providers and Radiologists. 2020;215:929–33. 10.2214/AJR.19.22270. [DOI] [PubMed]
- 11.Yu J-PJ, Kansagra AP, Mongan J. The Radiologist’s Workflow Environment: Evaluation of Disruptors and Potential Implications. J Am Coll Radiol 2014;11:589–93. 10.1016/j.jacr.2013.12.026. [DOI] [PubMed]
- 12.Yacoub JH, Swanson CE, Jay AK, Cooper C, Spies J, Krishnan P. The Radiology Virtual Reading Room: During and Beyond the COVID-19 Pandemic. J Digit Imaging 2021:1–12. 10.1007/s10278-021-00427-4. [DOI] [PMC free article] [PubMed]
- 13.Rosenkrantz AB, Sherwin J, Prithiani CP, Ostrow D, Recht MP. Technology-Assisted Virtual Consultation for Medical Imaging. J Am Coll Radiol. 2016;13:995–1002. doi: 10.1016/j.jacr.2016.02.029. [DOI] [PubMed] [Google Scholar]