Implementation of a Software Distribution Intervention to Improve Workload Balance in an Academic Pediatric Radiology Department

Abstract

In our pediatric radiology department, radiographs (XR) are the shared responsibility of the body section and interpreted in addition to modality or site-specific assignments. Given an unequal contribution amongst radiologists to the XR workload, a software solution was developed to distribute radiographs and improve workload balance. Metrics to evaluate the intervention’s effectiveness were compared before and after the intervention. Data was retrieved from the radiology analytics platform, scheduling software, and the peer learning database. Metrics were compared 12 months pre (March 2018–February 2019) and 6 months post (March 2019–August 2019) intervention on non-holiday weekdays, 7 am–5 pm. To evaluate the intervention’s effectiveness, variance between radiologists’ contributions to XR volume was assessed using Levene’s and Fisher’s tests. Changes in turnaround times (TATs) and error rates pre- and post-intervention were evaluated as secondary metrics. Following the intervention, the average number of XR interpreted on target rotations increased by 8.9% (p = 0.011) while the departmental volume of radiographs increased only 4.5%. The variance between radiologists’ daily XR contribution was 21.3% (p < 0.0001) higher prior to the intervention. Days where target rotations read fewer than 5 XR decreased from 17.8 to 1.1% (p < 0.0001) after the intervention. Days in which more than 75% of all XR had a TAT less than 60 min improved from 26.8 to 39.7% (p = 0.017) after the intervention. There was no statistically significant difference in error frequency (error rate 2.49% pre and 2.72% post, p = 0.636). In conclusion, the software intervention improved XR workload contribution with decreased variability. Despite increased volumes, there was an improvement in turnaround times with no effect on error rates.

Background

In our pediatric radiology department radiographs (XR) are the shared responsibilities of body section radiologists. Daily radiograph volume varies, ranging between 400 and 700 studies on daytime weekdays. At least one radiologist is assigned to primarily interpret XR, but all body radiologists are expected to interpret XR in addition to their daily modality, subspecialty, or site-specific assignments.

All radiologists were expected to interpret at least 20 radiographs in addition to their primary responsibilities. Despite this, complaints about XR workload distribution were frequent, for example, some radiologists routinely interpreted 20 exams early in the day and none later in the day, regardless of the unread XR worklist length. There were also complaints of cherry picking by preferentially reading less time-consuming studies. Finally, some radiologists felt obliged to interpret a large number of radiographs above the 20 XR expectation if the unread daily volume was high, while others capped at 20 XR regardless of the unread volume. Consequently, an intervention to more fairly balance radiologists’ contributions to the daily radiograph workload was sought.

To this end, a software solution was created to automatically distribute radiographs to existing worklists and balance each radiologist’s contribution to the daily XR workload. The primary metric used to evaluate the intervention’s effectiveness was variability in individual radiologists’ contributions to radiograph volume, before and following the intervention. Additionally, turnaround times (TATs) and reported error rates were compared pre- and post-intervention.

Methods

Study Protocol

This quality improvement project was exempt from institutional review board approval and is presented according to the Standards for Quality Improvement Report Excellence Guidelines, V.2.0 [1]. Body radiologists staff 11 weekday rotations between the hours of 7 am and 5 pm. Daily workflow is managed by PowerScribe Workflow Orchestration (Nuance Communications, Burlington, MA) which integrates with the departmental PACS system (IntelliSpace PACS, Philips Healthcare, Andover, MA), and the Radiology Information System (RIS) (Epic Radiant, Epic Systems Corporation, Verona, WI). Data was retrieved from our radiology analytics platform (RadMetrix, Nuance Communications, Burlington, MA), radiologist scheduling software (QGenda, Atlanta, GA), and peer learning database (PowerConnect Peer Learning, Nuance Communications, Burlington, MA).

Study periods were divided into pre- and post-implementation of the radiograph distribution system. The pre-implementation period is defined as the non-holiday weekdays from March 5, 2018, to March 4, 2019. The post-implementation period is defined as the non-holiday weekdays from March 12, 2019, to September 11, 2019. The distribution software was deployed March 5, 2019. March 5, 2019, through March 11, 2019, was excluded from both periods to allow time for distributor rollout and bug fixes.

Pre-intervention Workflow

Prior to the intervention, body radiologists utilized worklists matching their rotation (e.g., inpatient ultrasound, outpatient ultrasound, etc.) One radiologist was also assigned to the “All XR” worklist as their primary rotation, which constituted all unread radiographs; all other body radiologists were expected to read at least 20 studies from the “All XR” worklist in addition to their primary worklist (Fig. 1).

Fig. 1.

Screenshot of the typical daily worklists using the worklist manager. Radiologists select their primary assignment (*) and also are expected to opt-in (denoted by checkmark) to all XR (arrow). Subspecialty rotations which were not targeted for the distribution intervention are marked by the orange box

Intervention Planning and Description of the Intervention

We preferred to work with our existing worklist manager vendor and software (Nuance PowerScribe Workflow Orchestration) to develop a software solution to equitably distribute radiographs, rather than purchasing an off-the-shelf solution. Developing a solution with our existing vendor was preferred for two primary reasons. First, deploying existing off the shelf distribution managers would require a significant change in our workflow and additional costs and system integration. Second, we needed a software distribution solution that would allow radiologists to retain flexibility and autonomy, while at the same time managing our variable daily workflow by assigning exams.

The solution was developed as a new feature component within PowerScribe Workflow Orchestrator. It was developed collaboratively by team members from the local radiology department (project manager, pediatric body radiology lead, MS, and Chief of Radiology Informatics, ACS), Information Services (Imaging Informatics analyst), and the vendor (director of product management, AS and software developer, EO). Key features that were considered in designing and deploying the software are summarized in Table 1.

Table 1.

Selected key features which were considered when developing the software intervention

Feature requirements for end users	Feature requirements for IT department
Create no unnecessary new work for radiologists Solution: Add distributed exams to existing in use worklists	Need to be able to audit and report assigned exams Solution: Create reporting and auditing tool accessible to administrators
Allow users to ignore/skip exams if necessary Solution: Exclude specialized XR. Allow release of assigned exams to general worklist if unread for end user defined time interval	Should function automatically but be modifiable, with minimal input from IT Solution: Add schedule functionality, including predetermined changes for holidays and immediate day of control for unexpected events (such as natural disasters or mass casualty scenarios)
Show users which exams were assigned to them Solution: Add icon to assigned exams (Fig. 3)	Minimize need for server-side overhaul or new client install Solution: Implement as integrated application in existing worklist manager software

The developed software solution maintains and updates a database of unread radiographs via integration with the RIS. At set time intervals determined by the software logic, the database is queried for unread radiographs and distributes them in a round-robin manner to the body worklists, in effect “automatically assigning” radiographs to specific worklists and simultaneously removing them from the global “All XR” worklist (Fig. 2).

Fig. 2.

Within the existing software manager, a new filter is created to include all exams that should be distributed. At our facility, we excluded stat, urgent, and inpatient exams, as well as specialized exams such as genetic bone surveys and leg lengths. The software rule is configured to take exams from the source filter and distribute them to existing target worklists

All stat, urgent, and inpatient radiographs were excluded from distribution to mitigate potential negative effects where a delay in interpretation might cause patient harm. The exams’ meeting criteria for distribution were intended to be radiographs that any pediatric radiologist would not consider onerous to interpret. Therefore, certain types of radiographs (e.g., genetic bone surveys) were excluded from distribution. User-configurable options allowed for choosing which days, times, and rotations received distributed studies, as well as how many and how often the studies would be distributed.

Post-intervention Workflow

Based on historical volumes, indicating a capacity to interpret more radiographs on certain worklists, five out of ten daytime rotations were targeted to be automatically assigned distributed radiographs. The five rotations that were not targeted were all subspecialty rotations. Based on concern that adding volume to subspecialty rotations may dilute time available for radiologists covering these rotations, no studies were assigned to these worklists. On the worklists where radiographs were automatically assigned, the radiographs appeared on the targeted worklists and were designated with a symbol to indicate they had been automatically distributed (Fig. 3). A cap was set to each worklist, limiting the maximum number of studies that could be automatically assigned via the distribution software. The distribution process continued every 10 min during hours of distribution until worklist caps were met (Table 2).

Fig. 3.

Worklist screenshot showing the arrow icon that was added to distributed exams to allow radiologist to know a distributed exam was their responsibility. For example, in this case, the exam with the arrow would only be visible to the radiologist interpreting off the “Woodlands Body” primary assignment, while the other XR on the list are on the “ALL XR” list

Table 2.

The body section work assignments which were affected by distribution, including maximum number of assigned exams (20 for each rotation), and hours in which exams were assigned via the software

Rotation	Distributor times	Max assigned exams
Ultrasound Inpatient	8:30 am–11:30 am 1:30 pm–3:00 pm	10 10
Ultrasound Outpatient	8:30 am–11:30 am 1:30 pm–4:00 pm	10 10
Community Hospital 1	8:30 am–11:30 am 1:30 pm–3:30 pm	10 10
Community Hospital 2	8:30 am–11:30 am 1:30 pm–3:30 pm	10 10
Body CT/MR	1:30 pm–4:30 pm*	20

^*This rotation was scheduled 12:00–8:00 pm, but with additional responsibilities to support after-hours workflow after 5 pm. Consequently, in contrast to the other rotations where end time was 1 h prior to shift end, the end time was chosen based on evening rotational workflows

The time periods during which exams were distributed were selected both based on operational needs and subjective preferences of the radiology group. The hours selected were known to be peak hours where radiograph volumes were historically higher. However, we also elected not to automatically distribute exams to worklists over the noon hour (with a half hour buffer on each side, for a total of two hours break from distribution 11:30 am to 1:30 pm). Furthermore, distribution was stopped one hour before end of the rotation to allow radiologists to catch up if they were behind on their worklist and mitigate the potential that distributing exams would lead to longer workdays.

Statistical Analysis

Improving equitable contribution to the radiograph workload was assessed in two ways. First, the overall change in contribution to the unread radiograph volume by radiologists working on target rotations pre- and post-intervention was analyzed. Histograms of the number of radiographs read by radiologists per shift on target rotations were qualitatively compared pre- and post-distributor. The mean and variance of these distributions were compared with the t test and the Brown-Forsythe variant of Levene’s test, respectively, and reported as standard deviation in number of interpreted radiographs. Second, because it was conceivable that the intervention may only correct one or two outlier radiologists, we also evaluated ineffectual contribution before and after distributor by each radiologist, defined as a radiologist interpreting fewer than 5 XR per day. This was compared pre- and post-intervention using Fisher’s exact test and the binomial test was used to analyze the result.

Since we were concerned that the distribution solution may increase turnaround times (TAT) by shunting studies from all radiologists to specific radiologists, we evaluated (TAT) as a secondary metric. Using percentage of exams under a target turnaround time has been advocated as a way to provide consistency, improving customer service [2, 3]. For this reason, percentage of days in which turnaround times on 75% of radiographs were under specific thresholds was compared. As only routine exams were distributed, the effect on overall radiograph turnaround times, stat, and routine exams were each separately evaluated. TAT was defined as on-list time to final report time. Thresholds of under 30 min for stat exams, under 90 min for routine, and under 60 min for combined stat and routine exams were targeted goals. Days in each period where over 25% of studies did not meet these goals were quantified. The proportion of these days pre- and post-distributor was compared using Fisher’s exact test.

Because the intervention assigned exams to radiologists to interpret, there was concern that error rates could increase. For example, if a radiologist did not feel comfortable interpreting an assigned exam, they may be more likely to interpret the study incorrectly. At our institution, peer learning is accomplished through a worklist manager add-on that randomly assigns radiologists two exams daily to peer review. As a measure of quality, the rates of errors submitted to the peer learning system were compared pre- and post- intervention. For all analyses, a p value of less than 0.05 was considered significant.

Results

Detailed breakdown of data used in analysis is provided in Table 3. In total, 22 different radiologists worked during the study period. Twenty-one worked at least one target rotation prior to the distributor intervention, and 19 after the intervention.

Table 3.

Source data used in the statistical analysis

Metric	Pre-intervention (12 months)	Post-intervention (6 months)
Number XR on-list and interpreted on target rotations	48,083	25,968
Number XR on-list and interpreted on any rotation*	113,871	59,852
Number of STAT and non-STAT XR on-list and interpreted on any rotation	STAT: 25,450 Non-STAT: 88,421	STAT: 13,551 Non-STAT: 46,301
Number of non-holiday weekdaysa	252 days	127 days
Number of target shiftsa	1260 target shifts	635 target shifts
Number of non-holiday weekdays	252 days	127 days

^*This includes exams in this table, first line (number XR on-list and interpreted on target rotations), which were also separately analyzed based for target rotations

^aTwo days each in the pre-distribution period and post-distribution period were excluded because more than one radiologist was assigned to the same distributed rotation or one radiologist was assigned to two different distributed rotations

The primary qualitative observation from the histograms (Fig. 4) is that the distribution prior to the intervention is skewed to the left (0–4studiesinterpretedonarotation), and has two major peaks, while after the intervention there is only one major peak, indicating a distribution closer to a bell shaped curve (as expected for a normal distribution).

Fig. 4.

Prior to the intervention, there were many days where the radiologist interpreted fewer than five exams. After the intervention, the number of exams read by radiologists per shift were more centrally concentrated with a significantly lower number of shifts with fewer than 5 exam read (see Fig. 5 for a breakdown of this last change by radiologist)

The average number of radiographs interpreted per radiologist on the targeted rotations increased after the intervention by 8.9% (from 35.1 radiographs to 38.2, p = 0.011), while the average daily number of radiographs available to read increased (attributable to increased institutional volume) by 4.5% (from 454.7 to 475.0). The variation in the number of interpreted radiographs per target rotation was 21.3% higher before the intervention, with the standard deviation decreasing from 28.4 studies before the intervention to 23.4 studies after (p < 0.0001).

Prior to the intervention radiologists read fewer than five radiographs on 17.8% (224 of 1260) of targeted shifts. Following the intervention, there was a statistically significant decrease to 1.1% of targeted shifts (7 of 635, p < 0.0001). This improvement was not limited only to improvement by a subset of radiologists (Fig. 5). Of the 19 radiologists who interpreted exams on the target rotations following the intervention, 17 radiologists participated in at least five target rotations both pre- and post-intervention and were analyzed here. Of these 17, 13 (76.5%) radiologists had at least 1 day where they interpreted < 5 radiographs prior to the intervention. All (100%) had a lower percentage of days in which they interpreted < 5 radiographs after the intervention (p < 0.001), with 9 of the 13 (69.2%) decreasing to 0% of days interpreting < 5 radiographs post-intervention. Four of 17 (23.5%) radiologists never read < 5 radiographs during a normal daily assignment either prior to or following the intervention.

Fig. 5.

Prior to the intervention, a few radiologists frequently interpreted fewer than 5 XR (despite expectation of interpreting 20). Following the intervention, the frequency of interpreting fewer than 5 XR improved for all radiologists

For TATs, there was a statistically significant improvement in the percentage of days in which greater than 75% of all X-rays had a TAT less than 60 min and all STAT x-rays had a TAT less than 30 min after the intervention. The percentage of days in which greater than 75% of non-STAT X-rays had a TAT less than 90 min also trended lower post-intervention (Table 4).

Table 4.

Turnaround time targets on stat and all XR improved after the intervention, with a statistically significant improvement in both STAT and all XR turnaround times

TAT targets	Percent days TAT meet targets	P value
75% of STAT XR < 30 min	Pre-intervention: 94.8% Post-intervention: 98.9%	p = 0.017
75% of all XR < 60 min	Pre-intervention: 60.3% Post-intervention: 73.2%	p = 0.041
75% of routine XR < 90 min	Pre-intervention: 88.2% Post-intervention: 82.9%	p = 0.227

Finally, the reported error rate was 2.49% prior to the intervention (81 exams scored as a miss, and 3172 scored as agree) compared with 2.72% following the intervention (47 miss and 1680 agree). The difference was not found to be statistically significant (p value = 0.636).

Discussion

Our software intervention successfully improved workload balance amongst radiologists by distributing routine outpatient radiographs to existing modality or site-specific assignment worklists. Despite increased volumes and the potential to shunt exams from a primary reading worklist to worklists where radiologists had additional responsibilities, there was an improvement in turnaround times, with decreased variance in number of exams interpreted by radiologists, and no effect on error rates.

The radiograph distribution software solution achieved its goal: radiologists more equitably contributed to the radiograph workload. Integral to the desire for equitable contribution is the concept of fairness. While fairness is difficult to define, it deserves attention as the perception of unfairness is one of the originally described six factors that may contribute to burnout [4]. This is pertinent, as burnout is prevalent amongst radiologists [5–8].

A sense of control is also one of the six factors which contribute to burnout [4]. The effect of our intervention on radiologists’ sense of control is uncertain. On one hand, by assigning exams to a physician’s worklist, the automated distribution may adversely affect a physician’s sense of autonomy. On the other hand, the physician may experience increased control over the workday in the form of predictability and defined expectations. For example, by using the automated software, the radiologist can know hours and frequency in which exams will populate their worklist. The extent to which this affects individuals is uncertain, with likely some radiologists feeling less control and others more.

While emphasis on fast turnaround times has been discouraged [9], turnaround times are an accepted quality metric [10–15]. Specifically, imaging turnaround times can be used for the value-based payments mandated by the Medicare Access and CHIP Reauthorization Act (MACRA) [16, 17]. Here, we saw an improvement in turnaround times as a secondary effect of the intervention.

Additionally, we evaluated overall error rates, as assigning exams to a radiologist could cause the radiologist to less safely interpret studies. For example, when assigned to the outpatient ultrasound rotation, the radiologist may dictate the ultrasounds too rapidly if they are pressured to also read the automatically distributed radiographs. Alternatively, if the radiologist is uncomfortable with one of the assigned exams, they could incorrectly interpret the assigned study. Given a preexisting workflow where radiologists are encouraged to consult with colleagues as needed, as well as the exclusion of more specialized exams from distribution, we did not anticipate an uptick in misinterpretations. Accordingly, the lack of identifiable change in error rates suggests the intervention did not have a measurable effect on our error rate. There are known limitations to using the peer learning system as a surrogate for errors, including underreporting of errors [18, 19] and bias in which studies are submitted to peer learning or peer review systems [20]. However, the peer learning system is independent of the distributor solution, and there was no reason to suspect that rates of potential underreporting and biased submission were different pre- and post-intervention.

Practically, the software intervention is sustainable, requiring no input from personnel on a daily basis. Routine maintenance is carried out by IS in keeping with their other scheduled maintenance. Depending on the extent of changes requested, revisions to the distribution program can either be made locally or require vendor support, similar to other software upgrades.

Limitations

Definitively isolating the intervention’s effects is difficult. For example, differences in the number of studies read between radiologists could be due to other factors, including those both specific to individual radiologists (such as life stressors outside work) or systemic changes (e.g., changes in ordering patterns by referring providers). Consequently, the observed decrease in TATs following the intervention could be confounded. While the use of a control group could help delineate the effects of the intervention, use of a control group was considered impractical and counter to the desired need for immediate operational improvement.

The method by which we chose to evaluate turnaround times may not be the one selected by other institutions. However, there are no universally accepted standards, with both the Joint Commission and the American College of Radiology emphasizing that patient care is best when imaging results are available by the time referring providers need the results to make treatment decisions [21, 22]. The intent of our analysis was to use TAT metrics that could evaluate for both consistent and timely reports.

The intervention we employed was custom designed for our setting which may limit generalizability to other institutions. The types of rotations to which exams were distributed could be less relevant in other practices. Another limitation of is that the solution is not intended as a complete, fully automated, distribution system for all exams. Such a system would need to take into account additional considerations including the variable complexity of exams. Furthermore, the relationship between the intervention and mitigating burnout has not been fully studied, though is a topic for further study. Nevertheless, the concept underlying the distribution solution has the potential for wide applicability, and the software solution is easily modifiable to account for local priorities. More practically, radiologists’ acceptance of the distribution solution is evidenced in action: since this analysis, the number of rotations targeted by the distributor software and also the number of exams auto-assigned to worklists was increased by group consensus. In informal feedback, radiologists have affirmed that the automated solution has improved their workdays by defining expectations and helping to mitigate an accumulation of unread studies throughout the workday.

Conclusion

In response to increasing volumes and dissatisfaction with unequal contributions to the daily radiograph workload, a software solution was developed to automatically distribute routine, outpatient radiographs to existing modality or location based worklists between set weekday hours. Following the deployment of the software intervention, radiologists more equitably contributed to the department’s workload. The distributor solution had no effect on frequency of errors, and turnaround times improved. The perceptions of fairness and control in the workplace affect burnout. Further analysis of software-driven operational workload management techniques, such as this, that may improve radiologists’ sense of fairness and/or control may be considered for future studies.

Declarations

Conflict of Interest

Authors Marla Sammer and Andrew Sher are members of Nuance Communications, Inc., reference and advocacy connection group, for which they receive no financial compensation. Authors Andrew Stahl and Eray Ozkan are employees of Nuance Communications, Inc., which provided no financial assistance.