Abstract
Background
Children’s hospitals often do not have a high enough volume to justify providing radiologist staffing overnight, leading to hospitals employing teleradiology services to offer preliminary reports. There is limited literature related to discrepancies between preliminary teleradiology pediatric radiologists and final interpretations.
Purpose
The purpose of this study is to examine discrepancy rates for teleradiologists preliminarily interpreting pediatric exams at a children’s hospital.
Material and Methods
Eight thousand seven hundred seventy-eight consecutive preliminary reports issued by pediatric teleradiologists were reviewed. The hospital utilized a system in which local onsite radiologists rated the preliminary reports of teleradiologists following the interpretations as part of standard operating procedure. Discrepancies were also rated according to whether the discrepancy was actionable (judged to alter patient management by the final rater) or not. Rates were stratified by modality, preliminary teleradiologist reader, and final rater and compared to each using a normal approximation. The mean discrepancy rates were compared using a z test for proportions. Linear regression was applied to the effect of years of radiologist experience on the total and actionable discrepancy rates.
Results
The overall actionable discrepancy rate was 1.6%, similar to inter-observer discrepancy rates reported in other studies. There were no significant differences in the actionable discrepancy rates among teleradiologists. There was no correlation between years of experience and discrepancy rate for either the teleradiologists or the final raters.
Conclusion
Pediatric subspecialty teleradiologists issue reports that mirror discrepancy rates typical of radiologists who issue reports for emergent adult studies. Years of radiologist experience is not a predictor of discrepancy rate.
Keywords: Residents, performance, discrepancies, on call, teleradiology, pediatrics
Introduction
The concept of overnight attending coverage of radiology services has emerged as a hot topic in radiology with increasing trends toward 24-h onsite coverage.1 Those in favor of 24-h coverage cite patient safety concerns, as a delayed diagnosis may result in morbidity or mortality to the patient. At the same time, many hospitals are experiencing increased utilization of emergent imaging after hours. As emergency departments become more crowded, ordering providers are less able to review their patients’ imaging and are thus more reliant on accurate, final radiology reports.
As most children’s hospitals are affiliated with academic centers, many children’s hospitals have been traditionally covered by radiology residents. The actionable discrepancy rate between radiology residents and faculty has been reported to be between 0.33% and 1.9%2 with improvement seen during training.3 The discrepancy rate of subspecialty-trained radiologists compared to residents was previously assessed by Branstetter et al.4
Most large children’s hospitals remain affiliated with radiology residency programs to some extent; however, large children’s hospitals tend to be free-standing private facilities. Many have grown at rates that no longer allow for radiology residents to staff after-hours services exclusively, and shortfalls in the number of fellows pursuing pediatric radiology5 have placed increased pressure on these facilities. Many children’s hospitals have shifted toward staffing with independent attending radiologists after hours that provide final interpretations.6 Radiology has seen marked growth in the teleradiology sector to meet staffing needs, yet discrepancy rates of preliminary teleradiology reports are not prominent in the radiology literature. This paper specifically examines subspecialty pediatric teleradiology discrepancy rates at a children’s hospital.
Material and Methods
This retrospective review was performed following an exemption from full review by the IRB. The study site was a freestanding 400-bed children’s hospital offering onsite attending pediatric radiologist coverage from 0800 to 2200 all days of the week. All onsite radiologists had formal fellowship training in pediatric radiology or neuroradiology and had 1–33 years of experience at the time of the study onset. For the purposes of this study, the time period between 2200 and 0800 is referred to as after-hours and did not have routine onsite radiologist coverage. Unrestricted access to CT and ultrasound was available after-hours at the facility, and preliminary interpretations by a pediatric teleradiologist were given for these studies as part of standard operating procedure. All pediatric teleradiologists had formal training in pediatric radiology and had 0–29 years of experience. MRI coverage was available on an emergent basis only. Radiography was available overnight; however, after-hours interpretations of radiographs were only provided if requested by the ordering provider.
Rating of the overnight reports was performed according to local standard operating procedure without new intervention. At the beginning of each workday, a local pediatric radiologist or pediatric neuroradiologist provided a final report for each preliminary report issued by the teleradiologist after-hours. The local rater had access to the preliminary report at the time of interpretation. Each report was scored as either agreement or discrepancy. A discrepancy was further classified as actionable if it was felt by the local radiologist that the change in the report from preliminary to final would result in a change in management of the patient. Examples of actionable discrepancies include unrecognized appendicitis or intracranial hemorrhage. Actionable discrepancies were called to the ordering provider on the following morning. For the purposes of this study, the local radiologist issuing the final report was referred to as the rater, and the teleradiologist issuing the preliminary report was referred to as the original reader.
In this study, 8778 consecutive teleradiology reports issued over a 30-month time course during after-hours coverage were compared to the final reports issued by the local radiologist using standard local operating procedures. All reports were rated. The discrepancy rates according to exam type are described in Table 1 with the 95% confidence intervals computed for each exam type using a normal approximation of the data. The overall and actionable discrepancy rates were assessed for 16 teleradiologists (Table 2) and 15 local raters (Table 3). Preliminary readers and final raters who read or reviewed fewer than 10 examinations are not included in this analysis, though such encounters did not result in any discrepancies. Inter-teleradiologist and inter-rater comparisons were made by comparing the proportion means using the confidence intervals computed by the normal approximation. Correlation between years of experience after residency training for teleradiologists and local raters for both total and actionable discrepancy rate was determined using a least squares linear regression.
Table 1.
Discrepancy rates by exam type.
| Exams | Total discrepancy rate (95% CI) | Actionable discrepancy rate (95% CI) | |
|---|---|---|---|
| CT | |||
| Abdomen | 1083 | 0.240 (0.215–0.266) | 0.031 (0.021–0.042) |
| Chest | 63 | 0.254 (0.146–0.361) | 0 |
| Extremity | 41 | 0.146 (0.038–0.254) | 0.049 (0–0.115) |
| Face | 681 | 0.163 (0.135–0.191) | 0.019 (0.009–0.029) |
| Head | 2938 | 0.155 (0.142–0.168) | 0.016 (0.011–0.020) |
| Neck | 272 | 0.221 (0.171–0.270) | 0.018 (0.002–0.034) |
| Spine | 544 | 0.118 (0.091–0.145) | 0.015 (0.005–0.024) |
| MRI | |||
| Brain | 69 | 0.290 (0.183–0.397) | 0.029 (0–0.069) |
| Spine | 31 | 0.161 (0.032–0.291) | 0.032 (0–0.095) |
| Abdomen | 2 | 0 | 0 |
| Extremity | 2 | 0 | 0 |
| Ultrasound | |||
| Abdomen | 297 | 0.141 (0.102–0.181) | 0 |
| Abd limited | 1616 | 0.078 (0.065–0.091) | 0.011 (0.006–0.016) |
| Chest | 50 | 0.100 (0.017–0.183) | 0 |
| Extremity | 129 | 0.194 (0.126–0.262) | 0.016 (0–0.037) |
| Head | 142 | 0.099 (0.050–0.148) | 0.007 (0–0.021) |
| Pelvis | 226 | 0.062 (0.031–0.093) | 0.004 (0–0.013) |
| Scrotum | 150 | 0.133 (0.079–0.188) | 0.020 (0–0.042) |
| Vascular | 134 | 0.090 (0.041–0.138) | 0.007 (0–0.022) |
| Radiographs | |||
| Abd chest | 141 | 0.121 (0.067–0.174) | 0.021 (0–0.045) |
| Extremity | 167 | 0.084 (0.042–0.126) | 0 |
Table 2.
Discrepancy rate by preliminary teleradiologist years of experience.
| Teleradiologist experience | Exams | Total discrepancy rate (95% CI) | Actionable discrepancy rate (95% CI) |
|---|---|---|---|
| 22 | 34 | 0.088 (0–0.183) | 0.029 (0–0.086) |
| 17 | 1806 | 0.119 (0.104–0.134) | 0.014 (0.008–0.019) |
| 26 | 92 | 0.120 (0.053–0.186) | 0.011 (0–0.032) |
| 28 | 45 | 0.133 (0.034–0.233) | 0.022 (0–0.065) |
| 12 | 830 | 0.133 (0.109–0.156) | 0.022 (0.012–0.032) |
| 1 | 94 | 0.138 (0.069–0.208) | 0.021 (0–0.050) |
| 5 | 858 | 0.147 (0.123–0.171) | 0.009 (0.003–0.016) |
| 1 | 2572 | 0.150 (0.136–0.164) | 0.016 (0.011–0.021) |
| 0 | 102 | 0.157 (0.086–0.227) | 0.010 (0–0.029) |
| 29 | 1447 | 0.159 (0.140–0.178) | 0.015 (0.008–0.021) |
| 23 | 135 | 0.163 (0.101–0.225) | 0.015 (0–0.035) |
| 18 | 56 | 0.179 (0.078–0.279) | 0.018 (0–0.053) |
| 7 | 217 | 0.189 (0.137–0.241) | 0.032 (0.009–0.056) |
| 14 | 154 | 0.195 (0.132–0.257) | 0.013 (0–0.039) |
| 4 | 270 | 0.196 (0.149–0.243) | 0.030 (0.009–0.050) |
| 11 | 49 | 0.224 (0.108–0.341) | 0 |
Note: Teleradiologists with fewer than 10 exams were excluded from this analysis.
Table 3.
Discrepancy rate by rating local radiologist years of experience.
| Local rater experience | Exams | Total discrepancy rate (95% CI) | Actionable discrepancy rate (95% CI) |
|---|---|---|---|
| 6 | 920 | 0.048 (0.034–0.061) | 0.005 (0–0.010) |
| 19 | 414 | 0.051 (0.030–0.072) | 0.005 (0–0.011) |
| 16 | 331 | 0.063 (0.037–0.090) | 0.012 (0–0.024) |
| 2 | 430 | 0.067 (0.044–0.091) | 0.009 (0–0.018) |
| 39 | 415 | 0.077 (0.051–0.103) | 0.012 (0.002–0.022) |
| 13 | 370 | 0.081 (0.053–0.109) | 0.008 (0–0.017) |
| 22 | 408 | 0.115 (0.084–0.146) | 0.012 (0.002–0.023) |
| 21 | 518 | 0.124 (0.095–0.152) | 0.008 (0–0.015) |
| 1 | 1031 | 0.138 (0.117–0.159) | 0.018 (0.010–0.027) |
| 10 | 977 | 0.145 (0.123–0.167) | 0.017 (0.002–0.023) |
| 4 | 410 | 0.149 (0.114–0.183) | 0.002 (0–0.007) |
| 24 | 849 | 0.166 (0.141–0.191) | 0.027 (0.016–0.038) |
| 2 | 301 | 0.213 (0.166–0.259) | 0.017 (0.002–0.031) |
| 33 | 515 | 0.326 (0.286–0.367) | 0.008 (0–0.015) |
| 9 | 600 | 0.328 (0.291–0.366) | 0.055 (0.037–0.073) |
Note: Raters with fewer than 10 exams were excluded from this analysis.
Results
The overall discrepancy rate was 14.4%, and the actionable discrepancy rate was 1.6%. Among CT exams, CT of the chest had the highest discrepancy rate; however, no actionable discrepancies were identified for any of the chest CT’s performed. CT of the head was the most common CT requested, and discrepancies in interpretations of these examinations mirrored the overall rates for this study.
Discrepancies in interpretation of brain MRI’s were higher than the overall rate; however, a low number of MRI’s were requested after-hours which makes direct comparison to the other modalities difficult. The highest ultrasound discrepancy rate was among extremity ultrasounds, and the highest actionable discrepancy rate in this modality was ultrasound of the scrotum. Ultrasound of the pelvis yielded the lowest discrepancy rates in the ultrasound category.
The highest discrepancy rate among teleradiologists was 22.4%; however, a proportion comparison test showed an insignificant difference compared to the median teleradiologist with one year of experience and the highest number of examinations read (p = 0.152). This radiologist did not have any actionable discrepancies. There were no outlying actionable discrepancy rates among the teleradiologists.
There was greater variability in the rating radiologists with a wide range of total discrepancies from 4.8% to 32.8%, but only one of the raters cited a significantly higher proportion of actionable discrepancies compared to the median rater (p < 0.001).
The r-squared correlation coefficient between years of experience and discrepancy rate was 0.101 for overall discrepancies and 0.001 for actionable discrepancies among teleradiologists. The r-squared correlation coefficient between years of experience and discrepancy rate was 0.006 for overall discrepancies and 0.010 for actionable discrepancies among local raters.
Discussion
Data presented here demonstrate an actionable discrepancy rate between pediatric teleradiologists and final interpreting radiologists at a children’s hospital which mirrors that of other studies. A 2013 meta-analysis describing 58 studies evaluated 388,123 adult CT examinations showed a major discrepancy rate of 2.4% with a 95% confidence interval of 1.7%–3.2%.7 With regard to a large sample outsourced teleradiology services, Wong et al. uncovered a rate of 1.1%.8 Cheng et al. reported a clinically significant error rate of 1% among preliminary reports issued for adult patients by an overnight teleradiology service.9 The findings here confirm that reinterpretation of pediatric exams initially read by pediatric teleradiologists yields similar discrepancy rates in comparison to studies analyzing rates for studies of adult patients.
There was no effect of the years of experience on the discrepancy rate with respect to either the teleradiologists or the rating local radiologists. This provides validity to the training required to become a specialized radiologist. In 2012, Eakins et al. showed that the actionable discrepancy rates when examinations interpreted at non-pediatric institutions were 12.6% and 32.6% for neuroimaging and general pediatric radiology examinations, respectively, when a second opinion was provided at the children’s hospital in which a child was transferred to.10 The fact that the teleradiology group performed similarly lends considerable credibility to the validity of the teleradiology group hired by this hospital, especially since the teleradiologists were less familiar with local equipment and sonographer-related variables. Likewise, the teleradiologists are subject to workflow that spans across multiple institutions submitting studies simultaneously throughout the night. Nonetheless, the low actionable discrepancy rate and relatively uniform inter-teleradiologist performance is remarkable in light of this.
There was inter-rater variability regarding the total discrepancy rate even though the inter-teleradiologist variability was minimal. This may suggest that the rating radiologists may have greater variation in what they determine to represent a discrepancy.
There was a higher discrepancy rate for MRI; however, the sample size for this modality was much lower since MRI was performed on an emergent basis only. Weinberg et al.3 saw a similar increased disparity rate for MRI examinations. There are likely to be nuances in MRI that local radiologists are more familiar with, and despite these elements, the actionable discrepancy rate was only 2.9% for brain MRI and 3.2% for spine MRI. There were only two emergent body and two musculoskeletal MRI examinations requested during the period study, none of which demonstrated a discrepancy.
Strengths of this study include the fact that this investigation was able to assess the variability of teleradiology interpretations within a specific subspecialty of radiology. The teleradiology vs. onsite radiologist variable was the major variable tested. Large sample sizes add strength to the conclusions, and this study adds to the literature that children’s hospitals can use in justifying the use of preliminary pediatric teleradiologists. These findings may also foster the market for pediatric teleradiology services. As teleradiology companies continue to grow, the addition of pediatric radiologists to their labor forces may provide value if these companies can assure children’s hospitals that pediatric teleradiology services are of similar quality, especially in light of the findings of Eakins et al.10 that general radiology interpretations of pediatric studies can result in higher rates of actionable discrepancies.
Some limitations to this study are noteworthy. This study, just as the others referenced, does not assess accuracy or error, as the gold standard was considered to be the final rater’s interpretation. It is possible that in a given examination, the teleradiologist uncovered a finding that would not have eventually been perceived by the final interpreting radiologist since this was not a double-blinded experiment, though this could be an additional benefit to hiring a teleradiology company. Future studies could focus on independently interpreted examinations on a prospective basis. The relatively low number of radiograph interpretations requested during the study limits the generalizability of this modality, though the discrepancy rates were nonetheless representative of the overall findings.
In conclusion, the actionable discrepancy rate between pediatric teleradiologists and final attending radiologist interpreters is similar to those described in other studies comparing inter-radiologist variability.
Footnotes
Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) received no financial support for the research, authorship, and/or publication of this article.
ORCID iDs: Cory M Pfeifer https://orcid.org/0000-0002-2220-9230
Mary L Dinh https://orcid.org/0000-0002-4318-7595
References
- 1.Hammer MM, Shetty AS, Cizman Z, et al. Results of the 2015 survey of the American Alliance of Academic Chief Residents in Radiology. Acad Radiol 2015; 22:1308–1316. [DOI] [PubMed] [Google Scholar]
- 2.Bruno MA, Duncan JR, Bierhals AJ, et al. Overnight resident versus 24-hour attending radiologist coverage in academic medical centers. Radiology 2018; 289:809–813. [DOI] [PubMed] [Google Scholar]
- 3.Weinberg BD, Richter MD, Champine JG, et al. Radiology resident preliminary reporting in an independent call environment: multiyear assessment of volume, timeliness, and accuracy. J Am Coll Radiol 2015; 12:95–100. [DOI] [PubMed] [Google Scholar]
- 4.Branstetter BF, Morgan MB, Nesbit CE, et al. Preliminary reports in the emergency department: is a subspecialist radiologist more accurate than a radiology resident? Acad Radiol 2007; 14:201–206. [DOI] [PubMed] [Google Scholar]
- 5.Pfeifer CM. Declining interest in pediatric radiology prompts a call to action. J Am Coll Radiol 2018;15:490–492. [DOI] [PubMed]
- 6.Donaldson JS, Thakrar KH. 24/7 pediatric radiology attending coverage: times are changing. Pediatr Radiol 2017; 47:803–807. [DOI] [PubMed] [Google Scholar]
- 7.Wu MZ, McInnes MDF, Macdonald DB, et al. CT in adults: systematic review and metaanalysis of interpretation discrepancy rates. Radiology 2013; 270:717–735. [DOI] [PubMed] [Google Scholar]
- 8.Wong WS, Roubal I, Jackson DB, et al. Outsourced teleradiology imaging services: an analysis of discordant interpretation in 124,870 cases . J Am Coll Radiol 2005;2:478–484. [DOI] [PubMed] [Google Scholar]
- 9.Cheng T, Dumire R, Golden S.et al. Impact on patient care of discordance in radiology readings between external overnight radiology services and staff radiology readings at a level 1 trauma center. Am J Surg 2013; 205:280–282. [DOI] [PubMed] [Google Scholar]
- 10.Eakins C, Ellis WD, Pruthi S, et al. Second opinion interpretations by specialty radiologists at a pediatric hospital: rate of disagreement and clinical implications. AJR Am J Roentgenol 2012; 199:916–920. [DOI] [PubMed] [Google Scholar]