An analysis of radiographer preliminary image evaluation – A focus on common false negatives

Abstract

Introduction

Preliminary image evaluation (PIE) is a mechanism whereby radiographers provide a preliminary evaluation of whether pathology is present in their radiographs, typically acquired within the emergency department (ED). PIE provides referrers with a timely communication of pathology prior to the availability of a radiology report. The purpose of this study was to determine the most common radiographer PIE false‐negative interpretations.

Methods

Each month over a two‐year period, 100 PIEs of adult and paediatric patients were randomly reviewed in a metropolitan hospital ED. The radiographer’s PIE was compared with the radiologist’s report and categorised into basic quality indicators; true positive, true negative, false positive and false negative. The anatomical regions which most commonly indicated a false‐negative interpretation were further analysed.

Results

2402 cases were reviewed which resulted in an overall PIE accuracy of 88.7%. Wrists, hands, phalanges (upper), ankles, feet and phalanges (lower) reporting the highest false‐negative or false‐negative/true‐positive interpretations (60/116). Of the 60 false‐negative PIEs, 68 pathologies were identified. 41.1% (28/68) of the pathology not identified were in the phalanges. Within these regions, examinations with multiple injuries commonly reported false negatives (17/60).

Conclusions

This study demonstrated the most common false‐negative radiographer PIEs were within the upper and lower distal extremities. Specifically, the phalanges and examinations demonstrating multiple injuries reported high levels of misinterpretation. The misinterpretation in multi‐injury examinations could be attributed to ‘Subsequent Search Miss (SSM)’ error. These results provide valuable insights into areas of emphasis when providing image interpretation education.

Preliminary image evaluation (PIE) is a mechanism, whereby radiographers provide a preliminary evaluation on whether pathology is present in the radiographs acquired, typically within the emergency department (ED). Over a two‐year period, 100 PIEs from a metropolitan hospital ED were audited and categorised into the basic quality indicators true positive, true negative, false positive and false negative. 2402 cases were reviewed, with wrists, hands, phalanges (upper), ankles, feet and phalanges (lower) reporting the highest false‐negative or true‐positive/false‐negative interpretations (60/116).

Introduction

‘Preliminary Image Evaluation’(PIE)¹ or ‘Preliminary Clinical Evaluation’(PCE)² is the Australian and United Kingdom contemporary terminology for what was originally termed ‘Radiographer Commenting’.¹, ², ³ PIE is an abnormality detection and description system performed by a radiographer, to provide emergency referrers with an informal opinion on radiographic pathology.⁴ PIE is a further development of the Red Dot system,² which involved the radiographer denoting a radiograph with a red dot if an abnormality was visualised.⁵ This further developed system provides more information to referrers, such as location, type and appearance of the abnormality.⁶ This leads to less ambiguous communication of pathology, as a written description is provided, compared to that of the Red Dot system.⁵

Radiologist interpretation and clinical reports are used by clinicians to guide diagnostic and treatment decisions. This interpretation is not always available in time for patient treatment, following general radiography.⁷ A study by Eastgate et al.,⁷ conducted at a Queensland tertiary hospital, demonstrated that the average time between image acquisition and a final radiological report was 84 hours for general radiography. For Australian Emergency Departments (ED), the goal is for 90 per cent of patients presenting for treatment to be discharged from or admitted to the hospital within four hours of arrival.⁸ This is called the National Emergency Access Target.⁸ Therefore, the majority of the patients who undergo general radiography in the ED will likely not have a radiologist report available by the time they are discharged/treated, consequently meaning only the patient’s ED treating team will have reviewed the images. The informal opinion provided by radiographer PIE is particularly important when a radiology report is unavailable, as some ED referrers, such as junior doctors, have limited skills in radiographic interpretation.⁹ This inexperience may reflect limited radiological teaching time during medical school.¹⁰

PCE/PIE is used widely amongst UK hospitals, but infrequently in Australia. Using Queensland as an example, in 2013, only 16% (4/25) of public hospitals were reported as using a Radiographer Abnormality Detection System, with three quarters using a ‘red dot’ style approach and one quarter using verbal communication with referrers.¹¹ Multiple studies have identified barriers with the implementation of PIE.¹², ¹³, ¹⁴ These barriers include access to education, low radiographer confidence in abnormality detection and communication of pathology, lack of time, and the perception that PIE is encroaching on the radiologist’s role.¹², ¹³, ¹⁴ A study by Neep et al. in 2014, found that the greatest perceived barrier by radiographers who undertook their survey was their ability to access education specifically targeted to image interpretation.¹² The Medical Radiation Practice Board of Australia (MRPBA) is the national regulatory body responsible for registration of medical radiation professionals and the development of standards within these professions.¹⁵ The MRPBA’s professional capability requirements include the capability to ‘identify any urgent and/or unexpected findings’¹⁶ highlighting that all radiographers are expected to be competent in image interpretation and pathology identification. Considering that pathology identification is a requirement by the national regulatory body, it is surprising that these barriers are still present creating a paucity in the implementation of PIE.

As education has been identified as a barrier to PIE, there is a need for more research into radiographer‐specific image interpretation. Existing literature has investigated radiographers’ performance undertaking radiographic image interpretation such as, Red Dot, PCE/ PIE and formal radiographer reporting.⁹, ¹⁷ However, there is a paucity of evidence that has focussed on PIE specific studies that utilise large sample sizes to report the specific regions misinterpreted when radiographers examine trauma radiographs.

This study’s purpose was to address this knowledge gap, by identifying the anatomical regions that elicited the most common false‐negative radiographer PIEs. Additionally, it aims to report the specific bones and types of injuries involved in the false‐negative interpretations. The overarching outcome of this study could lead to more focussed education, so that radiographers are able to communicate ‘urgent and unexpected findings’ to referrers, assisting in the treatment of patients and reducing missed pathology.

Method

Ethics

Ethical approval was granted from Metro South Human Research Ethics Committee and Queensland University of Technology Human Research Ethics Committee.

Design

This retrospective clinical audit was conducted over a two‐year period from January 2017 to December 2018.

Study setting and participants

This study was set in a metropolitan hospital of South East Queensland. At this site, PIE is undertaken 24 hours a day, 7 days a week, within the ED. All radiographers are required to participate in PIE when rostered in the ED. Every month, over the two‐year period, one radiographer randomly selected 100 PIE examinations. The same radiographer collected the data each month. At the study site, the scope of PIE included bony fractures, dislocation/subluxations, knee lipohaemarthrosis, elbow joint effusions, foreign bodies, pneumothorax and pneumoperitoneum.

Sample size calculation

The research team took a pragmatic approach to developing a suitable method to calculate the sample size of radiographic examinations to be audited per month. The key objectives were to obtain a random sample representative of a variety of radiographic examinations including different anatomical regions, different times of the day and a variety of radiographers. The size of the monthly sample also needed to be sustainable as a long‐term quality assurance audit. The Royal College of Radiologists suggests that a sample size of 100 consecutive referrals is adequate for a local clinical audit.¹⁸ A monthly sample of 100 PIE examinations for the study period was collected for review by choosing a letter of the alphabet and then selecting the patients with a last name of that letter. On average, this equated to using four consecutive letters to obtain the 100 cases. The following month in the audit cycle, the subsequent letters of the alphabet were used to collect and randomise the 100‐case sample.

Procedure

The PIE description for each examination was compared with the radiologist’s report to determine the category of the evaluation. The categories correspond with previous literature.⁶ These eight categories are as follows:

• True Positive (TP)

• True Negative (TN)

• False Positive (FP)

• False Negative (FN)

• False Negative/True Positive (FN/TP)

• True Positive/False Positive (TP/FP)

• Unsure

• No Participation

A TP was assigned when the patient had an abnormality and was correctly identified by the radiographer. If the patient had no abnormality and the radiographer stated no abnormality present, it was deemed a TN. A FP was assigned when the radiographer indicated an abnormality was present, but the patient had no pathology. Any PIEs that indicated no abnormality when one was present were assigned a FN. A partially correct PIE was assigned the FN/TP category. For example, where a fracture of the fourth metacarpal was identified but it was in fact a fracture of the fifth metacarpal. This FN/TP category included cases where two or more pathologies were present, but only one was identified. The TP/FP was assigned when multiple pathologies were identified by the radiographer, but only one was present. For example, ‘fracture of the left distal radius and ulna’ when only a distal radius fracture was present. If the radiographer was unsure the PIE was categorised as ‘unsure’. If no PIE was made, it was labelled ‘No Participation’. The 'Unsure' and ‘No Participation’ categories were combined to form the ‘other’ category throughout this study. These categories were combined for analysis as both involved a lack of decisive interpretation made by the radiographer.

Data analysis

The 100 cases each month were entered into a Google form¹⁹ recording anatomical region imaged, radiographer PIE and the category of the PIE (e.g. TP, TN). These data were culminated into a spreadsheet, which was exported into Microsoft Excel²⁰ for data analysis. These categorical data were then analysed to determine the distribution and overall PIE accuracy. The first category counted was the result of the PIE (e.g. TP, TN), to demonstrate the overall distribution. Further analysis was undertaken to determine the specific regions with the highest number of FNs. The radiologist’s reports for all the FN and FN/TP within the focused regions were then interrogated to identify the specific traumatic pathology and these findings were grouped into categories. The number of responses coded into each category was then recorded and expressed as a percentage of total cases.

To demonstrate reliability of the audit methodology, inter‐rater and intra‐rater reliability were calculated on a subset of the sample prior to analyses. This involved an additional radiographer and the auditor re‐marking 5% of the total sample (n = 120). Favourable inter‐rater reliability (kappa > 0.85 for all cases) and intra‐rater reliability (kappa > 0.90 for all cases) were observed, indicating a reliable audit process.

Results

The total number of PIEs provided over the two‐year study period was 52 900. This clinical audit reviewed 2402 cases. Table 1 presents the demographics of the anatomical regions within this study. The regions are displayed as combined areas, with upper extremity including humerus to phalanges and lower extremity including femur to phalanges. The overall accuracy of radiographer PIEs was 88.7% for the study period. Table 2 demonstrates the number of PIE for each region of the 2402 cases. The two categories that recorded the highest interpretive errors by radiographers were upper extremities with 33.5% (804/2402) and lower extremities with 28.5% (685/2402) of the total examinations reviewed. Within these categories, phalanges (hands and feet), foot, ankle, hand and wrist contained the highest number of FNs. Within the 2402 examinations, the majority of PIE interpretations were TNs, representing 70.5% (1694/2402), where radiographers correctly identified normal examinations. Figure 1 demonstrates the distribution of categories of the total sample. There were 130 examinations that were categorised as an incorrect interpretation (FP, FN, FN/TP and TP/FP). Of these examinations, 58.8% (76/130) were FN and 30.8% (40/130) were FN/TP. FNs of the upper extremity accounted for 34.2% (26/76) of the total FNs within this study. The lower extremity category amounted for 39.4% (30/76) of the FNs. Within the upper and lower extremities, there was a total of 86 FNs and FN/TP. 69.7% (60/86) of these were within the regions of the phalanges (hands and feet), feet, ankles, hands and wrists (see Table 3). Of these 60 examinations, 35% (21/60) were FN/TP.

Table 1.

Demographics of data for each anatomical region of the body.

Anatomical Region	TP†	TN‡	FP¶	FN^	TP†/FP¶	TP†/FN^	Other	Total
Upper Extremities	236	492	5	26	0	20	25	804
Lower Extremities	103	504	6	30	1	10	31	685
Skull	2	16	0	0	0	1	1	20
Spine (inc. ST§ neck)	4	88	0	4	0	0	12	108
Chest & Abdo	14	387	1	6	0	1	64	473
Pelvis	21	123	1	6	0	2	9	162
Shoulder Girdle	47	84	0	4	0	6	9	150
Total	427	1694	13	76	1	40	151	2402

^§ Soft Tissue.

^† True Positive.

^‡ True Negative.

^¶ False Positive.

^{^} False Negative.

Table 2.

Anatomical breakdown audited examinations.

Region	Number of PIEs
Abdomen	19
Ankle	219
Calcaneus	5
Cervical Spine	31
Chest	449
Clavicle	18
Elbow	126
Femur	24
Finger	121
Foot	173
Forearm	120
Hand	211
Hip	30
Humerus	19
Knee	179
Lumbar Spine	51
Mandible	5
Orthopantomography (OPG)	13
Orbits	2
Pelvis	132
Ribs	2
Sacrum	1
Shoulder	132
Soft Tissue Neck	8
Sternum	3
Thoracic Spine	17
Thumb	42
Tibia/Fibula	64
Toes	21
Wrist	165
Total	2402

Figure 1.

Categorisation of preliminary image evaluations

Table 3.

Number of false negatives for each region within the distal upper and lower extremities.

Anatomical Region	Number of False Negatives
Ankle	12
Finger	11
Foot	11
Hand	11
Thumb	1
Toes	3
Wrist	11
Total	60

Figure 2 demonstrates the number of pathologies not identified by radiographer PIE for each upper extremity region. As demonstrated, the phalanges of the upper extremity accounted for over half the upper extremity pathology (51%, 20/39). The fifth phalanx had the highest number of incorrectly interpreted pathologies with 35% (7/20). Four of these were from fractures of the middle phalanx. In the phalanges, 80% (16/20) of the pathologies were incorrectly interpreted fractures. The middle phalanx represented 50% (8/16) of the phalangeal fractures. In the wrist, 75% (3/4) of the ulna fractures not identified were of the ulnar styloid process. Both carpal pathologies involved the scaphoid, with a scaphoid fracture and a scapholunate joint subluxation not interpreted. For the metacarpals, there was one fracture not detected for each metacarpal excluding the third. There were also two foreign bodies not identified within the metacarpals, one each at the fourth and fifth metacarpal.

Figure 2.

Upper extremity pathology missed

Figure 3 demonstrates the number of pathologies not interpreted for each lower extremity region. The fibula accounted for the most pathology not identified for a single bone in this region. Of the tarsal injuries, 66.6% (4/6) were pathology involving the talus. All four of these fractures to the talus were described as avulsions within the radiologist report. The missed pathology of the navicular was a comminuted fracture, which involved both the proximal and distal articulating surfaces. 50% (4/8) of the fractures of the lower extremity phalanges were of the fifth phalanx. Of the remaining four, three involved the hallux, and one involved the second phalanx.

Figure 3.

Lower extremity pathology missed

Of the 60 FN interpretations of the distal upper and lower extremities, 28.3% (17/60) contained multiple injuries, where one was a FN (did not identify any pathology present) and 16 were FN/TP. The examination with the FN interpretation contained a comminuted fracture of the navicular and a fracture of the talus. Of the 16 FN/TPs, 15 contained fractures that were not interpreted and one contained an unidentified foreign body. Table 4 demonstrates the pathology that was not identified in the 16 multi‐injury cases (FN/TP). The remaining five of the 21 FN/TP examinations were PIEs that did not identify single pathology, compared to multiple (see Table 5). These were categorised as FN/TP as the PIEs lacked specificity or documented the incorrect bone.

Table 4.

The pathology present for False‐negative/ True‐positive category PIEs for multiple injuries.

	Pathology Identified	Pathology Missed
Multiple Injuries	# 4th metatarsal	# 2nd and 3rd metatarsals
	# 2nd, 3rd, 4th metatarsal	# 1st Cuneiform
	# distal radius	# ulna styloid process
	# distal radius	# ulna styloid process
	# distal phalanx, index phalange	# middle phalanx, index phalange
	# distal phalanx, ring phalange	# proximal interphalangeal joint, ring phalange
	# distal ulna	# distal radius
	# talus	# distal fibula
	Lucencies proximal metatarsal † , ‡	# 2nd, 3rd, 4th metatarsals
	Comminuted # index phalange†‡	Comminuted # distal and middle phalanx ring phalange.
	# distal radius	# distal ulna
	# 5th metacarpal	# 4th metacarpal
	# distal radius	# ulna styloid process
	# 4th metacarpal	Foreign body
	# thumb, distal phalanx	Subluxation of thumb. # 5th metacarpal
	Dislocation 5th proximal interphalangeal joint (hand)	# 5th middle phalanx (hand)

The word fracture has been replaced with the conventional # within the tables

^† Denotes comments which either specified the wrong bone or were not specific enough as to location or pathology type.

^‡ Statements in left hand column contain the pathology the radiographer commented on.

^§ CMCJ = Carpometacarpal Joint, # = Fracture, MC = Metacarpal

Table 5.

The pathology present for False‐negative/True‐positive PIE for singular injuries

	Pathology Present	Radiographer comment
Single Injuries	#(undisplaced) 1st phalange, distal phalanx† (Foot)	‘multiple fragments at proximal part of right 1st distal phalanx’
	#(undisplaced) fibula†	‘distal left bony fragment noted at inferior lateral malleolus’
	Subluxation of 1st carpometacarpal joint	’dislocation of 1st CMCJ. Nil #s’§
	# fifth phalange, middle phalanx† (Hand)	’5th Phalanx;’
	# second metacarpal†	’Oblique/spiral # of 1st MC’§

The word fracture has been replaced with the conventional # within the tables.

^† Denotes comments which either specified the wrong bone or were not specific enough as to location or pathology type.

^‡ Statements in left hand column contain the pathology the radiographer commented on.

^§ CMCJ = Carpometacarpal Joint, # = Fracture, MC = Metacarpal

Discussion

This is the largest study, to date, to report the specific type of injuries misinterpreted by radiographers performing PIE. This audit reviewed 2402 radiographic examinations over a two‐year period that demonstrated an overall PIE accuracy of 88.7%. The most common regions misinterpreted were ankle, foot, wrist, hand and phalanges (upper and lower extremity). The phalanges, in particular, produced the greatest number of pathologies incorrectly interpreted within PIEs. Another key finding identified was that not all pathologies were accurately located in examinations with multiple pathology. These findings are beneficial to education providers and radiographers as they highlight the anatomical regions and pathology commonly misinterpreted by radiographers undertaking PIE.

Phalanges (hands and feet) proved to be an area of frequent misinterpretation. Previous studies, including radiographer PIE and generalised image interpretation research, found phalanges to have one of the highest, if not the highest, number of incorrectly interpreted pathology.²¹, ²², ²³, ²⁴, ²⁵ This is consistent with the results achieved in this study. Within this study, radiographic examinations of a hand included the phalanges, so phalangeal pathology was unintentionally recorded within data collected for the hand. Therefore, the regions of FNs may have been influenced by the anatomical region requested by the referrer. The data demonstrated that phalanx pathology was found on hand requests. The centring point and positioning for a hand examination are different to that of a finger. Traditionally, a lateral hand is positioned with the phalanges superimposed, increasing the potential for phalangeal fractures to be missed. The centring point is important as correct centring is the best technique to demonstrate a fracture.²² Correct centring reduces distortion and allows visualisation through narrow joint spaces, for example interphalangeal joints.²⁶ This implies that performing a radiographic examination of the hand for phalangeal pathology may lead to missed fractures, as the centring is not optimal.²² Thus, it attributes to the increased number of missed phalangeal fractures. This theory can also be applied to feet/phalanges. These findings could be used to promote more specific requesting of radiological examinations by ED referrers.

Regions with multiple injuries were present in over a quarter of the FN and FN/TP examinations. This was higher than results reported in existing literature.²² Guly²² conducted a study involving ED doctors over a 6‐month period and found 134 patients had missed diagnoses, with 13 missed due to multiple injuries. Guly²² concluded that had the participants continued to search after they had identified an initial pathology, further injuries would have been identified in multi‐injury cases. Whilst Guly²² is referring to doctors, this reasoning could be applied to radiographers in their interpretation of the examinations with multiple pathology. Evidenced in the current study, 15 (of 17) cases identified one fracture correctly but missed further fractures or pathology. The importance of ongoing searching for abnormalities is not a new concept, with errors relating to multiple injury examination being termed ‘Satisfaction of Search’ or Subsequent Search Miss (SSM).²⁷, ²⁸ In an Australian publication,²⁸ the SBASS systematic search strategy was introduced with a step incorporated to specifically remind health professionals to search for multiple pathology. This strategy captures the key components to search when interpreting radiographs. The S stands for ‘Soft Tissue’, B for ‘Bones’, A for ‘Alignment of Joints’ and SS for ‘Satisfaction of Search’.²⁸ The ‘Satisfaction of Search’ or SSM component of this strategy is there to remind the interpreter not to conclude searching for abnormalities once an initial one is visualised.²⁸ Therefore, the findings from the current study further demonstrate the importance of the ‘Satisfaction of Search’ element within the SBASS strategy. Whilst the aim of this research was to demonstrate the common interpretive errors within the specific anatomical regions, a main cause of these errors may be unidentified subsequent injuries rather than the misinterpretation of pathology in a specific region.

The wrist was identified as a common region for missed pathology in this study. Of the 11 FN wrist examinations reviewed, five cases contained fractures of both the radius and ulna. It is interesting to note, in all five cases, the fracture of only one bone was identified. For example, the radius fracture was identified; however, the ulna fracture was not and vice versa. All FNs attributed to the ulna were the result of an unidentified second injury. Three (of four) accompanying ulnar fractures were of the ulnar styloid process. A fracture of the ulna styloid is common as it is located within the zone of vulnerability of the wrist created by the ligamentous attachments in this region.²⁹ The radioulnar ligament has two further ligaments within this zone that attach to it. It then attaches to the ulna styloid process.²⁹ When a radius injury occurs, this ligament tends to avulse the ulnar styloid process. As this styloid injury was repeatedly missed in fractures of the radius (three of the seven radius injuries), it can be suggested that this type injury needs further attention when assessing radiographs that have identified a fracture of the radius.²⁹ This is an important finding as it highlights a region that could be emphasised when providing education and training, so these pathologies are not missed in future image interpretation.

There are strengths and limitations of this study that warrant consideration. A strength of this study is the large sample size that was collected over a two‐year period. A limitation of this study was the inability to determine whether the PIE was describing pathology of a paediatric or adult patient. The lack of distinguishability between paediatrics and adults could have influenced the results as multiple studies have reported paediatrics to be a region of difficulty, due to lack of knowledge and familiarity.¹⁷, ²², ²³ As such, future research would warrant including the collection of patient age in the data.

Conclusion

This is the largest study, to date, to report the specific type of injuries misinterpreted by radiographers performing PIE. This audit reviewed 2402 radiographic examinations over a two‐year period that demonstrated an overall PIE accuracy of 88.7%. This study achieved its intended aims, demonstrating that wrists, hands, phalanges (upper and lower), ankles and feet were the regions commonly misinterpreted by radiographers undertaking PIE. Additionally, this study reported the specific type of injuries commonly misinterpreted by radiographers performing PIE. Another key finding of this study was the frequency of interpretive errors reported in cases that contained multiple pathologies. These types of errors are likely to be attributed to ‘Subsequent Search Miss (SSM)’ error. The results of this study can be used to highlight areas of emphasis for targeted curriculum, education and training. These findings may be of great relevance to healthcare providers, emergency and medical imaging departments seeking to improve radiographers’ and other health professionals’ image interpretation ability.

J Med Radiat Sci. 68(2021) 237–244