Radiologist variability in assessing the position of the cavoatrial junction on chest radiographs

Abstract

Objective:

To assess the variability in identifying the cavoatrial junction (CAJ) on chest X-rays (CXRs) amongst radiologists.

Methods:

23 radiologists (13 consultants and 10 trainees) assessed 25 posteroanterior erect CXRs (including 8 duplicates) and marked the positions of the CAJ. Differences in the CAJ position both within and between observers were evaluated and reported as limits of agreement (LOA), repeatability coefficients (RCs) and intraclass correlation coefficients and were displayed graphically with Bland–Altman plots.

Results:

The mean difference for within-observer assessments was −0.2 cm (95% LOA, −1.5 to +1.1 cm) and between observers, it was −0.3 cm (95% LOA, −2.5 to +1.8 cm). Intraobserver RCs were marginally lower for consultants than for trainees (1.1 vs 1.5). RCs between observers were comparable (2.1 vs 2.2) for consultants and trainees, respectively.

Conclusion:

This study detected a large interobserver variability of the CAJ position (up to 4.3 cm). This is a significant finding considering that the length of the superior vena cava is reported to be approximately 7 cm. We conclude that there is poor consensus regarding the CAJ position amongst radiologists.

Advances in knowledge:

No comparisons exist between radiologists in determining CAJ position from CXRs. This report provides evidence of the large observer variability amongst radiologists and adds to the discussion regarding the use of CXRs in validating catheter tip location systems.

INTRODUCTION

Peripherally inserted central catheters are frequently being used for long-term venous access to administer drugs such as antibiotics¹and chemotherapy,² as well as for the delivery of total parenteral nutrition.³ Peripherally inserted central catheters are often left in position for several weeks or months; it is therefore vital that the catheter tip is sited in an optimum position within the central circulation.^4,5 Techniques are available which can help reduce the incidence of catheter tip malposition, including X-ray fluoroscopy.^4–7 Fluoroscopy has limitations; it is an expensive resource,^8,9 has risks from the use of ionizing radiation⁹ and is impractical for patients who are critically ill.⁵ A newer and more popular alternative to fluoroscopic guidance is the use of electromagnetic tracking and intracavity electrocardiography.^6,10

The successful introduction of catheter tip positioning systems within clinical practice has relied on validation against a “gold standard”, a post-insertion chest X-ray (CXR). Reports of technical success do vary; in a report by Johnston et al,⁵ catheter malposition rates, defined using a post-insertion CXR, have been reported. When an adequate position was defined as low superior vena cava (SVC) or cavoatrial junction (CAJ), 134 (56.1%) catheters [95% confidence interval (CI) 50–62%] were malpositioned. A separate study by Lelkes et al⁷ reported more favourable outcomes, where 375 (97.7%) of 384 patients had the catheter tip positioned appropriately; again, this was defined by post-insertion CXR.

Validation of catheter tip positioning systems using CXR is in our opinion problematic. It is widely speculated that assessment of catheter tip position on CXR is inaccurate and subject to interobserver variability.^11–15 For CXR to be a valid tool it would require that a radiologist is able to reliably identify the CAJ position. To our knowledge, the accuracy of this task, in this specifically trained group, has not been assessed. The aim of our study was to assess intraobserver and interobserver variability in identifying the CAJ using adult CXRs.

METHODS AND MATERIALS

Radiologists (consultants and trainees) from a single university hospital were invited to take part in this study. Recruitment was aimed at participants with general radiology experience and those with a specific interest in chest radiology were asked not to take part. 13 radiology consultants and 10 trainees volunteered. 17 randomly selected posteroanterior chest radiographs were collected from a picture archiving and communication system. All images had been previously acquired as part of an anonymized teaching archive and therefore, no formal ethical approval was sought. The CXRs were labelled with numbers 1–17. 8 of these 17 CXRs were randomly selected and duplicated. These images were then subsequently labelled as images 18–25 and had deliberate alterations to the shuttering borders and image annotations in order to reduce the chances of the duplicate images being detected by the observers. The decision regarding the number of images was based on the need to assess intraobserver and interobserver variability and the estimated time required by the observers to complete the task. The sample size used in this study was consistent with those used in similar studies reported in the literature.^12,16 All CXRs were acquired to a standard technique¹⁷ and acceptable image quality was verified by two of the study authors.

Each participant was asked to retrospectively indicate the position of the CAJ on each of the 25 CXR images, independently, using a hospital laptop. The laptop, usually used by on-call radiologists to report scans remotely, had a 1920 × 1200-pixel 17-inch screen running Microsoft Powerpoint^® 2007 (Microsoft^® Corp., Redmond, WA). It was considered that the reporting laptop provided acceptable image quality for the purposes of this research. All images were checked for quality on the laptop by two study authors. Also, if any participant felt that there were image quality issues which prohibited identification of the CAJ position, then they could move on to the next image. Furthermore, the laptop also conformed to the Royal College of Radiologists minimum specification for primary diagnostic display devices used for clinical image interpretation.¹⁸

Each of the radiologists were given basic instructions regarding the study and asked to place an arrow where they thought the position of the CAJ was in the craniocaudal plane (Figure 1). A research assistant was present at all times during the assessment in order to ensure each radiologist understood the instructions and that the viewing conditions remained consistent. After annotating each image with an arrow, the image was saved and the observer then moved on to the next image. Participants were not permitted to make changes to the windowing or magnification settings nor adjust the image post-processing parameters.

Figure 1.

A posteroanterior chest X-ray image illustrating an example of an observer annotating the craniocaudal position of the cavoatrial junction using an arrow tip (arrow). The 10-cm vertical scale used in the calibration process is present on the right side of the image.

Following data collection from all 23 radiologists, the annotated images were analyzed by a study researcher. A horizontal line was placed on each image to provide a horizontal reference point on the image which was in a superior position to the CAJ. The horizontal reference point selected was the superior border of the aortic arch and remained in a fixed position on each of the 17 original images. The vertical distance from the tip of the observer-placed arrow to the horizontal reference line (aortic arch) was measured on each CXR. On each CXR, there was a 10-cm scale on the right side of the image. This allowed the distance between the horizontal reference line and the tip of the manually placed arrow to be correctly calibrated. Calibration was based on distances at the image receptor surface.

Measurements between the observer annotations (arrows) and the horizontal reference line were undertaken using 400% magnification; this was selected to minimize any measurement errors. Each measurement was then repeated three times by the same study researcher and the mean value was recorded. Measurements were then entered into a Microsoft Excel^® (Microsoft^® Corp., Redmond, WA) spreadsheet. Measurements were compared with repeat measurements by the same observer and then repeat measurements between observers. Full details of the measurement and calibration processes are illustrated in Figure 2.

Figure 2.

Graphical illustration of the measurement and calibration processes: using the calibration scale on the right of the image, 10 cm radiographically equates to 6 cm on the image. The calibration factor (10.0/6.0 cm) equals 1.67 and is used to convert the 4.3 cm (aortic arch) to radiologist-applied cavoatrial junction (CAJ) marker (arrow) to its respective radiographic distance (4.3 × 1.67 cm = 7.2 cm). As a result, in this example, the radiologist has indicated that the CAJ is 7.2 cm inferior to the superior border of the aortic arch.

Statistical analysis

Several methods have been proposed for the evaluation of observer variability data. It is believed by many authors^15,19 that for the analysis of measurement studies, it is desirable to report the degree of agreement using multiple statistical methods as no single method is perfect and each has its own limitations. First, the method described by Bland and Altman²⁰ was used to assess the intraobserver and interobserver variability of CAJ position assessments. For the assessment of intraobserver variability, the difference in position between each of the eight paired images by the same observer was calculated (first CAJ assessment minus the second CAJ assessment). Using these data, the mean difference (between the repeat CAJ positions) and standard deviation were calculated, as well as the 95% limits of agreement (LOA). LOA are a simple method of estimating the agreement interval within which 95% of the differences of the second measurement, when compared with the first, would fall. For interobserver variability, the mean difference, together with the LOA, was calculated in a similar manner compared with the first observer (observer one) but excluding the eight repeated images.

Coefficients of repeatability (RC) were calculated for the intraobserver and interobserver variability. The RC, as defined by Bland and Altman,²¹ is based on the one-way analysis of variance with the subject as the factor and provides a measure of precision that represents the value below which the absolute difference between repeat measurements is expected to lie with a 95% probability after extracting biologic variability. To calculate RC, firstly, the within-subject variance $\left({{\displaystyle S_{\text{w}}}}^2\right)$ is calculated. Two CAJ identifications by the same/different observers will then be within $1.96\sqrt{2S_{\text{w}}}$ or 2.77S_w for 95% of the participants and this is the resultant RC value.

Intraclass correlation coefficients (ICCs) were also used to report the degree of agreement within and between observers. A number of different models can be used for computing the ICC value.²² In this study, to report the observer variability, a two-way random model²³ was used since the set of images presented to the radiologists were a random subset of images from a large teaching archive and the radiologists were also randomly selected from the population of radiologists. Different guidelines exist for the interpretation of ICC: it has been suggested that an ICC value of <0.40 indicates poor reproducibility, ICC values in the region of 0.40–0.75 indicate fair to good reproducibility and an ICC value of >0.75 shows excellent reproducibility.²⁴

RESULTS

A total of 184 paired images (23 observers; 8 duplicate observations) were assessed for intraobserver variability and the CAJ position was indicated on each of these images using a horizontal arrow. When comparing intraobserver variability for all observers, the mean difference in CAJ position was −0.2 cm [95% LOA (−1.5, +1.1 cm)]. 26 (14%) of the intraobserver paired differences were >1.0 cm. A more detailed analysis of intraobserver variability is presented in Table 1 and Figures 3 and 4, together with a breakdown by observer type (consultant vs trainee).

Table 1.

Results for the assessment of intraobserver variability in determining cavoatrial junction (CAJ) position on chest X-ray (CXR)

Variability measures	All n = 23	Consultants n = 13	Trainees n = 10
n	184	104	80
Mean difference (cm)	−0.2	−0.2	−0.3
SD (cm)	0.7	0.6	0.8
Lower 95% LOA	−1.5	−1.3	−1.8
Upper 95% LOA	1.1	0.9	1.2
RC	1.3	1.1	1.5
>1 cm (n) (%)	26 (14%)	10 (10%)	16 (20%)
>2 cm (n)(%)	2 (1%)	0 (0%)	2 (2%)

LOA, limits of agreement; n, number of paired measurements; RC, coefficient of repeatability; SD, standard deviation.

Mean difference refers to the mean distance between the CAJ position for all of the paired CXRs.

Figure 3.

Box-and-whisker plot providing an illustration of the median, interquartile range and minimum and maximum differences for the assigned cavoatrial junction positions between observer groups for image 1.

Figure 4.

Intraobserver variability of cavoatrial junction (CAJ) identification on chest X-rays for both consultant radiologists and trainees. Intraobserver variability refers to the differences between repeat CAJ positions by the same observer (within observer). The difference between the two positions has been plotted against the mean distance in the CAJ position from the horizontal reference line. cm, centimetre; SD, standard deviation.

For the assessment of interobserver variability, a total of 391 images (23 observers; 17 observations) were assessed and the CAJ position was indicated on each of the images. When comparing CAJ positions between all observers, the mean (interobserver) difference was −0.3 cm [95% LOA (−2.5, +1.8 cm)]. A total of 124 (33%) of the paired differences between observers were >1.0 cm. A more detailed analysis of interobserver variability is presented in Table 2 and Figure 5, including analysis between observer types. Upon review of Figure 5, there was some linearity for paired differences between consultants and a distinct small cluster of paired differences above the upper LOA for trainees. The linearity could be explained by more senior observers identifying the CAJ as an area on the image and not as a finite point, whereas the small cluster could represent a small number of more novice trainees.

Table 2.

Results for the assessment of interobserver variability in determining cavoatrial junction position on chest X-ray

Variability measures	All n = 23	Consultants n = 13	Trainees n = 10
n	374	204	170
Mean difference (cm)	−0.3	−0.5	−0.1
SD (cm)	1.1	1.1	1.1
Lower 95% LOA	−2.5	−2.6	−2.4
Upper 95% LOA	1.8	1.5	2.1
RC	2.2	2.1	2.2
>1 cm (n) (%)	124 (33%)	71 (35%)	53 (31%)
>2 cm (n) (%)	37 (10%)	22 (11%)	15 (9%)

LOA, limits of agreement; n, number of paired measurements; RC, coefficient of repeatability; SD, standard deviation.

Mean difference refers to the differences between Observer 1 measurements and the remaining observers for each of the 18 images.

Figure 5.

Interobserver variability of cavoatrial junction (CAJ) identification on chest X-rays for both consultant radiologists and trainees. Interobserver variability refers to the differences in CAJ position between multiple observers. These differences are plotted against the mean of the two CAJ positions relative to the horizontal reference line. All calculations for interobserver variability were based on the CAJ positions by Observer 1. cm, centimetre; SD, standard deviation.

The variability within observers (intraobservers) and the between-observer variability (interobservers) were further assessed using an ICC. Overall, the mean ICCs for the overall cohort were 0.901 (95% CI 0.849–0.927) and 0.347 (95% CI 0.200–0.467) for intraobserver and interobserver variability, respectively. Different guidelines exist for the interpretation of ICC: it has been suggested that an ICC value of <0.40 indicates poor reproducibility, ICC values in the region of 0.40–0.75 indicate fair to good reproducibility and an ICC value of >0.75 shows excellent reproducibility.²⁴ The ICC values across the different observer types are displayed in Table 3.

Table 3.

Intraclass correlation coefficients (ICCs)

Observer type	Within observers				Between observers
	n	ICC	95% CI		n	ICC	95% CI
All	184	0.901	0.849	0.927	374	0.347	0.200	0.467
Consultants	104	0.917	0.878	0.944	204	0.355	0.151	0.511
Trainees	80	0.882	0.816	0.924	170	0.354	0.125	0.522

CI, confidence interval; n, number of paired measurements.

Measurement differences in CAJ position were based on adjustment for magnification at the image receptor surface. The CAJ is not in direct contact with the image receptor and will, therefore, be subject to radiographic magnification (RM). As a result, measurement differences between and within observers are likely to be influenced by the degree of magnification (CAJ to image receptor distance). RM can be quantified using the following equation:²⁵

$$\text{RM}=\frac{\text{Source\hspace{0.17em}to\hspace{0.17em}image\hspace{0.17em}receptor\hspace{0.17em}distance}}{\text{Source\hspace{0.17em}to\hspace{0.17em}image\hspace{0.17em}receptor\hspace{0.17em}distance}\mathrm{-}\text{CAJ\hspace{0.17em}to\hspace{0.17em}image\hspace{0.17em}receptor\hspace{0.17em}distance}}$$

Depending on the distance between the CAJ and the image receptor surface, measurements would need to be adjusted for magnification.

DISCUSSION

Catheter tip location systems are now available and are able to provide an indication of central venous catheter (CVC) tip position. In order to compare the results of catheter tip location systems, a reference standard must be available. In recent studies, electromagnetic detection systems have been compared against chest radiography.^7,26 However, in recent years, several authors have questioned the value of a CXR in defining tip position, arguing that for CXR images to be an acceptable standard, they would need to be consistent and accurate in identifying tip position.^27,28 Studies have shown that there can be constant disagreement as to the ideal position of a CVC on CXR.^9,14 There is, however, some consensus that CVC tips should be located at the CAJ.²⁹

For the CAJ to be a sound reference point would require that this anatomical landmark can be repeatedly and consistently identified from CXRs. According to the work by Aslamy et al,³⁰ the CAJ is defined as the caudal margin of the SVC at the level below which the SVC flares into the right atrial chamber. Radiographically, the CAJ has often been considered to be the right superior heart border in the plane of the SVC as an approximation.³¹ The report by Aslamy et al³⁰ correlated radiographic landmarks with MRI scans and demonstrated that the right superior border of the heart on a CXR is composed of the left, rather than the right, atrium in 38% of patients. From this, they and others have argued that the cardiac silhouette on a CXR in the region of the SVC is an unreliable indicator of CAJ.^30,29

To our knowledge, our study is the first report on the variability of the CAJ position assessed by radiologists using CXRs. When comparing repeat measurements by the same observer (within subject), 95% of CAJ positions were within 2.6 cm of each other. Variation was marginally smaller for consultant radiologists than for trainees. This feature was also experienced in the study by Wirsing et al,¹⁴ who compared senior and junior radiologists in determining CVC tip malposition. For the study group as a whole, over three-fourth of within-subject CAJ position assessments were <1 cm apart. This suggests that observers are consistent when invited to undertake repeat assessments of CAJ position. Results are likely to reflect an individual's consistency in applying internal definitions when asked to provide an opinion on the CAJ position on CXRs.

When comparing the determination of CAJ position between observers, the agreement was lower. For the cohort as a whole, 95% of paired CAJ assessments were within 4.3 cm of each other. This equated to around two-third of paired assessments being within 1 cm or less of each other. Comparison between observer types also demonstrated that more senior observers were marginally more consistent in their assessment of CAJ position. On the whole, there was a higher disagreement in the assessment of CAJ position between observers and this may be due to a lack of accepted radiological landmarks and definitions within the radiological community.

ICCs can provide a useful tool for the assessment of observer variability. Within our study, ICC values for the assessment of intraobserver variability were above 0.88 and based on Rosner's²⁴ work, this can be interpreted as excellent reproducibility. When interpreting ICC values, there were some evidence of intraobserver differences when separating consultants from trainees (ICC 0.92 vs 0.88, respectively). Both groups can, however, be categorized as excellent for intraobserver variability. For assessments between observers, the ICC values were lower; the group as a whole generated an ICC value of 0.35, which can be classified as poor agreement. There was little difference between consultants and trainees (ICC 0.36 and 0.35, respectively). ICC values are limited in that they are coefficients and do not provide information regarding whether any agreement or disagreement is clinically acceptable.

It has been observed that between 20 and 47% of CVCs are incorrectly classified to be in an intra-atrial position.³⁰ Aslamy et al,³⁰ in a report in 1998, suggested that the effects of parallax and variations in radiographic technique may lead to erroneous reporting of malposition. An additional factor that may have contributed to this figure is the lack of agreement regarding the radiological landmarks for the CAJ. Our study goes some way in proving that there is a lack of accepted landmarks between radiologists for identifying the CAJ. Even with standardization, based on the study by Aslamy et al, a CXR is unlikely to be insufficient for allowing the precise identification of CAJ position. Other methods such as transoesophageal echocardiography (TOE) are likely to be superior. Confirming this, in a recent study comparing TOE with CXR, the sensitivity and specificity for CXR, in determining catheter malpositioning, were 47% and 66%, respectively.¹⁴ However, the use of TOE to replace chest CXR in determining catheter malpositioning for all CVC placements will have significant resource implications, is not practical and would be unpopular with patients.

When reporting this study, we accept that there are limitations. Both radiographic technique and parallax are likely to affect an observer's ability to localize the CAJ. The adequacy of CXR images included in this study was determined by two co-authors. Measurement variability may have been different if a wider range of CXRs was included. A further limitation of this study was the lack of a definitive indicator of actual CAJ position. One option was to use CT images and generate a RaySum-style CXR image,³² from which observers could locate the CAJ. This was not considered to be a viable option, since there are large differences in image quality between a conventional CXR and those generated from CT data. In addition, CT images are almost always generated in the supine position with arms raised above the head. This is a totally different position to that of a typical CXR and the resultant differences in apparent CAJ position would need to be quantified.

Radiologists were invited to participate from a single UK hospital. Participation was voluntary following an e-mail invitation; this may have introduced some bias in that radiologists who had concerns regarding their ability to precisely identify the CAJ may not have opted to take part. Due to the voluntary nature of this study, the true variability CAJ assessments could be greater than reported if those who considered themselves to be poor at this task opted not to take part. We, however, do feel that this is unlikely to be a factor since observer assessments were anonymized from the outset and recruitment was not an issue. Observations were also undertaken on a hospital laptop and not on a typical reporting grade picture archiving and communication system workstation. This is again unlikely to be significant as the laptop was used in image interpretation, images were checked for both anatomical content and quality and the laptop specification met national standards.¹⁸

RM is also a consideration when interpreting measurement differences. Digital radiographs have scales located on the image, which provide an indication of distance measurements calibrated to those on the surface of the image receptor. The CAJ sits within the thorax and will be a distance away from the image receptor surface and will, therefore, be subject to magnification. By way of an example, a 2.0-cm² region at a posteroanterior tissue depth of 4 cm would cast a 2.1-cm² area on the resultant radiograph. At a depth of 8 cm, this would increase to 2.2 cm² and as such, the CAJ will not be a finite point on a CXR but will correspond to an area, the size of which will depend on the distance away from the image receptor.

Based on results from this study, there is a need for further work. One option is the role of training in reducing observer variability. Within our study, we purposefully opted not to provide any training on the identification of CAJ position as we sought to capture the current levels of variability. We accept that it would be useful to ascertain the performance of assessing CAJ position following a period of training. In order to achieve this, it is important to gain a consensus on the radiological landmarks which promote accurate delineation of the CAJ position.

CONCLUSION

Accurate assessment of CVC tip position is essential in order to ensure adequate line function together with long-term patient safety. The limitations of chest radiography, in providing precise tip position, have been previously identified. This problem is further exacerbated by a lack of consistency amongst trained radiologists in the localization of the CAJ. Currently, the consensus between radiologists is that the CAJ position sits within a 4.3-cm craniocaudal region within the mediastinum. This is a significant finding considering that the length of the SVC is reported to be approximately 7 cm.