Intra- and interobserver reliability analysis of digital radiographic measurements for pediatric orthopedic parameters using a novel PACS integrated computer software program

Abstract

Background

The between-observer reliability of repeated anatomic assessments in pediatric orthopedics relies on the precise definition of bony landmarks for measuring angles, indexes, and lengths of joints, limbs, and spine. We have analyzed intra- and interobserver reliability with a new digital measurement system (TraumaCad Wizard™).

Methods

Five pediatric orthopedic surgeons measured 50 digital radiographs on three separate days using the TraumaCad system. There were 10 anterior–posterior (AP) pelvic views from developmental dysplasia of the hip (DDH) patients, 10 AP pelvic views from cerebral palsy (CP) patients, 10 AP standing view of the lower limb radiographs from leg length discrepancy (LLD) patients, and 10 AP and 10 lateral spine X-rays from scoliosis patients. All standing view of the lower limb radiographs were calibrated by the software to allow for accurate length measurements, using as reference a 1-inch metal ball placed at the level of the bone. Each observer performed 540 measurements (totaling 2,700). We estimated intra- and interobserver standard deviations for measurements in all categories by specialists and nonspecialists. The intraclass correlation coefficient (ICC) summarized the overall accuracy and precision of the measurement process relative to subject variation. We examined whether the relative accuracy of a measurement is adversely affected by the number of bony landmarks required for making the measurement.

Results

The overall ICC was >0.74 for 13 out of 18 measurements. Accuracy of the acetabular index for DDH was greater than for CP and relatively low for the center–edge angle in CP. Accuracy for bone length was better than for joint angulations in LLD and for the Cobb angle in AP views compared to lateral views for scoliosis. There were no clinically important biases, and most of the differences between specialists and nonspecialists were nonsignificant. The correlation between the results according to the number of bony landmarks that needed to be identified was also nonsignificant.

Conclusions

Digital measurements with the TraumaCad system are reliable in terms of intra- and interobserver variability, making it a useful method for the analysis of pathology on radiographs in pediatric orthopedics.

Introduction

The reliability of radiographic angles, ratio, and length measurements is a key factor for clinical decisions on treatment plans in pediatric orthopedic pathologies, such as hip dysplasia, limb deformities, and scoliosis. A 1998 German study on conventional hip X-rays concluded that the center–edge angle of Wiberg, Sharp’s acetabular angle, and acetabular index of the weight-bearing surface are more reproducible than other measurements, such as the acetabular index of depth to width and neck shaft angle [1]. The reason for the unsatisfactory results was difficulties in defining the landmarks necessary for the various angle determinations [1]. Other studies on conventional radiographs of the hip found the center–edge angle of Wiberg and the modification of this angle according to Ogata (extending the measurements to the lateral edge of the condensed acetabular root, the sourcil) to be reliable intra- and interobserver measurement methods, with an accuracy of 2° [2, 3]. An earlier paper from Norway reported that the interobserver reliability of the center–edge angle of Wiberg and femoral head coverage was better than that in the Catterall, Salter Thompson, and Herring lateral pillar classifications in Perthes’ disease [4]. In 2004, a study from the USA found moderate intra-observer and poor interobserver reliability for measurements of the tibiofemoral angle using mid-diaphyseal lines on conventional long-standing X-rays and short-knee anterior–posterior (AP) views prior to knee surgery [5]. In a 2005 paper from Austria which compared mechanical axis measurements between conventional and digital standing view of the lower limb X-rays, the Spearman correlation coefficient was 0.74 and 0.91 for manual and digital measurements, respectively, and showed the digital measurements to be more accurate [6]. The axis lines were drawn from the mid-femoral head to the center of the knee joint and from there to the center of the ankle. Another paper from Germany published in 2006 found higher intra-observer reliability of digital versus conventional measurements of leg length and of mLPFA, mLDFA, mMPTA, mLDTA, and JLCA (the “Baltimore numbers”) [7]. The dedicated software allowed for calibration of the digital X-rays, greater accuracy of length measurements, and significantly shorter time for evaluation than conventional radiographs [7]. The intra- and interobserver reliability of evaluating acetabular dysplasia on digital X-rays was proven to be equal to conventional radiographs in a 2006 paper from the USA [8]. A more recent paper from the USA concluded that there was limited reliability in intra- and interobserver measurements of a great number of radiographic features of hip dysplasia and femoral acetabular impingements. The problem was that, although the observers agreed on the structural definition of normal anatomy, the method of measurements was left to the individual observer, thus leading to poorer reliability [9].

The integration of the PACS into multiple medical institutions created the possibility of taking numerous digital measurements for the assessment of anatomic and deformity parameters of the torso and limbs. There is now a number of commercial digital software programs to provide the medical practitioner with even more tools to assess normal and pathological anatomic features using specific bony landmarks. For example, the anatomic features of the hip joint, the length and alignment of the lower limbs, and the curvature of the spine can be measured with these digital software tools. These angles, indexes, and length differences are crucial for decision-making in pediatric orthopedics.

We set out to design a reproducible technique of carrying out measurements on digital X-rays of various anatomic parameters in the field of pediatric orthopedics. The pediatric section of the TraumaCad software (TraumaCad version 2.0, Orthocrat™) was designed so, that when the anatomy of the hip, long leg, spine, or foot and ankle is analyzed, an illustration corresponding to various conventional measuring tools would appear at the bottom of the page so the position of markers on specific anatomic locations can be reliably located (Fig. 1). Some of the illustrations contain a short text for providing a more exact definition of the anatomic landmarks. In addition, a dedicated wizard was developed to guide the observer during acetabular and lower limb length and angle analysis in order to mark anatomic landmarks for carrying out the various measurements.

Fig. 1.

Hip joint radiograph of a developmental dysplasia of the hip (DDH) patient; acetabular index tool

The full spectrum of the TraumaCad applications includes ten planning modules for adult hip joint replacement, pediatric hip joint, deformity correction, spine, foot and ankle, adult knee joint replacement, upper limbs, trauma, and 3-D implant visualization with computed tomography (CT) imaging.

The purpose of the current study was to assess intra-observer reproducibility and interobserver reliability of the TraumaCad software for conducting measurements on PACS digital radiographs. Anatomical angles, indexes, and length of bones in four common pediatric orthopedics conditions (hip dysplasia, cerebral palsy, lower limb length inequality, and scoliosis) were measured by five experienced specialist observers during three separate sessions. The data were analyzed statistically in order to assess intra-observer (error) and interobserver (doctor variance) reliability.

Materials and methods

Study design

All relevant radiographs were imported from our medical center’s PACS into the integrated TraumaCad software. Twenty hip joint radiographs (AP views) of patients being followed up for developmental dysplasia of the hip (DDH, n = 10) and cerebral palsy (CP, n = 10) were measured by means of the TraumaCad software hip joint wizard (Figs. 1–5). Ten digital long leg standing (AP views) radiographs of patients followed up for lower limb deformities were calibrated with a standard size metal ball (i.e., 1 inch = 25.4 mm). The marker was placed at the level of the bone of interest at the same distance from the detector plate, whether it is DR or CR (Fig. 6). The lower limbs were measured for bone length and joint orientation (“Baltimore numbers”) using the TraumaCad LLD wizard (Figs. 7 and 8). The Cobb angle was measured using the TraumaCad software in ten spinal scoliosis radiographs (AP and lateral views) of patients followed up for scoliosis (Figs. 9 and 10). All of the measurements were carried out by five senior orthopedic surgeons after a consensus conference for deciding upon the measuring technique. Only one of them (ES) was a frequent user in applying the TraumaCad software for all of the measurement fields provided by the system. This was the first exposure for the other four participants (Table 1).

Fig. 5.

Hip joint radiograph of a CP diplegic patient; lines and angles drawn

Fig. 6.

Standing view radiograph of a leg length discrepancy (LLD) patient; calibration with the metal ball

Fig. 7.

Long-standing radiograph of an LLD patient; deformity and length analysis [11]

Fig. 8.

Standing view of lower limbs radiograph of an LLD patient; simulation of osteotomy and lengthening

Fig. 9.

Spine radiographs of a scoliosis patient; frontal curve analysis with the Cobb Angle tool

Fig. 10.

Spine radiographs of a scoliosis patient; sagittal curve analysis with Thoracic Kyphosis Angle tool

Table 1.

List of measurements on ten radiographs on three separate sessions by the five observers

1.	Acetabular index Rt + Lt (DDH)
2.	Acetabular index Rt + Lt (CP)
3.	Center–edge angle Rt + Lt (CP)
4.	Reimer’s index Rt + Lt (CP)
5.	Femur length Rt + Lt (LLD)
6.	Tibia length Rt + Lt (LLD)
7.	mLDFA Rt + Lt (LLD)
8.	mMPTA Rt + Lt (LLD)
9.	Cobb angle AP + lateral (scoliosis)

DDH developmental dysplasia of the hip, CP cerebral palsy, Rt right, Lt left, AP anterior–posterior, LLD leg length discrepancy, mLDFA mechanical lateral distal femoral angle, mMPTA mechanical medial proximal tibial angle

Each session of measurements by each observer included: 20 acetabular index angles of 10 DDH patients (right and left), 60 acetabular index angles, center–edge angles of Wiberg and Reimer’s index on 10 CP patients (right and left), 40 lengths of tibiae and femora, 40 distal femur and proximal tibia mechanical angles of 10 LLD patients (right and left), and 20 Cobb angles of 10 scoliosis patients (AP and lateral) (Table 1). The total number of measurements for an observer in each session was 180.

Intra-observer reliability was tested by repeating the same set of measurements during three different sessions with an interval of at least 30 days between them; each observer made 180 measurements in a single session, with a total of 540. The total number of measurements for the five observers was 2,700. A single high-resolution computer was used for all measurements by all observers.

One specialist observer was designated as an “expert” for each measurement. The “expert” was always the observer with the most extensive clinical experience in treating the conditions for which the specific measurement is most relevant. For example, the director of the neuromuscular clinic was the “expert” for measurements of DDH and CP pelvic X-rays, the director of the LLD clinic was the “expert” for measurements of standing view of the lower limb X-rays, and the director of the spinal clinic was the “expert” for the scoliosis measurements.

We examined the hypothesis that the relative accuracy of a measurement is adversely affected by the number of bony landmarks that must be identified to carry out that measurement, i.e., 4 landmarks for DDH (Figs. 2 and 3), 10 for CP (Figs. 4 and 5), 16 for LLD (Figs. 7 and 8), 4 for scoliosis AP view and 4 for scoliosis lateral view (Figs. 9 and 10). The study was designed and analyzed in consultation with the Laboratory of Statistics at our university.

Fig. 2.

Hip joint radiograph of a DDH patient; acetabular index wizard, for bony landmarking

Fig. 3.

Hip joint radiograph of a DDH patient; acetabular index, lines, and angles drawn

Fig. 4.

Hip joint radiograph of a cerebral palsy (CP) diplegic patient, with ten bony landmarks [10]

Statistical considerations

Random effects analysis of variance (ANOVA) models were fitted to each measure to estimate the intersubject, interobserver (between doctors), and intra-observer (measurement error) variances and standard deviations. Estimates were computed for the five observers combined, as well as for the measurements taken by “nonexperts” of a given specialty. We also compared the “nonexperts” measurements to those of the “expert” and examined the results for any significant differences using a paired t-test based on the average measurements for “experts” and “nonexperts” for each radiograph. The intraclass correlation coefficient (ICC) was used to summarize the overall accuracy of the measurement process relative to variations among subjects in each category. ICC values ranged from 0 to 1, with higher values reflecting more reliable measurements. An ICC value of 0.0 indicates that the variance is entirely due to measurement error and that none is the result of interpatient variability, whereas a value of 1.0 implies that the variance is due entirely to intersubject variability. An ICC value >0.75 was taken to indicate satisfactory measurement reliability (Figs. 1–4).

Source of funding

There was no external source of funding for this study; all measurements were done on our medical center computers, using installed software.

Results

Table 2 displays means and standard deviations of the right and left sides for the four categories of orthopedic patients (DDH, CP, LLD, and scoliosis), the intra-observer standard deviation, the interobserver standard deviation, and the ICC for each group of measurements.

Table 2.

Digital measurements for all observers i.e., intra-observer, interobserver results and intraclass correlation

Patient	Measurements	Mean	SD	SD (pat)	SD (error)	SD (doc)	ICC	ICC*	Dif*	P value**
1	Ac index Rt (DDH)	25.48	5.51	5.04	2.57	0.97	0.77	0.78	0.65	0.376
2	Ac index Lt (DDH)	25.53	6.35	6.11	2.56	0.00	0.85	0.84	−0.17	0.755
3	Ac index Rt (CP)	20.13	5.52	4.78	2.96	1.12	0.69	0.72	1.88	0.0205
4	Ac index Lt (CP)	18.20	6.02	5.36	3.18	0.27	0.74	0.73	1.17	0.0512
5	CE angle Rt (CP)	27.91	8.81	7.08	5.04	2.92	0.60	0.60	−1.31	0.2041
6	CE angle Lt (CP)	30.79	10.70	9.61	4.81	3.08	0.74	0.72	1.76	0.1754
7	Reimer’s index Rt (CP)	26.36	18.68	17.75	7.91	1.04	0.83	0.80	−0.70	0.6205
8	Reimer’s index Lt (CP)	18.67	11.65	11.69	3.31	1.03	0.92	0.92	−2.42	0.0163
9	Femur length Rt (LLD)	280.01	58.22	58.71	15.78	4.57	0.93	0.92	5.78	0.0454
10	Femur length Lt (LLD)	278.99	65.87	67.21	15.15	4.65	0.95	0.94	6.22	0.0133
11	Tibia length Rt (LLD)	226.06	61.21	61.62	17.30	3.02	0.92	0.91	4.26	0.0653
12	Tibia length Lt (LLD)	227.42	64.12	65.65	14.00	3.61	0.95	0.95	4.43	0.0354
13	mLDFA Rt (LLD)	87.54	4.73	3.89	2.93	0.29	0.63	0.62	−0.01	0.9866
14	mLDFA Lt (LLD)	93.43	12.77	13.07	2.81	0.59	0.95	0.95	0.92	0.0618
15	mMPTA Rt (LLD)	85.95	3.59	3.08	2.05	0.20	0.69	0.70	0.6	0.1846
16	mMPTA Lt (LLD)	88.94	5.07	4.77	2.20	0.49	0.82	0.81	0.7	0.0324
17	Cobb Angle AP (scoliosis)	31.15	17.80	17.86	5.60	1.01	0.91	0.91	2.64	0.0055
18	Cobb angle Lat (scoliosis)	40.15	18.41	16.22	8.48	5.95	0.71	0.68	−7.31	0.0345

* Difference to measurements without the “expert”

* ICC without the “expert”

** P values for comparing the all observers to without “expert” averages

There was an overall ICC >0.74 in 13 out of 18 measurements, with a greater level of accuracy for the acetabular index for DDH compared with CP. There was relatively low accuracy for the CE angle in CP and greater accuracy for bone length than for joint angulations in LLD. Accuracy for the Cobb angle was greater in AP views than for lateral views for scoliosis.

There were no clinically significant differences in the measurements of all observers taken together and the measurements when those of the “expert” were excluded from the analysis. There was also no significant difference in any of the results in association with the number of bony landmarks that needed to be identified according to the system protocol.

Discussion

Orthopedic surgery has always relied heavily on the interpretation of plain radiographs, with the measurements of sizes, angles, and indexes providing vital information to supplement the clinical examination. Orthopedic conditions that may appear at birth, such as DDH and scoliosis, or develop later in life, such as Perthes’ disease, slipped capital femoral epiphysis, limb deformities due to metabolic condition, and others, are even classified according to radiographic appearance. The severity of these diseases, the natural history, indications for surgery, and the follow-up of surgical results are based on specific radiographic measurements between defined landmarks for assessing hip joint development, lower limb length discrepancies and alignments, scoliosis curves, and more.

Conventional X-rays were, for decades, the main source of data for the definition and classification of these orthopedic conditions. The observer drew lines between previously defined landmarks that were based on common knowledge in order to measure lengths, angles, and various indexes according to the anatomic structure in question. Measurements of angles and indexes were shown to have higher interobserver reliability than radiographic classifications [4], but they might have not been accurate when anatomic landmarks were not clearly defined. The traditional measurements on conventional radiographs had worked well for the more common angles [1], but the definition and marking of bony landmarks may have been inaccurate and have probably led to less reliable results for the more complex angles and indexes. The introduction of the PACS to a growing number of medical institutions created new challenges and new opportunities. Most PACSs contain basic tools for measurements, such as rulers, angle tools, parallel lines, etc., but there are no specific guidelines for the bony landmarks in orthopedics. Dedicated digital software tools have been developed by companies worldwide in an effort to overcome these problems. Recent publications that compared manual measurements on conventional radiographs to digital measurements done on the PACS images with the aid of designated software (IMPAX. Agfa-Gevaert, Belgium; MediCAD Hectec GmbH, Altfraunhofen, Germany; Horizon Dx View 3.2.3.1, McKesson, Alpharetta, GA, USA) for lower limb alignment or acetabular morphology found that digital results were as reliable or even better than the manual measurements [6–8] and that they demanded less time for the process.

The TraumaCad software provides tools and templates for the preplanning of joint replacement and trauma surgery. During its development, we were able to specify our requirement for measurements that were relevant to pediatric orthopedics. The current version of the software allows for the calibration of radiographs using metal balls as well as any other radio-opaque object of a known size (Fig. 6). The software contains tools designated for the measuring of various pediatric orthopedic pathologies, amongst which tools for hip morphology analysis (Fig. 1): Reimer’s subluxation and acetabular indexes for DDH and CP, VCA angle of Lequesne, epiphyseal and tibiofemoral indexes, neck shaft and slip angle, metaphyseal diaphyseal angle (Fig. 4), limb length and angular measurements, including mechanical lateral proximal femoral, mechanical lateral distal femoral, mechanical medial proximal tibial, mechanical lateral distal tibial, and joint lateral conversion angles (Figs. 7 and 8), spinal measurements for scoliosis, i.e., Cobb angle, coronal and sagittal balance, sacral obliquity, pelvic radius angle, spondylolisthesis, thoracic kyphosis and trunk shift, and lumbar lordosis angle (Figs. 9 and 10). These designated tools are presented together with an illustration to remind the observer of the bony landmarks being used for each measurement. The hip morphology and the limb length and deformity analysis tools are constructed with a wizard that guides the observer to the order and location of the bony landmarks linked to the illustrative drawings that appear on the side of the screen (Figs. 9 and 10). The software allows, after limb length and deformity measurements, to perform center of rotation and angulation analysis and to simulate the planned osteotomy with deformity correction and lengthening for restoring normal limbs anatomy (Fig. 8).

In order to validate the reliability of these tools, five experienced pediatric orthopedic surgeons were recruited to take the measurements. Each observer performed three separate sessions of hip, lower limb, and spine measurements in our patients who were being followed up for DDH, CP, LLD, and scoliosis. The intra-observer error was 2°–3° for acetabular indexes in DDH and CP, 4°–5° for the CE angle in CP, 3–7% for Reimer’s index in CP, 14–17 mm for bone length in LLD, 2°–3° for joint orientation in LLD, and 5°–8° for the Cobb angle in scoliosis. The interobserver variance was 0°–1° for acetabular indexes in DDH and CP, 0.7°–3° for the CE angle in CP, 1% for Reimer’s index in CP, 3–4.7 mm for bone length in LLD, 0°–0.6° for joint orientation in LLD, and 1°–6° for the Cobb angle in scoliosis.

The ICC was >0.74 for 13 of 18 categories of measurements, and <0.74 for the right acetabular index in CP, the right CE angle in CP, the right mLDFA in LLD, and the right mMPTA in LLD. The error for these categories was in the range 3°–5° and the interobserver variance was 0°–2.9°.

We attribute the relatively high error and intra-observer variance values (14–17 mm) for bone length in LLD to the need to calibrate each image with the 1-inch metal ball before proceeding with the measurements. The manual calibration process created another source of error before initiating the actual length measurements. The recent introduction of an automatic calibration function (already installed in the new version of the software) after locating the metal ball in the field is expected to improve the level of accuracy. The relatively high error and interobserver variance values (5.9°–8.5°) for the lateral Cobb angle in scoliosis is probably due to difficulties in identifying the top end of the T3 vertebra. Lateral X-rays of the spine, especially in the upper thoracic area, are difficult to interpret.

We believe that these results establish that the TraumaCad system, which consists of specially designed digital measurement tools in the field of pediatric orthopedics, has high intra-observer reproducibility and interobserver reliability.

We also examined and failed to find any influence of expertise in a specialized area of pediatric orthopedics on the observer’s skill in measuring the various parameters described above. Moreover, the number of landmarks that were correctly identified on each radiograph or on the distribution of results was not affected either. We attribute these findings to the built-in system of wizards and graphical presentations that created good uniformity within and between the observers in the placement of the landmarks.

The incorporation of the designated software into the PACS will carry with it many advantages for the orthopedic surgeon. All digital images may be calibrated online using metal objects of a known size in order to increase the accuracy of measuring lengths and sizes. All measurements of angles and indexes may be matched online to illustrative figures with agreed upon clearly identified landmarks, and a written definition of each landmark can be added to the relevant page. The system will bring about better conformity for the individual observer, for a number of observers, and between different medical centers worldwide. A considerable amount of time will be saved by using compatible software: the various built-in tools have been constructed according to common orthopedic consensus and in collaboration with software developers, the PACS producers, and the orthopedic community at large. These tools can be continuously updated as needed. The images are measured on the PACS itself, with no need to export images to external software. All measured images can be stored in the patient’s file on the PACS for future reference. Cooperation between digital software developers, medical practitioners, and medical center administrations will greatly improve the accuracy, speed, and ease of common orthopedic digital measurements wherever they are needed.