Reliability of the Pediatric Specific Musculoskeletal Ultrasound Scoring Systems for the Elbow, Wrist and Finger Joints

Abstract

Background:

Musculoskeletal ultrasound (MSUS) is increasingly being utilized in the evaluation of pediatric musculoskeletal diseases. In order to provide objective assessments of arthritis, reliable MSUS scoring systems are needed. Recently, joint-specific scoring systems for arthritis of the pediatric elbow, wrist and finger joints were proposed by the Childhood Arthritis and Rheumatology Research Alliance (CARRA) MSUS workgroup. This study aims to assess reliability of these scoring systems when used by sonographers with different levels of expertise.

Methods:

Members of the CARRA MSUS workgroup attended training sessions for scoring the elbow, wrist and finger. Subsequently, scoring exercises of B-mode and Power Doppler (PD)-mode still-images for each joint were performed. Inter-reader reliability was determined using two-way single score intra-class correlation coefficients (ICC) for synovitis and Cohen’s kappa for tenosynovitis.

Results:

Seventeen pediatric rheumatologists with different levels of MSUS expertise (1 – 15 years) completed a 2-hour training session and calibration exercise for each joint. Excellent reliability (ICC>0.75) was found after the first scoring exercise for all of the finger and elbow views evaluated on B-mode and PD-mode, and for all of the wrist views on B-mode. After a second training session and a scoring exercise the wrist PD-mode views reached excellent reliability as well.

Conclusion:

The preliminary CARRA MSUS scoring systems for assessing arthritis of the pediatric elbow, wrist and finger joints demonstrate excellent reliability among pediatric MSUS sonographers with different levels of expertise. This reliable joint-specific scoring system could serve as a clinical tool and scientific outcome measure with further validation.

Introduction

Juvenile idiopathic arthritis (JIA) is a significant cause of morbidity worldwide¹. Persistent joint inflammation can lead to functional limitations and lower health-related quality of life¹. Clinical evaluation of JIA disease activity includes the assessment of active joint count (AJC), physician global assessment (PGA), parent/patient global assessment, presence and duration of morning stiffness and biologic markers of inflammation². While these variables are included in validated composite outcome measures such as the Juvenile Arthritis Disease Activity Score (JADAS)³, recent guidelines acknowledge the need for further standardization of the patient/parent and PGA assessments². The PGA can have poor interrater reliability among providers, particularly in patients with low disease activity or inactive disease⁴. In addition, the reliability of AJC is limited⁵ and cannot always adequately identify joints with synovitis⁶.

Musculoskeletal ultrasound (MSUS) is increasingly being utilized in children⁷. It is well tolerated, readily available and relatively inexpensive compared to other imaging modalities. Normal age-related findings and definitions of pediatric synovitis on MSUS have been developed^8–11. MSUS can provide point of care information including the identification of subclinical disease^6,12,13. In order to provide objective assessments of arthritis, reliable scoring systems are necessary. They exist for rheumatoid arthritis^14,15 but in light of the unique sonographic features of the pediatric joint, specific MSUS scoring systems for JIA are needed. Our group recently proposed a joint-specific MSUS scoring system for the assessment of arthritis of the pediatric elbow, wrist and finger joints which demonstrated excellent reliability when used by experienced ultrasonographers (>7 years of experience)¹⁶. As MSUS use increases in pediatric rheumatology, reliable scoring systems for different levels of experience are needed. The objective of this study was to assess the interreader reliability of a B-Mode and Power Doppler (PD) mode scoring system for arthritis of the pediatric elbow, wrist and finger¹⁶ between sonographers with different levels of experience.

Methods

Seventeen pediatric rheumatology providers who are members of the Childhood Arthritis and Rheumatology Research Alliance (CARRA) MSUS workgroup participated. All providers had prior formal training in pediatric MSUS with 1 to 15 years of subsequent clinical experience in pediatric MSUS. For the analysis, participants were divided into 2 groups: an expert group, defined as participants with > 5 years of experience in MSUS and > 10 MSUS studies/week in children (n = 5), and a non-expert group (n = 12). This study was approved by the Cincinnati Children’s Hospital Medical Center Institutional Review Board (approval number: 2018-7939). Written assent and consent to participate was obtained from all children whose images were used in the scoring exercises. Written informed consent for publication was obtained from all the study participants.

For each of the joints (elbow, wrist and finger) participants received an initial 2-hour online virtual training session from an expert who had contributed to the preliminary CARRA scoring system (PVF, EO, JR)¹⁶. The training session included reviews of normal sonoanatomy, pathologic findings in JIA, the preliminary CARRA semiquantitative scoring system and case-based examples of scoring including pitfalls. Participants then took part in a calibration exercise using still images. Through a subsequent debrief, any remaining questions were addressed.

The preliminary CARRA scoring system for the elbow, wrist and finger joints consists of a semiquantitative grading from 0 to 3 (0- being normal or no pathology, 3- being severe pathology) for both B-mode and PD-mode images; tenosynovitis is assessed through a binary system with 0 (no pathology) or 1 (presence of pathology) in B- and PD-mode (Supplementary Table 1–4)¹⁶. Anonymized MSUS images of children 2 to 17 years of age were used, and the age of the patient was available to the participants.

Scoring exercises of both B-mode and PD-mode were completed for each joint. Interreader reliability was estimated using two-way single score intraclass correlation coefficients (ICC), a validated statistical measure of interreader reliability when variables in a study are rated by multiple coders¹⁷. An ICC was considered excellent for values of 0.75–1.00, good 0.60–0.74, fair 0.40–0.59, and poor <0.40¹⁸. As a nominal variable, agreement for the scoring system of the extensor tendons of the wrist in transverse view was assessed using Cohen’s Kappa coefficient. Kappa values from 0.0 to 0.2 indicates slight agreement, 0.21 to 0.40 indicates fair agreement, 0.41 to 0.60 indicates moderate agreement, 0.61 to 0.80 indicates substantial agreement, and 0.81 to 1.0 indicates almost perfect or perfect agreement¹⁷. For those views which did not attain excellent reliability (ICC) or moderate agreement (Kappa) for all participants for the lower end of the 95% confidence interval (CI), the participants underwent a subsequent round of calibration and scoring exercises using a different set of B-mode and PD-mode images. Statistical analysis software used was SAS v9.4©, Cary, NC.

Results

A total of 300 still-images were used for the first round of calibration and scoring exercises. These images were obtained in children aged 2–18 years, distributed equally across this age range, and including a broad number of images for each of the grade 0–3 categories (Supplementary Table 5). Interreader reliability results for the entire group of raters are shown in the table. In general, experts and non-experts combined demonstrated excellent interreader agreement for all B-mode and PD-mode views of the elbow and finger joints as well as the distal radioulnar and midline radiocarpal views of the wrist. For all participants, the view of the radiocarpal joint in ulnar probe position reached excellent reliability in B-mode but only good reliability in PD-mode for the lower limit of the CI (excellent for the ICC itself). Images of the extensor tendons of the wrists demonstrated moderate agreement in B-mode (kappa criteria, see materials and methods section) but only fair agreement in PD-mode for all participants Given the overall excellent reliability for synovitis and moderate agreement for tenosynovitis in B-mode, instead of proposing a modification of the scoring system, the PD mode scoring was repeated for the ulnar view of the radiocarpal joint and the extensor tendons following a second training session with special focus on the distinction of physiologic and pathologic findings. Regardless of level of expertise, excellent interrater reliability of 0.96 (95% CI 0.94–0.97) and moderate agreement of 0.72 (95% CI 0.61 – 0.83) were obtained following the second training and scoring exercise. Separate results for the expert and non-expert group are shown in Supplementary Table 6 and Supplementary Table 7.

Table.

Interreader Reliability Exercise for the Pediatric Elbow, Wrist, and Finger Joints

Joint		Exercise 1 ICC (95% CI)1 (n= 300 images )		Exercise 2 ICC (95% CI)1 (n= 28 images )
		B-Mode	PD-mode	B-Mode	PD-mode
Elbow	Anterior humeroradial and humeroulnar joint recesses in longitudinal view	0.97 (0.96–0.98)	0.89 (0.84–0.92)	N/A	N/A
	Posterior humeroulnar joint recess in longitudinal view	0.96 (0.94–0.97)	0.90 (0.86–0.92)	N/A	N/A
Wrist	Distal radioulnar joint recess in transverse view	0.94 (0.92–0.96)	0.96 (0.94–0.97)	N/A	N/A
	Dorsal radiocarpal joint recess in midline longitudinal view	0.93 (0.91–0.94)	0.97 (0.96–0.98)	N/A	N/A
	Dorsal radiocarpal joint recess in ulnar longitudinal view	0.87 (0.81–0.93)	0.83 (0.61–0.91)	N/A	0.96 (0.94 – 0.97)
	Extensor tendons in transverse view2	0.67 (0.53 – 0.81)	0.51 (0.37 – 0.65)	N/A	0.72 (0.61 – 0.83)
Finger	MCP dorsal joint recess in longitudinal view	0.93 (0.91–0.97)	0.97 (0.96–0.98)	N/A	N/A
	MCP volar joint recess in longitudinal view	0.93 (0.91–0.94)	0.87 (0.84–0.90)	N/A	N/A
	PIP volar long joint recess in longitudinal view	0.96 (0.95–0.97)	0.89 (0.86–0.91)	N/A	N/A
	PIP dorsal long joint recess in longitudinal view	0.96 (0.94–0.97)	0.95 (0.93–0.97)	N/A	N/A

CI: confidence interval. MCP: metacarpophalangeal joint, PIP: proximal interphalangeal joint.

¹Intraclass correlation coefficient (ICC) was based on a two-way random effects model for a single measure. Excellent ICC was defined to be between 0.75–1.00, good 0.60–0.74, fair 0.40–0.59 and poor <0.4¹⁸.

²Kappa values from 0.0 to 0.2 indicates slight agreement, 0.21 to 0.40 indicates fair agreement, 0.41 to 0.60 indicates moderate agreement, 0.61 to 0.80 indicates substantial agreement, and 0.81 to 1.0 indicates almost perfect or perfect agreement¹⁷.

Discussion

This study demonstrated excellent reliability of the preliminary semiquantitative CARRA MSUS scoring system for the pediatric elbow, wrist, and finger joints for providers with different levels of expertise. In addition, advanced MSUS concepts were successfully taught in a virtual format. By demonstrating the reliability of this semiquantitative measurement instrument, our study supports the potential use of MSUS scoring systems as an objective outcome measure at bedside and in further research studies.

Several MSUS scoring systems have been published in recent years^16,19–22. Only a few of these scoring systems have been evaluated with sonographers of variable experience. The pediatric MSUS scoring system of the knee proposed by the CARRA group was tested in pediatric rheumatology providers from the CARRA JIA Ultrasound workgroup (n=16) with <1 to 10 years of US experience. This exercise demonstrated good to excellent reliability for B-mode and PD-mode views²². Most recently, Rossi-Semerano et al²³ reported the reliability of the OMERACT pediatric US synovitis scoring system among 13 pediatric ultrasonographers of diverse subspecialty backgrounds: nine rheumatologists, two pediatricians, and two radiologists with varying degrees of experience. This group used a total of 75 images to evaluate the reliability of the most representative view of the wrist, elbow, MCP II, knees, and ankle joints. For the scoring systems of the MCP II, wrist, and elbow, they found fair to good reliability for the B-mode and excellent reliability for PD-mode. However, the scoring system used was not joint-specific, which the authors acknowledged²³. Our study used a larger sample of normal and pathologic images (n=300), including all views recommended in the evaluation of JIA and involved 17 pediatric rheumatology sonographers with different levels of expertise. Our joint-specific scoring system is based on one-plane view per area, using the most representative view to capture pathology, with any abnormal findings confirmed in a second plane.

Since all views reached excellent reliability¹⁸ after the second training session, the differences in the ICC noted after the first scoring exercise may have resulted from variability in the experience level of the participants interpreting MSUS rather than intrinsic issues with the scoring system. The lower ICC levels and 95% CI range in the MCP and PIP volar views in PD-mode for the expert group was a function of the number of positives rating being small. Reliability coefficients assume an equal distribution of positive and negative findings²⁴. The cutoffs used for differentiating the various levels of agreement (poor to excellent) were based on Cicchetti et al¹⁸. Other authors (Koo and Li²⁵) have proposed slightly higher cutoffs. It is important to note, that no universal agreement exists on how to define these levels. The ICC ranges between 0.00 and 1.00, with values closer to 1.00 representing stronger reliability. Given that most of our ICC values are 0.9 or above with the lower end of the range being very close to it, the results suggest very good reliability independent of the specific cut-offs used. We also considered the lower end of the 95% CI for the decision of whether agreement was good enough and did not just base this decision on the ICC value itself.

The value of training sessions that include a review of the definitions of key MSUS findings in healthy children and in children with arthritis, normal sonoanatomy, the scoring system, sonographic images with pathology, and a calibration session, was demonstrated. Participant feedback following the first and second training exercises noted that careful review of the normal spectrum of sonographic findings related to the degree of skeletal maturation was most helpful. Given the in-person meeting limitations imposed by the COVID-19 pandemic, this project was conducted in an online virtual setting. A major benefit of the change in format included the opportunity for the recording of the sessions and the possibility to reach out to a larger group. The session recordings were used by participants unable to attend the virtual meetings. The excellent reliability reached in this project supports the use of an online virtual format as an effective method for pediatric MSUS training, including training for scoring exercises.

Additional exercises of the reliability of the proposed MSUS scoring systems in real-time ultrasound images among different patient age groups, for instance via patient-based exercise, may follow. Future studies will also need to assess the construct and predictive validity of this preliminary MSUS scoring system.

Conclusion

A novel MSUS scoring system for B-mode and PD-mode of the pediatric elbow, wrist and finger showed excellent reliability among pediatric rheumatology ultrasonographers with varying levels of expertise. This was supported by an in-depth virtual training format. This joint-specific scoring system for pediatric arthritis could serve as a clinical and scientific outcome measure following further refinement and validation.