Skip to main content
PMC home page
Annals of the Rheumatic Diseases logoLink to Annals of the Rheumatic Diseases
. 2005 Jan 7;64(7):1043–1049. doi: 10.1136/ard.2004.030387

Interobserver reliability of rheumatologists performing musculoskeletal ultrasonography: results from a EULAR "Train the trainers" course

A Scheel 1, W Schmidt 1, K Hermann 1, G Bruyn 1, M D'Agostino 1, W Grassi 1, A Iagnocco 1, J Koski 1, K Machold 1, E Naredo 1, H Sattler 1, N Swen 1, M Szkudlarek 1, R Wakefield 1, H Ziswiler 1, D Pasewaldt 1, C Werner 1, M Backhaus 1
PMCID: PMC1755572  PMID: 15640263

Abstract

Objective: To evaluate the interobserver reliability among 14 experts in musculoskeletal ultrasonography (US) and to determine the overall agreement about the US results compared with magnetic resonance imaging (MRI), which served as the imaging "gold standard".

Methods: The clinically dominant joint regions (shoulder, knee, ankle/toe, wrist/finger) of four patients with inflammatory rheumatic diseases were ultrasonographically examined by 14 experts. US results were compared with MRI. Overall agreements, sensitivities, specificities, and interobserver reliabilities were assessed.

Results: Taking an agreement in US examination of 10 out of 14 experts into account, the overall κ for all examined joints was 0.76. Calculations for each joint region showed high κ values for the knee (1), moderate values for the shoulder (0.76) and hand/finger (0.59), and low agreement for ankle/toe joints (0.28). κ Values for bone lesions, bursitis, and tendon tears were high (κ = 1). Relatively good agreement for most US findings, compared with MRI, was found for the shoulder (overall agreement 81%, sensitivity 76%, specificity 89%) and knee joint (overall agreement 88%, sensitivity 91%, specificity 88%). Sensitivities were lower for wrist/finger (overall agreement 73%, sensitivity 66%, specificity 88%) and ankle/toe joints (overall agreement 82%, sensitivity 61%, specificity 92%).

Conclusion: Interobserver reliabilities, sensitivities, and specificities in comparison with MRI were moderate to good. Further standardisation of US scanning techniques and definitions of different pathological US lesions are necessary to increase the interobserver agreement in musculoskeletal US.

Full Text

The Full Text of this article is available as a PDF (286.2 KB).

Figure 1.

Figure 1

Open in a new tab

 Shoulder joint. (A and B) Humeral head erosions. (A) In MRI, multiple erosions can be seen from the anterior and posterior sides of the humeral head as bone defects with sharp margins (arrows). (B) Distinct bone defects below the bone surface (erosions, arrows) can also be detected by US. This image is taken from the anterior side with maximum inner rotation (transverse scan). (C and D) Glenohumeral joint synovitis. (C) In MRI, contrast enhancement clearly depicts a subdeltoid/subacromial bursitis (arrows) and synovitis within the joint. (D) The US image shows a lateral longitudinal scan of the shoulder joint. Subdeltoid bursitis can be visualised as an anechoic area below the deltoid muscle (arrows).

Figure 2.

Figure 2

Open in a new tab

 Finger joint (MCP II). (A) The MR image shows the MCP joints II–V in transverse section. Focusing on MCP joint II shows slight contrast enhancement from the dorsal and palmar aspects, representing synovitis (arrowheads). Also, tenosynovitis is seen at the flexor tendons (arrow). (B) The US longitudinal image from the palmar side displays an anechoic to hypoechoic area at the region of the diaphysis reflecting synovitis (arrows). Also, there is tenosynovitis along the flexor tendon (upper arrows).

Figure 3.

Figure 3

Open in a new tab

 Knee joint. (A) MRI shows some contrast agent enhancement in the suprapatellar recess, reflecting inflammatory effusion (two arrows). (B) US also clearly depicts the effusion in the suprapatellar recess (arrows). (C) In MRI, a popliteal cyst is visualised in the sagittal view with a deep part (arrowheads) and a superficial part (arrows). (D) Both parts can also clearly be detected by US as anechoic areas (arrows).

Figure 4.

Figure 4

Open in a new tab

 Ankle/toe joints. (A) MRI of the ankle shows contrast enhancement in the tibiotalar joint from anterior and posterior aspects (arrows). (B) The longitudinal US image is an example of the anterior side of the tibiotalar joint. The anechoic area displays effusion (anechoic) and synovitis (hypoechoic; arrows).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.

  1. Backhaus M., Burmester G. R., Gerber T., Grassi W., Machold K. P., Swen W. A., Wakefield R. J., Manger B., Working Group for Musculoskeletal Ultrasound in the EULAR Standing Committee on International Clinical Studies including Therapeutic Trials Guidelines for musculoskeletal ultrasound in rheumatology. Ann Rheum Dis. 2001 Jul;60(7):641–649. doi: 10.1136/ard.60.7.641. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Backhaus M., Kamradt T., Sandrock D., Loreck D., Fritz J., Wolf K. J., Raber H., Hamm B., Burmester G. R., Bollow M. Arthritis of the finger joints: a comprehensive approach comparing conventional radiography, scintigraphy, ultrasound, and contrast-enhanced magnetic resonance imaging. Arthritis Rheum. 1999 Jun;42(6):1232–1245. doi: 10.1002/1529-0131(199906)42:6<1232::AID-ANR21>3.0.CO;2-3. [DOI] [PubMed] [Google Scholar]
  3. Balint P. V., Sturrock R. D. Intraobserver repeatability and interobserver reproducibility in musculoskeletal ultrasound imaging measurements. Clin Exp Rheumatol. 2001 Jan-Feb;19(1):89–92. [PubMed] [Google Scholar]
  4. Carotti M., Salaffi F., Manganelli P., Salera D., Simonetti B., Grassi W. Power Doppler sonography in the assessment of synovial tissue of the knee joint in rheumatoid arthritis: a preliminary experience. Ann Rheum Dis. 2002 Oct;61(10):877–882. doi: 10.1136/ard.61.10.877. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. D'Agostino Maria-Antonietta, Said-Nahal Roula, Hacquard-Bouder Cécile, Brasseur Jean-Louis, Dougados Maxime, Breban Maxime. Assessment of peripheral enthesitis in the spondylarthropathies by ultrasonography combined with power Doppler: a cross-sectional study. Arthritis Rheum. 2003 Feb;48(2):523–533. doi: 10.1002/art.10812. [DOI] [PubMed] [Google Scholar]
  6. Grassi W., Filippucci E., Farina A., Cervini C. Sonographic imaging of tendons. Arthritis Rheum. 2000 May;43(5):969–976. doi: 10.1002/1529-0131(200005)43:5<969::AID-ANR2>3.0.CO;2-4. [DOI] [PubMed] [Google Scholar]
  7. Grassi W., Filippucci E., Farina A., Salaffi F., Cervini C. Ultrasonography in the evaluation of bone erosions. Ann Rheum Dis. 2001 Feb;60(2):98–103. doi: 10.1136/ard.60.2.98. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Hermann Kay-Geert A., Backhaus Marina, Schneider Udo, Labs Karsten, Loreck Dieter, Zühlsdorf Svenda, Schink Tania, Fischer Thomas, Hamm Bernd, Bollow Matthias. Rheumatoid arthritis of the shoulder joint: comparison of conventional radiography, ultrasound, and dynamic contrast-enhanced magnetic resonance imaging. Arthritis Rheum. 2003 Dec;48(12):3338–3349. doi: 10.1002/art.11349. [DOI] [PubMed] [Google Scholar]
  9. Iagnocco A., Coari G., Palombi G., Valesini G. Sonography in the study of metatarsalgia. J Rheumatol. 2001 Jun;28(6):1338–1340. [PubMed] [Google Scholar]
  10. Koski J. M. Detection of plantar tenosynovitis of the forefoot by ultrasound in patients with early arthritis. Scand J Rheumatol. 1995;24(5):312–313. doi: 10.3109/03009749509095169. [DOI] [PubMed] [Google Scholar]
  11. Koski J. M. Ultrasonography of the subtalar and midtarsal joints. J Rheumatol. 1993 Oct;20(10):1753–1755. [PubMed] [Google Scholar]
  12. Lassere Marissa, McQueen Fiona, Østergaard Mikkel, Conaghan Philip, Shnier Ron, Peterfy Charles, Klarlund Mette, Bird Paul, O'Connor Philip, Stewart Neal. OMERACT Rheumatoid Arthritis Magnetic Resonance Imaging Studies. Exercise 3: an international multicenter reliability study using the RA-MRI Score. J Rheumatol. 2003 Jun;30(6):1366–1375. [PubMed] [Google Scholar]
  13. Manger B., Kalden J. R. Joint and connective tissue ultrasonography--a rheumatologic bedside procedure? A German experience. Arthritis Rheum. 1995 Jun;38(6):736–742. doi: 10.1002/art.1780380603. [DOI] [PubMed] [Google Scholar]
  14. Naredo E., Iagnocco A., Valesini G., Uson J., Beneyto P., Crespo M. Ultrasonographic study of painful shoulder. Ann Rheum Dis. 2003 Oct;62(10):1026–1027. doi: 10.1136/ard.62.10.1026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Ostergaard M., Klarlund M., Lassere M., Conaghan P., Peterfy C., McQueen F., O'Connor P., Shnier R., Stewart N., McGonagle D. Interreader agreement in the assessment of magnetic resonance images of rheumatoid arthritis wrist and finger joints--an international multicenter study. J Rheumatol. 2001 May;28(5):1143–1150. [PubMed] [Google Scholar]
  16. Ribbens Clio, André Béatrice, Marcelis Stefaan, Kaye Olivier, Mathy Luc, Bonnet Valérie, Beckers Catherine, Malaise Michel G. Rheumatoid hand joint synovitis: gray-scale and power Doppler US quantifications following anti-tumor necrosis factor-alpha treatment: pilot study. Radiology. 2003 Sep 11;229(2):562–569. doi: 10.1148/radiol.2292020206. [DOI] [PubMed] [Google Scholar]
  17. Scheel Alexander K., Backhaus Marina. Ultrasonographic assessment of finger and toe joint inflammation in rheumatoid arthritis: comment on the article by Szkudlarek et al. Arthritis Rheum. 2004 Mar;50(3):1008–1009. doi: 10.1002/art.20202. [DOI] [PubMed] [Google Scholar]
  18. Schmidt W. A., Völker L., Zacher J., Schläfke M., Ruhnke M., Gromnica-Ihle E. Colour Doppler ultrasonography to detect pannus in knee joint synovitis. Clin Exp Rheumatol. 2000 Jul-Aug;18(4):439–444. [PubMed] [Google Scholar]
  19. Szkudlarek M., Court-Payen M., Strandberg C., Klarlund M., Klausen T., Ostergaard M. Power Doppler ultrasonography for assessment of synovitis in the metacarpophalangeal joints of patients with rheumatoid arthritis: a comparison with dynamic magnetic resonance imaging. Arthritis Rheum. 2001 Sep;44(9):2018–2023. doi: 10.1002/1529-0131(200109)44:9<2018::AID-ART350>3.0.CO;2-C. [DOI] [PubMed] [Google Scholar]
  20. Szkudlarek Marcin, Court-Payen Michel, Jacobsen Søren, Klarlund Mette, Thomsen Henrik S., Østergaard Mikkel. Interobserver agreement in ultrasonography of the finger and toe joints in rheumatoid arthritis. Arthritis Rheum. 2003 Apr;48(4):955–962. doi: 10.1002/art.10877. [DOI] [PubMed] [Google Scholar]
  21. Terslev L., Torp-Pedersen S., Savnik A., von der Recke P., Qvistgaard E., Danneskiold-Samsøe B., Bliddal H. Doppler ultrasound and magnetic resonance imaging of synovial inflammation of the hand in rheumatoid arthritis: a comparative study. Arthritis Rheum. 2003 Sep;48(9):2434–2441. doi: 10.1002/art.11245. [DOI] [PubMed] [Google Scholar]
  22. Wakefield R. J., Gibbon W. W., Conaghan P. G., O'Connor P., McGonagle D., Pease C., Green M. J., Veale D. J., Isaacs J. D., Emery P. The value of sonography in the detection of bone erosions in patients with rheumatoid arthritis: a comparison with conventional radiography. Arthritis Rheum. 2000 Dec;43(12):2762–2770. doi: 10.1002/1529-0131(200012)43:12<2762::AID-ANR16>3.0.CO;2-#. [DOI] [PubMed] [Google Scholar]
  23. Wakefield Richard J., Brown Andrew K., O'Connor Philip J., Emery Paul. Power Doppler sonography: improving disease activity assessment in inflammatory musculoskeletal disease. Arthritis Rheum. 2003 Feb;48(2):285–288. doi: 10.1002/art.10818. [DOI] [PubMed] [Google Scholar]
  24. Østergaard Mikkel, Peterfy Charles, Conaghan Philip, McQueen Fiona, Bird Paul, Ejbjerg Bo, Shnier Ron, O'Connor Philip, Klarlund Mette, Emery Paul. OMERACT Rheumatoid Arthritis Magnetic Resonance Imaging Studies. Core set of MRI acquisitions, joint pathology definitions, and the OMERACT RA-MRI scoring system. J Rheumatol. 2003 Jun;30(6):1385–1386. [PubMed] [Google Scholar]
  25. Østergaard Mikkel, Wiell Charlotte. Ultrasonography in rheumatoid arthritis: a very promising method still needing more validation. Curr Opin Rheumatol. 2004 May;16(3):223–230. doi: 10.1097/00002281-200405000-00010. [DOI] [PubMed] [Google Scholar]

Articles from Annals of the Rheumatic Diseases are provided here courtesy of BMJ Publishing Group

RESOURCES