Abstract
Background. Clinical, laboratory, and radiologic parameters are used for diagnosis and classification of spondyloarthritis (SpA). Magnetic resonance imaging (MRI) of sacroiliac (SI) joints is being increasingly used to detect early sacroiliitis. We decided to evaluate the interobserver agreement in MRI findings of SI joints of SpA patients between a local radiologist, a rheumatologist, and an expert radiologist in musculoskeletal diseases. Methods. 66 MRI images of the SI joints of patients with established diagnosis of SpA were evaluated. Agreement was expressed in Cohen's kappa. Results. Interobserver agreement between a local radiologist and an expert radiologist was fair (κ = 0.37). Only acute findings showed a moderate agreement (κ = 0.45), while chronic findings revealed 76.5% of disagreement (κ = 0.31). A fair agreement was observed in acute findings (κ = 0.38) as well as chronic findings (κ = 0.38) between a local radiologist and a rheumatologist. There was a substantial agreement between an expert radiologist and a rheumatologist (κ = 0.73). In acute findings, a 100% agreement was achieved. Also chronic and acute plus chronic findings showed high levels of agreement (κ = 0.73 and 0.62, resp.). Conclusions. Our study shows that rheumatologists may have similar MRI interpretations of SI joints in SpA patients as an expert radiologist.
1. Background
The spondyloarthritis (SpA) is a group of interconnected inflammatory arthritides that share multiple clinical features and common genetic factors [1]. Men with a clinical onset before 50 years are more frequently affected [2, 3]. The main clinical manifestations are inflammatory back pain, peripheral arthritis, enthesitis, and uveitis, while other organ manifestations are rare [4].
Clinical, laboratory, and radiologic parameters are used for its diagnosis and classification. During the last 30 years, multiple classification criteria have been proposed; however one of the most accepted and used is the European Spondyloarthropathy Study Group (ESSG). The ESSG categorizes SpA in Ankylosing Spondylitis (AS), Psoriatic Arthritis (PsA), Reactive Arthritis (ReA), Arthritis associated with Inflammatory Bowel Disease (AIBD), and Undifferentiated Spondyloarthritis (uSpA) [5].
Recently, the Assessment of SpondyloArthritis International Society (ASAS) developed new classification criteria for the early recognition of SpA, which differentiates two predominant manifestations of SpA: axial (axSpA) and peripheral (pSpA) disease [6, 7]. According to ASAS, axSpA is defined as the presence of sacroiliitis by radiography or magnetic resonance imaging (MRI) plus at least one SpA feature (imaging arm) or the presence of HLA-B27 plus at least two SpA features (clinical arm) [6].
However, the recognition of sacroiliitis on conventional radiographs can be challenging because of the anatomic complexity of the sacroiliac (SI) joints, which leads to misinterpretations [8]. This is demonstrated in previous studies where interobserver (κ = 0.19 to 0.79) and intraobserver (κ = 0.07 to 1.0) variations differ widely [9–13]. Even more so, a study by van Tubergen demonstrated that rheumatologist as well as radiologist showed modest sensitivity and specificity for sacroiliitis on conventional radiographs, which did not improve significantly with neither individual training nor workshops [14].
Different outcomes are demonstrated in similar studies with the use of MRI. Acute and chronic findings in SI joints showed improvement in interobserver variations (κ = 0.38 to 0.80) [13, 15–17]. Also, when comparing conventional radiographs and MRI for detection of chronic structural changes in SI joints, MRI showed better sensitivity and specificity (84% and 61%, resp.) with low agreement concerning definite erosions (κ = 0.11), moderate agreement for definite subchondral sclerosis (κ = 0.46) and definite joint space abnormalities (κ = 0.41), and almost perfect agreement for joint ankylosis (κ = 0.85) [13]. Also, even better interobserver agreements for SI erosion on MRI have been reported by Weber et al. with kappa values above 0.70 [15].
Initial diagnostic approach is made by primary care physicians, family physicians, or internal medicine specialists, which relied on an accurate reading of SI images. In daily practice, readings of the SI images are made by local radiologists. However, radiographic interpretations tend to differ according to expertise, as it is demonstrated in a study by Geijer et al. where the presence of sacroiliitis in computed tomography (CT) showed a good interobserver agreement between two expert radiologists (κ = 0.67), but it decreased when comparing an expert radiologist reading and a local radiologist reading (κ = 0.46) [18]. Therefore, the aim of the present study is to establish the degree of interobserver variation of the MRI readings of acute and chronic changes of SI joints in patients with SpA between local radiologists, an expert musculoskeletal radiologist, and a rheumatologist.
2. Methods
2.1. Patients
In total, 66 MRI images of the SI joints of patients with established diagnosis of SpA according to the ESSG criteria [5], who attended spondyloarthritis ambulatory clinic of the Hospital Militar Central, between January and December of 2015, were evaluated. All patients were assessed under a previously validated structured protocol. Also, patients were characterized according to the ASAS criteria for further analysis [6, 7]. Exclusion criteria included subjects under 18 years and individuals with malignancies or other rheumatic diseases.
The study was approved by the Ethics Committee of the Hospital Militar Central. All studied subjects provided written informed consent and confidentiality was strictly maintained. The study followed norms established by the Helsinki Declaration, The Guidelines for Good Clinical Practice, and Resolution 8430 (1993) of the Colombian Ministry of Social Protection.
2.2. Assessment of Clinical Rheumatologic Parameters
Disease activity was assessed by Bath Ankylosing Spondylitis Disease Activity Index (BASDAI) [19], erythrocyte sedimentation rate (ESR), and high sensitivity C-reactive protein (hsCRP). Functionality was assessed by Bath Ankylosing Spondylitis Functionality Index (BASFI) [20].
2.3. Evaluation of MRI
Semicoronal and axial MR slices of the SI joints with T1, T2 sequence, fat suppression technique, and Short Tau Inversion Recovery (STIR) with axial T1 and T2 protocol were taken for each patient in the radiology department of the Hospital Militar Central. The same General Electric MR450 1.5 tesla equipment was used in all patients. All images were read by four blinded local radiologists in charge of all readings of the Hospital Militar Central, without the use of any reading protocol. The readings by the local radiologist included the following findings: normal (no findings), acute (bone marrow oedema), chronic (sclerosis, erosions, fat infiltration, ankylosis, and bony formations), and acute plus chronic (acute plus any chronic finding), which were assessed in a dichotomous way. An expert radiologist in musculoskeletal disease (EC) and a rheumatologist (JL) also read the images blinded to the diagnosis and the local radiologists readings, using the previous mentioned categories (normal, acute, chronic, and acute plus chronic) in order to make comparisons.
Local radiologists are graduated radiologists with no formal experience in musculoskeletal radiology. The expert radiologist (EC) is a radiologist with 30-year experience in musculoskeletal radiology with exclusive dedication to interpreting images in rheumatology, specially in SpA. The rheumatologist (JL) is a professor and coordinator of an outpatient clinic of SpA in a University Hospital (Hospital Militar Central). Both EC and JL have attended and fulfilled several EULAR and ASAS courses on SpA imaging.
2.4. Statistical Analysis
Results of the categorical variables were expressed in contingency and frequency tables. For numerical variables measures of central tendency and dispersion that included mean, standard deviation (SD), median, and interquartile range were used.
Agreement was calculated using cross-tabulation expressed in Cohen's kappa [21, 22] for the following comparisons: acute findings between local radiologist, expert radiologist, and a rheumatologist; chronic findings between local radiologist, expert radiologist, and a rheumatologist; and acute plus chronic findings between local radiologist, expert radiologist, and a rheumatologist.
All kappa values were interpreted according to the standards proposed by Landis and Koch, as follows: values 0–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and 0.81–1.0 indicating almost perfect agreement [23]. Values between minus 1 and 0 the same categories apply as above 0, but for disagreement. Also percentage of agreement was calculated.
SPSS software version 19.0 for Windows was used for the statistical analysis.
3. Results
3.1. Characteristics of SpA Patients
The mean age of the 66 patients included in the study was 34.35 ± 11.76 years. Most patients were male (n: 41; 62.7%). ReA was the most frequent diagnosis (n: 35; 52.5%) and only 13 patients (25.6%) completed the New York criteria. Table 1 shows the general characteristics of the studied population.
Table 1.
Characteristics of the study population.
| ESSG | ASAS | ||||
|---|---|---|---|---|---|
| AS | ReA | uSpA | axSpA | pSpA | |
| (n: 20; 30.5%) | (n: 35; 52.5%) | (n: 11; 16.9%) | (n: 50; 76.3%) | (n: 55; 83.1%) | |
| Age of onset, years (mean; SD) | 25.2 ± 8.6 | 27.9 ± 6.8 | 31.4 ± 10.3 | 31.45 ± 11.0 | 38.7 ± 10.1 |
| Sex (%) | |||||
| Male | 12 (60.0) | 28 (80.0) | 5 (44.0) | 24 (47.6) | 26 (46.9) |
| Female | 8 (40.0) | 7 (20.0) | 6 (56.0) | 26 (52.4) | 29 (53.1) |
| HLA-B27 (%) | 16 (82.3) | 21 (61.5) | 5 (48.9) | 38 (75.2) | 36 (65.3) |
| Symptom duration (median; interquartile range) | 6.2 (1.4–12.1) | 1.17 (0.2–4.9) | 3.0 (1.4–10.1) | 3.79 (1–11.6) | 3.0 (1–10.7) |
| ESR (mean; SD) | 21.8 ± 17.5 | 25.6 ± 22.5 | 16.9 ± 14.8 | 22.1 ± 19.2 | 21.4 ± 18.0 |
| hsCRP (median; interquartile range) | 0.54 (0.2–4.1) | 2.48 (0.2–15.9) | 0.38 (0.1–0.5) | 0.50 (0.1–2.7) | 0.50 (0.2–2.4) |
| BASDAI (mean; SD) | 5.3 ± 2.6 | 5.1 ± 2.9 | 5.8 ± 2.2 | 5.5 ± 2.5 | 5.66 ± 2.4 |
| BASFI (mean; SD) | 5.1 ± 2.5 | 4.9 ± 2.9 | 5.1 ± 2.3 | 5.1 ± 2.4 | 5.1 ± 2.4 |
ESSG: European Spondyloarthropathy Study Group; ASAS: Assessment of SpondyloArthritis International Society; AS: Ankylosing Spondylitis; ReA: Reactive Arthritis; uSpA: Undifferentiated Spondyloarthritis; axSpA: axial spondyloarthritis; pSpA: peripheral spondyloarthritis; SD: standard deviation; ESR: erythrocyte sedimentation rate; hsCRP: high sensitivity C-reactive protein; BASDAI: Bath Ankylosing Spondylitis Disease Activity Index; BASFI: Bath Ankylosing Spondylitis Functional Index.
3.2. Interobserver Agreement between Local Radiologist and Expert Radiologist
In general, interobserver agreement between a local radiologist and an expert radiologist was fair (κ = 0.37). Only acute findings showed a moderate agreement (κ = 0.45), while chronic findings revealed high levels of disagreement (κ = 0.31) (Table 2).
Table 2.
Interobserver agreement between local radiologist and expert radiologist.
| Expert radiologist | Local radiologist | ||||
|---|---|---|---|---|---|
| Agreement | Disagreement | Total | κ | p | |
| n (%) | n (%) | ||||
| Normal | 24 (96) | 1 (4) | 25/66 | 0.39 | 0.000 |
| Acute | 3 (50) | 3 (50) | 6/66 | 0.45 | 0.000 |
| Chronic | 4 (23.5) | 13 (76.5) | 17/66 | 0.31 | 0.000 |
| Acute plus chronic | 7 (38.9) | 11 (61.1) | 18/66 | 0.34 | 0.003 |
| Total | 38 (57.7) | 28 (43.3) | 66 | 0.37 | 0.000 |
3.3. Interobserver Agreement between Local Radiologist and Rheumatologist
A fair agreement was observed in acute findings (κ = 0.38; p = 0.071) as well as chronic findings (κ = 0.38; p = 0.071) between local radiologist and a rheumatologist. An even worse agreement was found in acute plus chronic findings with a κ value of 0.19 indicating a slight agreement (p = 0.502).
3.4. Interobserver Agreement between Expert Radiologist and Rheumatologist
There was a substantial agreement between an expert radiologist and a rheumatologist (κ = 0.73). In acute findings, a κ value of 0.69 was achieved. Also chronic and acute plus chronic findings showed high levels of agreement (κ = 0.73 and 0.62, resp.) (Table 3).
Table 3.
Interobserver agreement between expert radiologist and rheumatologist.
| Rheumatologist | Expert radiologist | ||||
|---|---|---|---|---|---|
| Agreement | Disagreement | Total | κ | p | |
| n (%) | n (%) | ||||
| Normal | 22 (89) | 3 (11) | 25/66 | 0.89 | 0.000 |
| Acute | 6 (100) | 0 (0) | 6/66 | 0.69 | 0.001 |
| Chronic | 15 (89) | 2 (11) | 17/66 | 0.73 | 0.001 |
| Acute plus chronic | 16 (89) | 2 (11) | 18/66 | 0.62 | 0.005 |
| Total | 59 (89) | 7 (11) | 66/66 | 0.73 | 0.000 |
4. Discussion
To our knowledge this is the first study to evaluate variations in MRI of SI joints in SpA patients between local radiologists, an expert radiologist, and a rheumatologist. In our study, local radiologists are in charge of interpreting every day studies. This includes plain radiographs, MRIs, CTs, and ultrasonography. Their knowledge in musculoskeletal imaging is limited to what was learned during their radiology training. Some will have more experience than others; however none have specific preparation in musculoskeletal imaging. This could be the reason why we found a fair interobserver agreement of imaging findings between local radiologist and an expert radiologist in musculoskeletal diseases.
A similar study found slightly higher interobserver agreements. In the mentioned study, the original reports of SI joints evaluated with CT were compared to readings from two other expert observers. A moderate agreement was found between the original reports and the readings from the two observers (κ=0.46 and κ = 0.44) [18].
Other studies have compared readings of radiologists and rheumatologist with different levels of expertise. Also, higher interobserver agreements were found. Van Den Berg et al. found a moderate agreement (κ = 0.55) on readings of pelvic radiographs of SI joints of patients with suspicion of SpA between rheumatologists or radiologists and experts [12]. A study with SI joint MRIs found an even higher interobserver agreement (κ = 0.70) between readings of local radiologists and rheumatologists and two experts [16]. However, these studies compared the readings of experts and “nonexperts” in musculoskeletal imaging with the knowledge of being in a study when the readings were made.
The expertise in reading MRIs of SI in SpA patients and higher levels of agreement is also corroborated in a study by Arnbak et al. where intra- and interobserver agreement between experts was substantial to almost perfect (κ = 0.61 and κ = 0.79, resp.) [24]. These findings suggest that the expertise, understood as specific training in musculoskeletal imaging, increases the correct interpretation with almost perfect reproducibility. However, a study in detection of sacroiliitis in plain radiographs of SI joints demonstrated that no improvement in performance was found after individual training sessions and workshops [14]. They explained that possibly the participants after training sessions changed the attitude towards interpreting the radiographs.
This change of attitude towards interpretation in studies can be explained by the Hawthorne effect. Participants tend to be more careful and concerned with accuracy and exactness, a phenomenon of altered behavior resulting from the awareness of being part of an experimental study [25, 26]. This could explain our lower interobserver agreement between experts and “nonexperts.”
Interestingly, our study showed a much better performance of a rheumatologist in the interpretation of MRI of the SI joints when compared to an expert radiologist, achieving substantial agreement throughout the whole gamut of findings. Agreement between an expert radiologist and an expert rheumatologist trained by specialized courses is high and similar for both acute and chronic lesions. There was no relevant difference in agreement for acute (κ = 0.69) and chronic lesions (κ = 0.73). However, when comparing between local radiologist and a rheumatologist the interobserver agreements were fair at best. Although the Hawthorne effect can explain these results, we also consider that the improved interpretation of SI joints by a rheumatologist is accomplished by specific training during fellowship and constant readings and exposure to musculoskeletal imaging.
It is important to emphasize that the broad community of radiologists (and possibly also many rheumatologists) would need specific training in assessment of axial SpA, as it has become so important for diagnosis/classification and treatment decisions. In most countries, both specialities have no such education in their professional curriculum.
Our study has limitations. First, no specific reading protocol of MRI was used. This can lead to interpretation bias and lack of standardization. Second, two different timelines for data collection were compared. The MRIs interpretations from local radiologists were retrospectively collected whereas rheumatologist and expert radiologist interpretations were prospectively collected. This could increase the Hawthorne effect because local radiologists were unaware that their interpretations were to be used in a study, while the rheumatologist and expert radiologist were aware. Third, a bigger sample size would increase the statistical power.
Finally, more studies are required with strict methodological rigor as well as more sample size to confirm our findings. Also, a study similar to the one by van Tubergen should be performed using MRIs in order to confirm whether proper training improves or not interpretation.
5. Conclusions
Our study shows that rheumatologists may have similar MRI interpretations of SI joints in SpA patients as an expert radiologist. We believe that in some degree these results may be altered by the Hawthorne effect. However, it is clear that expertise achieved during rheumatology training as well as constant reviewing of musculoskeletal imaging improves interpretation, which can be reflected in daily rheumatology practice.
Acknowledgments
The authors would like to thank the Department of Radiology of Universidad Nacional and Dr. Alba Rubio for their aid.
Ethical Approval
The authors declare that this study was approved by the Research Ethics Board of the Hospital Militar Central no. 2012-36123.
Consent
All studied subjects provided written informed consent and confidentiality was strictly maintained.
Disclosure
The study followed norms established by the Helsinki Declaration, The Guidelines for Good Clinical Practice, and Resolution 8430 (1993) of the Colombian Ministry of Social Protection.
Competing Interests
The authors declare no conflict of interests.
References
- 1.Ehrenfeld M. Spondyloarthropathies. Best Practice & Research Clinical Rheumatology. 2012;26(1):135–145. doi: 10.1016/j.berh.2012.01.002. [DOI] [PubMed] [Google Scholar]
- 2.Toussirot É. Diagnosis and Management of late-onset spondyloarthritis: implications of treat-to-target recommendations. Drugs & Aging. 2015;32(7):515–524. doi: 10.1007/s40266-015-0280-y. [DOI] [PubMed] [Google Scholar]
- 3.Olivieri I., D’Angelo S., Padula A., Leccese P., Palazzi C. Spondyloarthritis with onset after age 45. Current Rheumatology Reports. 2013;15, article 374 doi: 10.1007/s11926-013-0374-7. [DOI] [PubMed] [Google Scholar]
- 4.Zochling J., Brandt J., Braun J. The current concept of spondyloarthritis with special emphasis on undifferentiated spondyloarthritis. Rheumatology. 2005;44(12):1483–1491. doi: 10.1093/rheumatology/kei047. [DOI] [PubMed] [Google Scholar]
- 5.Dougados M., Linden S. V., Juhlin R., et al. The european spondylarthropathy study group preliminary criteria for the classification of spondylarthropathy. Arthritis & Rheumatism. 1991;34(10):1218–1227. doi: 10.1002/art.1780341003. [DOI] [PubMed] [Google Scholar]
- 6.Rudwaleit M., van der Heijde D., Landewe R., et al. The development of Assessment of SpondyloArthritis international Society classification criteria for axial spondyloarthritis (part II): validation and final selection. Annals of the Rheumatic Diseases. 2009;68(6):777–783. doi: 10.1136/ard.2009.108233. [DOI] [PubMed] [Google Scholar]
- 7.Rudwaleit M., van der Heijde D., Landewé R., et al. The Assessment of SpondyloArthritis international Society classification criteria for peripheral spondyloarthritis and for spondyloarthritis in general. Annals of the Rheumatic Diseases. 2011;70(1):25–31. doi: 10.1136/ard.2010.133645. [DOI] [PubMed] [Google Scholar]
- 8.Spoorenberg A., de Vlam K., van der Linden S., et al. Radiological scoring methods in ankylosing spondylitis. reliability and change over 1 and 2 years. Journal of Rheumatology. 2004;31(1):125–132. [PubMed] [Google Scholar]
- 9.Hollingsworth P. N., Cheah P. S., Dawkins R. L., Owen E. T., Calin A., Wood P. H. Observer variation in grading sacroiliac radiographs in HLA-B27 positive individuals. The Journal of Rheumatology. 1983;10(2):247–254. [PubMed] [Google Scholar]
- 10.Yazici H., Turunc M., Ozdogan H., Yurdakul S., Akinci A., Barnes C. G. Observer variation in grading sacroiliac radiographs might be a cause of 'sacroiliitis' reported in certain disease states. Annals of the Rheumatic Diseases. 1987;46(2):139–145. doi: 10.1136/ard.46.2.139. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Taylor H. G., Wardle T., Beswick E. J., Dawes P. T. The relationship of clinical and laboratory measurements to radiological change in ankylosing spondylitis. Rheumatology. 1991;30(5):330–335. doi: 10.1093/rheumatology/30.5.330. [DOI] [PubMed] [Google Scholar]
- 12.Van Den Berg R., Lenczner G., Feydy A., et al. Agreement between clinical practice and trained central reading in reading of sacroiliac joints on plain pelvic radiographs: results from the DESIR cohort. Arthritis and Rheumatology. 2014;66(9):2403–2411. doi: 10.1002/art.38738. [DOI] [PubMed] [Google Scholar]
- 13.Poddubnyy D., Gaydukova I., Hermann K.-G., et al. Magnetic resonance imaging compared to conventional radiographs for detection of chronic structural changes in sacroiliac joints in axial spondyloarthritis. Journal of Rheumatology. 2013;40(9):1557–1565. doi: 10.3899/jrheum.130141. [DOI] [PubMed] [Google Scholar]
- 14.van Tubergen A. Radiographic assessment of sacroiliitis by radiologists and rheumatologists: does training improve quality? Annals of the Rheumatic Diseases. 2003;62(6):519–525. doi: 10.1136/ard.62.6.519. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Weber U., Zubler V., Pedersen S. J., et al. Development and validation of a magnetic resonance imaging reference criterion for defining a positive sacroiliac joint magnetic resonance imaging finding in spondyloarthritis. Arthritis Care and Research. 2013;65(6):977–985. doi: 10.1002/acr.21893. [DOI] [PubMed] [Google Scholar]
- 16.Van Den Berg R., Lenczner G., Thévenin F., et al. Classification of axial SpA based on positive imaging (radiographs and/or MRI of the sacroiliac joints) by local rheumatologists or radiologists versus central trained readers in the DESIR cohort. Annals of the Rheumatic Diseases. 2015;74(11):2016–2021. doi: 10.1136/annrheumdis-2014-205432. [DOI] [PubMed] [Google Scholar]
- 17.Heuft-Dorenbosch L., Weijers R., Landewé R., van der Linden S., van der Heijde D. Magnetic resonance imaging changes of sacroiliac joints in patients with recent-onset inflammatory back pain: inter-reader reliability and prevalence of abnormalities. Arthritis research & therapy. 2006;8, article R11 doi: 10.1186/ar1859. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Geijer M., Göthlin G. G., Göthlin J. H. Observer variation in computed tomography of the sacroiliac joints: a retrospective analysis of 1383 cases. Acta Radiologica. 2007;48(6):665–671. doi: 10.1080/02841850701342146. [DOI] [PubMed] [Google Scholar]
- 19.Garrett S., Jenkinson T., Kennedy L. G., Whitelock H., Gaisford P., Calin A. A new approach to defining disease status in ankylosing spondylitis: the bath ankylosing spondylitis disease activity index. Journal of Rheumatology. 1994;21(12):2286–2291. [PubMed] [Google Scholar]
- 20.Calin A., Garrett S., Whitelock H., O'Hea J., Mallorie P., Jenkinson T. A new approach to defining functional ability in ankylosing spondylitis: the development of the bath ankylosing spondylitis functional index. Journal of Rheumatology. 1994;21(12):2281–2285. [PubMed] [Google Scholar]
- 21.Cohen J. Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin. 1968;70(4):213–220. doi: 10.1037/h0026256. [DOI] [PubMed] [Google Scholar]
- 22.Dice L. R. Measures of the amount of ecologic association between species. Ecology. 1945;26(3):297–302. doi: 10.2307/1932409. [DOI] [Google Scholar]
- 23.Landis J. R., Koch G. G. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–174. doi: 10.2307/2529310. [DOI] [PubMed] [Google Scholar]
- 24.Arnbak B., Jensen T. S., Manniche C., Zejden A., Egund N., Jurik A. G. Spondyloarthritis-related and degenerative MRI changes in the axial skeleton—an inter- and intra-observer agreement study. BMC Musculoskeletal Disorders. 2013;14, article 274 doi: 10.1186/1471-2474-14-274. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Vehmas T. Hawthorne effect: shortening of fluoroscopy times during radiation measurement studies. The British Journal of Radiology. 1997;70(838):1053–1055. doi: 10.1259/bjr.70.838.9404210. [DOI] [PubMed] [Google Scholar]
- 26.Parsons H. M. What Happened at Hawthorne?: new evidence suggests the Hawthorne effect resulted from operant reinforcement contingencies. Science. 1974;183(4128):922–932. doi: 10.1126/science.183.4128.922. [DOI] [PubMed] [Google Scholar]