Inter-rater agreement in evaluation of disability: systematic review of reproducibility studies

Abstract

Objectives To explore agreement among healthcare professionals assessing eligibility for work disability benefits.

Design Systematic review and narrative synthesis of reproducibility studies.

Data sources Medline, Embase, and PsycINFO searched up to 16 March 2016, without language restrictions, and review of bibliographies of included studies.

Eligibility criteria Observational studies investigating reproducibility among healthcare professionals performing disability evaluations using a global rating of working capacity and reporting inter-rater reliability by a statistical measure or descriptively. Studies could be conducted in insurance settings, where decisions on ability to work include normative judgments based on legal considerations, or in research settings, where decisions on ability to work disregard normative considerations.Teams of paired reviewers identified eligible studies, appraised their methodological quality and generalisability, and abstracted results with pretested forms. As heterogeneity of research designs and findings impeded a quantitative analysis, a descriptive synthesis stratified by setting (insurance or research) was performed.

Results From 4562 references, 101 full text articles were reviewed. Of these, 16 studies conducted in an insurance setting and seven in a research setting, performed in 12 countries, met the inclusion criteria. Studies in the insurance setting were conducted with medical experts assessing claimants who were actual disability claimants or played by actors, hypothetical cases, or short written scenarios. Conditions were mental (n=6, 38%), musculoskeletal (n=4, 25%), or mixed (n=6, 38%). Applicability of findings from studies conducted in an insurance setting to real life evaluations ranged from generalisable (n=7, 44%) and probably generalisable (n=3, 19%) to probably not generalisable (n=6, 37%). Median inter-rater reliability among experts was 0.45 (range intraclass correlation coefficient 0.86 to κ−0.10). Inter-rater reliability was poor in six studies (37%) and excellent in only two (13%). This contrasts with studies conducted in the research setting, where the median inter-rater reliability was 0.76 (range 0.91-0.53), and 71% (5/7) studies achieved excellent inter-rater reliability. Reliability between assessing professionals was higher when the evaluation was guided by a standardised instrument (23 studies, P=0.006). No such association was detected for subjective or chronic health conditions or the studies’ generalisability to real world evaluation of disability (P=0.46, 0.45, and 0.65, respectively).

Conclusions Despite their common use and far reaching consequences for workers claiming disabling injury or illness, research on the reliability of medical evaluations of disability for work is limited and indicates high variation in judgments among assessing professionals. Standardising the evaluation process could improve reliability. Development and testing of instruments and structured approaches to improve reliability in evaluation of disability are urgently needed.

Introduction

Many workers seek wage replacement benefits on the basis of disabling illness or injury, and over the past decade most countries of the Organisation for Economic Co-operation and Development (OECD) have experienced escalating rates of affected workers.^{1 2} Current estimates range from four to eight individuals per thousand per year,² corresponding to 16 000 newly affected workers/year for smaller countries like Switzerland and 1 700 000/year for countries like the US.

Both public and private insurance systems provide wage replacement benefits for employees whose impaired health prevents them from working, as long as eligibility criteria are met.¹ To inform this decision, insurers often arrange for evaluation of disability claims by medical professionals.^{3 4 5} Based on these evaluations, about half of all disability claims are declined.²

Equality before the law requires that claimants with similar health impairments and exposed to similar work demands should receive similar judgments of medical restrictions and limitations. Concerns have been raised, however, regarding low quality evaluations^{6 7 8} and poor reliability between medical experts.^{9 10 11 12 13 14} Evaluation of disability is a complex process that is affected by the skillset, attitudes, and beliefs of the expert, and few countries enforce standards of practice,^{3 5} which presents considerable challenges to reliability (box 1).^{15 16} We conducted a systematic review of reproducibility studies to summarise empirical evidence regarding the inter-rater reliability of global judgments on work disability and examined the hypothesis that studies using standardised assessments would show higher reliability.

Box 1: Sources of variation causing low inter-rater reliability in medical evaluations (modified from Kobak and colleagues¹⁶)

Interaction between expert and claimant

• Information variance

  • Experts obtain different information as a result of asking different questions

• Observation variance

  • Experts differ in what they notice and remember when presented with the same information

• Interpretation variance

  • Experts differ in the importance they attach to what is observed

• Criterion variance

  • Experts use different criteria to score the same information

Within subject and within expert

• Claimant variance

  • True differences exist in the claimant between testings when claimants say different things to each expert or when claimants truly change between a first and a second interview

• Expert variance

  • Experts differ in their understanding of the demands of a certain job on the workers’ capacities and of the consequences of functional limitations on work performance

  • Experts differ in their personal value system on what level of effort, endurance, and discomfort can reasonably be expected by a claimant

  • Experts differ in their understanding of the legal requirements on a medical expertise that could affect their medical judgments

Methods

We followed the standards set by the Guidelines for Reporting Reliability and Agreement Studies (GRRAS)¹⁷ and Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA)¹⁸ for the reporting of our study.

Eligibility criteria

We included reproducibility studies conducted in an insurance setting (evaluation of claimants) or in a research setting (evaluation of patients for work disability outside of actual assessments) in which two or more health professionals evaluated the work capacity of individuals claiming disability and reported inter-rater reliability on a global rating of work disability. Studies that reported only the inter-rater reliability of experts’ evaluation of specific physical or mental activities (such as lifting, conflict management) were excluded. All types of “subjects” qualified: real claimants, records of claimants, videotaped actors, vignettes, short case summaries.

Search strategy

We searched Medline, Embase, and PsycInfo from inception to 16 March 2016, without language restrictions. An experienced medical librarian (RC) developed database specific search strategies combining the following subject terms: reproducibility of results (MeSH, including reliability) and reliability statistics, disability or work capacity evaluation, and sick leave (see appendix 1 for the detailed search strategy). We screened the bibliographies of all included studies for additional relevant articles.

Study process

Three teams of paired reviewers (WdB, JWB, JH, SK, JS, RK) with expertise in medical evaluations and training in research methodology independently screened titles, abstracts, and full texts for eligibility, assessed generalisability, and collected data from each eligible study using standardised pilot tested forms with detailed instructions. Reviewers resolved disagreement through discussion or, if required, adjudication by a third reviewer (RK or WdB).

Quality appraisal

Quality appraisal of reproducibility studies includes methodological quality and generalisability to the setting in which the instrument will be used.^{17 19 20} To address the former, we assessed the blinding of raters to each other’s findings, the risk of order effects, and appropriateness of the statistical analyses following Quality Appraisal for Reliability Studies (QAREL) guidance. To address generalisability, we evaluated whether claimants, raters, and the performance of the disability evaluation were similar to the insurance context in which such evaluations take place.^{17 20}

As reliability is a product of the interaction between the performance of the test, the subjects/objects, and the context of the assessment, and as its estimate is affected by various sources of the variability in the measurement setting (that is, rater and subject characteristics, performance of the test,¹⁷box 1), we used an explicit and transparent process to evaluate generalisability. Based on the checklist of QAREL,¹⁹ GRRAS,¹⁷ and expert guidance,²⁰ we identified four claimant items and four expert items for defining greater generalisability:

• The recruitment strategy captures diverse cases as would present in actual evaluation of disability (in declining order: random, consecutive, other recruitment; not applicable to written cases or videos)

• Recruitment success (in declining order: >80%, 80-50%, <50%; not applicable to records of patients, videos, or written cases)

• Verisimilitude—that is, the extent that cases reflect the population in real life (in declining order: real claimants, including videotapes/audiotapes of real claimants, records of real claimants, videos with actors, hypothetical patients, written cases)

• Range of raters’ expertise in performing work disability evaluations (for example, wide range of experience that is comparable with the real world v narrow range of experience)

• Medical experts with formal training in disability evaluation (for example, licensed disability raters, rehabilitation specialists) or without any specific training (no formal requirement, family physicians certifying sick leave), where experts without formal training—as is the case in most countries—closer resemble real life

• No specific training for study purposes

• Number of cases that more closely resembles real life (in declining order: >100, 31-100, 11-30, 6-10; 1-5)

• Number of raters that more closely resembles real life (in declining order: >16, 11-15, 6-10, 3-5, 2)

We gave more weight to studies with a broader spectrum and a larger number of experts to reflect the wide variation among medical experts in actual disability assessment, which tends to contribute substantially to the measurement error.

Five reviewers (JB, RK, WdB, JS, JanHo), blinded to the study results, assessed generalisability of each study, independently and in duplicate. Given the lack of empirical evidence about the relative importance of each item we used a sequential approach from medical decision making²¹ to make the weighting of each item explicit (see appendix 2 for detailed description). This approach facilitated judgments regarding overall generalisability (that is, “generalisable,” “probably generalisable,” “probably not generalisable,” and “not generalisable”). We calculated the reviewers’ concordance in generalisability ranking using Kendall’s W (coefficient of concordance), which generates values between zero (no agreement) and one (perfect agreement).

We limited assessment of generalisability to studies performed in an insurance setting because studies conducted in a research setting, by definition (“normative or legal considerations not part of the judgment”, see data analysis), lack generalisability to real life assessments of disability.

Data collection

We extracted the following information from each eligible study:

• Study context—background and setting (insurance, rehabilitation, research)

• Patients’ characteristics (“cases”)—number of cases per study; presenting disorder(s) (mental disorder, musculoskeletal disease, mixed); course of disease or injury (acute, chronic)

• Expert characteristics (“raters”)—number of raters per study; number of cases per rater; number of raters per case; profession (primary or secondary care, occupational physician, insurance physician)

• Procedures—time frame before the evaluation for judging current health status and work disability; time frame for predicting global work disability (for both time frames, short term refers to less than six months; long term refers to more than six months; mixed); instrument (professional expertise with or without specific rating instrument) to support global rating of work disability and the related categories (for example, fully limited, partially limited, no limitations) or scales (for example, scale 0-100)

• Outcomes—global rating of work disability (for example, work capacity, sick leave, readiness for return to work, reduction in working hours); decisions on suitability for a specific job; occupational functioning); measure of reliability or agreement (intraclass correlation coefficient (ICC), κ statistic, or percentage agreement), including measure of precision, or descriptive measure (for example, frequency of judgments).

Data analysis

For three studies that reported on reduction of working hours,^{22 23 24} we calculated the κ statistics^{22 23} and intraclass correlation,²⁴ based on the raw data provided by authors.

We distinguished between studies conducted in an insurance setting or a research setting. In an insurance setting, health professionals make judgments on disability for work based on functional limitations that includes normative judgments from a societal perspective. An insurance setting does not imply any specific format of the claimant’s presentation in the study, which can range from a real patient to a written case (see also “generalisability, verisimilitude”). Researchers in a research setting who develop and/or validate instruments tend to standardise their research environment when judging occupational functioning. Normative (legal) considerations or a societal perspective are not part of their judgments.

We used studies conducted in a research setting to investigate the association between level of standardisation in the evaluation process and inter-rater reproducibility. Level of standardisation was considered as “not standardised” when medical experts in the insurance setting used only their professional expertise to elicit information and rate findings from the claimant; as “semi-standardised” when they used a structured instrument as one component of the evaluation; and as “fully standardised” when occupational functioning was primarily evaluated with a structured instrument.

Lack of information on variation associated with reproducibility statistics and heterogeneity of statistical measures and outcomes precluded pooling of the data across studies. Using a two tailed Fisher’s exact test, we explored whether objective (versus subjective) and acute (versus chronic) health conditions as well as higher levels of generalisability and/or higher levels of standardisation in the evaluation process were associated with a higher inter-rater reproducibility. We defined mental disorders as “subjective complaints” and somatic disorders as objective complaints, though we acknowledge the crude nature of this classification, and acute conditions shorter than six months and chronic conditions longer than six months. We excluded from our analysis three studies that did not specify the chronicity. Fisher’s exact test does not provide a test statistic, only whether the difference is significant or not.

For clinical interpretation of reliability measures, we used the thresholds established by Fleiss in 1981²⁵ to distinguish between poor, fair, good, and excellent inter-rater reliability.^{26 27 28} For κ, weighted κ, and intraclass correlation, the cut-off levels were <0.40 (poor), 0.40-0.59 (fair), 0.60-0.74 (good), and 0.75-1.00 (excellent); for percentage agreement, the levels were <70% (poor), 70-79% (fair), 80-89% (good), and 90-100% (excellent), taking into account that percentage agreement does not account for an agreement of raters by chance. Biometricians acknowledge that these guidelines are broadly accurate with some arbitrariness. Though at times they might come up with conflicting results, they have proved valuable in clinical application.²⁸

Patient involvement

No patients were involved in setting the research question, in developing plans for design, interpretation, reporting or implementation of the study. We plan to disseminate the results of this study to organisations supporting patients with disabilities.

Results

Study characteristics

Of 4562 potentially relevant citations identified, 101 reports proved potentially eligible after we had screened titles and abstracts. On full text screening, 23 studies,^{9 11 22 23 24 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46} including four non-English studies,^{9 39 40 41} proved eligible for analysis (fig 1). All studies were published from 1992 onwards and enrolled disability claimants from 12 countries in Europe, North America, Australia, the Middle East, and northeast Asia. Seven studies (30%) were conducted in the Netherlands. Seventy percent of the studies (16/23) were conducted in an insurance setting, with the remainder in a research setting. Investigators used a broad spectrum of designs, ranging from real life disability evaluations, videotapes with actors, and records of claimants to 10 line case vignettes, to perform reliability studies. Study size varied considerably with number of raters from two to 103 and number of patients from one and 3562 per study (tables 1 and 2).

Fig 1 Identification of studies assessing inter-rater agreement of evaluation of disability

Table 1.

Included studies on evaluation of disability from insurance setting^*

	Context of study	Disease/course	Retrospective time frame/prospective time frame for judgment
de Kort, 1992, Netherlands32	Random selection of 180 applicants (from national database of 101 754 individuals) for one of three jobs in public domain (administration, prison security, cleaning, catering) who underwent pre-employment assessment. 90 applicants (30 from each job category) had been judged (temporarily) unfit and 90 cases were used as reference cases with similar diagnosis. Study explores a) agreement among panel physicians and b) agreement between panel and decision of government occupational health and safety service	Mixed (individuals judged fit and unfit for job)/chronic	Unclear, not reported/unclear not reported
Dell-Kuster, 2014, Switzerland33	Single centre study on 3562 real life disability assessments about reliability of medical experts and family physicians of claimants judging work ability of claimants	Mixed conditions/chronic disease	Long term history/long term ≥6 months
Dickmann, 2007, Germany9	Fictional medical assessment of female administrative person with depression was videotaped and circulated to 22 psychiatric experts. Experts rated work ability in last job and in any suitable alternative job in context of claim for social security benefits	Mental disorder (depression)/chronic disease	Long term history/long term ≥6 months
Elder, 1994, UK34	35 medical experts attending scientific conference for medical experts rated 10 short case histories of real patients who applied for early retirement. Experts decided on acceptance or rejection of early retirement request	Mixed: psychiatric illness; somatic illness; somatic illness with psychiatric comorbities/mixed	Long term history/long term ≥ 6 months
Ikezawa, 2010, Canada37	In survey with 3 case vignettes from real cases, 36 clinicians made return to work recommendations. Clinicians worked in major rehabilitation facility operated by Alberta Workers Compensation Board	Fracture, dislocation and low back pain (musculoskeletal)/injury or work accident	Long term history/short term <6 months
Ingravallo, 2008, Italy11	4 medical commissions with 4 experts each reviewed 16 patients from database of 150 patients with different degrees of narcolepsy	Narcolepsy/chronic disease	Long term history/long term ≥6 months
Lax, 2004, US38	Single centre study in publicly funded occupational health centre where medical staff regularly evaluate patients for occupational illnesses compared 23 patients with mixed medical diagnoses who had recently undergone IME for evaluation of workers’ compensation claim with findings of health centre staff	Somatic disease, almost all occupational illnesses, including musculoskeletal disorders/injury or work incident	Long term history/long term ≥6 months
Lederer, 1998, Germany39	Quality assurance study in public administration in Germany, with public health physicians who regularly perform medical expertises on civil servants. These 103 physicians evaluated single claim file and reported their judgment on permanent work disability	Neurological disease (stroke with right sighted hemiplegia/chronic disease	Long term history/long term ≥6 months
Okpaku, 1994, US42	Independent rehabilitation team who was familiar in social security administration (SSA) assessments re-evaluated 158 adults with mental health problems, by imitating SSA approach. Cases all claimants or recipients of social security benefits. Judgments of rehabilitation team compared with those of SSA team	Mental disorder/chronic disease	Long term history/unclear, not reported, mixed
Rudbeck, 2011, Denmark43	11 medical specialists in social medicine rated work ability of eight 8 case histories of real patients	Mental or musculoskeletal; patients with mental problems can also report on back pain/chronic disease	Long term history/mix of short term and long term
Schellart, 2013, Netherlands44	43 Dutch insurance physicians applied insurance medicine guidelines for depression to four videos of actor claimants with depression. Insurance physicians used list of functional abilities before and after training to apply guidelines. Subanalysis of randomised controlled trial on training	Mental disorder (depression)/chronic disease	Long term history/long term ≥6 months
Schreuder, 2012, Netherlands45	5 occupational physicians assessed readiness to work of employees (n=132) who had been sicklisted for three weeks because of mental or musculoskeletal disorders	Mental and musculoskeletal disorders; other disorders/unclear or no information	Short term/short term ≥6 months
Slebus, 2010, Netherlands46	25 insurance physicians compared assessment of work ability in 5 patients with major depressive disorders with and without specific “checklist for work ability.” Subanalysis of randomised controlled trial on training	Mental disorders/chronic disease	Not reported/long term ≥6 months
Spanjer, 2008, Netherlands23	12 insurance physicians used disability assessment structured interview reports of 12 claimants with mental or physical disorder. Reports assessed on functional information system and mental ability list	Mental and somatic disorders/chronic disease	Long term history/long term ≥6 months
Spanjer, 2009, Netherlands22	Insurance physicians (n=27) assessed 30 claimants for disability benefits with musculoskeletal disorders. Different information about claimants was provided: exclusively medical; exclusively functional; mixed	Musculoskeletal disorders/chronic disease	Unclear, not reported, mixed/long term ≥6 months
Spanjer, 2010, Netherlands24	16 insurance physicians from Dutch social insurance office trained partly in disability assessment structured interview assessed 62 real cases. Subanalysis of randomised controlled trial on training	Musculoskeletal disorders/chronic disease	Not reported/long term ≥6 months

*In insurance setting, health professionals make judgments on work disability based on functional limitations, which include normative judgments, often from societal perspective.

Table 2.

Included studies on evaluation of disability in research setting^*

	Context of study	Disease/course	Retrospective time frame/prospective time frame for judgment
Berns, 2007, US29	2 practitioners rated 29 of larger sample with bipolar disorders with newly developed multidimensional scale of independent functioning (MSIF). Study conducted in single centre	Mental (bipolar disorders)/chronic disease	Short term/not reported
Chopra, 2002, Australia30	2 clinicians assessed feasibility and reliability of international classification of impairments, disability, and handicap (ICIDH-II) in 20 patients with psychotic disorders. Multicentre study	Mental (psychosis)/acute disease	Short term/short term <6 months
Daradkeh, 1994, United Arab Emirates31	2 psychiatrists with experience in rating disability assessment schedule (DAS, based on axis V of ICD-10, with one dimension of work) reviewed 42 psychiatric patients with different informants (such as family). Single centre study	Mixed mental inpatient and outpatients/mixed	Short term/not reported
Hannula, 2006, Finland35	Group of researchers developed rating scale based on social adjustment scale with focus on social and occupational functioning (SOFAS). Four clinically trained professionals administered rating scale to 39 videotaped interviews of consecutive patients from Helsinki Psychotherapy Study	Mental: anxiety and mood disorders/mixed	Short term/short term <6 months
Hill, 1989, UK36	Authors developed adult personality functioning assessment (APFA) with work as one subdomain. 3 raters used APFA for assessment 21 audiotaped standardised interviews with client	Mental/chronic disease	Unclear, not reported, mixed/unclear, not reported, mixed
Mundo, 2010, Italy40	18 raters assessed 180 inpatients with Kennedy Axis V (K Axis), which is equivalent to global assessment of functioning (GAF). One subscale covers occupational skills	Mixed mental disorders/unclear / no information	Short term/unclear, not reported, mixed
Nozu, 1995, Japan41	3 experts assessed schizophrenic outpatients who started occupational therapy at Tokyo Metropolitan Chubu Comprehensive Mental Health Centre with newly developed Work-Personality Insufficiency Rating Scale	Patients with schizophrenia/chronic disease	Unclear/unclear, not reported, mixed

*In research setting, researchers who develop instruments tend to standardise their research environment when judging occupational functioning. Normative considerations or societal perspective are not part of their judgments.

Methodological quality and generalisability

Assessment of methodological quality included blinding of raters to each other’s findings, presence of order effects, and appropriateness of the statistical analyses (table 3; appendix 2). The studies on the reproducibility between medical experts conducted in an insurance setting met 80% (31/39) of these items, 15% (6/39) remained unclear, and 5% (2/39) were not applicable. The methodological quality items did not fit the design of the studies that looked at the reproducibility between medical experts and health professionals. Studies conducted in a research setting met 52% (11/21) of the quality items; 33% (7/21) remained unclear and 14% (3/21) were not met (table 3).

Table 3.

Methodological quality of included studies

		Raters blinded to findings of others	Risk for order effect (sequence of examination)		Appropriate statistical measure of agreement
Reproducibility among experts in insurance setting
de Kort, 199232		Yes	No risk		Yes
Dickmann, 20079		Yes	No risk		NA
Elder, 199434		Unclear	No risk		Yes
Ikezawa, 201037		Yes	No risk		Yes
Ingravallo, 200811		Yes	No risk		Yes
Lederer, 199839		Yes	No risk		NA
Rudbeck, 201143		Yes	No risk		Yes
Schellart, 201344		Unclear	Unclear		Yes
Schreuder, 201245		Unclear	No risk		Unclear
Slebus, 201046		Yes	No risk		Yes
Spanjer, 200823		Yes	No risk		Yes
Spanjer, 200922		Unclear	No risk		Yes
Spanjer, 201024		Yes	No risk		Yes
Reproducibility among experts and health professionals in insurance setting
Dell-Kuster, 201433		NA	NA		Yes
Lax, 200438		NA	NA		NA
Okpaku, 199442		NA	NA		NA
Research setting
Berns, 200729	Unclear			Yes	Yes
Chopra, 200230	Probably yes			No	Yes
Daradkeh, 199431	Unclear			Unclear	Yes
Hannula, 200635	No			Yes	Yes
Hill, 198929	Yes			Unclear	Yes
Mundo, 201040	Yes			Unclear	Yes
Nozu, 199541	Unclear			Unclear	Yes

With regards to generalisability of the findings to real life disability evaluation, 44% (7/16) of studies in the insurance setting were rated as “generalisable,” 19% (3/16) as “probably generalisable,” and 37% (6/16) as “probably not generalisable” (table 4). Kendall’s W for reviewers’ concordance in ranking generalisability was 0.93, with a rank correlation of 0.89, confirming high agreement among the raters’ rankings.

Table 4.

Generalisability of study findings to real world of insurance medicine^*

	Recruitment strategy (for claimants)	Recruitment success	Verisimilitude	Range of experience in raters	Specific training for work capacity assessment	Training for study purposes	No of cases	No of raters	Generalisability
de Kort, 199232	Random sample	NA	Records of real patients	Narrow	Yes	No	180	5	Yes
Dell-Kuster, 201433	Consecutive sample	>80%	Real patients	Wide	No	No	3562	Unclear	Yes
Dickmann, 20079	NA	NA	Video case scenario	Narrow	Yes	No	1	22	Probably no
Elder, 199434	NA	NA	Written case scenarios	Unclear	Yes	No	10	35	Probably no
Ikezawa, 201037	NA	NA	Written case scenarios	Wide	Yes	No	3	36	Probably yes
Ingravallo, 200811	Random sample	>80%	Real patients	Narrow	Yes	Yes	15	16	Yes
Lax, 200438	Random sample	>80%	Records of real patient	Narrow	Yes	No	23	2	Yes
Lederer, 199839	Any other recruitment	NA	Records of real patients	Wide	Yes	No	1	103	Probably yes
Okpaku, 199442	Unclear, not reported	NA	Records of real patients	Narrow	Yes	Yes	144	6	Probably yes
Rudbeck, 201143	NA	NA	Written case scenarios	Unclear	Yes	No	8	11	Probably no
Schellart, 201344	NA	NA	Video case with actor	Wide	Yes	Yes	4	40	Probably no
Schreuder, 201245	NA	NA	Written case scenarios	Unclear	Yes	No	132	5	Probably no
Slebus, 201046	NA	NA	Written case scenarios	Narrow	Yes	Mixed, unclear	5	51	Probably no
Spanjer, 200823	Random sample	NA	Records of real patient	Wide	Yes	No	12	12	Yes
Spanjer, 200922	Random sample	NA	Records of real patients	Narrow	Yes	No	30	27	Yes
Spanjer, 201024	Any other recruitment	>50-80%	Real patients	Narrow	Yes	No	62	16	Yes

NA=not applicable.

*44% of studies rated as generalisable, 19% as probably generalisable, and 37% as probably not generalisable (see appendix 2 for details).

Studies conducted in insurance setting

In the insurance setting, 13 studies including 463 patients and 367 raters explored agreement between medical experts (two or more experts assessing the same patient) (table 1; appendix 4).^{9 11 22 23 24 32 34 37 39 43 44 45 46} Three studies including 3729 patients (with 3562 patients from a single centre³³) and eight raters (information was lacking from one study³³) explored agreement between medical experts and claimant’s treating physicians³³ or independent rehabilitation or occupational health teams with a mandate to care.^{38 42} The median number of patients per study was 13.5 (range 1-3562), and the median number of raters per study was 12 (2-103, excluding one study that did not report the number of raters³³). All but three studies^{24 3 42} used a fully crossed design (that is, all raters evaluated all patients), with a median of 11 patients (range 1-180) per rater and a median of 11.5 raters (2-103) per patient.

Table 5 summarises claimants’ characteristics. Studies focused on mental health (n=6), musculoskeletal disease (n=4), and mixed disorders (n=6). They enrolled patients with chronic diseases (n=11), chronic injuries (n=2), or mixed, acute, and chronic conditions (n=3). Most referred to a long term time frame before the evaluation for judging health status and work disability and predicted a long term perspective exceeding six months. Most studies used professional expertise only to generate a global rating of work ability (n=10). Six administered one or more specific rating instruments; five were referenced (appendix 3), and none was reported as validated.

Table 5.

Characteristics of studies investigating eligibility for work disability benefits

	Insurance setting (n=16)	Research setting (n=7)
Health conditions:
Mental disorders	38%9 11 39 42 44 46	100%29-31 35 36 40 41
Musculoskeletal disorders	25%22 24 37 38	—
Mixed (somatic and mental disorders)	38%23 32-34 43 45	—
Course of disease or injury:
Acute diseases	—	14%30
Acute and chronic diseases	6%34	28%31 35
Chronic diseases	75%9 11 22-24 32 33 39 42-44 46	43%29 36 41
Chronic injuries	13%37 38	—
No information/unclear	6%45	14%40
Composition of patient population:
Single disorders (such as narcolepsy, stroke, depression, low back pain, psychosis, depression, anxiety, schizophrenia)	31%9 11 24 39 46	71%29 30 35 36 41
Mixed disorders	69%22-24 32-34 37 38 42 43 45	29%31 40
Reference time frame before evaluation for judgments on health condition:
Short term period	6%45	71%29-31 35 40
Long term period	69%9 11 23 33 34 37-39 42-44	—
Not reported	25%22 24 32 46	29%36 41
Prognostic time frame:
Short term (<6 months)	13%37 45	29%30 35
Long term (≥6 months)	69%9 11 22-24 33 34 38 39 44 46	—
Mixed	6%43	—
Not reported	13%32 42	71%29 31 36 40 41
Use of tools to facilitate rating of work disability:
Professional expertise only	63%9 11 32-34 38 39 42 43 45	—
≥1 rating or reporting instruments	37%22-24 37 44 46	100%29-31 35 36 40 41

Work disability outcomes varied considerably between studies and included a broad spectrum of domains, definitions, and measurement approaches, ranging from work ability to the employee’s readiness and ability to return to work, the degree of disability or handicap, or reduction in working hours. Measurement approaches included scales, scores, and categories (table 6).

Table 6.

Outcomes used in insurance setting to assess global rating of disability for work

Outcome measure	Quantification
(Functional) work ability (n=5)
Global rating of work ability33	Scale from 100-0%
Work ability9	3 categories: >6 hours; 6-3 hours; <3 hours
Health related work ability43	4 categories: intact or slightly/much/extremely reduced
List of functional abilities44	Sum score; range not reported
Global rating of work ability46	Scale from 100% (status as before depressive disorder) to 0% (“inability to work”)
Fit for work recommendations (n=3)
Global rating of fit for work32	3 categories: fit/doubt fit for work/unfit for work
Recommend return to work37	3 categories: return to previous work/return to modified work/no return to work
Recommend fit for work39	2 categories: yes v no
Readiness and ability to return to work (n=1)
Readiness and ability of employee to return to work45	2 categories: high v low
Decisions on disability benefits (n=3)
Approval or decline of application for early retirement because of ill health34	4 categories: accept/reject/other action/no response
Decision on disability benefit11	Scale on % disability from 100-0%
Approval for social security benefit42	Social security administration—2 categories: yes v no. Team—4 categories: yes/maybe/no/undecided
Degree of disability or handicap (n=2)
Severity of handicap11	3 categories: no handicap/handicap/severe handicap
Agreement among occupational health professional and medical expert on 4 disability items38	3 categories: full/partial/disagreement
Reduction in working hours (n=3)
Reduction in working hours22-24	Hours/day

Studies conducted in research setting

Studies conducted in a research setting included 371 patients and 32 raters (table 2; appendix 4). Four studies reported on instrument development,^{29 35 36 41} and three studies validated existing instruments.^{30 31 40} The median number of patients per study was 39 (range 20-180), and the median number of raters per study was three (2-18). All but two studies^{29 40} used a fully crossed design, with a median of 21 patients (11-42) per rater and a median of two raters (2-4) per patient.

All studies were conducted with actual patients and focused on acute and chronic mental health conditions. Most used a short term time frame before the evaluation for judging occupational functioning, two provided a short term prognostic judgement on occupational functioning, and this information was missing in five studies (table 2). All seven studies used instruments of varying complexity to elicit or to report capacities or limitations to determine a global rating for occupational functioning (appendix 4). All studies generated global ratings on a range of outcomes for occupational functioning, such as “occupational functioning” or “remunerative employment” (table 7).

Table 7.

Outcomes used in research setting to assess global rating of disability for work

Outcome measure	Quantification
Functioning within work environment; occupational skills (n=5)
Global rating about functioning within work environment29	7 item Likert scale, 1 (normal functioning)-7 (total disability)
Adult personality functioning assessment36	6 point scale, 0-5, higher values indicate worse functioning
Occupational functioning31	6 point scale, “no dysfunction” to “maximum dysfunction”
Occupational functioning35	Scale 100-0, higher values indicate better functioning
Occupational skills40	Scale 100-0, higher values indicate better occupational skills
Remunerative employment, employability
Global rating for remunerative employment30	5 item scale: no to complete or extreme problem
Employability41	No information

Inter-rater reliability of ratings on disability for work and occupational functioning—insurance setting

Overall, across all conditions and outcomes, the median inter-rater reliability was 0.45, ranging from ICC of 0.86 (musculoskeletal disorders; reduction in working hours²²) to κ of –0.10 (narcolepsy; disability benefit¹¹) (table 8). Six studies reported excellent or good inter-rater reliability for a global rating of work disability, with ICCs of 0.64⁴⁶ and 0.65,⁴⁴ percentage agreement 82.4% (“return-to-work” recommendations³⁷), or κ of 0.80²³ and 0.86²² for reduction in working hours. One study presented mixed judgments in a single case, which we considered overall as “good agreement” based on the relative importance of the outcomes of functional ability to work (91.2% agreement on remaining work ability) and for work recommendations (86% agreement on limitations in work performance) over the outcome of readiness and ability to return to work (56% agreement on reduction in working hours).³⁹ All Dutch studies used one or more rating instruments for determining functional limitations.^{22 23 44 46} Two studies qualified as “generalisable,”^{22 23} two as “probably generalisable,”^{37 39} and two as “probably not generalisable.”^{44 46}

Table 8.

Reproducibility among experts stratified by level of inter-rater reliability

Study	Use of rating or reporting instrument	Outcome	Outcome measure and IRR findings	Generalisability to real world disability evaluation
Studies investigating reproducibility of work disability evaluations between experts (insurance setting)
Excellent to good
Schellart44	Yes	Functional work ability	ICC 0.65	Probably no
Slebus46	Yes	Functional work ability	ICC 0.64	Probably no
Ikezawa37	Yes	Recommend return to work	% agreement 82.4%	Probably yes
Spanjer 200823	Yes	Reduction in working hours	κ 0.8	Yes
Spanjer 200922	Yes	Reduction in working hours	κ 0.86	Yes
Lederer39	NR	Remaining work ability; limitations in work performance (single case)	Frequency of agreement: 91%; 86%	Probably yes
Fair to poor
De Kort32	NR	Fit for work	κ 0.38	Yes
Dickmann9	NR	Work ability in last job (single case): <3 hours; 3-6 hours; >6 hours	Frequency of agreement: 27%; 36%; 37%	Probably no
Elder34	NR	Early retirement	κ 0.24	Probably no
Ingravallo11	NR	Disability benefit	κ −0.10-0.35	Yes
Rudbeck43	NR	Health related work ability	κ 0.33	Probably no
Schreuder45	NR	Readiness and ability to return to work	κ 0.14	Probably no
Spanjer 201024	Yes	Reduction in working hours	ICC 0.53	Yes
Studies investigating reproducibility of work disability evaluations between experts and health professionals with mandate to care (insurance setting)
Dell-Kuster33	NR	Work ability: last job; alternative job	Agreement: 51%; 20%	Yes
Lax38	NR	Agreement on 4 disability items: full; partial; no agreement	Frequency of agreement*: 4%; 34%; 78%	Yes
Okpaku42	NR	Approval for social security benefits	Frequency of agreement: yes/no decisions 40%	Probably yes
Studies investigating reproducibility of work disability evaluation between researchers (research setting)
Excellent to good
Berns29	Yes	Functioning in work environment	ICC 0.86	NA
Chopra30	Yes	Remunerative employment	κ 0.62	NA
Hannula35	Yes	Occupational functioning	ICC 0.91	NA
Hill36	Yes	Dysfunctioning in work as social role	ICC 0.76	NA
Mundo36	Yes	Occupational skills	ICC 0.75	NA
Nozu41	Yes	Employability	ICC 0.88	NA
Fair to poor
Daradkeh31	Yes	Occupational functioning	κ 0.53	NA

NR=not reported; ICC=intraclass correlation; IRR=inter-rater reliability; NA=not applicable.

*Total >100%.

Seven studies reported fair or poor inter-rater reliability across all global ratings of work disability outcomes. All but one²⁴ based their judgments exclusively on professional expertise. One study presented discordant judgment on a single case⁹ (one third of experts each rated “full,” “partial,” or “no work ability” for the same patient). Three studies qualified as “generalisable” and four as “probably not generalisable.”

Reproducibility between experts and health professionals with a mandate to care

Overall, across conditions and outcomes, percentage agreement ranged from 51% (work ability in last job)³³ to 4% (somatic occupational disorders; four disability items)³⁸ (table 8). Three studies compared reproducibility of ratings on work disability between experts and health professionals with a mandate to care.^{33 38 42} One study reported poor agreement between experts and the claimants’ treating physicians.³³ Another study reported highly discordant judgments on disability between medical experts and health professionals of an occupational health centre.³⁸ The third study found poor agreement between the decisions of the social security administration and those of an independent rehabilitation team.⁴²

The direction of disagreement was mixed. Medical experts approved higher levels of work ability for claimants³³ or their recommendations and decisions favoured the insurer,³⁸ while in the third study, the rehabilitation team was more reluctant to grant disability benefits to patients with mental disorders than the social security administration.⁴² All studies based their judgments exclusively on professional expertise. Two studies qualified as “generalisable,”^{33 38} one as “probably generalisable.”⁴²

Inter-rater reliability of ratings on disability for work and occupational functioning—research setting

Overall, across conditions and outcomes, the median inter-rater reliability was 0.76, ranging from an ICC of 0.91 (anxiety and mood disorders; occupational functioning³⁵) to κ of 0.53 (mixed mental disorders; occupational functioning³¹).

Five of seven studies (71%) reported excellent (global) inter-rater reliability on work disability judgements with ICCs ranging from 0.75⁴⁰ to 0.91.³⁵ The remaining two studies^{30 31} reported agreement on single items: good agreement (κ 0.62) regarding the ability to engage in remunerative employment³⁰ and fair agreement (κ 0.53) for difficulties encountered in day-to-day work (occupational functioning).³¹

Impact of generalisability and level of standardisation on inter-rater reliability

Testing the relation between inter-rater reliability and subjective (versus objective) and chronic (versus acute) health conditions as well as the studies’ overall generalisability did not show any association (subjectivity, 23 studies, P=0.46; chronicity, 20 studies, P=0.45; generalisability, 16 studies, P=0.65). Testing the relation between the level of standardisation and inter-rater reliability in all 23 studies showed a highly significant association (P=0.006).

Discussion

Principal findings

Current evidence regarding reliability of disability evaluation is limited and shows highly variable agreement between medical experts. Higher agreement seems to be associated with the use of a standardised approach to guide judgment and studies in a research (manufactured) setting.

Strengths and limitations

Strengths of our study include broad inclusion criteria to define eligibility and inclusion of publications in any language, which increases confidence that we captured all studies eligible for our review. Our outcome—global rating of disability for work—is highly relevant to the practice of medical experts, disability insurers, and employers, which increases the practical implications of our findings. Further, we evaluated the generalisability of evidence by following international guidance for evaluating reliability studies^{17 19 20} and by using an explicit approach in eliciting reviewers’ judgments on the relative weights of the generalisability items. While the high agreement we found among reviewers strengthens the credibility of the results, this approach requires further validation. Some cut offs of the generalisability criteria (such as number of raters) are context specific and might not be applicable to settings other than assessment of disability. Furthermore, variability of study designs, measures of agreement, and outcomes precluded statistical pooling across studies.

Relevant literature

Disability evaluation is a poorly understood process^{14 15 16} that lacks any reference standard to confirm the validity of the findings. Health professionals who perform this task assess medical restrictions and limitations of claimants and are often asked to infer consequences on the ability to work. This, however, requires expertise in vocational rehabilitation, as medical restrictions do not correlate well with function and the ability to work.⁵ In such situations, reliability studies evaluate the measurement properties of observers.⁴⁷ At each step of disability evaluation, multiple sources of variation come into play (box 1),^{15 16} including experts’ personal attitudes, beliefs, and values towards disability, all of which affect the global judgment of work disability. Left unmanaged, these sources of variation can lead to low inter-rater reliability.

We found higher agreement when disability evaluation was guided by a standardised instrument. Instruments that standardise the collection, interpretation, and reporting of information are one promising approach to reduce variation.¹⁵ Five of the seven Dutch studies that used instruments to guide assessment of work disability all achieved fair to good reliability. As all Dutch insurance physicians undergo four years of specialty training in insurance medicine,⁴⁸ however, we cannot disentangle whether higher agreement is a result of use of a formal instrument or calibration by training, or both.

We did not detect any association between inter-rater reliability and subjectivity or chronicity of the health conditions, or overall generalisability to real world disability evaluation. The low number of studies in the analyses, however, precludes any premature conclusions that such associations do not exist.

Not all sources of variations are easily accessible to change. Other sources, in particular attitudes, beliefs, and value judgments, will require other approaches.⁴⁹ Implicit in the use of evaluations of disability by a third party is the concern that treating clinicians could have difficulty providing impartial assessments of their patients. Indeed, our findings suggest that medical experts (versus treating physicians) are more likely to conclude that claimants are capable of working. Claimant lawyers and patients’ organisations have raised concerns that experts who are paid to assess claimants for insurers might feel pressure to render opinions that favour the referral source.

Implications for practice

Our review suggests that use of standardised instruments could improve reliability in expert judgments on work disability. Appropriate instruments should therefore be considered in routine practice of disability evaluations (see table 8 and appendix 3 for examples). To ensure appropriate administration and interpretation of the findings, experts will need appropriate training and calibration on the use of such instruments. As most instruments reported in this review are available only in Dutch, other countries would need to develop their own instruments or translate instruments and accompanying manuals in national languages.

As few countries have standards to guide assessments, standardised instruments that improve reliability could become a target for change and parties ordering assessments should demand their use.

Unanswered questions and future research

Given the widespread use of evaluation of disability for work to determine claimants’ eligibility for work replacement benefits, our findings suggest that further research to improve reliability is urgently needed. Promising targets include formal training in evaluation of capacity to work,⁵⁰ use of standardised instruments to guide disability evaluations,⁵⁰ and addressing the conflict of interest that arises when insurers (or lawyers) select their own experts. Further, there might be greater need for strategies to improve agreement when patients present with subjective complaints. Ikezawa and colleagues found that different medical experts were able to agree on claimant’s ability to return to work in 97% of claims involving a fracture and 94% of claims involving a dislocation, but only 56% of claims because of chronic low back pain.³⁷ Our review further suggests that interventions should be validated in real insurance settings, as experimental settings could artificially inflate agreement.

Improved knowledge of individual factors that contribute to variability in evaluation of capacity to work is also needed. Promising targets could provide a starting point to develop and test focused strategies to reduce variability (for example, appropriate assessment tools, guidelines, standard cases). Guidance is also required to inform the required level of inter-rater reliability to ensure equal treatment of claimants. Any decision on what constitutes an appropriate threshold, which might be similar to thresholds for clinical medical tests,^{27 28} will require societal discussion on what constitutes acceptable differences in the treatment of claimants or align to standards set by professional organisations of psychology or education. To make evaluations on work disability fair and meaningful and thereby qualify for decisions on claimants’ disability benefits, however, we suggest a minimum intraclass correlation coefficient of 0.6 (the cut off between fair and good inter-rater reliability), with a sufficiently narrow 95% confidence interval (0.5 to 0.7) to exclude poor reliability.

Conclusions

Despite their widespread use, medical evaluations of work disability show high variability and often low reliability. Use of standardised and validated instruments to guide the process could improve reliability. There is an urgent need for high quality research, conducted in actual insurance settings, to explore promising strategies to improve agreement in evaluation of capacity to work.

What is already known on this topic

• Social and private disability insurers use medical experts to evaluate claimants with impaired health to determine eligibility for disability benefits

• Anecdotal evidence suggests that experts often disagree in their judgment of capacity to work when assessing the same claimant

What this study adds

• This systematic review of 23 reproducibility studies from 12 countries shows a lack of good quality data applicable to the real world of disability assessment

• In most studies, medical experts reached only low to moderate reproducibility in their judgment of capacity to work

• Studies reported higher reproducibility when experts used a standardised evaluation procedure

• These findings are disconcerting and call for substantial investment in research to improve assessment of disability

Web Extra.

Extra material supplied by the author

Appendix 1: Search strategy

Appendix 2: Determination of overall generalisability

Appendix 3: Studies reporting use of instrument with instrument referenced

Appendix 4: Full details of performance and findings of included studies