Assessing the return-to-work mode of precarious workers with mental health issues: reliability, validity, and usability of the REMODE-tool

Abstract

Mental health issues are highly prevalent among precarious workers, often leading to prolonged sickness absence and unemployment. A worker’s perceptions and attitudes about return-to-work are important determinants of work resumption and can be categorized into three modes: an expectant, an ambivalent-uncertain, and an active return-to-work mode. To support professionals in identifying these modes, we developed the REturn-to-work MODe Evaluation (REMODE) tool. This study evaluated REMODE’s inter-rater agreement, inter-item consistency, content validity, and usability. In a vignette study, 71 occupational health professionals from a Dutch social security institute viewed six videos of consultations between insurance physicians and precarious workers. They then used REMODE to assess the worker’s return-to-work mode and need for occupational support. Participants also rated REMODE’s validity and usability with 5-point Likert scale questions based on the Content Validity Index and System Usability Scale. We used a generalized linear mixed model to analyse inter-rater agreement and inter-item consistency. The professionals highly agreed on the REMODE-score [ICC 0.87 (95% CI 0.63–0.97)] and corresponding return-to-work mode [ICC 0.83 (0.54–0.95), Κω 0.75 (0.74–0.75)]. Their agreement on need for occupational support was moderate [ICC 0.65 (0.30–0.89), Κω 0.57 (0.56–0.57)]. REMODE’s internal consistency demonstrated excellence (Cronbach’s alpha 0.92), and the content validity index (0.83) and system usability scale (76) were acceptable. REMODE is a promising tool for occupational health professionals as it supports identification of the return-to-work mode of precarious workers with mental health issues. We propose a refined version of RE-MODE for use in occupational healthcare and research.

Introduction

Nearly half of all sick leave is attributed to mental health issues, often resulting in unemployment and work disability, particularly among workers without permanent contracts, referred to as precarious workers. For these individuals, work resumption is especially challenging, because alongside mental health treatment, they must obtain new employment before occupational rehabilitation can even begin [1, 2]. Precarious workers also face more psychosocial stressors and poorer health compared to individuals with permanent jobs [3], leading to longer sick leave and higher rates of long-term disability benefits [1]. This is concerning, as most people with mental health issues do wish to work and a successful return-to-work (RTW) can enhance well-being and support recovery [4, 5].

Therefore, tailored occupational healthcare is essential to facilitate RTW in precarious workers. Understanding factors affecting work resumption enables occupational health professionals to provide targeted RTW-interventions. An important factor is a worker’s attitude with underlying perceptions regarding RTW. Recognizing attitudes and perceptions hindering RTW is therefore important for identifying workers at risk of long-term sickness absence [6], and for providing appropriate occupational rehabilitation support [7, 8]. Drawing from our prior research, we identified that RTW-perceptions and attitudes can be categorized into three modes within the RTW process: the passive-expectant, ambivalent-uncertain, and active RTW-mode [7, 9]. Workers may shift between these modes, as RTW-perceptions and attitudes fluctuate during recovery and reintegration. Because ‘passive-expectant’ carries negative connotations for professionals and workers [unpublished data], we use ‘expectant’ RTW-mode throughout the article.

An expectant RTW-mode is characterized by a hesitant attitude, negative perceptions of workability, and fear of symptom relapse. While this mode seems common early in sickness absence due to impeding symptoms and treatment initiation, it may persist or (re-)emerge later because of psychological or contextual barriers. In contrast, the ambivalent-uncertain RTW-mode involves a desire to work, but uncertainty regarding workability and the steps needed for RTW. This mode may arise as symptoms begin to improve. Finally, the active RTW-mode reflects confidence in workability and presumed benefits of RTW, with workers typically seeking employment and nearing recovery [7]. These RTW-modes resemble the pre-contemplation, contemplation, and acting stages of the stages-of-change model [10] but incorporate perceptions and attitudes specific to precarious workers with mental health issues [7].

Identifying a worker’s RTW-mode enables occupational health professionals to implement tailored interventions that support workers throughout the RTW process [7, 8]. Workers with an active RTW-mode may benefit from targeted job application training, while those displaying an expectant RTW-mode may require a more comprehensive intervention, initially focused on addressing RTW-barriers, such as severe symptoms, limited social support, or challenging work conditions [6–8]. If unhelpful RTW-perceptions and attitudes themselves hinder functional recovery, interventions such as motivational interviewing [11] and work-focused cognitive behavioural therapy [12] may foster more adaptive responses, supporting RTW and reduce sickness absence. The RTW-mode reflects only the professional’s observation and is not a diagnostic or normative classification. The RTW-mode may diverge from a worker’s actual readiness for work; e.g. someone could display an active RTW-mode while not yet medically fit for work [13]. Overall, identifying the RTW-mode supports a biopsychosocial assessment of support needs in occupational healthcare.

However, recognizing RTW-perceptions and attitudes can be challenging, and existing instruments, mainly developed for fixed-contracted workers, often include items unsuitable for sick-listed precarious workers with mental health issues [14]. Furthermore, these instruments predominantly rely on self-report, while our previous study showed that professionals prefer personal contact to ask specific questions and observe non-verbal signs [7]. We, therefore, developed a professional checklist to assess the RTW-mode during consultations with precarious workers. From interviews with professionals [7] and precarious workers [9], we identified key perceptions and attitudes of each RTW-mode to develop the REMODE (REturn-to-work MODe Evaluation)-tool. In our previous unpublished study, professionals were interviewed after using REMODE during three real consultations with precarious workers. They found REMODE easy to use and believed it could enhance interprofessional communication regarding a worker’s RTW-readiness and support needs.

The current study aimed to further evaluate REMODE’s psychometric properties within an experimental context, an essential step for future implementation of newly developed tools in occupational healthcare. The findings may also support further development of REMODE and enhance its applicability in real-world practice. The primary objective was to evaluate agreement among occupational health professionals regarding the REMODE-score and RTW-mode classification of sick-listed precarious workers with mental health issues. Given the widely acknowledged variation in professional assessments of workability and deployment of RTW-interventions [15], this study also examined interprofessional agreement on a worker’s readiness for work and occupational support needs. Additionally, we assessed content validity, usability, and usefulness to evaluate REMODE’s relevance and readiness for real-world use.

Methods

Study design

This experimental cross-sectional study used six video case-vignettes to evaluate psychometric properties of the REMODE-tool, simulating real occupational health consultations [16]. Occupational health professionals viewed the vignettes in random order to control for order effects [17]. After each video, they used REMODE and subsequently evaluated REMODE via questionnaire. We used the STROBE checklist for reporting observational studies [18] and the COSMIN guideline for reporting studies on measurement properties of patient-reported outcome measures, including specific recommendations on content validity and reliability [19]. The Medical Ethics Committee of the Academic Medical Center of Amsterdam UMC determined that no ethics committee approval was required for this study (reference number W22_283 # 22.343). All participants signed informed consent before participation.

Setting and recruitment

Between February and June 2023, we recruited 75 occupational health professionals working at The Dutch Social Security Institute: The Institute for Employee Benefits Schemes (UWV) by advertising about this study via e-mail and UWV intranet. Interested professionals provided their e-mail address to the researchers and received additional study information. After giving informed consent, they received a personal Survalyzer login to watch the videos and fill in questions. Completing the full questionnaire, including watching the videos, required approximately 60 min. Participants could pause and resume, but not rewatch the videos. Upon completing the questionnaire, participants received a €20 voucher.

Participants

To obtain a heterogeneous sample, we applied purposive sampling based on age, work experience, and professional type, including labour experts, case workers, RTW-coordinators, social nurses, psychologists, and (insurance) physicians. Professionals were eligible if they had ≥3 years of experience in occupational healthcare for precarious workers and provided informed consent. Participants were excluded from data-analyses if they did not complete the questionnaire, finished in <45 min, skipped videos, or provide largely identical answers. Based on the calculated sample size for reliability analyses [20] ≥50 participants were required.

Video case-vignettes

Six video case vignettes were developed using the checklist of Hillen et al. [16], portraying consultations between insurance physicians and sick-listed precarious workers with various mental health issues and diversity in RTW-perceptions and attitudes (Supplementary Table S1). The consultations were fictional, but the cases were derived from extensive experience of the researchers: an occupational health physician (F.S.), an insurance physician (Y.S.), and a principal investigator in mental health and work research (K.N.). To reflect real-life consultations, each video started with a written summary of the consultation reason and the worker’s characteristics.

Development of REMODE

REMODE is a tool designed to assist occupational health professionals in identifying the RTW-mode of sick-listed precarious workers with mental health issues. Supplementary File S1 includes REMODE (Supplementary Fig. S1) and a flow chart of its development (Supplementary Fig. S2). REMODE compromises 12 perception-related items and eight attitude-related items, based on key themes from previous studies [6, 8], which are recognized determinants for RTW [8, 14]. Each item is scored as: ‘present’, ‘occasionally present’, or ‘absent’. Total scores range from 20 to 60, indicating an expectant (20–33), ambivalent-uncertain (34–47), or active (48–60) RTW-mode.

Study questionnaire

After viewing each video, participants first indicated the worker’s RTW-mode without using REMODE, followed by scoring REMODE’s items. They then assessed the worker’s readiness for RTW and need for occupational support using a four-point scale: (1) ready to work, (2) ready to work but needs re-employment support, (3) not yet ready to work but can start with prevocational training, and (4) not yet ready for work or prevocational training. After completing all vignettes, participants answered 10 system usability scale (SUS) questions [21], and six perceived-usefulness questions adapted from the behavioural intentions of the Technology Acceptance Model (TAM), all on five-point Likert scales [22]. Finally, participants rated item relevance on a four-point Likert scale for the Content Validity Index (CVI) [23] and could suggest missing items. The complete questionnaire is provided in Supplementary File S1.

Outcomes

The primary outcomes were the inter-rater reliability of the REMODE-score (intraclass correlation coefficient, ICC) and RTW-mode classification (Cohen’s weighted Kappa, Κω, ICC). Secondary outcomes included inter-rater reliability on RTW-readiness and support needs (Κω, ICC), inter-rater reliability on individual item scores (Κω, ICC), inter-item consistency (Cronbach’s Alpha), and each item’s contribution to the total REMODE-score (beta coefficient, β), and RTW-mode classification (odds ratio, OR). Additionally, we examined correlations between the REMODE-score, RTW-mode, and RTW-readiness and support needs (Spearman’s rho, β), assessed item- and scale-level content validity (I-CVI, S-CVI), calculated the SUS-score, and summarized responses to usefulness questions descriptively.

Item revision

Items with an I-CVI < 0.70 were excluded from the refined version of REMODE [23]. Items scoring 0.70–0.78 were revised based on their contribution to the REMODE-score (β) and RTW-mode (OR), inter-rater reliability (ICC, Κω), impact on inter-item consistency (α) if deleted, participants’ comments, and insights from our unpublished studies on workers’ and professionals’ perspectives.

Statistical analyses

Descriptive statistics were applied to summarize participants’ characteristics and their absolute agreement on the REMODE-score, RTW-mode, items scores, and RTW-readiness and support needs, as well as their responses to the SUS, TAM, and CVI-related questions. For the SUS-score we used the recommended formula [21].

We built a generalized linear mixed model (GLMM) to analyse relative inter-rater agreement, inter-item consistency, and each item’s contribution to the total score, based on the repeated measurements in our fully crossed design [24, 25]. We began by fitting a simple GLMM and gradually increased its complexity, comparing models using likelihood ratio tests. In the final model, we treated raters as repeated measures and incorporated participants and vignettes as random effects to account for between-participant and between-vignette variability. Professional type and work experience were included as fixed effects due to their strong association with the REMODE-score. Subsequently, we visually assessed models’ assumptions using plots and correlation matrices of residuals and random effects. Levene’s test was applied to evaluate homogeneity of rater variances (heteroscedasticity) [26]. Finally, we used variance components to calculate ICCs, Cronbach’s Alpha, and correlations coefficients.

Relative agreement on the RTW-mode classification, items scores, and RTW-readiness and support needs, was assessed using median Kωs across rater pairs, with quadratic weights to capture disagreement magnitude. ICCs were also calculated to assess rating consistency while accounting for data hierarchy and participant characteristics.

ICC values were interpreted as: <0.5 poor, 0.5–0.75 moderate, 0.75–0.90 good, and ≥0.90 excellent agreement [24]; Κω values as: <0 poor, 0–0.20 slight, 0.21–0.4 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and >0.80 almost perfect agreement [27].

Supplementary File S1 provides a detailed description of all analyses performed in IBM SPSS Statistics for Windows, Version 28.0 (IBM Corp, Armonk, NY, USA).

Results

Of the 75 occupational health professionals recruited, four were excluded: two did not meet the inclusion criteria, and two completed the questionnaire <45 min. Table 1 presents the characteristics of the remaining 71 participants, yielding a total of 2485 rater pairs for reliability analyses.

Table 1.

Participant characteristics (N = 71)

Characteristic	Category	N	%
Age	20–29 years	6	9
	30–39 years	24	34
	40–49 years	18	26
	50–59 years	19	27
	≥60 years	4	6
Gender	Female	55	78
	Male	16	23
	Other	0	0
Education	Primary vocational or prevocational education	3	4
	Secondary vocational or pre-university education	11	16
	University BSc or higher vocational education	21	30
	University MSc or MD	17	24
	Postgraduate, including medical specialization	17	24
	Doctorate, PhD	2	3
Type of occupational health professional	Case worker	15	21
	RTW-coordinatora	11	16
	Labour experta	10	14
	Social nurse	6	9
	Psychologist	3	4
	Physician	7	10
	Insurance physician	18	25
	Other	1	1
Work experience	0–5 years	28	39
	5–10 years	24	34
	10–15 years	8	11
	15–20 years	2	3
	>20 years	9	13

^aNo access to workers’ medical or psychological data in Dutch occupational healthcare and in this study.

Visual inspection revealed approximately normally distributed REMODE-scores, except for vignette E, indicating a ceiling effect. Figure 1 displays boxplots of the scores (20–60) for each vignette.

Figure 1.

Distribution of professionals’ REMODE-scores across video case-vignettes. Boxplots showing professionals’ REMODE-scores across six video case-vignettes of sick-listed precarious workers with mental health issues. REMODE-score interpretation: 20–33 expectant, 34–47 ambivalent-uncertain, 48–60 active RTW-mode.

Inter-rater agreement

Relative agreement on the REMODE-score (ICC 0.87) and RTW-mode classification (ICC 0.83; Κω 0.75) was good (Table 2), with only minor variability observed in individual case-vignettes (ICC 0.80–0.91, Supplementary Table S2). Pre- vs. post-test analyses revealed significantly higher agreement on the RTW-mode when REMODE was used (Κω difference +0.02, Z 2.54, P .015), despite potential learning effect from repeated use of REMODE. Agreement on RTW-readiness and support needs assessment was moderate (ICC 0.65; Κω 0.57). Item-level agreement ranged from ICC 0.54 to 0.83 and Κω 0.34 to 0.80. Table 3 and Supplementary Table S3 show all item-level reliability results.

Table 2.

Scale-level reliability outcomes of REMODE (K = 71 raters, N = 2485 rater pairs)

		Absolute agreement %	ICC	95% CI	Κω	95% CI	Interpretation
Inter-rater agreement	REMODE-score	–	0.87	0.63–0.97	–	–	Good
	RTW-mode before using REMODEa	78	0.83	0.58–0.94	0.73b	0.69–0.74	Good
	RTW-mode based on REMODE-score	80	0.83	0.54–0.95	0.75b	0.74–0.75	Good
	RTW-readiness and support needs	57	0.65	0.30–0.89	0.57	0.56–0.57	Moderate
Inter-item consistency (Cronbach’s alpha)	Full scale	–	0.92	0.77–0.98	–	–	Excellent
	Refined scalec	–	0.93	0.78–0.98	–	–	Excellent

^aIncludes a potential learning effect in five out of six randomly assigned vignettes.

^bPre- vs. post-test using Wilcoxon signed-rank test P = .02.

^cRefined version of REMODE includes all original items, except item 9 and item 15, which were excluded due to their I-CVI <0.70.

Table 3.

Item-level scores for content validity, inter-rater reliability, and contribution to inter-item consistency, REMODE-score, and RTW-mode

K = 71 raters, N = 2485 rater pairs	Contribution to REMODE scale							Inter-rater agreement					Interpretation
Item	I-CVI	α a	95% CI	β total score	95% CI	OR RTW-mode	95% CI	%	ICC	95% CI	Κω	95% CI	Overall score
RTW perception-related items
1. ‘Considers themselves able to (return to) work’	0.96	0.91	0.75–0.97	2.01	1.60–2.43	26.07	7.58–89.66	81	0.83	0.58–0.94	0.68	0.66–0.70	Excellent
2. ‘Beliefs themselves able to independently obtain employment’	0.87	0.92	0.75–0.98	0.80	0.49–1.12	1.36	0.51–3.65	71	0.75	0.38–0.92	0.75	0.73–0.75	Good
3. ‘Perceives themselves as actively seeking employment’	0.89	0.92	0.75–0.98	1.37	1.05–1.68	3.62	1.44–9.10	75	0.77	0.47–0.94	0.80	0.78–0.81	Good
4. ‘Assesses their labour market prospects positively’	0.90	0.91	0.75–0.97	1.66	1.33–1.99	15.21	5.15–45.03	73	0.80	0.46–0.95	0.68	0.67–0.69	Good
5. ‘Expects that returning to work will enhance overall wellbeing’	0.89	0.92	0.75–0.98	1.54	1.25–1.84	2.73	1.26–5.92	71	0.71	0.41–0.90	0.52	0.16–0.53	Good
6. ‘Evaluates past work experiences positively’	0.75	0.93	0.78–0.98	1.39	1.17–1.62	5.27	2.52–11.02	62	0.54	0.19–0.86	0.34	0.32–0.37	Moderate
7. ‘Perceives encouraging RTW-advice from others (family, friends, caretakers)’	0.70	0.92	0.92–0.93	1.25	1.00–1.51	3.49	1.57–7.75	74	0.68	0.34–0.92	0.67	0.66–0.67	Moderate
8. ‘Perceives difficulties in their personal life’	0.89	0.92	0.77–0.98	1.39	1.33–1.81	6.10	2.83–13.14	72	0.76	0.33–0.94	0.73	0.72–0.74	Good
9. ‘Perceives current vocation support as hindering work resumption’	0.61	0.92	0.78–0.98	1.25	0.83–1.37	1.58	0.78–3.21	78	0.63	0.41–0.93	0.77	0.73–0.78	Poor
10. ‘Experiences fear of losing benefits’	0.70	0.93	0.78–0.98	1.57	1.47–2.04	3.11	1.42–6.82	80	0.80	0.54–0.93	0.63	0.56–0.66	Moderate
11. ‘Perceives major impairments in daily functioning’	0.89	0.92	0.76–0.98	1.80	1.52–2.08	3.46	1.62–7.43	64	0.67	0.32–0.90	0.69	0.67–0.70	Good
12. ‘Beliefs that rest is necessary and/or that current daily activities reflect their maximum capacity’	0.80	0.92	0.76–0.98	1.62	1.32–1.93	3.42	1.55–7.55	68	0.72	0.33–0.91	0.57	0.56–0.58	Good
RTW attitude-related items
13. ‘Demonstrates a focus on returning to work’	0.89	0.91	0.75–0.97	2.31	1.94–2.69	3.95	1.56–1.00	75	0.70	0.41–0.90	0.62	0.62–0.64	Good
14. ‘Demonstrates a focus on symptoms and/or treatment’	0.86	0.92	0.76–0.98	1.50	1.24–1.77	2.65	1.35–5.21	64	0.59	0.23–0.87	0.60	0.59–0.62	Good
15. ‘Expresses strong emotions or grief related to recent job loss’	0.54	0.92	0.77–0.98	1.19	0.92–1.46	2.67	1.41–5.05	68	0.74	0.35–0.91	0.50	0.48–0.52	Poor
16. ‘Appears motivated for returning to work’	0.96	0.92	0.75–0.98	3.08	2.69–3.47	29.10	9.88–85.66	75	0.77	0.41–0.94	0.70	0.68–0.70	Excellent
17. ‘Displays severe impairments in personal and social functioning’	0.93	0.89	0.88–0.89	1.34	1.06–1.61	3.28	1.64–6.54	61	0.57	0.24–0.85	0.62	0.61–0.64	Good
18. ‘Displays despair or pessimism about returning to work’	0.87	0.92	0.76–0.97	1.80	1.51–2.08	7.31	3.49–15.32	69	0.70	0.62–0.89	0.55	0.53–0.56	Good
19. ‘Displays avoidance or resistance about returning to work’	0.90	0.92	0.76–0.98	1.14	0.88–1.40	2.47	1.31–4.68	60	0.54	0.25–0.84	0.58	0.56–0.59	Good
20. ‘Appears anxious, frozen, or distressed about returning to work’	0.85	0.92	0.76–0.98	1.49	1.25–1.73	2.17	1.14–4.14	61	0.56	0.24–0.84	0.51	0.49–0.52	Good

^aCronbach’s alpha if item deleted, α total scale: 0.92 (95% CI 0.77–0.98).

Correlation between REMODE-score, RTW-mode, and RTW-support needs

The mean REMODE-score (rho −0.68, P < .001) and RTW-mode score (rho −0.64, P < .001) demonstrated strong negative linear correlations with RTW-readiness and support needs, indicating that a more active RTW-mode, corresponded with lower support needs (β −0.46, P < .001; Supplementary Fig. S3).

Internal consistency

REMODE demonstrated excellent internal consistency: Cronbach’s alpha 0.92 (Table 2), with minimal changes upon item deletion (Table 3). The potential subscales perceptions (items 1–12) and attitudes (items 13–20) showed lower internal consistency, and, therefore, are not considered separate dimensions (Supplementary Table S3).

Contribution of each item to the total score of REMODE

Each item was significantly associated with both the REMODE-score and RTW-mode classification (Table 3). In the multivariable GLMM, all items remained significant predictors of the REMODE-score. Under the proportional odds assumption, all items, except 2 and 9, remained also associated with the RTW-mode classification. Item 1, 4, 13, and 16 showed the strongest associations with both outcomes.

Content validity

The S-CVI of REMODE was 0.83 based on the average across all items (S-CVI/Ave) and 0.75 based on the proportion of items with an I-CVI ≥0.78 (S-CVI/UA), indicating substantial scale-level content validity. Exclusion of items 9 and 15 based on their poor relevance (I-CVI <0.70) improved the S-CVI/Ave to 0.86 and raised the S-CVI/UA to an excellent score of 0.83. Table 3 and Supplementary Fig. S4 present the items’ relevance ratings and I-CVIs. Items 1 and 16 demonstrated the highest content validity, both with an I-CVI of 0.96. Items 6, 7, and 10 scored moderately between 0.70 and 0.78.

Item revision

Most items required no revision given their strong psychometric properties. Items 9 and 15 were excluded due to their low I-CVI score, while items 6, 7, and 10 were revised based on their moderate I-CVI values. Both Cronbach’s alpha and S-CVI values improved after excluding items 9 and 15 (Table 2 and Supplementary Table S3).

Usability and usefulness

REMODE’s SUS-score of 76, along with the mean agreement level of 4.02 out of 5 points, indicated good acceptability. Participants most strongly agreed that REMODE is easy to use (mean 4.59) and does not require personal technical support (mean 4.62). All other SUS-questions received substantial agreement (mean scores 3.55–4.27), except for ‘I think that I would like to use REMODE frequently’, showing neutral agreement (mean 3.13). This is consistent with their responses to related TAM-derived questions. However, they sufficiently agreed on ‘I intend to USE REMODE’ (mean 3.66) and the participant comments learned that REMODE may not be useful when a worker’s is clearly not yet able to work because of serious mental health issues.

Other TAM-derived questions showed that REMODE was found useful (mean 3.80), by contributing to the RTW-readiness and support needs assessment (mean 3.56), and inter-professional communication (mean 3.90). Supplementary Table S4 presents the results for all SUS- and TAM-derived questions.

Subgroup analyses

Professional groups did not differ in their agreement on the REMODE-score (ICC). However, higher education, moderate work experience, and medical-file access were associated with greater agreement on the RTW-mode and support needs. Supplementary Table S5 presents all subgroup reliability, usability, and content validity scores.

Sensitivity analyses

Additional explorative sensitivity analyses showed consistent results with our primary GLMM analyses of inter-rater agreement on the REMODE-score.

Discussion

Main findings

This study evaluated the reliability, validity, and usability of REMODE, a newly developed tool for occupational health professionals to assess the RTW-mode in sick-listed precarious workers with mental health issues. The findings showed good inter-rater reliability for both the total REMODE-score and RTW-mode classification, indicating consistent professional agreement on whether a worker’s RTW-mode was active, ambivalent-uncertain, or expectant. Agreement improved significantly by using REMODE, despite potential learning effects from repeated use. The RTW-readiness and support needs assessment showed moderate reliability but correlated strongly with both the REMODE-score and RTW-mode. Additionally, REMODE demonstrated excellent internal consistency, good scale-level content validity (S-CVI), and good usability (SUS). Based on item-level content validity (I-CVI) and participant feedback, we proposed a refined version of REMODE.

Comparison with literature

To our knowledge, REMODE is the first professional tool for assessing the RTW-mode in sick-listed precarious workers with mental health issues. Nonetheless, REMODE demonstrates similarly high inter-rater agreement compared to instruments measuring related constructs, such as the work-ability support scale (WSS) showing ICCs between 0.78 and 0.96 [28]. While some WSS-items are less applicable to precarious workers with mental health issues, items addressing contextual factors overlap with REMODE. The excellent inter-item consistency of REMODE is consistent with, or even exceeds, the results of the WSS [29] and other comparable instruments [30–32].

Moderate interprofessional agreement regarding workers’ RTW-readiness and support needs aligns also with prior research, showing even lower agreement without standardized instruments [15]. Although the study design did not allow testing whether REMODE improved agreement, strong correlations with the RTW-mode suggest that higher RTW-mode consensus may reflect greater alignment on readiness and support needs. This contributes to the assumption that identifying RTW-perceptions and attitudes, and clustering workers by support needs, enables more tailored occupational rehabilitation [33].

Lastly, item 1 (‘Considers themselves able to RTW’), one of REMODE’s strongest items regarding content validity, reliability, and contribution to the RTW-mode score, is closely related to perceived workability. This construct is also captured in other instruments like the widely used work ability index (WAI) and previously discussed WSS, both demonstrating strong reliability [28, 34]. Some researchers even argue that a single item about perceived workability may suffice for assessing work readiness, though this is generally based on its predictive value for RTW in employees [14, 35, 36]. Furthermore, perceived workability is self-reported, while REMODE adds professionals’ perspectives, which prior research has deemed essential [7, 34, 35]. Professionals also rated other REMODE items as (very) relevant, including item 4 (‘Assesses their labour market prospects positively’) and item 16 (‘Appears motivated for returning to work’) both strongly associated with the RTW-mode, supporting their continued inclusion.

Strengths and limitations

A major strength of our study is the extensive number of rater pairs, allowing for robust inter-rater reliability assessment. Furthermore, the raters assessed the vignettes in random order to diminish order effect and they were blinded for the ratings of other raters. We also employed extensive reliability analyses with a GLMM to adjust for the hierarchical structure. Furthermore, we assessed REMODE’s usability and usefulness, both important for evaluation and development of professional tools. Lastly, we incorporated unpublished qualitative data from our related studies to enhance interpretation of the findings.

However, this study has certain limitations. First, using case-vignettes rather than real consultations may have reduced representativeness. To enhance realism, we developed video consultations using Hillen et al.’s checklist [16], included diverse worker characteristics, and used real insurance physicians alongside trained patient actors. Nevertheless, a real-world evaluation of REMODE, including workers’ perspectives, is forthcoming.

Second, the sample may not fully represent the broader population of occupational health professionals, given the predominance of female participants with limited professional experience, and underrepresentation of psychologists and social nurses. Nonetheless, all relevant professional disciplines in Dutch occupational healthcare for precarious workers were included, with diverse experience and training. We also adjusted our GLMM for key professional characteristics and conducted subgroup and sensitivity analyses. Therefore, we consider our findings sufficiently informative for real-world occupational healthcare.

Recommendations for research and practice

The refined version of REMODE (Supplementary File S1) offers a promising instrument for assessing the RTW-mode in sick-listed precarious workers with mental health issues, while new research on evaluation of REMODE in real-world occupational health practice is recommended. Understanding the RTW-mode can help professionals determine if, when, and what type of support is needed. For example, in Dutch occupational healthcare, professionals may be better equipped to choose between the commonly used ‘fit-for-work’-trajectories (enhancing self-efficacy), ‘ready-for-work’-trajectories (supporting job search), or other barrier-targeted interventions [37]. More broadly, identifying RTW-perceptions and attitudes may support professionals decide whether a worker could benefit from work-focused cognitive behavioural therapy addressing unhelpful RTW-cognitions [12, 38], or if interventions targeting other RTW-barriers are indicated.

However, further research is required to evaluate REMODE’s impact on occupational rehabilitation. We are therefore continuing our qualitative study on workers’ perspectives to refine REMODE. Identifying barriers and facilitators for implementation is also important, as previous research highlighted several challenges for integrating new tools in occupational healthcare [37, 39], including professionals’ positive beliefs toward new procedures [40]. Given some professional hesitations about using REMODE in daily practice, we will develop and evaluate a training program to support implementation and strengthen non-normative interpretation of the RTW-mode. Evaluating REMODE’s impact on occupational counselling will be central to the forthcoming study. Finally, research should also examine the RTW-mode and REMODE’s applicability in other worker populations, including those with physical health issues.

Conclusions

This study demonstrates that REMODE is a promising tool for occupational health professionals in assessing the RTW-mode in sick-listed precarious workers with mental health issues. REMODE showed good reliability, content validity and usability, and can support professionals in providing tailored occupational rehabilitation to facilitate work resumption. Future research should examine REMODE’s impact on occupational rehabilitation outcomes, and its implementation in practice, incorporating workers’ perspectives.

Supplementary data

Supplementary data are available at EURPUB online.

Conflict of interest

The first author Y.S. is employed as researcher at Amsterdam UMC and as insurance physician at UWV. All authors declare no conflict of interest in the analyses, interpretation of data, in writing the article, and in the decision to submit the article for publication.

Funding

This study was supported financially by The Dutch Institute for Employee Benefit Schemes (Uitvoeringsinstituut Werknemersverzekeringen: UWV). The funding organization played no role in the analysis, interpretation, writing, or decision to submit an article for publication.

Data availability

The datasets generated and analysed during the current study are available from the corresponding author on reasonable request.

Ethics

The Medical Ethics Review Committee of the Academic Medical Center of Amsterdam UMC declared/determined that no ethics committee approval was required for this study (trial number W22_283 # 22.343). All procedures followed were in accordance with the ethical standards of the responsible committee on research involving human subjects (institutional and national) and with the 1964 Helsinki declaration and its later amendments. All participants of the included studies signed informed consent.

Key points.

• REMODE is a valuable tool for professional assessments of the RTW-mode of sick-listed precarious workers with mental health issues.

• The tool demonstrates satisfactory inter-rater reliability, inter-item consistency, content validity, and usability.

• A refined version of REMODE is proposed for use in future research and occupational health practice.