Diagnostic accuracy of the Single‐item Measure of Burnout (Japanese version) for identifying medical resident burnout

Abstract

Background

Burnout is a psychological syndrome consisting of emotional exhaustion, cynicism, and decreased professional efficacy. The Maslach Burnout Inventory (MBI) is widely used as the standard measure. However, the MBI is lengthy and not free to use, which makes it a less than ideal tool for regularly assessing burnout. The single question burnout measure (SMB) is a novel and simple measure of burnout, which is associated well with emotional exhaustion and has sufficient diagnostic performance for burnout. This study aimed to evaluate the concurrent and convergent validity of the Japanese version of the single‐item measure of burnout (SMB‐J) compared with the MBI.

Methods

Ninety‐four medical residents volunteered to complete the MBI‐General Survey (MBI‐GS) and the SMB‐J. We assessed the concurrent (sensitivity and specificity) and convergent validity of the SMB‐J compared with the MBI‐GS.

Results

The sensitivity for identifying burnout using the SMB‐J was 53.8%, and the specificity was 88.2%. The area under the receiver operating characteristic curve (AUC) was 0.71. MBI‐GS scores on the subscales of Emotional Exhaustion (r = 0.509, p < 0.0001) and Cynicism (r = 0.57, p < 0.0001) strongly correlated with the SMB‐J scores.

Conclusions

We concluded that for identifying burnout among Japanese medical residents, the psychometric properties of the SMB‐J are comparable to those of the original version of the SMB. Although the SMB‐J has low sensitivity to detect burnout, it is more convenient to use than the MBI.

1. INTRODUCTION

Burnout is a psychological syndrome caused by chronic work stress and consists of three components: emotional exhaustion, cynicism (depersonalization), and feelings of reduced professional efficacy (personal accomplishment). ¹ Burnout is prevalent among physicians and medical residents, which can impair their health and job satisfaction, and reduce the quality of medical care. ² , ³ In recent years, Japan has adopted work‐style reforms that pay more attention to the mental health of workers, including medical professionals. ⁴ To prevent burnout, hospital administrators should regularly assess employee burnout, intervene at an individual level, and improve the work environment. ⁵

Although several assessment instruments have been developed to measure burnout, the Maslach Burnout Inventory (MBI) is recognized as the reference standard. ³ The MBI measures all three components of burnout, using 16 items for the General Survey (MBI‐GS), or 22 items for the Human Services Survey for Medical Personnel (MBI‐HSS MP). However, its use is limited by the relatively large number of questions (it takes a long time to answer them) and the license fee for using it. In contrast, the single‐item measure of burnout (SMB) has recently gained attention as another means of measuring burnout. The SMB consists of only one question rated on a 5‐point scale to determine whether the respondent has self‐defined burnout. ⁶ , ⁷ The question is included in the Mini‐Z 2.0, which is used to assess physician well‐being. ⁸ The advantages of the SMB over the MBI are that it is a one‐item questionnaire with a short response time, and it is free.

There have been several reports on the psychometric properties of the SMB. In a study of 303 physicians, Rohland et al. observed that the SMB score was highly correlated with the Emotional Exhaustion (EE) subscale score of the MBI (r = 0.64, p < 0.001). ⁹ Similar results were obtained in several studies. ¹⁰ , ¹¹ The diagnostic accuracy (i.e., sensitivity and specificity) of the SMB to identify burnout have also been evaluated in previous literature. In a study including 1010 participants, the SMB was able to identify burnout with 50.4% sensitivity, and the absence of burnout with 94.7% specificity in primary care physicians, and with 58.6% sensitivity and 94.7% specificity in medical staff, compared to the MBI diagnosis. ¹² In a study conducted among COVID‐19 frontline healthcare workers in Singapore, the diagnostic performance of the SMB was 80.6% sensitivity and 78.7% specificity to identify burnout. ¹³ Dolan et al. found that the SMB identified a positive result for one of the major items of EE in the MBI, with 83.2% sensitivity and 87.4% specificity. ¹⁴

The SMB's diagnostic performance suggests that the score correlates well with the MBI EE subscale score, and may have high specificity but low sensitivity for diagnosing burnout, as measured by the MBI. Since the SMB reflects the degree of emotional exhaustion and is free and easy to administer, it has become widely used as an alternative to the MBI. ⁸ , ¹⁵ , ¹⁶

In Japan, there are few studies on burnout, and only one nationwide survey on medical resident burnout has been published. ¹⁷ However, if the SMB were available and shown to accurately assess burnout, it would facilitate research on burnout and its assessment in medical institutions. To encourage the use of the SMB in Japan, a Japanese version was needed. The Physicians' Well‐being Committee of the Japanese chapter of the American College of Physicians translated the Mini‐Z 2.0 in 2020. ¹⁸ Since the SMB was included in the Mini‐Z 2.0, the Japanese version of the SMB (SMB‐J) was created at the same time. Burnout identified using the SMB item correlated with the other Mini‐Z 2.0 items that measure satisfaction, value alignment, work control, and stress. However, the MBI was not administered, and the psychometric properties of the SMB for measuring burnout were not evaluated in the study.

Therefore, our research objectives were to:

1. Compare the Japanese version of the SMB with the MBI—the reference standard for identifying burnout—among medical residents in Japan.

2. Assess the psychometric properties of the SMB‐J, including diagnostic accuracy in terms of sensitivity and specificity (i.e., concurrent validity), and convergent validity in relation to the MBI subscales.

2. METHOD

2.1. Procedure

This multicenter cross‐sectional study was part of a prospective national survey on medical resident stress that began in April 2021. Researchers approached all hospitals designated for physician residency training in Japan and scheduled to recruit medical residents in April 2021, to support the national survey. Of the 947 training hospitals, 272 confirmed that they would support the research. Medical residents at the 272 hospitals were invited to participate in the survey.

A Web‐based questionnaire combining the SMB‐J and MBI‐GS was made available to prospective participants during July 2021. No financial incentives for participation were provided to the training hospitals or medical residents. An appropriate research ethics committee approved the study. The survey was conducted anonymously, and consent was obtained if the questionnaire was completed.

2.2. Participants

The target population included medical residents in Japan. After completing six years of medical school education, medical residents in Japan enter a two‐year postgraduate training program that addresses basic knowledge and general medical skills. ¹⁹ Medical residents are required to rotate through seven specialties (internal medicine, emergency medicine, community medicine, surgery, pediatrics, psychiatry, and obstetrics and gynecology) under the guidance of a supervising physician. The Ministry of Health, Labour, and Welfare regulates the training programs.

Following recruitment from 272 training hospitals, 554 medical residents agreed in April 2021 to participate in the study. The questionnaire was made available to the 554 volunteers. Participants with missing SMB or MBI responses were excluded from the analysis.

2.3. Measures

2.3.1. The Japanese version of the Single‐item Measure of Burnout

The SMB‐J is a nonproprietary, single‐question instrument with five response options, used to diagnose burnout as defined by the respondents themselves. The SMB was translated into Japanese along with the Mini‐Z 2.0. ¹⁸ Standard forward and backward translation methods were used. No changes were made to the question for cultural adaptation. In addition to the cognitive debriefing by ten physicians and medical residents, the back‐translated version was reviewed and officially approved by the original author. The back‐translated Japanese version starts with “Please circle the option that best describes your situation based on your definition of “burnout,” and asks the respondent to select one of the following response options:

• 1 = “You feel totally burned out. You are at a point where you may need some help”

• 2 = “You always have symptom(s) of burnout. You are often worried about stress from work”

• 3 = “You are beginning to burnout and have at least one symptom of burnout (e.g., emotional exhaustion)”

• 4 = “You feel under stress. You are not always full of energy, but have never felt burned out”

• 5 = “You enjoy working. You have never felt burned out”

In the original version, respondents who choose option 3 (beginning of burnout) or lower are identified as having a burnout, which is the same as in the Japanese version. ⁸

2.3.2. The Maslach Burnout Inventory‐General Survey

The Japanese version of the Maslach Burnout Inventory‐General Survey (MBI‐GS) was used as the reference standard for diagnosing participants with burnout. ²⁰ , ²¹ This 16‐item questionnaire consists of three subscales: Emotional Exhaustion (EE), Cynicism (CY), and Professional Efficacy (PE). Respondents rated how often they experienced each from a list of symptoms on a 7‐point scale ranging from 0 (never) to 6 (every day). Subscale scores were calculated as the average score across all items of each subscale. In this study, we used the cutoff points defined by Kalimo et al ²² : high EE as >3.5, high CY as >3.5, and low PE as <2.5. Respondents who scored high on EE and either high on CY or low on PE were categorized as having burnout, resulting in a binary variable for burnout. ²³ We obtained a license to use the MBI‐GS (www.mindgarden.com).

We also asked about the characteristics of the participants, such as age, sex, and type of hospital (community hospital, university hospital, or university branch hospital).

2.4. Sample size calculation

The SMB is known to have low sensitivity and high specificity for diagnosing burnout, and we expected the SMB‐J to have similar results. The sample size of this study was set with the main purpose of detecting high specificity. We set the prevalence of burnout at 30%, α at 0.05, and β at 0.20. ²⁴ For specificity, 100 participants were needed to confirm that 80% specificity was statistically different from 50% specificity. ²⁵ In addition, for convergent validity, 47 participants were needed to confirm that a correlation of 0.4 or higher was statistically different from a correlation of zero. Finally, the sample size of this study was set at 100.

2.5. Data analysis

Using the diagnosis of burnout by the MBI as the reference standard, we calculated the sensitivity and specificity, the positive and negative predictive values, and the receiver operating characteristic curve (AUC) of the diagnosis of burnout by the SMB‐J (cutoff ≤3). We also evaluated the sensitivity and specificity for SMB‐J cutoffs ≤2 or ≤4 and computed the AUC. The discriminatory performance indicated by AUC is as follows: 0.9–1.0 is excellent, 0.8–0.9 is very good, 0.7–0.8 is good, 0.6–0.7 is sufficient, 0.5–0.6 is bad, and <0.5 indicates that the instrument is not useful. ²⁶ The SMB‐J scores and the scores of the three subscales of the MBI‐GS (EE, CY, and PA) were evaluated using Pearson's correlation to test the convergent validity of the SMB‐J. The rules specified by Cohen were used to determine the size of the correlations: r < 0.30 as minor, r = 0.30–0.49 as moderate, and r ≥ 0.50 as strong. ²⁷ All analyses were conducted using STATA software, version 15 (STATA Corporation).

3. RESULTS

3.1. Descriptive findings

Nine‐four residents completed the online questionnaire (17.0% response rate). Eleven participants submitted incomplete demographic data. The demographic characteristics of the participants are presented in Table 1. Of the participants with complete data, 8.5% were over 30 years old, 29.7% were women, and 54.3% were affiliated with university hospitals.

TABLE 1.

Participant demographics and burnout levels

	n	%
All respondents	94	100
Demographics a
Age (years)
<25	25	26.6
≥25 and <30	50	53.2
≥30	8	8.5
Missing	11	11.7
Sex
Male	53	56.4
Female	28	29.7
Not answered	2	2.1
Missing	11	11.7
Hospital types
University	51	54.3
University branch	0	0.0
Community	32	34.0
Missing	11	11.7
Burnout level
Maslach Burnout Inventory b
High emotional exhaustion (EE)	37	39.8
High cynicism (CY)	18	19.4
Low professional efficacy (PE)	44	47.3
Burnout (by exhaustion +1 criterion)	26	28.0
Single‐item measure of burnout
5‐never felt burned out	11	11.8
4‐under stress	61	65.6
3‐beginning to burnout	19	20.4
2‐always have symptoms of burnout	2	2.2
1‐totally burned out	1	1.1

Abbreviations: CY, cynicism; EE, emotional exhaustion; PE, professional efficacy.

Eleven participants submitted incomplete data.

Using the exhaustion +1 criterion, high EE +high CY, or low PE is diagnosed as burnout by the Maslach Burnout Inventory‐General Survey.

3.2. Diagnostic accuracy

The level of burnout among the sample of 94 medical residents was 28.0% using the MBI‐GS and 23.6% using the SMB‐J (cutoff ≤3; Table 1). The MBI‐GS diagnosis of burnout was used as the reference standard. SMB‐J diagnosis of burnout had a sensitivity of 53.8%, specificity of 88.2%, and AUC of 0.71, using the same cutoff as the original version (Table 2). The positive and negative predictive values were 64.6% and 83.3%, respectively. When the cutoff was changed to ≤4, it resulted in 100% sensitivity, however, specificity dropped to 13.2%. When the cutoff was reduced further to ≤2, it resulted in 100% specificity; however, sensitivity dropped to 11.5%.

TABLE 2.

The diagnostic accuracy of the Japanese version of the Single‐item Measure of Burnout

	Sensitivity	Specificity	AUC
Cutoff ≤4	92.3%	13.2%	0.5277
Cutoff ≤3 (standard)	53.8%	88.2%	0.7104
Cutoff ≤2	11.5%	100.0%	0.5577

The diagnosis of burnout using the Maslach Burnout Inventory was the reference standard. The original version of the Single‐item Measure of Burnout had a cutoff of <3, although we analyzed it at different cutoff points.

Abbreviation: AUC, area under the receiver operating characteristic curve

3.3. Convergent validity

Strong correlations were observed between the SMB‐J and EE (r = 0.509, p < 0.0001) and CY (r = 0.57, p < 0.0001) subscales of the MBI‐GS (Table 3). However, no or minimal correlation was found between the SMB‐J and the PE subscale of the MBI‐GS (r = 0.101, p = 0.331).

TABLE 3.

Correlation between the Japanese version of the Single‐item Measure of Burnout and each subscale of the Maslach Burnout Inventory

	The Japanese version of the Single‐item Measure of Burnout					r
	Never felt burned out (n = 11)	Under stress (n = 61)	Beginning to burnout (n = 19)	Always have symptoms of burnout (n = 2)	Totally burned out (n = 1)
The Maslach Burnout Inventory	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)
Emotional exhaustion	5.6 (7.6)	7.6 (6.4)	16.3 (5.3)	26.5 (3.0)	30.0 (0.0)	0.509*
Cynicism	11.6 (4.8)	12.9 (6.2)	21.4 (8.1)	26.0 (3.5)	30.0 (0.0)	0.570**
Professional efficacy	15.0 (8.2)	15.5 (6.7)	15.7 (8.8)	7.0 (1.0)	6.0 (0.0)	0.101***

Abbreviations: M, mean; r, Pearson's correlation coefficient; SD, standard deviation.

*p < 0.0001.

**p < 0.0001.

***p = 0.331.

4. DISCUSSION

We examined the psychometric properties of the SMB‐J in relation to the MBI‐GS when identifying burnout among medical residents in Japan. The strong correlation between the SMB‐J and EE and CY subscales as measured by the MBI‐GS was comparable to the results obtained in previous studies on the original version. ⁸ , ⁹ , ¹² The SMB‐J has psychometric properties comparable to the original version in diagnosing burnout.

Rohland et al. first assessed the validity of the original version of the SMB by analyzing correlations. ⁹ Subsequent studies have continued to show a strong correlation between the SMB and MBI scores for EE and possibly CY (depersonalization). ⁸ , ⁹ , ¹² However, no correlation has been found between SMB and low PE (personal accomplishment). We found a strong correlation between the SMB‐J score and the MBI‐GS scores for EE and CY, confirming previous results, and substantiating the convergent validity of the Japanese version.

Regarding diagnostic characteristics, such as sensitivity and specificity, past studies have found a wide range of sensitivity (approximately 50%–80%) and high specificity (approximately 80%–90%). ¹² , ¹³ Our findings are similar to those of Knox et al., suggesting that the Japanese version has the same level of diagnostic performance as the original SMB. ¹² However, the Knox et al. study further pointed out that the SMB may estimate a lower prevalence of burnout than the MBI. ¹² The prevalence of burnout identified by the SMB‐J was approximately 5% lower in our study than what was identified by the MBI‐GS. This could be because the SMB and SMB‐J cannot detect PE (personal accomplishment) as accurately as the MBI does. Among the participants in our study, 47.3% (n = 93) showed a decrease in MBI‐GS PE scores. A large burnout study of medical residents conducted in Japan also found that 52.0% of participants experienced low PE on the MBI‐GS. ²⁴ Therefore, when the SMB‐J is used with Japanese medical residents, it may lead to a lower burnout rate estimate than the MBI‐GS, as the SMB‐J may not detect a decline in PE. One possible reason why the SMB and the SMB‐J did not successfully detect a decrease in PE is that the core symptoms of burnout are EE and CY (depersonalization). ²⁸ Decreased PE is thought to be a late symptom of burnout and depends on the personality traits of the individual. In several studies, employees who received psychological interventions for burnout had higher EE and CY scores, but a less noticeable decrease in PE scores. ²⁹ , ³⁰ In general, people seem to recognize burnout by the worsening EE and CY, which may be the reason why the SMB‐J could not detect low PE in this study.

The advantage of the SMB and SMB‐J over the MBI is that they are free of charge and consist of only one question with a simple response scale. This means that they are recommended for use in surveys and research where budgets are limited or when long time scales are not preferable. It is not appropriate to use the SMB‐J as a screening test for burnout, because it does not measure all three dimensions of burnout. Consequently, it is likely that burnout will be underestimated. Despite these limitations, the SMB has logistic strengths, and it has been used widely, for example, in the large MEMO study. ³¹ As a result, it was included in the Mini‐Z survey. ⁸

Ideally, researchers should develop a simple burnout scale with better diagnostic performance than the SMB. The first attempt to accomplish this was the development of a version of the MBI with a reduced number of questions (1, 2, or 12). ³² , ³³ , ³⁴ These shortened versions have high diagnostic performance but also require a license fee.

There are other simple scales unrelated to the MBI, such as the Rapid Burnout Screening Tool (RBST), which is a free instrument developed by Ong et al. ¹³ The RBST consists of four questions, one for each of the three dimensions of burnout and one for self‐assessment of burnout. The RBST has a higher diagnostic performance than the SMB. ¹³ However, since the instrument is new and has only been validated among medical staff on the COVID‐19 frontline in Singapore, evaluation of its validity and diagnostic properties in a wider range of target populations and regions is needed.

An untested option that could improve the SMB by enhancing its diagnostic properties is adding a question that detects a decrease in PE. In its current form, the SMB identifies burnout from a score of three or less, but many respondents in our study answered two (stressed). If respondents who answered two were also diagnosed with burnout when their professional efficacy declined, then the sensitivity of the SMB to diagnose burnout would increase. Although the SMB is currently considered one of the best simple measures of burnout, the development of a simple instrument with higher performance should be encouraged.

5. LIMITATIONS

Our study has several limitations. We included only medical residents, which limits the generalizability of the findings about the SMB‐J. Since the instrument is expected to be used for the assessment of other healthcare workers and nonhealthcare workers, further studies using a wider range of target groups are necessary. However, in a previous study including physicians and other medical staff, the psychometric properties of the SMB were similar, and there may not be a significant difference between occupations. ¹²

Only 94 medical residents were included in our study, which is not a representative sample. Although data about the age and sex of all medical residents are not directly published in Japan, the Japan Residency Matching Program announced that 38.9% and 38.1% of medical residents were in residency at university hospitals in 2020 and 2021, respectively. ³⁵ Thus, medical residents who belong to university hospitals were over‐represented in our study.

In terms of sex, women accounted for about one‐third of the total number of doctors who passed the national examination in Japan during the past five years. The same ratio of women participated in our study.

The level of burnout ranged from 18% to 33% in other studies among Japanese medical residents, ²⁴ , ³⁶ , ³⁷ which is comparable to the levels found in our study. Nonrespondent bias because of the low response rates (17%) could have influenced our findings. Medical residents who did not respond may not have been concerned about stress or, conversely, may have had severe stress problems. Only three medical residents selected a rating of four or five in the SMB‐J.

There are also methodological limitations. The number of the participants was 94, which was slightly less than the calculated sample size. In addition, we have not conducted a reliability assessment using the test–retest method.

6. CONCLUSION

The SMB‐J has psychometric properties similar to that of the original version of the SMB when assessing burnout among Japanese medical residents. Although it has some limitations, we hope that this simple and free instrument will facilitate the measurement of burnout and lead to new studies about burnout in Japan.

CONFLICT OF INTEREST

The authors declare that no conflicts of interest exist.

Nagasaki K, Seo E, Maeno T, Kobayashi H. Diagnostic accuracy of the Single‐item Measure of Burnout (Japanese version) for identifying medical resident burnout. J Gen Fam Med. 2022;23:241–247. 10.1002/jgf2.535