Abstract
The augmentation of clozapine with electroconvulsive therapy (ECT) has been an optimal treatment option for patients with treatment- or clozapine-resistant schizophrenia. Using data from the Research on Asian Psychotropic Prescription Patterns for Antipsychotics survey, which was the largest international psychiatry research collaboration in Asia, our study aimed to develop a machine learning algorithm-based substantial prediction model for the augmented use of clozapine with ECT in patients with schizophrenia in terms of precision medicine. A random forest model and least absolute shrinkage and selection operator (LASSO) model were used to develop a substantial prediction model for the augmented use of clozapine with ECT. Among the 3744 Asian patients with schizophrenia, those treated with a combination of clozapine and ECT were characterized by significantly greater proportions of females and inpatients, a longer duration of illness, and a greater prevalence of negative symptoms and social or occupational dysfunction than those not treated. In the random forest model, the area under the curve (AUC), which was the most preferred indicator of the prediction model, was 0.774. The overall accuracy was 0.817 (95% confidence interval, 0.793–0.839). Inpatient status was the most important variable in the substantial prediction model, followed by BMI, age, social or occupational dysfunction, persistent symptoms, illness duration > 20 years, and others. Furthermore, the AUC and overall accuracy of the LASSO model were 0.831 and 0.644 (95% CI, 0.615–0.672), respectively. Despite the subtle differences in both AUC and overall accuracy of the random forest model and LASSO model, the important variables were commonly shared by the two models. Using the machine learning algorithm, our findings allow the development of a substantial prediction model for the augmented use of clozapine with ECT in Asian patients with schizophrenia. This substantial prediction model can support further studies to develop a substantial prediction model for the augmented use of clozapine with ECT in patients with schizophrenia in a strict epidemiological context.
Keywords: schizophrenia, clozapine, electroconvulsive therapy (ECT), augmentation, machine learning, precision medicine
1. Introduction
Approximately one-third of patients with schizophrenia respond poorly to standard antipsychotic medication treatments [1,2]. Although clozapine has been commonly considered an effective antipsychotic medication option for patients with treatment-resistant schizophrenia (TRS) [3,4], approximately half to three-quarters of these patients are resistant to clozapine treatment [3,5]. In addition, electroconvulsive therapy (ECT) has been an effective treatment option for catatonia, suicidality, aggression, and neuroleptic malignant syndrome [6]. In terms of an augmentation of clozapine with ECT, clozapine may reduce the seizure threshold [7]. The combination of clozapine and ECT is considered one of the effective and safe treatment options for TRS (especially clozapine-resistant schizophrenia [CRS]) [8] according to the following reasons: despite its poor understanding, increased monoamine release; the stimulation of pituitary secretion of thyroid-stimulating hormone, adrenocorticotropic hormone, endorphins, and prolactin; and increased neurogenesis have been proposed as mechanisms for the augmentation of clozapine with ECT [9]. In addition, a systematic review and meta-analysis of open and randomized controlled trials, retrospective chart reviews, case series, and case reports demonstrated that approximately two-thirds (n = 83, 66%) of 126 patients with TRS met the response criterion to a combination of clozapine and ECT. In contrast, 14% of patients have shown adverse events (i.e., cognitive dysfunction, postictal confusion, delirium, seizure) [10]. In addition, a randomized single-blind 8-week study of patients with CRS reported that approximately half met the response criterion in the ECT augmentation group, whereas none met the criterion in the clozapine group. No differences between the two groups were reported in terms of side effects and cognitive effects [8]. Furthermore, the observational studies presented that an augmentation of clozapine with ECT can rapidly and substantially improve the psychotic symptoms of patients with TRS [11,12].
Given the potential biomarkers, clinical characteristics, and cognitive features of schizophrenia in terms of neural circuit theory, the four-stage model was proposed by Insel [13]. The characteristics of the four stages are as follows: Stage I is characterized by genetic vulnerability and environmental exposure, which can be diagnosed with genetic sequence and family history, and few cognitive deficits. Stage II is characterized by cognitive and behavioral deterioration, and help seeking; it can be diagnosed with cognitive assessment, neuroimaging, and the Structured Interview for Prodromal Syndrome and can be supported by polysaturated fatty acid consumption and social support. Stage III is characterized by features of abnormal thought and behavior and a relapsing–remitting course, which can be diagnosed through clinical interviews and treated with medications and psychosocial interventions. Stage IV is characterized by continued deterioration, unemployment, homelessness, medical complications, and incarceration, which can be treated with medication, psychosocial interventions, and rehabilitation services. However, in terms of precision medicine in psychosis, to the best of our knowledge, few clinically valid and reliable biomarkers of its onset, course, and outcome have been identified [14]. Although it has been expected that the use of clozapine augmentation with ECT can be optimized in terms of the four-stage model of schizophrenia proposed by Insel [13], its use is still based only on the clinical characteristics and cognitive features of individual patients. In addition, it is necessary to develop a machine learning-based substantial prediction model for the augmentation of clozapine with ECT based on the sociodemographic, clinical, and symptomatic characteristics of patients with schizophrenia.
It is hypothesized that the use of clozapine augmentation with ECT is rare in patients with schizophrenia. In addition, machine learning can provide meaningful insights and applications by offering promising solutions to harness big data [15]. However, to the best of our knowledge, substantial prediction models based on machine learning for the use of a combination of clozapine and ECT have rarely been developed. In terms of precision medicine, using data from the Research on Asian Psychotropic Prescription Pattern for Antipsychotics, phase 4 (REAP-AP4) [16,17,18], which has been the largest international collaborative study in the realm of psychiatry in Asia, we aimed to develop a machine learning-based substantial prediction model for the augmented use of clozapine with ECT in Asian patients with schizophrenia. However, data from the REAP-AP survey have been obtained temporally but not longitudinally. Therefore, we aim to achieve a substantial prediction model.
2. Materials and Methods
2.1. Study Overview and Participants
As described in previous studies [16,17,18], the REAP-AP4 survey aimed to examine the patterns of psychotropic medication use and their clinical correlates and explore methods to improve the patterns of psychotropic medication use in Asian patients with schizophrenia. A total of 3744 patients with schizophrenia were consecutively recruited from March to June 2016 from 71 survey centers in 15 Asian countries and special administrative regions (SARs), including Bangladesh, China, Hong Kong, India, Indonesia, Japan, Korea, Malaysia, Myanmar, Pakistan, Singapore, Sri Lanka, Taiwan, Thailand, and Vietnam. Study participants were enrolled using convenience sampling. The Institutional Review Board of Taipei City Hospital, Taipei, Taiwan (No. TCHIRB-10412128-E), and other survey hospitals approved the study protocol and informed consent forms. All study participants completed the written informed consent forms before enrolment. A conference meeting to increase the consistency of data collection and confirm the diagnosis of schizophrenia among the survey centers was held prior to the initiation of the study. As per the protocol, detailed sociodemographic, clinical, symptomatic, and psychiatric treatment-related characteristics were collected by the study coordinators trained and supervised by clinical psychiatrists, at the survey centers. Data collected from the study participants were stored on the REAP survey website.
To examine psychotropic medication use patterns in Asian patients with schizophrenia in the real world, the inclusion and exclusion criteria were defined. The inclusion criteria were as follows: (i) diagnosis of schizophrenia, based on the Diagnostic and Statistical Manual of Mental Disorders, fifth edition (DSM-5) [19], confirmed by clinical psychiatrists at the survey centers; (ii) treatment with antipsychotic medications (N05A), defined by the Anatomical Therapeutic Chemical classification index [20]; and (iii) availability of information on current clozapine medication and ECT application. The exclusion criteria were as follows: (i) comorbidity of severe physical disease and (ii) inability to read or write.
2.2. Variable Profiles for the Substantial Prediction Model
In the study, we used sociodemographic data (i.e., age, sex, body mass index [BMI]), clinical data (i.e., outpatient or inpatient status, duration of illness [<3 months, 3–6 months, 6–12 months, 1–5 years, 5–10 years, 10–20 years, or >20 years], clinical course for the past 1 year [remission or persistent symptoms]), and current psychopathological symptoms (i.e., delusion, hallucination, disorganized speech, grossly disorganized or catatonic behavior, negative symptoms, social or occupational dysfunction, verbal aggression, physical aggression, significant affective symptoms). A predictive utility of BMI for metabolic syndrome was approved among Japanese patients with schizophrenia [21]. BMI, which might be inversely associated with homocysteine level, was proposed to indicate clinical symptoms and glucose and lipid levels among Chinese patients with schizophrenia [22]. BMI was positively associated with positive symptoms among antipsychotic-naïve schizophrenia patients [23]. In addition, a differential association between BMI and fronto-limbic white matter microstructure was found among patients with first-episode schizophrenia spectrum disorders [24]. Furthermore, BMI and age had the moderating effects of an association between a history of suicidal attempts and COVID-19 infection among patients with schizophrenia or schizoaffective disorder [25]. Therefore, BMI and other sociodemographic and clinical data were included as one of the variable profiles for the substantial prediction model. The current symptoms were defined based on the DSM-5 [19] and evaluated using a dichotomous value manner (presence or absence).
2.3. Data Processing and Machine Learning
All patients with missing data were excluded from the analysis. Furthermore, all data were divided into training (0.7) and test (0.3) sets in a 7:3 ratio of 3744 patients. A 10-fold cross-validation was used to train the entire training dataset within the training set. Because the augmentation of clozapine with ECT is rarely used, the synthetic minority oversampling technique was used [26]. In addition, the area under the curve (AUC) of the receiver operating characteristic curve was used to determine the main performance. Moreover, the overall accuracy, sensitivity, specificity, negative predictive value, and positive predictive value were calculated. The classification and regression training package of R was used to perform hyperparameter tuning, confusion matrix composition, and AUC assessment [27,28]. The random forest model and least absolute shrinkage and selection operator (LASSO) model were used to compare substantial prediction models for a combination of clozapine and ECT among Asian patients with schizophrenia.
First, a random forest model was used as the algorithm in this analysis. Although the decision tree model can indicate variable importance and classification mechanisms in an intuitive manner, and its computing cost is low, it is vulnerable to overfitting. Thus, the decision tree model is rarely used as an algorithm. Random forest is the ensemble model used to overcome the limitations of the decision tree model [29]. Its predictive power is maximized under the condition that new data (i.e., test set) are provided rather than used for training by minimizing the overfitting of the decision tree. According to the principle of “mean decreases in accuracy,” the variable importance was calculated to identify the variables that significantly contributed to the optimization of the predictive model [30]. The importance of variable j was estimated as follows: Several cases were not sampled and were called out-of-bag because bagging permitted cases to be sampled more than once for the same classifier. After handing down the out-of-bag samples to the tree, the prediction accuracy decreased with training using one of the trees. Subsequently, the change in variable j was conducted in the out-of-bag samples of trees, and the accuracy was calculated again. Before and after the permutation over all trees, the raw score for the importance of variable j was calculated by computing the average gap in out-of-bag errors. The return of the score to normal was calculated using the standardized deviation of the difference. Finally, the score was reduced so that the minimum and maximum values were set to 0 and 100, respectively. Using care, the attainment of a confusion matrix, hyperparameter tuning, and performance measurements were conducted. The prediction performance of the out-of-bag portion of the data was recorded for each tree. The measurement and recording of the prediction performance were conducted each time, while alternative permutations of the predictors were conducted. Based on all the trees, the average value of the difference between these two prediction performances was obtained and normalized. The final prediction performance was measured using test data, which were not used in the training phase.
Second, the LASSO model was also used in this analysis. The LASSO model used a regularization term λE(ω) = λΣ |ωk| [31]. The LASSO model can be practically used as a feature reduction method, as the coefficients of weak predictive variables decrease to zero. In this analysis, the hyperparameter, which inversely reflected the strength of the regularization parameter λ, was set to 0.0076. In addition, the penalty option was set to “l1,” and other hyperparameters were set to default in the logistic regression scikit-learn library.
3. Results
3.1. General Characteristics of the Study Participants
Among the 3744 Asian patients with schizophrenia, 1.3% (n = 47) were treated with a combination of clozapine and ECT. As shown in Table 1, the patients who were treated with an augmentation of clozapine with ECT were characterized by significantly greater proportions of females (χ2 = 5.142, p = 0.023) and inpatients (χ2 = 39.942, p < 0.0001), a longer duration of illness (χ2 = 19.253, p = 0.004), and a greater prevalence of negative symptoms (χ2 = 8.571, p = 0.003) and social or occupational dysfunction (χ2 = 3.960, p = 0.047) than those treated without clozapine augmentation with ECT.
Table 1.
General description of the study participants (n = 3744).
| Total (n = 3744) | Augmentation of Clozapine with ECT | Statistical Coefficient | p-Value | ||
|---|---|---|---|---|---|
| Yes (n = 47) | No (n = 3697) | ||||
| Country/SAR | χ2 = 19.616 | <0.0001 | |||
| Bangladesh, n (%) | 99 (2.6) | 0 (0.0) | 99 (2.7) | ||
| China, n (%) | 160 (4.3) | 20 (42.6) | 140 (87.5) | ||
| Hong Kong, n (%) | 31 (0.8) | 0 (0.0) | 31 (0.8) | ||
| India, n (%) | 479 (12.8) | 2 (4.3) | 477 (12.9) | ||
| Indonesia, n (%) | 581 (15.5) | 10 (21.3) | 571 (15.4) | ||
| Japan, n (%) | 229 (6.1) | 2 (4.3) | 227 (6.1) | ||
| Korea, n (%) | 131 (3.5) | 0 (0.0) | 131 (3.5) | ||
| Malaysia, n (%) | 305 (8.1) | 4 (8.5) | 301 (8.1) | ||
| Myanmar, n (%) | 164 (4.4) | 0 (0.0) | 164 (4.4) | ||
| Pakistan, n (%) | 298 (8.0) | 0 (0.0) | 298 (8.0) | ||
| Singapore, n (%) | 171 (4.6) | 2 (4.3) | 169 (4.6) | ||
| Sri Lanka, n (%) | 97 (2.6) | 1 (2.1) | 96 (2.6) | ||
| Thailand, n (%) | 322 (8.6) | 2 (4.3) | 320 (8.7) | ||
| Taiwan, n (%) | 403 (10.8) | 2 (4.3) | 401 (10.8) | ||
| Vietnam, n (%) | 274 (7.3) | 2 (4.3) | 272 (7.4) | ||
| Age, mean (SD) years | 39.5 (13.2) | 39.3 (13.6) | 39.5 (13.1) | t = −0.109 | 0.913 |
| Sex | χ2 = 5.142 | 0.023 | |||
| Male, n (%) | 2199 (58.7) | 20 (42.6) | 2179 (58.9) | ||
| Female, n (%) | 1545 (41.3) | 27 (57.4) | 1518 (41.1) | ||
| BMI, mean (SD) kg/m2 | 23.9 (4.7) | 23.2 (3.4) | 23.9 (4.7) | t = −1.376 | 0.319 |
| Hospitalization | χ2 = 39.942 | <0.0001 | |||
| Outpatient, n (%) | 1793 (47.9) | 1 (2.1) | 1792 (48.5) | ||
| Inpatient, n (%) | 1951 (52.1) | 46 (97.9) | 1905 (51.5) | ||
| Duration of illness | χ2 = 19.253 | 0.004 | |||
| <3 months, n (%) | 161 (4.3) | 6 (12.8) | 155 (4.2) | ||
| 3–6 months, n (%) | 125 (3.3) | 0 (0.0) | 125 (3.3) | ||
| 6–12 months, n (%) | 199 (5.3) | 1 (2.1) | 198 (5.4) | ||
| 1–5 years, n (%) | 794 (21.2) | 5 (0.6) | 789 (21.3) | ||
| 5–10 years, n (%) | 729 (19.5) | 8 (17.0) | 721 (19.5) | ||
| 10–20 years, n (%) | 971 (25.9) | 10 (21.3) | 961 (26.0) | ||
| >20 years, n (%) | 765 (20.4) | 17 (36.2) | 748 (20.2) | ||
| Clinical course for the past 1 year | |||||
| Remission, n (%) | 1262 (33.7) | 18 (38.3) | 1244 (33.6) | χ2 = 0.449 | 0.503 |
| Persistent symptoms, n (%) | 1917 (51.2) | 20 (42.6) | 1897 (51.3) | χ2 = 1.423 | 0.233 |
| Current symptoms | |||||
| Delusion, n (%) | 1599 (42.7) | 19 (40.4) | 1580 (42.7) | χ2 = 0.101 | 0.750 |
| Hallucination, n (%) | 1752 (46.8) | 19 (40.4) | 1733 (46.9) | χ2 = 0.776 | 0.378 |
| Disorganized speech, n (%) | 1110 (29.6) | 12 (25.5) | 1098 (29.7) | χ2 = 0.387 | 0.534 |
| Grossly disorganized or catatonic behavior, n (%) | 666 (17.8) | 7 (14.9) | 659 (17.8) | χ2 = 0.273 | 0.601 |
| Negative symptom, n (%) | 1313 (35.1) | 26 (55.3) | 1287 (34.8) | χ2 = 8.571 | 0.003 |
| Social or occupational dysfunction, n (%) | 1693 (45.2) | 28 (59.6) | 1665 (45.0) | χ2 = 3.960 | 0.047 |
| Verbal aggression, n (%) | 942 (25.2) | 9 (19.1) | 933 (25.2) | χ2 = 0.913 | 0.339 |
| Physical aggression, n (%) | 780 (20.8) | 11 (23.4) | 769 (20.8) | χ2 = 0.191 | 0.662 |
| Significant affective symptoms, n (%) | 425 (11.4) | 3 (6.4) | 422 (11.4) | χ2 = 1.168 | 0.280 |
BMI, body mass index; ECT, electroconvulsive therapy; SAR, special administrative region; SD, standard deviation.
3.2. Substantial Prediction Model Performance and Variable Importance: Random Forest Model
The substantial prediction model was developed for 47 patients treated with a combination of clozapine and ECT among the 3744 patients in this analysis. As shown in Figure 1, the AUC, which was the most preferred indicator of the prediction model, was 0.774 in the random forest model to predict the use of a combination of clozapine and ECT. The overall accuracy, sensitivity, specificity, positive predictive value, and negative predictive value were 0.817 (95% confidence interval [CI], 0.793–0.839), 0.049, 0.991, 0.821, and 0.556, respectively. As shown in Figure 2, inpatient status was the most important variable in the substantial prediction model, followed by BMI, age, social or occupational dysfunction, persistent symptoms, illness duration > 20 years, and others.
Figure 1.
Area under the curve of the receiver operating characteristic curve for the augmented use of clozapine with electroconvulsive therapy in 3744 patients with schizophrenia (random forest model).
Figure 2.
The importance of variables in the prediction model for the augmented use of clozapine with electroconvulsive therapy in 3744 patients with schizophrenia (random forest model). BMI, body mass index.
3.3. Substantial Prediction Model Performance and Variable Importance: LASSO Model
As shown in Figure 3, the AUC was 0.831 in the LASSO model to predict the use of a combination of clozapine and ECT. The overall accuracy, sensitivity, specificity, positive predictive value, and negative predictive value were 0.644 (95% CI, 0.615–0.672), 0.033, 0.993, 0.722, and 0.643, respectively. As shown in Figure 4, inpatient status was the most important variable in the substantial prediction model, followed by significant affective symptoms, illness duration > 20 years, disorganized speech, persistent symptoms, social or occupational dysfunction, and others.
Figure 3.
Area under the curve of the receiver operating characteristic curve for the augmented use of clozapine with electroconvulsive therapy in 3744 patients with schizophrenia (LASSO model). LASSO, least absolute shrinkage and selection operator.
Figure 4.
The importance of variables in the prediction model for the augmented use of clozapine with electroconvulsive therapy in 3744 patients with schizophrenia (LASSO model). BMI, body mass index.
4. Discussion
A substantial prediction model has been developed for the augmented use of clozapine with ECT with 47 of the 3744 Asian patients with schizophrenia using machine learning algorithms of both the random forest and LASSO models. In the random forest model, most importantly, the finding that inpatient status is the most important variable in the prediction model has several clinical implications. First, it is hypothesized that recruitment as “inpatient” may be associated with a “moderately ill” or “markedly ill” severity level of schizophrenia because of the following reasons: in a prospective and randomized study of participants administered ECT plus clozapine (n = 20) [7], the mean total and psychotic symptom subscale scores on the Brief Psychiatric Rating Scale were 45.7 (standard deviation [SD] = 1.9) and 16.6 (SD = 0.9), respectively, which indicate “moderately ill” severity according to the framework by Leucht et al. [32]. In addition, in study participants receiving the combination of clozapine and ECT (n = 14), the mean total score on the Positive and Negative Syndrome Scale was 101.7 (SD = 13.9), which indicates a “markedly ill” severity based on the implication by Leucht et al. [33]. Moreover, negative symptoms and social or occupational dysfunction were significantly more prevalent in the group with the augmented use of clozapine with ECT than in the group without. Second, the “inpatient” status may be associated with TRS because of the following reasons: according to the definition of the Treatment Response and Resistance in Psychosis group [34,35], TRS denotes the nonresponse to a trial of two antipsychotics for 6 or more weeks each at a dose equivalent to more than 600 mg of chlorpromazine, which is the daily dose among patients with schizophrenia. TRS is clinically characterized by a longer duration of illness, more severe negative symptoms, poor premorbid social adjustment, and lack of illness insight [36,37,38], which are consistent with the findings. In addition, clozapine is considered the most effective pharmacological treatment option for TRS [39]. Third, the “inpatient” status may be associated with CRS because of the following reasons: CRS denotes the nonresponse to 200–300 mg of clozapine daily with blood levels > 350 ng/mL for 2–3 months among patients with schizophrenia [40,41,42,43]. CRS is considered the most severely ill condition of schizophrenia and is clinically characterized by a greater severity of positive and negative symptoms and lower quality of life [44,45]. Common treatments for CRS include antipsychotics [45], mood stabilizers or antidepressants [46,47,48,49,50], and ECT [3,4,5,6,7,8,9,10,11,12] augmentations. Herein, the “inpatient” status may be associated with a “moderately ill” or “markedly ill” severity level of TRS or CRS, which corresponds to stage IV of Insel’s four-stage model [13]. In the random forest model, other predictive variables for the augmented use of clozapine were BMI, age, social or occupational dysfunction, persistent symptoms, and duration of illness > 20 years. The BMI was mainly affected by clozapine-induced weight gain. A recent meta-analysis showed that clozapine can result in weight gain of approximately 3 kg after a median duration of 6 (range, 2–13) weeks [51]. Weight gain is clinically associated with lower quality of life, increased morbidity and mortality, and patient nonadherence [52]. In several studies on the efficacy of clozapine augmentation with ECT, the mean duration of clozapine treatment ranged from 3.9 (SD = 3.4) to 6.4 (SD = 6.1) years [11,12,48,49]. In addition, age, social or occupational dysfunction, persistent symptoms, and illness duration >20 years were all associated with the clinical characteristics of TRS or CRS. As previously mentioned, social or occupational dysfunction and persistent symptoms can be associated with the clinical characteristics of TRS or CRS [28,29,34,35,36,37]. In addition, several studies assessing the efficacy of the augmented use of clozapine with ECT presented that CRS patients’ mean duration of illness ranged from 15.0 (SD = 8.6) to 22.8 (SD = 14.2) years [11,12,53,54]. In patients with TRS or CRS, age may be affected by a longer duration of illness and longer duration of untreated psychosis, and social or occupational dysfunction and persistent symptoms may be affected by clinical characteristics.
In the random forest model, regarding the performance of the substantial prediction model for the augmented use of clozapine with ECT, the AUC and accuracy were 0.774 and 0.817, respectively. In general, AUCs of 0.5, 0.7–0.8, 0.8–0.9, and >0.9 are considered no discrimination, acceptable, excellent, and outstanding, respectively [55]. Thus, the performance of our substantial prediction model is considered acceptable. In addition, an AUC of 0.714 corresponds to Cohen’s d of 0.800 [56]. However, the low sensitivity value needs to be overcome in future studies. Since the augmentation of clozapine with ECT is rarely used in clinical psychiatric practice, to the best of our knowledge, its rate has rarely been reported in previous epidemiological studies. Thus, although the sensitivity value is insufficient as a par value, using a machine learning-based algorithm, our findings allow the reporting of the specificity value for the substantial prediction model of the augmented use of clozapine with ECT.
In the LASSO model, inpatient status is also the most important variable for the substantial prediction model for using a combination of clozapine and ECT, significant affective symptoms, illness duration > 20 years, disorganized speech, persistent symptoms, social or occupational dysfunction, and others. Although subtle differences presented themselves in the rank orders, the important variable profiles for the LASSO regression-based model tended to be similar to those for the random forest model. In the LASSO model, significant affective symptoms were the second most important variable for the substantial prediction model. Greater affective components can be one of the symptoms in less rigorously defined TRS [57]. In terms of the prediction of the substantial prediction model, the AUC (0.831) is considered an excellent level [55]. Although the AUC of the LASSO model is subtly higher than that of the random forest model, the accuracy of the LASSO model (0.644) is less than that of the random forest model (0.817). Despite the subtle differences in both the AUC and overall accuracy of the random forest model and LASSO model, the important variables were commonly shared by the two models.
Our study has some limitations. First, the study participants of the REAP-AP4 survey were not recruited using a convenience sampling method but rather using a strict epidemiological context. We found a trend for psychotropic drug use patterns, including the augmented use of clozapine with ECT in many Asian countries. However, the significance of the findings from the REAP-AP4 survey might be limited in a strict epidemiological context. Second, current symptoms (i.e., delusion, hallucination, disorganized speech, grossly disorganized or catatonic behavior) were evaluated using a dichotomous value, not a continuous value. Thus, the importance of current symptoms in the substantial prediction model may be affected by their manner of evaluation. Third, we cannot exclude the possibility that our findings were affected by a selection bias. The rate of the augmented use of clozapine with ECT has significantly differed across the 10 Asian countries or SARs in our study. In addition, although the REAP survey is the largest international collaborative psychiatry study in Asia, it was conducted mainly in university-affiliated or training hospitals. Thus, actual clinical psychiatric situations in Asia may be limited to the findings of the REAP-AP survey. Fourth, study participants’ previous treatments with some other medication or ECT have been included as variables for the substantial prediction model. Thus, the generalizability of the substantial prediction model may be limited. Fifth, the duration or treatment of the augmented use of clozapine with ECT was not evaluated in our study. Thus, the substantial prediction model of the treatment outcome of the augmented use of clozapine with ECT was not developed in our study. Sixth, the operational definition of treatment resistance or clozapine resistance was not used in our study. On the contrary, the augmented use of clozapine with ECT was alternatively used. To overcome these limitations, further studies are required to use structured assessment tools to examine psychotropic drug use patterns and current symptoms in Asia in a strict epidemiological context.
5. Conclusions
Despite the limitations of the present study, using the machine learning algorithm, our findings allow for the development of a substantial prediction model for the augmented use of clozapine with ECT in Asian patients with schizophrenia. The substantial prediction model may be the basis of the clinical applicability for indicating the augmented use of clozapine with ECT in terms of precision medicine, although augmentation has rarely been used in real clinical practice. Although the model needs to be improved for sensitivity, in the context of precision medicine for patients with TRS or CRS, the substantial prediction model can be used to elaborate psychopharmacological treatment options. This substantial prediction model can be a basis for further studies to develop the substantial prediction model for the augmented use of clozapine with ECT in patients with schizophrenia.
Acknowledgments
The authors are grateful to the REAP-AP colleagues. We would also like to thank Jinseob Kim for his valuable help with the machine learning.
Author Contributions
Conceptualization, C.-H.T., N.S. (Norman Sartorius), N.S. (Naotaka Shinfuku), Y.C.P. and S.-C.P.; methodology, C.-H.T., N.S. (Norman Sartorius), N.S. (Naotaka Shinfuku), Y.C.P. and S.-C.P.; software, H.S.O. and S.-C.P.; validation, H.S.O. and S.-C.P.; formal analysis, H.S.O. and S.-C.P.; investigation, S.-C.P., H.S.O., B.J.L., Y.S.L., O.-J.J., Y.N., T.I., T.A.K., K.S., M.-Y.C., S.-K.L., T.S., Y.-T.X., A.A., S.G., R.A.K., P.P., K.Y.C., A.J.T., G.R., A.J., K.S., W.A.M., T.V.C., Y.W., S.K. (Shigenobu Kanba), C.-H.T., N.S. (Norman Sartorius), S.K. (Samudra Kathiarachchi), N.S. (Naotaka Shinfuku), Y.C.P. and S.-C.P.; resources, S.-C.P., H.S.O., B.J.L., Y.S.L., O.-J.J., Y.N., T.I., T.A.K., K.S., M.-Y.C., S.-K.L., T.S., Y.-T.X., A.A., S.G., R.A.K., P.P., K.Y.C., A.J.T., G.R., A.J., K.S., W.A.M., T.V.C., Y.W., S.K. (Samudra Kathiarachchi), C.-H.T., N.S. (Norman Sartorius), S.K. (Samudra Kathiarachchi), N.S. (Naotaka Shinfuku), Y.C.P. and S.-C.P.; data curation, S.-C.P., H.S.O., B.J.L., Y.S.L., O.-J.J., Y.N., T.I., T.A.K., K.S., M.-Y.C., S.-K.L., T.S., Y.-T.X., A.A., S.G., R.A.K., P.P., K.Y.C., A.J.T., G.R., A.J., K.S., W.A.M., T.V.C., Y.W., S.K. (Samudra Kathiarachchi), C.-H.T., N.S. (Norman Sartorius), S.K. (Samudra Kathiarachchi), N.S. (Naotaka Shinfuku), Y.C.P. and S.-C.P.; writing—original draft preparation, H.S.O. and S.-C.P.; writing—review and editing, all authors; visualization, H.S.O. and S.-C.P.; supervision, H.S.O. and S.-C.P.; project administration, all authors; funding acquisition, S.-C.P. All authors have read and agreed to the published version of the manuscript.
Institutional Review Board Statement
The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board of Taipei City Hospital, Taipei, Taiwan (receipt number: TCHIRB-10412128-E).
Informed Consent Statement
Informed consents were obtained from all subjects involved in the study.
Data Availability Statement
Data sharing not applicable.
Conflicts of Interest
All authors have declared no competing or potential conflict of interest.
Funding Statement
This work was supported by the research fund of Hanyang University (HY- 202100000001303).
Footnotes
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Joo S.W., Kim H., Jo Y.T., Ahn S., Choi Y.J., Choi W., Park S., Lee J. Comparative effectiveness of antipsychotic monotherapy and polypharmacy in schizophrenia patients with clozapine treatment: A nationwide, health insurance data-based study. Eur. Neuropsychopharmacol. 2022;59:36–44. doi: 10.1016/j.euroneuro.2022.03.010. [DOI] [PubMed] [Google Scholar]
- 2.Kanahara N., Nakamura M., Shiko Y., Kawasaki Y., Iyo M. Are serum oxytocin concentrations lower in patients with treatment-resistant schizophrenia?: A 5-year longitudinal study. Asian J. Psychiatry. 2022;6:103157. doi: 10.1016/j.ajp.2022.103157. [DOI] [PubMed] [Google Scholar]
- 3.Kane J., Honigfeld G., Singer J., Melter H. Clozapine for the treatment resistant schizophrenic: A double-blind comparison with chlorpromazine. Arch. Gen. Psychiatry. 1988;45:789–796. doi: 10.1001/archpsyc.1988.01800330013001. [DOI] [PubMed] [Google Scholar]
- 4.Breier A., Buchanan R.W., Kirkpatrick B., Davis O.R., Irish D., Summerfelt A., Carpenter W.T., Jr. Effects of clozapine on positive and negative symptoms in outpatients with schizophrenia. Am. J. Psychiatry. 1994;151:20–26. doi: 10.1176/ajp.151.1.20. [DOI] [PubMed] [Google Scholar]
- 5.Melzer H.Y., Bastani B., Kwon K.Y., Ramirez L.F., Burnett S., Sharpe J. A prospective study of clozapine in treatment-resistant schizophrenic patients, I: Preliminary report. Psychopharmacology. 1989;99:S68–S72. doi: 10.1007/BF00442563. [DOI] [PubMed] [Google Scholar]
- 6.Weiner R.D., Coffey C., Fochtmann L. The Practice of Electroconvulsive Therapy: Recommendations for Treatment, Training, and Privileging: A Task Force Report of the American Psychiatric Association. 2nd ed. American Psychiatric Publishing; Washington, DC, USA: 2001. [Google Scholar]
- 7.Petrides G., Malur C., Braga R.J., Bailine S.H., Schooler N.R., Malhotra A.K., Kane J.M., Sanghani S., Goldberg T.E., John M., et al. Electroconvulsive Therapy Augmentation in Clozapine-Resistant Schizophrenia: A Prospective, Randomized Study. Am. J. Psychiatry. 2015;172:52–58. doi: 10.1176/appi.ajp.2014.13060787. [DOI] [PubMed] [Google Scholar]
- 8.Kim Y.S., Chung I.W., Kang U.G., Ahn Y.M., Youn T. Principle and Practice of Electroconvulsive Therapy. Seoul National University Press; Seoul, Korea: 2019. p. 133. [Google Scholar]
- 9.Manubens L.K., Urbina D.L., Aceituno D. Is electroconvulsive therapy effective as augmentation in clozapine-resistant schizophrenia? Medwave. 2016;16((Suppl. 5)):e6577. doi: 10.5867/medwave.2016.6577. [DOI] [PubMed] [Google Scholar]
- 10.Lally J., Tully J., Robertson D., Stubbs B., Gaughran F., MacCabe J.H. Augmentation of clozapine with electroconvulsive therapy in treatment resistant schizophrenia: A systematic review and meta-analysis. Schizophr. Res. 2016;171:215–224. doi: 10.1016/j.schres.2016.01.024. [DOI] [PubMed] [Google Scholar]
- 11.Kim H.S., Kim S.H., Lee N.Y., Youn T., Lee J.H., Chung S., Kim Y.S., Chung I.W. Effectiveness of Electroconvulsive Therapy Augmentation on Clozapine-Resistant Schizophrenia. Psychiatry Investig. 2017;14:58–62. doi: 10.4306/pi.2017.14.1.58. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Kim J.H., Youn T., Choi J.G., Jeong S.H., Jung H.Y., Kim Y.S., Chung I.W. Combination of Electroconvulsive Therapy and Clozapine in Treatment-Resistant Schizophrenia. Psychiatry Investig. 2018;15:829–835. doi: 10.30773/pi.2018.05.15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Insel T.R. Rethinking schizophrenia. Nature. 2010;468:187–193. doi: 10.1038/nature09552. [DOI] [PubMed] [Google Scholar]
- 14.Tognin S., Van Hell H.H., Merritt K., Rossum I.W.-V., Bossong M.G., Kempton M.J., Modinos G., Fusar-Poli P., Mechelli A., Dazzan P., et al. Towards Precision Medicine in Psychosis: Benefits and Challenges of Multimodal Multicenter Studies—PSYSCAN: Translating Neuroimaging Findings From Research into Clinical Practice. Schizophr. Bull. 2019;46:432–441. doi: 10.1093/schbul/sbz067. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Grzenda A., Kraguljac N.V., McDonald W.M., Nemeroff C., Torous J., Alpert J.E., Rodriguez C.I., Widge A.S. Evaluating the Machine Learning Literature: A Primer and User’s Guide for Psychiatrists. Am. J. Psychiatry. 2021;178:715–729. doi: 10.1176/appi.ajp.2020.20030250. [DOI] [PubMed] [Google Scholar]
- 16.Choi J., Yoon H.-J., Park J.H., Nakagami Y., Kubota C., Inada T., Kato T.A., Yang S.-Y., Lin S.-K., Chong M.-Y., et al. Network Analysis-Based Disentanglement of the Symptom Heterogeneity in Asian Patients with Schizophrenia: Findings from the Research on Asian Psychotropic Prescription Patterns for Antipsychotics. J. Pers. Med. 2022;12:33. doi: 10.3390/jpm12010033. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Park S.-C., Kim G.-M., Kato T.A., Chong M.-Y., Lin S.-K., Yang S.-Y., Avasthi A., Grover S., Kallivayalil R.A., Xiang Y.-T., et al. Dyskinesia is most centrally sit-uated in an estimated network of extrapyramidal syndrome in Asian patients with schizophrenia: Findings from Research on Asian Psychotropic Prescription Patterns for Antipsychotics. Nordic. J. Psychiatry. 2021;75:9–17. doi: 10.1080/08039488.2020.1777462. [DOI] [PubMed] [Google Scholar]
- 18.Park S., Lee B.J., Park J.H., Kawasaki H., Avasthi A., Grover S., Tanra A.J., Lin S., Javed A., Tan C.H., et al. QT interval prolongation noted in one percent of 2553 Asian patients with schizophrenia: Findings from the REAP-AP survey. Kaohsiung. J. Med. Sci. 2020;36:1030–1037. doi: 10.1002/kjm2.12280. [DOI] [PubMed] [Google Scholar]
- 19.American Psychiatric Association . Diagnostic and Statistical Manual of Mental Disorders. 5th ed. American Psychiatric Association; Washington, DC, USA: 2013. [Google Scholar]
- 20.World Health Organization Anatomical Therapeutic Chemical (ATC) Classification System. [(accessed on 18 January 2020)]. Available online: https://www.whocc.no/atc/structure_and_principles/
- 21.Sugawara N., Yasui-Furukori N., Yamazaki M., Shimoda K., Mori T., Sugai T., Matsuda H., Suzuki Y., Ozeki Y., Okamoto K., et al. Predictive Utility of Body Mass Index for Metabolic Syndrome Among Patients with Schizophrenia in Japan. Neuropsychiatr. Dis. Treat. 2020;16:2229–2236. doi: 10.2147/NDT.S269619. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Huang Y., Wu K., Li H., Zhou J., Xiong D., Huang X., Li J., Liu Y., Pan Z., Mitchell D.T., et al. Homocysteine level, body mass index and clinical correlates in Chinese Han patients with schizophrenia. Sci. Rep. 2020;10:16–18. doi: 10.1038/s41598-020-72934-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Lin S.H., Tseng H.H., Tsai H.C., Chi M.H., Lee I.H., Chen P.S., Chen K.C., Yang Y.K. Positive symptoms in antipsychotic-naïve schizophrenia are associated with increased body mass index after treatment. Clin. Psychopharmacol. Neurosci. 2021;19:155–159. doi: 10.9758/cpn.2021.19.1.155. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Luckhoff H., du Plessis S., Scheffler F., Phahladira L., Kilian S., Buckle C., Smit R., Chiliza B., Asmal L., Emsley R. Fronto-limbic white matter fractional anisotropy and body mass index in first-episode schizophrenia spectrum disorder patients compared to healthy controls. Psychiatry Res. Neuroimag. 2020;305:111173. doi: 10.1016/j.pscychresns.2020.111173. [DOI] [PubMed] [Google Scholar]
- 25.Okusaga O.O., Kember R.L., Peloso G.M., Peterson R.E., Vujkovic M., Mitchell B.G., Bernard J., Walder A., Bigdeli T.B. History of Suicide Attempts and COVID-19 Infection in Veterans with Schizophrenia or Schizoaffective Disorder: Moderating Effects of Age and Body Mass Index. Complex Psychiatry. 2021;392:1789–1795. doi: 10.1159/000521230. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Chawla N.V., Bowyer K.W., Hall L.O., Kegelmeyer W.P. SMOTE: Synthetic Minority Over-sampling Technique. J. Artif. Intell. Res. 2002;16:321–357. doi: 10.1613/jair.953. [DOI] [Google Scholar]
- 27.Cho S.-E., Geem Z.W., Na K.-S. Development of a Suicide Prediction Model for the Elderly Using Health Screening Data. Int. J. Environ. Res. Public Health. 2021;18:10150. doi: 10.3390/ijerph181910150. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Cleophas T.J., Zwinderman A.H. Machine Learning in Medicine – A Complete Review. 2nd ed. Springer; New York, NY, USA: 2020. [Google Scholar]
- 29.Breiman L. Random forest. Mach. Learn. 2001;45:5–32. doi: 10.1023/A:1010933404324. [DOI] [Google Scholar]
- 30.Hastie T., Tibshirani R., Freedman J. The Elements of Statistical Learning. 2nd ed. Springer; New York, NY, USA: 2017. pp. 593–594. [Google Scholar]
- 31.Sicotte X.B. Lasso Regression: Implementation of Coordinate Descent. Data Science, Machine Learning and Statistics, Implemented in Python. 2018. [(accessed on 3 June 2022)]. Available online: https://xavierbourretsicotte.github.io/lasso_implementation.html.
- 32.Leucht S., Kane J.M., Kissling W., Hamann J., Etschel E., Engel R. Clinical implications of Brief Psychiatric Rating Scale scores. Br. J. Psychiatry. 2005;187:366–371. doi: 10.1192/bjp.187.4.366. [DOI] [PubMed] [Google Scholar]
- 33.Leucht S., Kane J.M., Kissling W., Hamann J., Etschel E., Engel R.R. What does the PANSS mean? Schizophr. Res. 2005;79:231–238. doi: 10.1016/j.schres.2005.04.008. [DOI] [PubMed] [Google Scholar]
- 34.Correll C.U., Howes O.D. Treatment-resistant schizophrenia: Definition, predictors, and therapy options. J. Clin. Psychiatry. 2021;82:MY20096AH1C. doi: 10.4088/JCP.MY20096AH1C. [DOI] [PubMed] [Google Scholar]
- 35.Keepers G.A., Fochtmann L.J., Anzia J.M., Benjamin S., Lyness J.M., Mojitabai R., Servis M., Walaszek A., Buckley P., Lenzenweger M.L., et al. The American Psychiatric Association Guideline for the Treatment of Patients with Schizophrenia. 3rd ed. American Psychiatric Association Publishing; Washington, DC, USA: 2021. [Google Scholar]
- 36.Carbon M., Correl C.U. Clinical predictors of therapeutic response to antipsychotics in schizophrenia. Dialogues Clin. Neurosci. 2014;16:505–524. doi: 10.31887/DCNS.2014.16.4/mcarbon. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Ortiz B.B., Higuchi C.H., Noto C., Joyce D.W., Correll C.U., Bressan R.A., Gadelha A. A symptom combination predicting treatment-resistant schizophrenia–A strategy for real-world clinical practice. Schizophr. Res. 2020;218:195–200. doi: 10.1016/j.schres.2020.01.002. [DOI] [PubMed] [Google Scholar]
- 38.Elkis H., Buckley P.F. Treatment-resistant schizophrenia. Psychiatry Clin. N. Am. 2016;39:239–265. doi: 10.1016/j.psc.2016.01.006. [DOI] [PubMed] [Google Scholar]
- 39.Huhn M., Nikolakopoulou A., Schneider-Thoma J., Krause M., Samara M., Peter N., Arndt T., Bäckers L., Rothe P., Cipriani A., et al. Comparative efficacy and tolerability of 32 oral antipsychotics for the acute treatment of adults with multi-episode schizophrenia: A systematic review and network meta-analysis. Lancet. 2019;394:939–951. doi: 10.1016/S0140-6736(19)31135-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Tranulis C., Mouaffak F., Chouchana L., Stip E., Gourevitch R., Poirier M.F., Olié J.P., Lôo H., Gourion D. Somatic aug-mentation strategies in clozapine resistance—What facts? Clin. Neuropharmacol. 2006;29:34–44. doi: 10.1097/00002826-200601000-00010. [DOI] [PubMed] [Google Scholar]
- 41.Mouaffak F., Tranulis C., Gourevitch R., Poirier M.-F., Douki S., Olié J.-P., Lôo H., Gourion D. Augmentation Strategies of Clozapine With Antipsychotics in the Treatment of Ultraresistant Schizophrenia. Clin. Neuropharmacol. 2006;29:28–33. doi: 10.1097/00002826-200601000-00009. [DOI] [PubMed] [Google Scholar]
- 42.Lee J., Takeuchi H., Fervaha G., Sin G.L., Foussias G., Agid O., Farooq S., Remington G. Subtyping Schizophrenia by Treatment Response: Antipsychotic Development and the Central Role of Positive Symptoms. Can. J. Psychiatry. 2015;60:515–522. doi: 10.1177/070674371506001107. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Howes O.D., McCutcheon R., Agid O., De Bartolomeis A., Van Beveren N.J., Birnbaum M.L., Bloomfield M., Bressan R.A., Buchanan R.W., Carpenter W.T., et al. Treatment-Resistant Schizophrenia: Treatment Response and Resistance in Psychosis (TRRIP) Working Group Consensus Guidelines on Diagnosis and Terminology. Am. J. Psychiatry. 2017;174:216–229. doi: 10.1176/appi.ajp.2016.16050503. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Neto J.H., Elkis H. Clinical aspects of super-refractory schizophrenia: A 6-month cohort observational study. Rev. Bras. Psiquiatry. 2007;29:228–232. doi: 10.1590/S1516-44462007000300007. [DOI] [PubMed] [Google Scholar]
- 45.Buckley P., Miller A., Olsen J., Garver D., Miller D.D., Csernansky J. When Symptoms Persist: Clozapine Augmentation Strategies. Schizophr. Bull. 2001;27:615–628. doi: 10.1093/oxfordjournals.schbul.a006901. [DOI] [PubMed] [Google Scholar]
- 46.Pai N.B., Laidlaw M., Vella S.-C. Augmentation of clozapine with another pharmacological agent: Treatment for refractory schizophrenia in the ‘real world’. Acta Psychiatr. Scand. 2012;126:40–46. doi: 10.1111/j.1600-0447.2012.01854.x. [DOI] [PubMed] [Google Scholar]
- 47.Tiihonen J., Wahlbeck K., Kiviniemi V. The efficacy of lamotrigine in clozapine-resistant schizophrenia: A systematic review and meta-analysis. Schizophr. Res. 2009;109:10–14. doi: 10.1016/j.schres.2009.01.002. [DOI] [PubMed] [Google Scholar]
- 48.Zheng W., Xiang Y.T., Xiang Y.Q., Li X.B., Ungvari G.S., Chiu H.F., Correll C.U. Efficacy and safety of adjunctive topiramate for schizophrenia: A meta-analysis of randomized controlled trials. Acta Psychiatry Scand. 2016;134:385–398. doi: 10.1111/acps.12631. [DOI] [PubMed] [Google Scholar]
- 49.Sommer I.E., Begemann M.J.H., Temmerman A., Leucht S. Pharmacological Augmentation Strategies for Schizophrenia Patients with Insufficient Response to Clozapine: A Quantitative Literature Review. Schizophr. Bull. 2011;38:1003–1011. doi: 10.1093/schbul/sbr004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Veerman S.R.T., Schulte P.F.J., Begemann M.J.H., de Haan L. Non-Glutamatergic Clozapine Augmentation Strategies: A Review and Meta-Analysis. Pharmacopsychiatry. 2014;47:231–238. doi: 10.1055/s-0034-1385930. [DOI] [PubMed] [Google Scholar]
- 51.Pillinger T., McCutcheon R., Vano L., Mizuno Y., Arumuham A., Hindley G., Beck K., Natesan S., Efthimiou O., Cipriani A., et al. Comparative effects of 18 antipsychotics on metabolic function in patients with schizophrenia, predictors of metabolic dysregulation, and association with psychopathology: A systematic review and network meta-analysis. Lancet Psychiatry. 2019;7:64–77. doi: 10.1016/S2215-0366(19)30416-X. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Haddad P. Weight change with atypical antipsychotics in the treatment of schizophrenia. J. Psychopharmacol. 2005;19:16–27. doi: 10.1177/0269881105058378. [DOI] [PubMed] [Google Scholar]
- 53.Youn T., Jeong S.H., Kim Y.S., Chung I.W. Long-term clinical efficacy of maintenance electroconvulsive therapy in patients with treatment-resistant schizophrenia on clozapine. Psychiatry Res. 2019;273:759–766. doi: 10.1016/j.psychres.2019.02.008. [DOI] [PubMed] [Google Scholar]
- 54.Havaki-Kontaxaki B.J., Ferentinos P.P., Kontaxakis V.P., Paplos K.G., Soldatos C.R. Concurrent Administration of Clozapine and Electroconvulsive Therapy in Clozapine-Resistant Schizophrenia. Clin. Neuropharmacol. 2006;29:52–56. doi: 10.1097/00002826-200601000-00012. [DOI] [PubMed] [Google Scholar]
- 55.Hosmer D.W., Lemeshow S. Applied Logistic Regression. 2nd ed. John Wiley and Sons; New York, NY, USA: 2000. pp. 160–164. Chapter 5. [Google Scholar]
- 56.Rice M.E., Harris G.T. Comparing effect sizes in follow-up studies: ROC Area, Cohen’s d, and r. Law Hum. Behav. 2005;29:615–620. doi: 10.1007/s10979-005-6832-7. [DOI] [PubMed] [Google Scholar]
- 57.Sinclair D.J., Zhao S., Qi F., Nyakyoma K., Kwong J.S., Adams C.E. Electroconvulsive therapy for treatment-resistant schizophrenia. Cochrane Database Syst. Rev. 2019;3:CD011847. doi: 10.1002/14651858.cd011847.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Data sharing not applicable.