Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Jan 12.
Published in final edited form as: Fam Process. 2016 Feb 15;56(2):436–444. doi: 10.1111/famp.12208

Integrity of Literature on Expressed Emotion and Relapse in Patients with Schizophrenia Verified by a p-Curve Analysis

MARC J WEINTRAUB *, DANIEL L HALL *,†, JULIA Y CARBONELLA *, AMY WEISMAN DE MAMANI *, JILL M HOOLEY ‡
PMCID: PMC5765756  NIHMSID: NIHMS932054  PMID: 26875506

Abstract

There is growing concern that much published research may have questionable validity due to phenomena such as publication bias and p-hacking. Within the psychiatric literature, the construct of expressed emotion (EE) is widely assumed to be a reliable predictor of relapse across a range of mental illnesses. EE is an index of the family climate, measuring how critical, hostile, and overinvolved a family member is toward a mentally ill patient. No study to date has examined the evidential value of this body of research as a whole. That is to say, although many studies have shown a link between EE and symptom relapse, the integrity of the literature from which this claim is derived has not been tested. In an effort to confirm the integrity of the literature of EE predicting psychiatric relapse in patients with schizophrenia, we conducted a p-curve analysis on all known studies examining EE (using the Camberwell Family Interview) to predict psychiatric relapse over a 9- to 12-month follow-up period. Results suggest that the body of literature on EE is unbiased and has integrity, as there was a significant right skew of p-values, a nonsignificant left skew of p-values, and a nonsignificant test of flatness. We conclude that EE is a robust and valuable predictor of symptom relapse in schizophrenia.

Keywords: Expressed Emotion, Psychiatric Relapse, Schizophrenia, p-Curve Analysis

INTRODUCTION

Expressed Emotion (EE) is a construct that reflects potentially problematic aspects of the family environment, particularly for people with psychiatric disorders (Hooley, 2007). EE is a measure of how critical, hostile, or overly involved a family member is toward an identified patient, and is thought to represent relational disturbances and transactional patterns between the family member and that patient. EE is considered to be a psychosocial stressor that interacts with patients’ diatheses, eventually culminating in relapse (Hooley & Gotlib, 2000). The traditional method of measuring EE dichotomizes family members into either high or low EE groupings based on the degree to which the family member makes critical, hostile, or overinvolved comments regarding the patient (Vaughn & Leff, 1976a).

Hearing critical or hostile comments or having an overly involved family member is not a pleasant experience for anyone. However, as noted above, individuals with schizophrenia appear to be at elevated risk for relapse if they live in family environments characterized by high levels of these common attitudes (Butzlaff & Hooley, 1998). When patients with schizophrenia return home from the hospital to live with a high-EE family, they are more than twice as likely to relapse compared to patients who return home to low-EE families (Hooley, 2007). The strength of this effect is estimated to be small to medium (effect size: r = .31; Butzlaff & Hooley, 1998). This effect size is based on studies that spanned twelve countries and four continents, suggesting cross-cultural relevancy. However, although the EE-relapse link is widely accepted as valid, no study to date has confirmed the evidential value or integrity of this body of literature. In other words, although many studies have found a link between EE and relapse in patients with schizophrenia, are these effects true and unbiased or do they reflect “selective reporting” (e.g., only publishing results that reach statistical significance; Simonsohn, Nelson, & Simmons, 2014)?

An unfortunate reality for the bio-behavioral sciences is the existence of false positives that result from statistical hypothesis testing. False positives are bound to happen on occasion; however, some researchers have presented findings to suggest that a majority of published findings are false and/or do not accurately represent reality (Ioannidis, 2005; Pashler & Harris, 2012). An example of this is seen in the ego depletion/self-control literature in social psychology, a literature that has about 200 publications of empirical support to show a seemingly strong and robust effect for the depletion effect (Hagger, Wood, Stiff, & Chatzisarantis, 2010). More recent investigation of this work has found, however, that the depletion effect is much smaller than originally believed and is task dependent (Carter, Kofler, Forster, & McCullough, 2015; Carter & McCullough, 2014). This work by Carter and McCullough highlights the importance of testing the validity of a body of literature, even a body of literature that is widely published and largely believed to be robust. Thus, in this paper, we present a novel method of testing the validity and integrity of a body of literature (through use of a p-curve analysis) on a widely published and acknowledged phenomenon—the link between EE and psychiatric relapse in patients with schizophrenia.

The result of false positives to the larger field of research can be devastating, as a variety of consequences result from these reported errors (Simmons, Nelson, & Simonsohn, 2011). For example, Simons and colleagues point out that, once reported, false positives remain persistent in the literature as studies that fail to replicate are often either rejected by prestigious journals or given less attention. False positives can have detrimental consequences for consumers as well (particularly in fields such as psychiatry or clinical psychology where clinical decisions often follow research findings). Finally, the scientific credibility of a given field can be compromised when false positives eventually surface.

Replication of a particular phenomenon helps to bolster the credibility of that finding, as repeatedly obtaining positive results decreases the chances of large “file-drawer” effects or failed attempts to replicate (Ioannidis, 2008). This surface-level analysis bodes well for the literature on EE, as the phenomenon has been replicated in over 50 years of research, and across multiple disorders. However, “researcher degrees of freedom” while collecting and analyzing data allow for a variety of post hoc decisions to be made that can lead to biased results (Simmons et al., 2011). For example, researchers can decide if more data should be collected, or whether certain observations should be excluded. This is sometimes referred to as “p-hacking” (Simonsohn et al., 2014). Even seemingly modest levels of p-hacking can result in statistical significance for nonexistent effects (Simonsohn et al., 2014).

To empirically test whether a body of literature has been p-hacked, Simonsohn et al. (2014) have developed a statistical test (a p-curve analysis) that examines the skewness of significant p-values within that body of literature. A p-curve is the distribution of statistically significant p-values for a body of literature. In the case of a nonexistent effect, every p-value is equally likely to be seen. In this situation when the null hypothesis is true (there are no differences between the groups), the p-curve should have a uniform distribution (i.e., a horizontal line of p-values). Just by chance any p-value is as likely as any other with a nonexistent effect. If the frequency of significant p-values within a body of literature is skewed left (i.e., there are many more p-values hovering close to .05 than .01), then there is reason to believe the literature may be p-hacked. On the other hand, if the frequency of p-values is skewed right (i.e., there are more p-values close to .01 and beyond than .05), this is indicative of an evidential body of literature. The reasoning behind this conclusion is because a real effect, when tested repeatedly, will show movement away from the .05 threshold. For example, if patients in high-EE families truly have more adverse outcomes than patients in low EE families, there is greater statistical likelihood that the results will show stronger evidence for this effect (p < .01) rather than weaker evidence (.04 < p < .05). As the power of the effect increases, there will be a greater right skew and p-values will be closer to zero (Simonsohn et al., 2014). Conversely, a body of literature that has been p-hacked is likely to have many more p-values that hover closer to the .05 threshold. In this case, if EE researchers are trying to push their “EE agenda” but there really is no effect, there are a couple reasons to explain the right skew. First, there is greater statistical likelihood that larger p-values are obtained for nonexistent findings. And second, researchers who are p-hacking are likely to stop p-hacking once a significant value is reached, rather than pursue the lowest p-value possible.

It is important to note that a p-curve analysis is quite different from and presents a very different message compared to a meta-analysis. Both types of analyses can point to whether an effect for a phenomenon is substantiated; however, a meta-analysis (like the one performed by Butzlaff & Hooley, 1998) calculates the size/magnitude of the effect found in a particular body of research. In other words, does the literature show a consistent relationship between variables and how strong is this relationship? So, do patients in high-EE families have different outcomes than patients in low EE families and, if so, are these differences worth “writing home about” (Cohen, 1988)? A p-curve analysis displays a different picture of the literature—one of statistical and empirical integrity rather than the magnitude of a phenomenon. As Simonsohn et al. (2014) note, researchers and journals tend to publish results that are statistically significant, so are these effects true and valid or do they reflect publication bias or statistical p-hacking? Both questions are critical to ask for clinicians and researchers, as the former indicates whether the magnitude of the effect is statistically and practically meaningful while the latter suggests that the research has been reported fairly and truthfully. The exploration of EE on psychiatric relapse has significantly slowed down since Butzlaff’s and Hooley’s seminal meta-analysis, probably because the meta-analysis found such consistently positive effects. However, before the scientific community “closes the door” on this research question, it is important to determine whether these results are overinflated due to p-hacking, or whether it is fair to take the results from that meta-analysis as representative of the true (or close to true) effect. In an effort to assess the integrity and evidential value of the EE literature, we conducted a p-curve analysis to examine the studies outlined by Butzlaff and Hooley’s (1998) seminal paper. This is the most frequently cited meta-analysis to examine the effect of EE on psychiatric relapse (as of January 30, 2016, this paper had been cited by 1138 studies in Google Scholar).

METHOD

Study Selection

This p-curve analysis analyzed findings from 29 studies investigating the relation between EE and relapse in schizophrenia. Study selection began by consulting Butzlaff and Hooley’s (1998) meta-analysis. Their meta-analysis included studies examining relapse among various psychiatric populations, including patients diagnosed with schizophrenia, mood disorders, or eating disorders. The vast majority of studies (27 in total) were specific to schizophrenia. These studies form the focus of the present p-curve analysis. We also located one recent publication with individuals from Poland (Cechnicki, Bielańska, Hanuszkiewicz, & Daren, 2013) as well as unpublished data from a study in the United States that examined EE as a predictor of relapse in schizophrenia patients (Hooley, Rosen, Leach, & Wilson, 1999). Aside from these two aforementioned data, no other data have been published that meet our criteria for this study (detailed below).

The focus of this study is on the validity of the original EE-relapse link research, which involves three major criteria: (1) patients who have schizophrenia, (2) the Camberwell Family Interview used as the method of assessing for EE, and (3) a relapse period of nine to twelve months (Butzlaff & Hooley, 1998). Therefore, all 29 studies used in this study shared these three key commonalities. First, samples included patients diagnosed with schizophrenia. Second, high versus low EE was measured using the Camberwell Family Interview (CFI; Vaughn & Leff, 1976b). Since its publication, the CFI has been recognized as the “gold standard” measure of EE (Hooley & Parker, 2006). Although many studies have used self-report measures of EE or the Five-Minute Speech Sample, these methods are problematic. The Five-Minute Speech Sample, for example, has been found to under-identify high-EE relatives. There are also mixed findings (as well as conceptual problems) concerning the predictive validity of self-report EE measures and their correlations with the CFI (see Hooley & Parker, 2006; see also Hooley & Richters, 1991). For these reasons, we only included data from studies that used the CFI. It is important to note that one study that used the CFI (Breitborde, López, Aguilera, & Kopelowicz, 2013) was excluded from the analysis because only the emotionally overinvolved component of EE was measured. Therefore, participants were not grouped into high versus low EE (like the other studies used in this paper), but rather as high versus low EOI. Third, all studies employed a time frame for measuring relapse that was between nine to twelve months. This is the modal period for measuring relapse within the EE literature (Butzlaff & Hooley, 1998). Finally, all studies utilized longitudinal designs. In studies testing an intervention, only results from the no treatment control condition were retained for use in the p-curve analysis because including intervention effects could cloud the findings.

Of the 29 studies included, 26 were published in peer-reviewed journals, two were presented at conferences as posters or papers, and one was never presented or submitted for publication. Not all studies reported a significant association between EE and relapse. In an attempt to locate more unpublished studies as well as those published since 1998, we searched PubMed and Google Scholar for published and unpublished manuscripts citing each study included in Butzlaff and Hooley’s (1998) meta-analysis. No additional studies were identified.

Statistical Analysis

The data for this manuscript were gathered from the original 29 manuscripts and are presented in the supplementary materials. The analyses presented in the original 29 articles ranged from chi-square tests, correlations, and Fisher exact tests. Then, the p-curve analytic software for public use (www.p-curve.com/app3) was utilized, based on Simonsohn et al.’s (2014) demonstration of p-curve practice and theory. This program allows for the original results to be plugged into the analyses, and two types of tests are performed on those original results to perform the p-curve analysis—binomial and continuous tests. The binomial tests compare the observed proportion of significant results that are p < .025 to the expected proportions when there is no effect and when studies have approximately one-third power (i.e., the chance of seeing an effect is there really is an effect). Power is typically expected to be at 80%, so one-third power would mean that the p-curve program gives a little slack to the data entered into the analyses. These values are simply the default in the p-curve analysis program and are the suggested values from the developers of the program; however, researchers can set other cutoff points should they desire. For more details about these default values, see Simonsohn et al. (2014). The continuous tests convert the p-values into Z scores, sum the Z scores, and divide by the square root of the number of p-values included in the analysis to achieve an overall Z score (known as the Stouffer’s Method). This overall Z score is then compared to the null (Z = 0) and the test when studies have approximately one-third power.

With these binomial test and continuous approaches, three parallel tests are conducted—a test of right skew (to determine whether the studies contain evidential value), left skew (to determine whether the studies show evidence of intense p-hacking), and a test of flatness (to determine whether the studies’ evidential value, if any, is inadequate). The p-curve analyses only use significant p-values to analyze skewness. This is because it is assumed that researchers would not tamper with their data or analyses to publish nonsignificant p-values. Therefore, nonsignificant findings are expected to be unbiased. However, significant p-values are suspect to bias, and it is this potential for bias that the p-curve analysis is testing.

RESULTS

Overall, 19 of the 29 studies reported significant p-values (alpha = .05). As depicted in Figure 1, almost two thirds of the studies reporting significant findings (63.2%) observed a p-value of .01 or less. About 15.8% percent of the 19 studies reported a p-value of approximately .02, 10.5% at p = .03, 5.3% at p = .04, and 5.3% at p = .05.

Figure 1.

Figure 1

Open in a new tab

p-curve results for cited literature examining expressed emotion and schizophrenia relapse. The observed p-curve includes 19 significant (p < .05) p-values. There were nine additional results entered but excluded from the p-curve because their results had p-values > .05.

Results indicate that the studies in this analysis cumulatively contain evidential value, indicated by a statistically significant right skew of p-values for both the binomial test (p = .0004) and the continuous test (Z = −7.61, p < .0001). Furthermore, the distribution did not contain a left skew according to the binomial test (p = .9999) and the continuous test (Z = 7.61, p > .9999), suggesting that the included studies were not intensely p-hacked. The test of flatness (flatter than 33% power), which would indicate that the evidential value is inadequate, were nonsignificant for both the binomial test (p = .9881) and the continuous test (Z = 4.47, p > .9999). Thus, the cumulative findings do not lack evidential value or exhibit evidence of intense p-hacking. Finally, the post hoc test of statistical power indicates that the average power of tests included in the p-curve (correcting for any publication bias) is 86%. The individual standardized results for all 29 studies are presented in Table 1. Table 1 presents the data from the 29 studies used in these analyses. The data were separated into 2 × 2 tables of counts—high versus low EE by relapsed versus not relapsed status. Also, a measure of the standardized effect size (r) for the effect found in each study is given to provide an easy method of comparison across studies.

Table 1.

Summary Table of Included Studies: Expressed Emotion (EE) and Schizophrenia

Source Relapse Information, No. Relapsed/Total No.
Low EE High EE ra
Arévalo and Vizcarro (1989) 5/13 8/18 .06*
Barrelet, Ferrero, Szigethy, Giddey, and Pellizzer (1990) 0/12 8/24 .35
Bertrando et al. (1992) 4/18 14/24 .38
Brown, Birley, and Wing (1972) 13/47 38/50 .53
Brown, Monck, Carstairs, and Wing (1962) 9/56 26/45 .47
Buchkremer, Stricker, Holle, and Kuhs (1991) N/Ab N/Ab .17*
Cechnicki et al. (2013) N/Ab N/Ab N/A*
Hooley et al. (1999) 0/4 27/40 .39
Ito and Oshima (1995) 3/37 16/35 .46
Ivanovic and Vuletic (1989) 2/31 19/29 .72
Karno et al. (1987) 7/27 10/17 .34
Köttgen, Sönnichsen, Mollenhauer, and Jurth (1984) 11/20 7/14 −.05*
Leff et al. (1990) 5/54 5/16 .27
Linszen et al. (1996) 0/13 6/26 .26*
MacMillan, Gold, Crow, Johnson, and Johnstone (1986) 14/34 26/38 .28
Moline, Singh, Morris, and Meltzer (1985) 4/13 10/11 .71
Montero, Ruiz, Puche, and Adam (1992) 7/31 8/28 .07*
Mozny and Votýpková (1992) 13/56 41/69 .38
Niedermeier, Watzl, and Cohen (1992) 6/21 16/28 .29
Nuechterlein, Snyder, and Mintz (1992) 0/12 12/31 .37
Parker, Johnston, and Hayward (1988) 9/15 20/42 −.11*
Phillips and Xiong (1995) 9/27 10/22 .12*
Rostworowska, Barbaro, and Cechnicki (1987) 1/11 15/25 .51
Stirling et al. (1991) 8/17 5/16 −.16*
Tanaka, Mino, and Inoue (1995) 6/28 14/24 .40
Tarrier et al. (1988) 4/19 14/29 .28*
Vaughan et al. (1992) 10/41 25/47 .30
Vaughn and Leff (1976a) 3/18 20/36 .39
Vaughn et al. (1984) 1/16 10/21 .48
Open in a new tab

Notes.

a

r is a measure of the standardized result that was calculated based on the researchers original published results.

b

authors did not provide breakdown of patients into high/low EE nor how many relapsed within each group.

*

indicates a nonsignificant p-value, and therefore was not included in the p-curve analysis.

DISCUSSION

Throughout a large portion of the psychiatric literature, EE has been identified as an important predictor of psychiatric relapse (e.g., Hooley & Gotlib, 2000). Butzlaff and Hooley’s (1998) meta-analysis of 27 longitudinal follow-up studies assessing EE via the Camberwell Family Interview (CFI) has become the seminal display confirming that EE is a valid predictor of psychotic relapse, with medium effect sizes across a worldwide sampling of studies. Since the publication of this meta-analysis (in which the authors called for an end to more research designed to simply replicate the EE-relapse association in schizophrenia patients), very few such studies have been published. In other words, this meta-analysis has long been considered conclusive with respect to establishing the validity of EE as a predictor of symptom relapse in schizophrenia patients.

Of course, Butzlaff and Hooley’s meta-analysis was published long before the idea of p-hacking came into existence. Nonetheless, this development provided us with a new technique to test the evidential value of this body of research. The results of our p-curve analysis suggest that (1) the body of literature that has examined EE as a predictor of relapse for patients with schizophrenia likely has integrity, and (2) EE appears to be a valid predictor of psychiatric relapse in these patients, as evidenced by the significant right skew in p-values. Furthermore, the literature does not appear to have been p-hacked, as there was no evidence for left skew in p-values. This finding is reassuring. It tells us that there is no evidence of significant bias or p-hacking in the literature on EE and psychiatric relapse. Furthermore, the significant right skew indicates that the findings in the EE literature contain evidential value, suggesting that the conclusion made by Butzlaff and Hooley (1998) is accurate—EE is a useful and powerful predictor of psychiatric relapse across a wide range of cultures, countries, and continents.

Research on EE had some of its earliest roots in the Family Process journal (e.g., Miklowitz et al., 1989; Strachan, Feingold, Goldstein, Miklowitz, & Nuechterlein, 1989). Therefore, the results from this report underscore a couple of important points in relation to this journal and its readership. First, it is encouraging that the researchers who have published on this topic appear to have done so with integrity. Furthermore, the readers of this journal, both clinicians and researchers, can feel more confident that their endeavors in working with EE as a point of psychological intervention and study are based on research that has evidential and integral backing. Researchers and clinicians should attempt to further test the evidential value and integrity with other bodies of clinical research. Investigating the evidential value and integrity (through such tests as the p-curve analysis) has been receiving increasing attention in the field of social psychology. It would behoove clinical scientists to examine our bodies of research with the same scrutiny, as our work has just as much, if not more, direct impact on the general public.

All studies have their limitations and this report is no exception. A primary concern is that the sampling framework of the studies examined in the meta-analysis may limit the generalizability of the EE-relapse association. For instance, only studies that used the CFI to operationalize EE and employed a nine to twelve month follow-up period were included. Expanding the framework to all existing measures of EE, or using a longer follow-up interval (e.g., predicting relapse over two to five years) could reveal a different pattern of results. Additionally, this study focused only on schizophrenia and did not examine the integrity of literature for EE in other (less frequently studied) conditions. Future directions could help confirm whether EE is similarly valid across multiple measures of EE as well as across diagnostic boundaries. Importantly, however, the present investigation demonstrates that high EE is a reliable predictor for relapse in schizophrenia across a nine to twelve month time frame. This is reassuring news for researchers and clinicians, as well as for patients receiving EE-based family interventions.

Acknowledgments

A special thanks to Michael McMullough, Ph.D., for his assistance on this manuscript.

References

  1. Arévalo J, Vizcarro C. “Emoción expresada” y curso de la esquizofrenia en una muestra Española. Análisis y Modificación de Conducta. 1989;15:3–23. [Google Scholar]
  2. Barrelet L, Ferrero F, Szigethy L, Giddey C, Pellizzer G. Expressed emotion and first-admission schizophrenia: Nine-month follow-up in a French cultural environment. The British Journal of Psychiatry. 1990;156(3):357–362. doi: 10.1192/bjp.156.3.357. [DOI] [PubMed] [Google Scholar]
  3. Bertrando P, Beltz J, Bressi C, Clerici M, Farma T, Invernizzi G, et al. Expressed emotion and schizophrenia in Italy: A study of an urban population. The British Journal of Psychiatry. 1992;161(2):223–229. doi: 10.1192/bjp.161.2.223. [DOI] [PubMed] [Google Scholar]
  4. Breitborde NJ, López SR, Aguilera A, Kopelowicz A. Perceptions of efficacy, expressed emotion, and the course of schizophrenia: The case of emotional overinvolvement. The Journal of Nervous and Mental Disease. 2013;201(10):833. doi: 10.1097/NMD.0b013e3182a5bf1d. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Brown GW, Birley JL, Wing JK. Influence of family life on the course of schizophrenic disorders: A replication. The British Journal of Psychiatry. 1972;121:241–258. doi: 10.1192/bjp.121.3.241. [DOI] [PubMed] [Google Scholar]
  6. Brown GW, Monck EM, Carstairs GM, Wing JK. Influence of family life on the course of schizophrenic illness. British Journal of Preventive and Social Medicine. 1962;16(2):55–68. [Google Scholar]
  7. Buchkremer G, Stricker K, Holle R, Kuhs H. The predictability of relapses in schizophrenic patients. European Archives of Psychiatry and Clinical Neuroscience. 1991;240(4–5):292–300. doi: 10.1007/BF02189543. [DOI] [PubMed] [Google Scholar]
  8. Butzlaff RL, Hooley JM. Expressed emotion and psychiatric relapse: A meta-analysis. Archives of General Psychiatry. 1998;55(6):547–552. doi: 10.1001/archpsyc.55.6.547. [DOI] [PubMed] [Google Scholar]
  9. Carter EC, Kofler LK, Forster DE, McCullough ME. A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. Journal of Experimental Psychology: General. 2015;144:796–815. doi: 10.1037/xge0000083. [DOI] [PubMed] [Google Scholar]
  10. Carter EC, McCullough ME. Publication bias and the limited strength model of self-control: Has the evidence for ego depletion been overestimated? Frontiers in Psychology. 2014;5:1–11. doi: 10.3389/fpsyg.2014.00823. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Cechnicki A, Bielańska A, Hanuszkiewicz I, Daren A. The predictive validity of expressed emotions (EE) in schizophrenia: A 20-year prospective study. Journal of Psychiatric Research. 2013;47(2):208–214. doi: 10.1016/j.jpsychires.2012.10.004. [DOI] [PubMed] [Google Scholar]
  12. Cohen J. Statistical power analysis for the behavioral sciences. 2. Hillsdale, NJ: Erlbaum; 1988. [Google Scholar]
  13. Hagger MS, Wood C, Stiff C, Chatzisarantis NL. Ego depletion and the strength model of self-control: A meta-analysis. Psychological Bulletin. 2010;136(4):495. doi: 10.1037/a0019486. [DOI] [PubMed] [Google Scholar]
  14. Hooley JM. Expressed emotion and relapse of psychopathology. Annual Review of Clinical Psychology. 2007;3:329–352. doi: 10.1146/annurev.clinpsy.2.022305.095236. [DOI] [PubMed] [Google Scholar]
  15. Hooley JM, Gotlib IH. A diathesis-stress conceptualization of expressed emotion and clinical outcome. Applied and Preventive Psychology. 2000;9(3):135–151. [Google Scholar]
  16. Hooley JM, Parker HA. Measuring expressed emotion: An evaluation of the shortcuts. Journal of Family Psychology. 2006;20(3):386. doi: 10.1037/0893-3200.20.3.386. [DOI] [PubMed] [Google Scholar]
  17. Hooley JM, Richters JE. Alternative measures of expressed emotion: A methodological and cautionary note. Journal of Abnormal Psychology. 1991;100:94–97. doi: 10.1037//0021-843x.100.1.94. [DOI] [PubMed] [Google Scholar]
  18. Hooley JM, Rosen LR, Leach K, Wilson A. EE as a predictor of symptom relapse in schizophrenia. Harvard University; Cambridge, MA: 1999. Unpublished data. [Google Scholar]
  19. Ioannidis JP. Why most published research findings are false. PLoS Medicine. 2005;2(8):e124. doi: 10.1371/journal.pmed.0020124. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Ioannidis JP. Why most discovered true associations are inflated. Epidemiology. 2008;19(5):640–648. doi: 10.1097/EDE.0b013e31818131e7. [DOI] [PubMed] [Google Scholar]
  21. Ito J, Oshima I. Distribution of EE and its relationship to relapse in Japan. International Journal of Mental Health. 1995;24(2):23–37. [Google Scholar]
  22. Ivanovic M, Vuletic Z. Expressed emotion in families of patients with frequent types of schizophrenia and influence on the course of illness: Nine months follow-up. Paper presented at XIX Congress of the European Association of Behavioural Therapy; Vienna, Austria. Sep 20, 1989. [Google Scholar]
  23. Karno M, Jenkins JH, de la Selva A, Santana F, Telles C, Lopez S, et al. Expressed emotion and schizophrenic outcome among Mexican-American families. The Journal of Nervous and Mental Disease. 1987;175(3):143–151. doi: 10.1097/00005053-198703000-00004. [DOI] [PubMed] [Google Scholar]
  24. Köttgen C, Sönnichsen I, Mollenhauer K, Jurth R. Families’ high-expressed-emotions and relapses in young schizophrenic patients: Results of the Hamburg Camberwell-Family-Interview study: II. International Journal of Family Psychiatry. 1984;5:83–94. [Google Scholar]
  25. Leff J, Wig NN, Bedi H, Menon DK, Kuipers L, Korten A, et al. Relatives’ expressed emotion and the course of schizophrenia in Chandigarh: A two-year follow-up of a first-contact sample. The British Journal of Psychiatry. 1990;156(3):351–356. doi: 10.1192/bjp.156.3.351. [DOI] [PubMed] [Google Scholar]
  26. Linszen D, Dingemans PMAJ, van der Does JW, Nugter A, Scholte P, Lenior RGMJ, et al. Treatment, expressed emotion and relapse in recent onset schizophrenic disorders. Psychological Medicine. 1996;26(02):333–342. doi: 10.1017/s0033291700034723. [DOI] [PubMed] [Google Scholar]
  27. MacMillan JF, Gold A, Crow TJ, Johnson AL, Johnstone EC. Expressed emotion and relapse. The British Journal of Psychiatry. 1986;148(2):133–143. doi: 10.1192/bjp.148.2.133. [DOI] [PubMed] [Google Scholar]
  28. Miklowitz DJ, Goldstein MJ, Doane JA, Nuechterlein KH, Strachan AM, Snyder KS, et al. Is expressed emotion an index of a transactional process? I. Parents’ affective style. Family Process. 1989;28(2):153–167. doi: 10.1111/j.1545-5300.1989.00153.x. [DOI] [PubMed] [Google Scholar]
  29. Moline RA, Singh S, Morris A, Meltzer HY. Family expressed emotion and relapse in schizophrenia in 24 urban American patients. American Journal of Psychiatry. 1985;142(9):1078–1081. doi: 10.1176/ajp.142.9.1078. [DOI] [PubMed] [Google Scholar]
  30. Montero I, Ruiz I, Puche E, Adam A. The influence of family expressed emotion on the course of schizophrenia in a sample of Spanish patients: A two-year follow-up study. The British Journal of Psychiatry. 1992;161(2):217–222. doi: 10.1192/bjp.161.2.217. [DOI] [PubMed] [Google Scholar]
  31. Mozny P, Votýpková P. Expressed emotion, relapse rate and utilization of psychiatric inpatient care in schizophrenia. Social Psychiatry and Psychiatric Epidemiology. 1992;27(4):174–179. doi: 10.1007/BF00789002. [DOI] [PubMed] [Google Scholar]
  32. Niedermeier T, Watzl H, Cohen R. Prediction of relapse of schizophrenic patients: Camberwell Family Interview versus content analysis of verbal behavior. Psychiatry Research. 1992;41(3):275–282. doi: 10.1016/0165-1781(92)90009-r. [DOI] [PubMed] [Google Scholar]
  33. Nuechterlein KH, Snyder KS, Mintz J. Paths to relapse: Possible transactional processes connecting patient illness onset, expressed emotion, and psychotic relapse. The British Journal of Psychiatry. 1992;18(3):387–425. [PubMed] [Google Scholar]
  34. Parker G, Johnston P, Hayward L. Parental expressed emotion as a predictor of schizophrenic relapse. Archives of General Psychiatry. 1988;45(9):806–813. doi: 10.1001/archpsyc.1988.01800330030003. [DOI] [PubMed] [Google Scholar]
  35. Pashler H, Harris CR. Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science. 2012;7(6):531–536. doi: 10.1177/1745691612463401. [DOI] [PubMed] [Google Scholar]
  36. Phillips MR, Xiong W. Expressed emotion in mainland China: Chinese families with schizophrenic patients. International Journal of Mental Health. 1995;24(3):54–75. [Google Scholar]
  37. Rostworowska M, Barbaro B, Cechnicki A. The influence of expressed emotion on the course of schizophrenia: A Polish replication. Poster presented at the 17th Congress of the European Association for Behaviour Therapy, Amsterdam; 1987. Aug, pp. 26–29. [Google Scholar]
  38. Simmons JP, Nelson LD, Simonsohn U. False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science. 2011;22(11):1359–1366. doi: 10.1177/0956797611417632. [DOI] [PubMed] [Google Scholar]
  39. Simonsohn U, Nelson LD, Simmons JP. P-curve: A key to the file-drawer. Journal of Experimental Psychology: General. 2014;143(2):534–547. doi: 10.1037/a0033242. [DOI] [PubMed] [Google Scholar]
  40. Stirling J, Tantam D, Thomas P, Newby D, Montague L, Ring N, Rowe S. Expressed emotion and early onset schizophrenia: A one year follow-up. Psychological Medicine. 1991;21(03):675–685. doi: 10.1017/s0033291700022315. [DOI] [PubMed] [Google Scholar]
  41. Strachan AM, Feingold D, Goldstein MJ, Miklowitz DJ, Nuechterlein KH. Is expressed emotion an index of a transactional process? II. Patient’s coping style. Family Process. 1989;28(2):169–181. doi: 10.1111/j.1545-5300.1989.00169.x. [DOI] [PubMed] [Google Scholar]
  42. Tanaka S, Mino Y, Inoue S. Expressed emotion and the course of schizophrenia in Japan. The British Journal of Psychiatry. 1995;167(6):794–798. doi: 10.1192/bjp.167.6.794. [DOI] [PubMed] [Google Scholar]
  43. Tarrier N, Barrowclough C, Vaughn C, Bamrah JS, Porceddu K, Watts S, Freeman H. The community management of schizophrenia: A controlled trial of a behavioural intervention with families to reduce relapse. The British Journal of Psychiatry. 1988;153(4):532–542. doi: 10.1192/bjp.153.4.532. [DOI] [PubMed] [Google Scholar]
  44. Vaughan K, Doyle M, McConaghy N, Blaszczynski A, Fox A, Tarrier N. The relationship between relative’s Expressed Emotion and schizophrenic relapse: An Australian replication. Social Psychiatry and Psychiatric Epidemiology. 1992;27(1):10–15. doi: 10.1007/BF00788950. [DOI] [PubMed] [Google Scholar]
  45. Vaughn CE, Leff JP. The influence of family and social factors on the course of psychiatric illness: A comparison of schizophrenic and depressed neurotic patients. The British Journal of Psychiatry. 1976a;129(2):125–137. doi: 10.1192/bjp.129.2.125. [DOI] [PubMed] [Google Scholar]
  46. Vaughn C, Leff J. The measurement of expressed emotion in the families of psychiatric patients. British Journal of Social and Clinical Psychology. 1976b;15(2):157–165. doi: 10.1111/j.2044-8260.1976.tb00021.x. [DOI] [PubMed] [Google Scholar]
  47. Vaughn CE, Snyder KS, Jones S, Freeman WB, Falloon IR. Family factors in schizophrenic relapse: Replication in California of British research on expressed emotion. Archives of General Psychiatry. 1984;41(12):1169–1177. doi: 10.1001/archpsyc.1984.01790230055009. [DOI] [PubMed] [Google Scholar]

RESOURCES