Are Community Studies of Psychological Trauma’s Impact Accurate? A Study among Jews and Palestinians

Abstract

We evaluated the accuracy of posttraumatic stress disorder (PTSD) and major depression (MD) diagnoses using brief assessment instruments conducted by phone. PTSD and MD were assessed by telephone interview in a randomly selected sample of Jewish and Palestinian residents of Jerusalem during a period of marked threat of terrorism and war (N=150). We utilized the PTSD Symptom Scale (Foa et al., 1993) and the Patient Health Questionnaire-9 (PHQ-9; Kroenke et al., 2001). We then conducted in-depth, in-person interviews within two weeks, assessing PTSD and MD using the Composite International Diagnostic Interview (CIDI; Kessler et al., 2004). The prevalence of PTSD and MD diagnosis ascertained by the two assessment modalities was similar. Indices of classification accuracy for the phone interview, using the in-person interview as the standard, ranged from modest to high. Brief phone and in-depth in-person measures of PTSD and MD also correlated similarly with other demographic, stress, and coping factors, suggesting convergent validity. Brief phone interviews appear useful for estimating the prevalence of psychological disorders in mass casualty contexts and may have a critical role in both epidemiologic work and guiding public health interventions.

Following the spread of terrorism around the world, and a spate of devastating natural disasters, there has been a rapid increase in research on communities affected by terrorism and mass casualty events (Bleich, Gelkopf, & Solomon, 2003; Bonanno, Galea, Bucciarelli, & Vlahov, 2006; Galea et al., 2002; Kessler et al., 2008; Miguel-Tobal et al., 2006; Norris et al., 2002; Schlenger et al., 2002; Schuster et al., 2001; Silver, Holman, McIntosh, Poulin, & Gil-Rivas, 2002). Such research is vital to our understanding of the public mental health impact of mass casualty and to assessing mental and physical health needs after these events (Kessler et al., 2008). Given the challenges that populations face in the aftermath of mass casualty events, brief, validated instruments that can assess psychopathology quickly and efficiently are essential (Connor, Foa, & Davidson, 2006).

In an important commentary on the topic, North and Pfefferbaum (2002) outlined key recommendations and criticisms for methodology of research on mass casualty events. They specifically suggested that high rates of reported PTSD and MD may be exaggerated by the superficial study methods. Principal among their concerns was that frequently used phone surveys, which almost exclusively rely on brief instruments, might be unreliable and invalid. Such brief surveys are typically administered by lay interviewers who are minimally trained and contain few or no probes to clarify the presence of psychological symptoms. Further, their brevity is achieved by asking a few comprehensive questions, rather than detailed multiple questions that are more nuanced. Because there have been literally hundreds studies using brief symptom and diagnostic assessment instruments (Maguen, Papa, & Litz, 2008; Norris et al., 2002), examining their validity addresses one of the foundational assumptions for a wide array of community research.

To examine the reliability and validity of phone interview methodology that relies on brief assessments of PTSD and depression in situations of community trauma and mass casualty threat, we compared a brief phone-based interview using brief scales to an intensive in-person structured diagnostic assessment of the same individuals. For the in-person structured assessment, we used the Composite International Diagnostic Interview (CIDI, Kessler et al., 2004), a well-established instrument for the epidemiological study of PTSD and depression in regions outside of the United States. We focused on the key issue of the assessment of diagnostic criteria, not mere presence of symptoms. We also compared both phone-based self-report and CIDI measures of PTSD and MD with other demographic, stress, and coping variables to examine whether the brief, phone-based and CIDI measures of PTSD and MD correlated similarly with these variables, as they should if the brief, phone-based and CIDI measures were accurately assessing similar constructs.

Method

We interviewed by phone and then in-person a randomly selected adult sample of Israeli citizens, Jewish and Arab (they refer to themselves as Palestinian) residents of Jerusalem and the immediate surrounding contiguous communities (e.g., Jerusalem metropolitan area). We first interviewed residents by phone using only brief measures, as commonly used in phone surveys. Then within a two week maximum time span, we re-interviewed the same individuals at their homes or in locations convenient to them using the Composite International Diagnostic Interview (CIDI; Kessler & Ustun, 2004), and the one brief general health item from the CIDI.

Procedures

Data for the demographic and all brief questionnaire measures (except the one item health question) were collected through telephone interviews with a random sample of Israeli residents between August 17^th and September 8^th, 2007. The institutional review boards of Kent State University, the University of Haifa, the University of Miami, and Rush University Medical Center approved the study, and oral and written informed consent was obtained from study participants.

Phone interviews were completed by skilled interviewers with extensive experience. They had undergone over 20 hours of interview training and ongoing supervision, and were part of a survey institute’s regular staff. Interviewers were fully fluent in the language in which interviews took place, shared the culture of the interviewees, and were trained in cultural sensitivity and interviewing skills. Home interviews were conducted by junior clinicians, enrolled in clinical training programs in psychology and social work. They were trained for over 40 hours and received ongoing supervision by CIDI-qualified trainers and spent many hours practicing interviews. Training focused on motivating full interviewee responses, not straying from the prescribed CIDI protocol, and projecting warmth and openness to ensure interviewee maximal comfort. CIDI-qualified trainers are trained at the University of Michigan in a multi-day program and must already have certified interview training. This is consistent with the international standards and protocol for training of CIDI interviewers (CIDI training, 2010).

Phone numbers were randomly dialed from among a list of residents of Jewish and Palestinian neighborhoods in Jerusalem and the contiguous suburbs of Jerusalem and adjacent villages. For the Jewish sample, contact was established with a random sample of 470 people by phone in order to achieve the target sample of 75 people to be interviewed both by phone and in their homes with the two-week time period. Although all people in this region had potential situational exposure to terrorism, this was not a criterion for participation in order to achieve an array of levels of exposure. For the Palestinian sample, a random sample of 450 people was contacted in order to achieve the same target sample of 75 participants. The final sample comprised 75 Jews, and 75 Palestinians who participated in both the phone and in-person interviews within a maximum two week time period. Interviews were conducted by trained native speakers and participants were paid the equivalent of $50 U.S.D. Phone interviews lasted about 25 minutes and home interviews required about two hours.

Study Instruments

Demographic variables included participants’ sex, age in years, educational attainment (less than high school, high school graduate, some college, and college graduate), income, and ethnicity (Jew, Palestinian). This information was collected during the phone interviews and rechecked during the home interviews.

The brief phone assessment of PTSD utilized the PTSD Symptom Scale interview format (PSS-I; Foa, Riggs, Dancu, & Rothbaum, 1993) assessing symptoms that were present for at least a month related to specific exposure to a terrorist attack. This instrument was found to have 94 percent agreement with diagnosis when compared to the Structured Clinical Interview for DSM-IV (SCID), a clinician administered structured clinical interview (Foa et al., 1993). The items assess symptom criteria indicated by Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV-TR) criteria for determining PTSD (Association, 2000). Item responses were made on a 4-point scale (0 = not at all to 3 = extremely) with scores of 1 or higher qualifying as present symptoms. Current PTSD was defined as the presence of at least one reexperiencing symptom (e.g., intrusive memories or distressing dreams), three avoidance symptoms (e.g., efforts to avoid thoughts or activities associated with the trauma), and two symptoms of hyperarousal (e.g., difficulty falling asleep or concentrating) in the past month. In addition, to meet criteria, respondents had to endorse that these symptoms were interfering with their functioning at home or at work.

Brief phone assessment of major depression (MD) relied on the 9-item Patient Health Questionnaire-9 (PHQ-9), a well-validated, highly sensitive instrument for identifying individuals with current MD (Kroenke, Spitzer, & Williams, 2001) used in Israeli Palestinian and Jewish populations (Hall, Hobfoll, Canetti–Nisim, Johnson, & Galea, 2009; Hall et al., 2010; Hobfoll, Canetti-Nisim, & Johnson, 2006; Hobfoll et al., 2008; Palmieri, Canetti-Nisim, Galea, Johnson, & Hobfoll, 2008) Items were rated on a 4-point likert scale (0 = not at all, 1 = several days, 2 = more than half the days, 3 = nearly every day). To be classified as having MD consistent with DSM-IV-TR, (Association, 2000) participants had to endorse at least five symptoms of major depression, including depressed mood or a lack of interest or pleasure in activities for more than half of the days in the past two-weeks, and endorse impairment in their functioning at home or at work.

For a structured clinical assessment of PTSD and MD, we utilized the CIDI (Kessler & Ustun, 2004). It is one of the most widely used, and the most widely translated, structured interviews for assessment of diagnosis of PTSD and depression. It has been validated in Hebrew and this region’s Arabic, which are the versions we utilized. The CIDI has been found to compare well with a clinician administered structured interview (SCID), showing diagnostic agreement with the SCID of between .8 to .9 across disorders (Haro et al., 2006; Kessler et al., 2004). The CIDI-PTSD assessment contains 281 items, as it follows through multiple traumas that may have occurred. Not all items are queried, depending on how root questions are answered. Symptoms are keyed to specific exposures, or multiple exposures. We targeted the Criterion A event(s) of exposure to political violence, terrorism or war, to make it comparable to the Criterion A event from the PSS. The CIDI-MD assessment includes 87 items, with many sub-items that are queried following certain responses in the root question. Diagnosis follows a specific algorithm set by the test originators. A major aspect of the CIDI is that all follow-up questions are built in to the scale, and that interviewers are trained not to deviate from the many item roots and stems that were specifically designed to insure uniformity.

Self reported physical health was measured using a single item from the CIDI interview. Participants were asked how they would rate their overall physical health. Response options ranged from 1 = poor to 5 = excellent.

Perceived self-efficacy was measured using the General Self-Efficacy scale (Schwarzer, Bäßler, Kwiatek, Schröder, & Zhang, 1997). This 10-item scale measures the self-perception that one can perform novel or difficult tasks, or cope with adversity in various domains of human functioning. Items were scored on a 4-point scale (1 = not at all true to 4 = exactly true) and were summed. Cronbach’s alpha for the scale was .93.

Psychosocial resource loss related to terrorism and war exposure was measured using a 9-item scale from the Conservation of Resources Evaluation (COR-E; Hobfoll & Lilly, 1993). This scale has been used previously in studies of terrorism in Israel (Hall et al., 2009; Hall et al., 2010; Hall et al., 2008; Hobfoll et al., 2006; Palmieri et al., 2008) and in the United States (Norris, 2001) and was found to be predictive of PTSD and MD. Sample items include: “Feeling that you are a person of great value to other people,” “stability of your family,” “intimacy with at least one friend,” “the feeling that you are a successful person,” “Sense of control in your life,” and “Hope.” For all items participants indicated the degree of their resource loss on a 4-point scale (1 = did not lose at all to 4 lost very much).

Stressful life events were assessed by asking participants whether they have experienced other difficult events during the past year (which were not related to rockets or terror attacks), including death, disease or injury of a close person, divorce, or a loss of work place/employment.

Social support from spouse and family were each assessed using one item from the Support Satisfaction Questionnaire (Sarason, Sarason, Shearin, & Pierce, 1987). The two items were prefaced with “how satisfied are you with the social support you receive from your… spouse/partner, family. Items were rated from 0 = not at all satisfied to 3 = very satisfied.

Results

PTSD Diagnosis

The prevalence of PTSD diagnosis in the brief phone interview assessment format compared to the structured clinical interview is shown in Table 1. Utilizing Z-tests for two proportions revealed that the prevalence estimates were not statistically different from one another. The prevalence of PTSD diagnosis in brief phone interview assessment format compared to structured, in-person CIDI interview format was 14% versus 15% for the full sample (z = 0.08, p = .94), 9% versus 7% for Jews (z = 0.15, p = .88), and 19% versus 23% for Palestinians (z = 0. 0.71, p = .69). Sensitivity for the full sample was .55, for Jews it was .80 and for Palestinians it was .47. Specificity for the full sample was .93, .96 for Jews, and .90 for Palestinians. The Positive predictive power (PPP) was .57 for the full sample, and for Jews and Palestinians. The Negative predictive value (NPV) was .92 for the full sample, .99 for Jews, and .85 for Palestinians. The overall correct classification rate was 87% for the full sample, 95% for Jews, and 67% for Palestinians. Kappa for the full sample was .48 (p < .001), .64 (p < .001) for Jews, and .39 (p < .001) for Palestinians. This indicates moderate agreement for the full sample, substantial agreement for Jews, and fair to moderate agreement for Palestinians (Landis & Koch, 1977).

Table 1.

Prevalence and classification accuracy for PTSD and MD for phone compared to CIDI interview

	Phone prevalence		In-person prevalence		Sensitivity	Specificity	Positive predictive power	Negative predictive power	Correct classification rate
PTSD	%	(N)	%	(N)
Total Sample	14%	(21)	15%	(22)	.55	.93	.57	.92	87%
Jews	9%	(7)	7%	(5)	.80	.96	.57	.99	95%
Palestinians	19%	(14)	23%	(17)	.47	.90	.57	.85	67%
MD
Total Sample	24%	(36)	29%	(43)	.70	.94	.83	.89	87%
Jews	20%	(15)	24%	(18)	.61	.93	.73	.88	85%
Palestinians	28%	(21)	33%	(25)	.76	.96	.90	.89	87%

Note: PTSD = posttraumatic stress disorder. MD = major depressive disorder. Classification accuracy statistics use the in-person interview as the criterion.

Several PTSD symptom items had poor concordance when all items were examined among Palestinians. Utilizing z-tests for comparing proportions, the following items were identified as significantly divergent for CIDI versus phone interview: Recurrent and intrusive distressing recollections of the event (CIDI = 32.9%/Phone = 16.4%; z = 0.2.11, p = .03; efforts to avoid thoughts, feelings, or conversations associated with the trauma (35.6%/15.1%; z = 2.67, p = .007); markedly diminished interest or participation in significant activities (18.3%/49.3%; z = 3.78, p < .001); restricted range of affect (16.4%/33.8%; z = 2.28, p = .02); irritability or outbursts of anger (30.7%/59.5%; z = 3.47, p < .001); difficulty concentrating (29.7%/48.6%; z = 2.19, p = .03); and hypervigilance (22.9%/52.7%; z = 3.56, p < .001).

MD Diagnosis

Prevalence of MD diagnosis in the brief phone interview format compared to structured, in person CIDI interview format was 24% versus 29% for the full sample (z = 0.85, p = .40), and 20% versus 24% for Jews (z = 0.70, p = .48) and 28% versus 33% for Palestinians (z = 0.81, p = .42), indicating that the prevalence estimates were not statistically for the two formats for MD (see Table 1). Sensitivity for the entire sample was .70, for Jews was .61 and for Palestinians was .76. Specificity was .94 overall, .93 for Jews and .96 for Palestinians. PPV was .83 for the entire sample, .73 for Jews and .90 Palestinians. NPV was high for the entire sample, .89, for Jews .88 and Palestinians .89. The overall correct classification rate was 87% overall, 85% for Jews, and 87% for Palestinians. Kappa for the entire sample was .67 (p < .001), for Jews was .57 (p < .001) and for Palestinians was .75 (p < .001). This indicates substantial overall agreement, moderate to substantial agreement for Jews and substantial agreement for Palestinians (Landis & Koch, 1977).

Comparable Convergent and Discriminant Relationships

Utilizing point-biserial correlations, we next compared the PTSD and MD diagnoses from the brief phone interviews and the CIDI with demographic, personal, social, and situational variables in order to examine whether the associations were similar to establish the comparability of convergent and divergent relationships. The question here is not whether PTSD and MD were correlated in a certain direction with these other variables, but whether the relationships were similar. Indeed, we would hope for similar relationships, even if that similarity reflected non-significant relationships for a given variable. As noted in Table 2, similar levels of correlations were found for both PTSD and MD in the two assessment formats with sex, age, education, income, self-assessed physical health, self-efficacy, spousal support, support from family, stressful life events, and loss of psychosocial resources. The magnitude of these relationships was similar for the demographic variables, with one exception for age. Older individuals had significantly greater PTSD on the phone assessment of PTSD, but PTSD and age were not significantly correlated on the CIDI assessment of PTSD. For the non-demographic variables, the brief phone and structured interview assessments were similarly correlated with them. In the few instances where statistical significance differed, the magnitude of the differences in the correlations were quite small, and when these correlations were subjected to comparisons using Fisher’s z-tests, none of the correlations were statistically different. Results of these comparisons are found in Table 2.

Table 2.

Correlations between brief-telephone and in-person assessments of PTSD and MD and demographic, health, self-efficacy, social support, and stress variables N = 150.

Variable	PTSD		Fisher’s z-test	MD		Fisher’s z-test
	CIDI	PSS-I		CIDI	PHQ-9
Sex (women)	−.027	.034	.052	.065	.007	.31
Age	.028	.193*	−1.44	−.069	−.072	.03
Education	−.170	−.103	−.59	−.021	−.062	.35
Income	.010	−.069	.68	.079	.049	.26
Overall physical Health (CIDI)	−.085	−.098	.11	−.233**	−.226**	−.06
Self-Efficacy	−.149	−.131	.43	−.130	−.192*	.55
Spousal Support	−.131	−.086	−.39	−.190*	−.194*	.04
Family Support	−.075	−.137	.54	−.177	−.144	−.29
Stressful events	.213**	.157	.50	.189*	.247**	−.52
Resource loss	.458**	.393**	.68	.043	.174*	−1.14

Note: PTSD = posttraumatic stress disorder. MD = major depressive disorder. PSS-I = Posttraumatic stress disorder symptom scale-interview format. PHQ-9 = patient health questionnaire – 9. CIDI = Composite International Diagnostic Interview. Phone = phone assessment.

^*p < .05.

^**p < .01.

Discussion

Overall, our results for the brief phone assessments compared favorably with the structured face-to-face CIDI interviews, suggesting that reports of high rates of PTSD and MD in community trauma studies are both reasonably accurate, and not exaggerated, as has often been suggested in the literature. The brief and in-depth diagnostic tools we examined resulted in comparable prevalence levels. Using the in-person interview as the standard, the phone interview showed good overall classification rates. Further, whereas some have feared that brief surveys might be at fault by overestimating psychopathology (North & Pfefferbaum, 2002), our results suggest instead that they result in prevalence rates for both PTSD and MD that were not statistically different.

A good standard for evaluating our results is comparing them to validation studies of brief instruments where they were compared to in-depth structured clinical assessments. Our results for PTSD in the Jewish subsample compare favorably to studies that compared the PSS in its various forms to the SCID and CIDI (Engelhard et al., 2007; Foa et al., 1993; Foa & Tolin, 2003; Wohlfarth et al., 2003). The results for Palestinians are more equivocal, and there is clearly less agreement on PTSD for brief phone interviews versus structured clinical interviews. For MD, the results for our brief phone survey compare favorably to studies where the PHQ-9 was compared to structured clinical interviews (Corapcioglu & Ozur, 2004; Lowe et al., 2004; Spitzer et al., 1999; Stafford et al., 2007). Sensitivity was somewhat lower for Jews in our study compared to prior studies, but specificity was comparable.

When we compared the relationships between our interview measures and other variables that have been studied in the past in trauma and mass casualty research, we found a similar pattern of correlations. Although some of these relationships are low, they are similar also to what is typically found. This suggests that the two methodologies are generally assessing the same constructs, despite disparate methods (Campbell & Fiske, 1959). This further attests to the functional usefulness of brief phone assessments of PTSD and MD and supports their use for theory and model testing.

It is also notable that our test conditions were highly demanding, with people confronted by terrorism, war, and economic deprivation. These conditions probably make individuals more volatile, which would also increase variability of their reactions at different times. As logistics demanded that re-interviews were on average a week apart, changes in PTSD and MD symptoms may have occurred between the two assessments. Both these factors might decrease the comparability of our two assessment formats, suggesting that our test is more stringent than ones conducted for prior studies where brief and structured interviews were given together.

Lower overall classification rates for PTSD for Palestinians compared to Jews (.67 versus .95) were not expected. Given that PTSD is directly related to traumatic political circumstances, Palestinian respondents may have been more reticent to answer with full disclosure in the more superficial phone interview, resulting in lower prevalence rates in this instance. Another possibility is that PTSD is a newer construct than depression, and has entered people’s cultural dialogue and culture less among Arabs. As such, the extra time and follow-up questioning taken by home interviewers in the CIDI’s highly detailed format may have been critical for PTSD.

Limitations and Strengths

This study has several limitations. The results are limited to the instruments we used and cannot be generalized to other disorders. Although studying Jews and Palestinians in Israel increases the strength of the study, as the results are robust across cultures where these tests did not originate, they must also be examined in other cultural and language contexts. The ongoing dialogue about trauma in the Israeli media may actually make individuals more aware of how they feel, and as such the results may not be generalized to non-chronic traumatic contexts were people have not been encouraged to think about their feelings. In this regard, culture and context both influence reporting of symptoms. Further, given that the phone and in-person interviews differed by the measure used, the exact time frame, and method of administration, we cannot pinpoint which of these contributed to any lack of concordance. Finally, although we believe that our sample size was reasonable for the questions we posed in this study, a larger sample size would have allowed testing of individual items and clusters of items that comprise both PTSD and MD. Also, although many people decided not to participate in the study, our range and levels of PTSD and MD are similar to those we found in large-scale epidemiological studies in Israel (Hobfoll et al., 2008; Palmieri et al., 2008). Hence, although the sample may have self-selected on some unknown factor, they represent a range of psychopathology that is typical of the larger population.

Conclusions

Although brief phone assessments of PTSD and MD may not be psychometrically equivalent with structured clinical interviews, our results suggest that even in mass casualty situations, and across cultures, brief phone assessments are good indicators of trends in psychopathology. As such they are useful and valuable measures of pathology in theoretical research on the impact of mass casualty and as a basis for making public health estimates as to the impact of mass casualty. This makes them constructive, practical tools for both research and intervention planning.

Building on past studies that found that telephone interviews produce similar results to in-person interviews (Aziz & Kenford, 2004), the current study advances the point that this also is the case when brief surveys administered by phone are compared to in-depth, in-person interviews. This finding is important, as it supports the use of brief measures in large-scale epidemiological research, as well as public health assessments made within mass casualty contexts. Brief phone assessments provide accurate estimates of diagnostic prevalence for PTSD and MD.