Skip to main content
PMC home page
VA Author Manuscripts logoLink to VA Author Manuscripts
. Author manuscript; available in PMC: 2024 Nov 11.
Published before final editing as: Psychol Trauma. 2023 May 11:10.1037/tra0001491. doi: 10.1037/tra0001491

Development and Refinement of the Mobile Anger Reduction Intervention for Veterans with Posttraumatic Stress Disorder

Kirsten H Dillon 1,2,3, Jeffrey A Hertzberg 1,2, Tiffany M Mosher 1,2, Ryan Levi 1,2, Eric B Elbogen 1,2,3,4, Patrick S Calhoun 1,2,4,5, Leslie A Morland 6,7, Jean C Beckham 1,2,3
PMCID: PMC10638458  NIHMSID: NIHMS1876828  PMID: 37166919

Abstract

Objective:

Problematic anger is commonly reported among veterans with posttraumatic stress disorder (PTSD) and is associated with numerous psychosocial impairments. There is a clear need to develop innovative and effective anger interventions. One of the cognitive mechanisms associated with anger is the hostile interpretation bias, which is the tendency to interpret ambiguous interpersonal situations as hostile. The current study presents a successive cohort design methodology to develop and refine a mobile treatment application, entitled Mobile Anger Reduction Intervention (MARI), which uses interpretation bias modification techniques to modify hostile interpretation bias.

Method:

Two cohorts (total N = 13) of veterans with PTSD and problematic anger used the MARI application for four weeks. After each cohort, qualitative and quantitative data were used to modify the MARI application. The intervention is described, as well as the qualitative and quantitative findings and subsequent changes made to the mobile application based on participant feedback.

Results:

Treatment adherence was high (90% of participants completed all sessions). Participants reported that they found the treatment helpful and easy to use, and experienced improvements in hostile interpretation bias and problematic anger.

Conclusions:

This study demonstrates the utility of a successive cohort treatment design for development of mobile interventions.

Keywords: posttraumatic stress disorder, anger, veterans, mobile interventions, interpretation bias modification

Clinical Impact Statement

Veterans with posttraumatic stress disorder often experience difficulty controlling their anger, which can negatively affect their lives. Many individuals with problematic anger have a tendency to think that others are behaving in a hostile way, when that it not necessarily the case. We have developed a mobile intervention (MARI) to reduce this tendency by helping veterans make benign interpretations to various situations. This study presents data from two groups of veterans who used MARI for four weeks and provided feedback on their opinions about the intervention. After each group used the intervention, changes were made to its design and content.

Problematic anger is the most commonly reported reintegration concern among combat veterans, especially those with posttraumatic stress disorder (PTSD; Sayer et al., 2010). Among veterans, anger has been associated with family discord (Galovski & Lyons, 2004), negative social and occupational outcomes (Frueh et al., 1997), aggression (Novaco & Chemtob, 2015), nonsuicidal self-injury (Dillon et al., 2021) and suicide risk (Doran et al., 2017;Wilks et al., 2019). In a survey of Iraq-Afghanistan-era veterans more than 80% of veterans with PTSD endorsed a desire for anger management treatment (Crawford et al., 2015). Veterans with PTSD and problematic anger are often offered evidence-based, trauma-focused treatments for PTSD (e.g., prolonged exposure, cognitive processing therapy). The effect sizes of these treatments on anger outcomes are often modest (Glassman et al., 2020; Resick et al., 2008; Schnurr & Lunney, 2015), and many patients continue to report clinically significant residual anger symptoms (Miles et al., 2020; Schnurr & Lunney, 2019). For example, across three studies that examined anger outcomes from trauma focused treatments, effect sizes were in the small to medium range (Cohen’s d = .23-.63; Glassman et al., 2020; Resick et al., 2008; Schnurr & Lunney, 2015).

Many of the evidence-based treatments for anger are based on cognitive-behavior therapy (CBT) and meta-analytic reviews of these interventions have found medium to large effect sizes for their effects on anger (Lee & DiGiuseppe, 2018). Trials of CBT for anger among veterans with PTSD have also found medium to large effect sizes (Morland et al., 2010; Shea et al., 2022; Van Voorhees et al., 2021). However, attrition rates for the CBT conditions have been as high as 50% (Shea et al., 2022) and 44% (Van Voorhees et al., 2021). The fact that so many veterans with PTSD desire anger management treatment (Crawford et al., 2015) and that attrition rates are so high, indicates that we may need to adjust the method or means by which these treatments are delivered in order meet the needs of our veterans. Veterans may be hesitant to see a therapist due to a variety of concerns (e.g., financial concerns, scheduling difficulties, fear of social stigma, privacy concerns). Additionally, many of the treatments discussed above (e.g., trauma-focused treatment, CBT for anger) require significant time and effort (i.e., usually consisting of twelve 60–90-minute weekly sessions, with between-session practice assignments) that may make it difficult for veterans to fully adhere. Offering veterans an effective, convenient, and efficient mobile intervention may help to overcome some of the barriers that have thus far kept them from engaging in, or benefitting from, more traditional therapy services.

Individuals with problematic anger tend to interpret ambiguous interpersonal situations as hostile, known as the hostile interpretation bias (Wilkowski & Robinson, 2008, 2010). Interpretation bias modification (IBM) techniques have been used to modify this cognitive bias in civilian samples without PTSD (Cougle et al., 2017; Hawkins & Cougle, 2013; Smith et al., 2018). IBM has primarily been delivered via computer and essentially helps participants to adopt more adaptive interpretational styles through repeated practice resolving ambiguous situations in a benign way. Recently, we have modified this intervention for veterans with PTSD and problematic anger (Dillon et al., 2020). In a small sample of veterans with PTSD and a high level of anger (N = 7), eight 15-min sessions of IBM, delivered via computer, helped reduce hostile interpretations and anger (Cohen’s d effect sizes = 1.03–1.96). Many of the participants in this pilot study indicated that veterans might be particularly interested in receiving treatments like this via mobile applications. Mobile interventions such as this may be used in populations of veterans who are not ready or willing to engage in more formal treatments or as an adjunctive treatment to existing trauma-focused interventions. Efficacy of a brief, mobile intervention to reduce anger could potentially have a significant impact on the anger and functioning of veterans with PTSD.

The goal of the current study was to develop and refine a mobile intervention that utilizes the same IBM task, entitled Mobile Anger Reduction Intervention (MARI). A successive cohort design was used, enabling iterative improvements of the intervention design based on qualitative data (Epstein et al., 2007). By eliciting feedback on the intervention from the participants for whom it was designed to help, we are confident that we will be able to design an intervention that is more likely to meet the needs of veterans. Our primary aim was to refine the intervention over two cohorts in order to finalize the intervention for larger scale evaluation. As such, the MARI intervention will be described, as well as the modifications made to the application.

Method

Design

The successive cohort design (SCD; Epstein et al., 2007) is an iterative process by which psychosocial treatments in the early stages of development are systematically refined and modified. In SCD, after theoretically appropriate treatment is selected, treatment elements are identified, and the initial treatment materials are developed. These materials are then tested in small cohorts of patients, with iterative revisions conducted after each cohort. Qualitative and quantitative data collected from these small cohorts is used to inform these revisions. Accordingly, we refined the treatment approach with two successive cohorts of veterans with PTSD and problematic anger. After the first cohort used the application for four weeks, each participant was interviewed and a qualitative and quantitative evaluation was conducted. Revisions were then made based on these evaluations before the next small cohort was treated. Again, following the second cohort an evaluation was completed and further revisions were made to the treatment.

Participants and Procedures

This study was approved by the Durham VA Health Care System and Duke University Medical Center Institutional Review Boards. Individuals enrolled in the Durham VA Health Care System with a diagnosis of PTSD were sent a letter introducing the study. This letter was followed by a phone call at least a week later to present the study and screen interested participants for eligibility. Participants who passed the phone screen and were interested in the study, attended an in-person or video teleconferencing visit to establish eligibility. Inclusion criteria were: (1) current DSM-5 PTSD (established by the Clinician-Administered PTSD Scale for DSM-5 [CAPS-5]; Weathers et al., 2013); (2) elevated anger, established by ≥ 12 on the 5-item Dimensions of Anger Reactions Scale (DAR-5; Forbes et al., 2014; and (3) reading ability of at least 6th grade level (ascertained during the informed consent process based on observation of whether the veteran was able to independently read the consent form). Exclusion criteria were: (1) expected to be unstable on their medication regimen during the study; (2) currently in a period of active psychosis or mania; (3) current prominent suicidal or homicidal ideation requiring immediate intervention; (4) receiving other anger-management psychotherapy during the study; (5) receiving evidence-based, trauma-focused therapy for PTSD (i.e., prolonged exposure therapy, cognitive processing therapy) during the study; or (6) being unable to complete the study procedures. In total 17 participants consented and completed screening procedures. Four were excluded for reasons that included not meeting criteria for PTSD (n = 2), not reporting elevated anger (n = 1), and being unable to complete study procedures (n = 1).

There were 13 participants enrolled in the study (Cohort 1, n = 5; Cohort 2, n = 8). Ten participants were men and three were women. Seven participants identified as Black, six identified as White, and one identified as multi-racial. Participants ranged in age from 40 to 74 (M = 58.62; SD = 11.09). They had a range of education levels, including high school (n = 2), some college (n = 3), an associate’s degree (n = 3), a bachelor’s degree (n = 2), and a graduate degree (n = 3). Three participants worked full-time, four worked part-time, and six were not working. The mean score on the DAR-5 was 17.00 (SD = 3.51).

Of note, Cohort 2 recruitment and data collection was ongoing in March 2020 when the COVID-19 pandemic began. Two of the participants in this cohort were lost to contact at this time and did not respond to attempts to convert the study assessments to video teleconference or return their study equipment. The final three participants enrolled in Cohort 2 completed the study procedures entirely remotely (via telephone and video teleconference). Overall, 11 participants completed the study (n = 5 in Cohort 1 and n = 6 in Cohort 2). The remainder of this manuscript will focus on study completers, as we do not have qualitative data or treatment outcome data for the two lost participants.

Qualitative Feedback

Think-aloud.

A think-aloud protocol was used during participants’ initial use of the MARI intervention (Jääskeläinen, 2010). The think-aloud protocol involves having participants think aloud as they use the application to assess their information needs, problems they encounter with the application interface, and any user or design problems. This method has been successfully utilized in previous application development (Jaspers, 2009). There was a script that was developed for this assessment. Participants were told, “Now I’d like you to talk me through your exact process as you… [turn on the device/find the app/turn on the app/etc.]. Think aloud while you are doing [insert task here]. Be as detailed as possible, including every thought you have and everything that you are trying.” These instructions were repeated as the participants examined each part of the application, completed a practice assignment, and a diary entry. If participants had difficulty narrating, clarifying questions were asked as needed.

Post-treatment interview.

At the post-treatment assessment, each veteran completed a qualitative interview and provided feedback on their experience using the MARI application. An interview guide was used for these interviews. This interview included open-ended questions to assess what participants liked and disliked about the intervention, their thoughts about the duration and number of sessions, situations covered, perceived benefits of their participation, comfort level, overall satisfaction, and suggestions for improvement. After the interview, participants were also asked closed-ended questions, which assessed how helpful, easy to understand, and easy to use they found the treatment to be; these questions were answered using a scale of 1 (lowest acceptability) to 9 (highest acceptability).

Intervention

Mobile anger reduction intervention (MARI).

The IBM treatment works as follows (see Figure 1): Participants are shown ambiguous, anger-relevant scenarios and instructed to imagine themselves in the described situation. For example, “Someone steps on your foot in a crowded place.” Next, another sentence appears that offers a benign, non-threatening interpretation of the scenario. For example, following this scenario, the sentence reads, “It was an acci_ent.” The participants fill in the missing letter of the word, forming the word “accident,” and assigning the benign interpretation. This interpretation is then reinforced by answering “yes” or “no” to a comprehension question (i.e., “Did this person mean to hurt you?”). In this example, they must answer “no” to proceed. Participants are given three tries to enter the correct missing letter and yes/no question before the program advances. Across treatment sessions, unique scenarios are presented (i.e., participants do not see the same scenario twice). Scenarios have been developed to capture a wide range of different themes that are likely to be anger-provoking. Themes include physical aggression, driving situations, irritating traits of others, thinking you are being ignored by others, feeling argued with or criticized, thinking someone is stealing from you, having people block you, thinking that others have hostile feelings, feeling disrespected, thinking that people will not help you, thinking that others do not appreciate you, and thinking that situations are unfair.

Figure 1.

Figure 1

Open in a new tab

Sample training scenario from MARI application

The computer-based intervention consisted of 8 sessions, each with 64 training scenarios, which took 15 minutes for participants to complete (see Dillon et al., 2020). The MARI application was designed to deliver the intervention more frequently for shorter durations (i.e., 10 minutes five times weekly for four weeks). Shorter treatment sessions are likely to be more convenient for the mobile application users, leading to greater engagement with the application. Additionally, participants have access to 200 minutes of training, which is a larger dose than the existing 120-minute computerized intervention and may lead to greater reductions in hostile interpretation bias and associated functional and anger outcomes. The application was programmed to include 20 treatment sessions that each included 42 training scenarios and took approximately 10 minutes to complete. No scenario was repeated across these sessions. To this end, additional training scenarios were developed for a total of 840. Consistent with the previous scenario development (see Dillon et al., 2020), nine PTSD experts reviewed the new scenarios to confirm that the content was relevant to this patient population. An online readability calculator (https://readabilityformulas.com/) was used to confirm that all scenarios were at a 6th grade reading level or less.

The mobile application allows participants to access their treatment sessions, track their progress, and complete a nightly diary (see Figure 2). The “My Progress” portion of the application displays how many sessions they have completed and their progress across sessions (e.g., number of scenarios resolved, time spent). When they start using the mobile application, participants are prompted in the “Reminders” section to select which days and times of day would be most convenient for them to complete their treatment sessions. Based on their selected schedule, there is an option to program the application to send them reminders to complete sessions. Each night, they were prompted to complete a “Nightly Diary” entry in which they reported for that day: 1) whether they used the application; 2) if used, how easy they found the application to use (scale of 0–10); 3) whether anything got in the way of using the application (checklist of barriers to select); 4) what their stress level was (0–10); and 5) how angry they felt (0–10).

Figure 2.

Figure 2

Open in a new tab

Home screen, Reminders section, and My Progress section of the MARI application

Several additional tools were programmed to enhance participants’ engagement with the application and use of the intervention. Specifically, the application is programmed to send participants notifications periodically. For example, when they adhered to the suggested treatment schedule (five sessions/week), they received a congratulatory notification at the end of that week encouraging them to keep up the frequency of sessions. Alternatively, participants who had not used the application for several days received a notification reminding them to do treatment sessions and offering suggestions for increasing adherence (e.g., blocking out 10 minutes each day, setting up reminders). Participants also gained badges of achievement by completing treatment sessions. This type of reward system has been suggested to enhance motivation in mHealth interventions (Lewis et al., 2016).

Measures

PTSD diagnosis and psychiatric disorders.

The CAPS-5 (Weathers et al., 2013) was used to confirm current PTSD diagnosis and the Structured Clinical Interview for DSM-5 (SCID-5; First et al., 2015) was used to assess other psychiatric disorders. The CAPS-5 has demonstrated strong interrater and test-retest reliability, internal consistency, convergent validity, and discriminant validity among military veterans (Weathers et al., 2018). The SCID-5 has demonstrated excellent reliability and high specificity and validity (Osorio et al., 2019).

Hostile interpretation bias.

Baseline hostile interpretation bias and changes in hostile interpretation bias were assessed with the Word Sentence Association Paradigm- Hostility scale (WSAP-H; Dillon et al., 2016). The WSAP-H consists of 16 ambiguous sentences (e.g., “Someone is in your way”) followed by a hostile (e.g., “inconsiderate”) or a benign word (e.g., “unaware”). Participants are asked to rate the similarity of the sentence and the word on a 1 (not at all similar) to 6 (extremely similar) scale. Each sentence is presented twice, once with the benign word and once with the hostile word. Benign and hostile subscales are calculated by taking the mean ratings for each type of word. The score range for the benign and hostile subscales is 1–6, with higher scores indicating greater tendencies to assign a benign or hostile interpretation to ambiguous events, respectively. Across several samples, the WSAP-H has demonstrated convergent and divergent validity and internal consistency (Dillon et al., 2016). In the current sample, internal consistency was α = .79 at pre-treatment and α = .94 at post-treatment for the hostile interpretations and α = .76 at pre-treatment and α = .99 at post-treatment for the benign interpretations.

Anger.

The Dimensions of Anger Reactions scale (DAR; Novaco, 1975) was used to measure anger. The DAR is a 7-item self-report scale that assesses the frequency, duration, and behavioral response to anger and anger-related functional impairment on relationships, work, and health. Items (e.g., I often find myself getting angry at people or situations) are rated on a scale of 0 (not at all) to 8 (exactly so). Possible total scores ranged from 0 to 56, with higher scores indicating greater anger difficulties. In a large sample of treatment-seeking soldiers who had deployed to Iraq or Afghanistan, the DAR was found to have concurrent and discriminate validity and to correlate with functional impairment measures (Novaco et al., 2012). In the current sample, internal consistency was α = .84 at pre-treatment and α = .94 at post-treatment.

Treatment satisfaction.

The Client Satisfaction Questionnaire (CSQ; Larsen et al., 1979), was used to assess participants’ level of satisfaction with the intervention they received. The CSQ is a commonly-used, 8-item self-report measure. Each item is rated on a scale of 1 to 4 and items are summed to create a total score. Possible total scores ranged from 8 to 32, with higher scores indicating greater satisfaction with treatment. It has been found to demonstrate good validity and reliability across several populations (Attkinsson & Greenfield, 2004). In the current sample, internal consistency was α = .95.

Data Analysis

Quantitative data.

Quantitative analyses aimed at describing the data. Descriptive statistics were generated for sample characteristics pretreatment and posttreatment mean values for outcome variables. Given the small sample size, statistical comparisons between pre- and post-treatment outcomes were not appropriate.

Qualitative data.

The goal of qualitative analysis was to identify elements for improvement within the study design and the application. Qualitative analysis was completed at the end of Cohorts 1 and 2. At the conclusion of each cohort, the post-treatment qualitative interviews were analyzed using rapid turn-around analysis (Hamilton, 2013). Rapid turn-around analysis involves developing a summary template that includes a neutral domain name (i.e., topic) that corresponds to each interview question (e.g., Strengths of App, Weaknesses of App). Next, this summary template is completed for each participant interview, summarizing the information learned for each neutral domain. This information is then transferred into a matrix (respondent × neutral domain). This matrix helps to quickly identify similarities, variations, and gaps in the data.

The think aloud interviews were not formally analyzed, but notes were taken by the study team regarding frequent difficulties that participants encountered or comments that they made when trying to use the application for the first time. This information was used to modify the application to be more user-friendly and easier to use.

Results

Treatment Outcomes

The study’s primary goals were to refine the application and establish the acceptability and feasibility of the intervention and study procedures. Treatment satisfaction and usability data can be found in Table 1. Overall, participants reported that they found the application helpful, easy to use and understand, and that they were satisfied with the program. Ratings for some aspects of the application were lower than others. For example, the lowest rating was for the notifications received through the application. This may be partially explained by the fact that due to an error these were not displayed for the first cohort of participants.

Table 1.

Treatment satisfaction and usability.

Treatment Acceptability Mean (SD) Response Range Possible Range
Overall, how helpful did you find the application you used? 7.09 (1.58) 4–9 1–9
Specifically, how helpful did you find the application you used in helping you manage your anger? 6.36 (1.50) 4–9 1–9
How easy to understand was the application that you used? 8.18 (0.81) 7–9 1–9
How easy to use did you find the app? 8.36 (0.81) 7–9 1–9
How helpful did you find the My Progress feature of the application (in terms of viewing your progress)? 6.18 (1.78) 3–9 1–9
How helpful was it to schedule treatment sessions and receive reminders through the app? 6.60 (2.46) 3–9 1–9
How helpful did you find the Nightly Diary feature of the app? 6.40 (2.32) 2–9 1–9
How helpful did you find the notifications you received from the app? 5.33 (1.63) 3–7 1–9
How helpful were the points and badges earned to motivate you to use the app? 5.91 (2.34) 2–9 1–9
Client Satisfaction Questionnaire
How would you rate the quality of the mobile intervention you received? 3.09 (0.70) 2–4 1–4
Did you get the kind of treatment you wanted? 3.09 (0.70) 2–4 1–4
To what extent has our mobile intervention met your needs? 2.73 (0.65) 2–4 1–4
If a friend were in need of similar help, would you recommend our mobile intervention to him/her? 3.73 (0.47) 3–4 1–4
How satisfied are you with the amount of help you received? 3.36 (0.67) 2–4 1–4
Has the mobile intervention you received helped you to deal more effectively with your problems? 3.36 (0.81) 2–4 1–4
In an overall, general sense, how satisfied are you with the mobile intervention you received? 3.09 (0.70) 2–4 1–4
If you were to seek help again, would you come back to our program? 3.45 (0.52) 3–4 1–4
Open in a new tab

Mean ratings on the DAR and WSAP-H indicated change in the improved direction. Mean pre-treatment anger (measured by the DAR) went from 36.64 (SD = 10.61) to 32.00 (SD = 13.37) at post-treatment (Cohen’s d = 0.40). Mean pre-treatment benign bias (measured by the WSAP-H) increased from 3.53 (SD = 0.66) to 3.91 (SD = 1.75) at post-treatment (Cohen’s d = 0.21). Mean pre-treatment hostile bias (measured by the WSAP-H) decreased from 4.12 (SD = 0.80) to 2.93 (SD = 1.07) at post-treatment (Cohen’s d = 1.06). Of note, participants went from having greater mean hostile bias scores than benign bias scores to having higher benign bias scores than hostile scores, which represents a shift in their tendency to assign benign vs. hostile interpretations to ambiguous situations.

Cohort 1 Findings and Modifications

All participants in Cohort 1 completed all 20 treatment sessions within the four-week study period. When asked about the application’s strengths, the majority of the participants reported that the application was easy to use. One participant said, “it’s elementary… if you can count 1 2 3 4 5, you can go along with what’s going on,” and another said, “all of it was pretty much user friendly.” They also shared that the application had helped them to calm down and change how they thought about situations. For example, one participant shared, “it made you look at somebody from their point of view… they are upset or they are having a bad day, whereas before I probably would not have thought about what they are going through. I just looked at their actions and reacted on them.” Another participant said, “it helped me for the simple fact when you go through the application and then a situation happened and you tell yourself remember some of this just ain’t that bad. It’s not as bad as you are trying to make it… it’s not them, it’s my mind.” And one simply said, “it enlightened me a lot.” One participant stated that they liked the consistency and repetition of the treatment. Regarding weaknesses of the app, some participants noted some glitches with the app. For example, if the nightly diary was completed after midnight, they would not be able to fill it out again the following night. One participant reported that they thought that they were supposed to do the treatment sessions at the same time each day (when the reminder went off), which had been disruptive to their schedule. Although some participants had appreciated the repetition of the scenarios, some found this to be a weakness. Participants had several suggestions for improvement, including: 1) giving the option of a few letters for the fill in the blanks, rather than the whole keyboard; 2) emphasizing that the sessions can be completed at any time during the day; 3) adding more “ding” when participants get the answers right; 4) including a report of accuracy (for spelling and comprehension questions) in the My Progress section; and 5) creating a “freezing point” where the application stops you if you keep getting answers wrong, as he reported that there had been times that he had tried to bypass the session by pushing random buttons.

Regarding the Treatment Sessions, participants reported that they liked the frequency and length of the sessions and found the content of the scenarios to be relevant and understandable. One participant reported that they would have liked to have more sessions available to do. Regarding the My Progress section, responses were mixed, with three participants reporting that it was motivating and helpful to see their progress and earn badges. The other two said they did not look at this part of the application that much. Overall, participants had positive responses about the Nightly Dairy, expressing that they found it helpful to reflect on their day. A few participants noted that they would have liked to answer more questions. None of the participants reported receiving pop-up messages through the application. The treatment modifications can be found in Table 2.

Table 2.

Application and study changes

Change Type Change to Application Reason for Change
After Cohort 1
Reminders Added a small explanation/introduction to this section on startup Participant confusion about this section of the app
Added a “Done” button for participants after setting reminders Observations from Think Aloud—participants did not realize how to save reminders
Treatment Sessions For fill-in-the-blanks, started giving an option of 5 letters instead of the whole keyboard Participant suggestion for people who struggle with spelling
Changed so that when participants make 6 errors in a row, a text box pops up asking them if they would like to save their progress and finish the session later Participant suggestion to prevent people from rushing through the treatment session
Made the sessions available for them to do again if they want to (after they have completed them all) Participant request to have more sessions available
Added a “ding” sound and a “Great job!” feedback message when they get the answer right Participant suggestion to increase positive reinforcement
Nightly Diary Fixed it so that entries made after midnight do not count for the next day and block the ability to complete an entry the next day Participant suggestion
My Progress Included a line graph of accuracy over time (session by session) Participant suggestion to increase motivation
Made badges more interesting and different from each other Participant suggestion to increase positive reinforcement
Notifications Fixed the notifications so that they would appear periodically during the 4-weeks of application use Participant report that they did not see these
Added additional messages to keep participants engaged Participant suggestion to increase motivation
Assessments Removed the weekly phone assessments during application use. Will call after first week of application use only to check in. Study team decision based on good compliance
After Cohort 2
Treatment Sessions Developed content for 8 more sessions of treatment so that it can be done daily for 4 weeks Participant suggestion to be able to do the treatment daily, if wanted
Nightly Diary Removed questions about application use, ease of application use, and barriers to application use. Study team decision
Added questions about other emotions/experiences (i.e., happy, content, sadness, pain). Participant suggestion to be asked more during the diary
Added questions about how helpful they found the application that day and whether the treatment had affected their thoughts or emotions that day. Participant request to be able to write about their day
Notifications Added more messages to keep engaged, especially because of new treatment sessions. Added one after completing all the sessions as well. Participant suggestions to increase motivation
Information Revised the treatment rationale to validate the experience of anger in some situations. Participant report that the rationale felt invalidating to their experience
My Progress Made the graph easier to read; labeled axes Participant suggestion
Open in a new tab

Cohort 2 Findings and Modifications

There was good treatment compliance among Cohort 2 as well. Five out of the six participants who returned their research phone had completed all 20 treatment sessions within the four-week study period. One participant had only completed five sessions. Again, many of the participants identified the simplicity and ease of the application as a strength, noting “the ease of it. Yeah, how simple it was,” “it’s easy to use and give you the scenarios that we often deal with on a day-to-day basis,” and “it’s easy to use because it’s intuitive.” Several reported that the application had helped them to change the way that they thought about things and had improved their angry thoughts. One participant said, “[the app] gives a way of looking at it from a perspective where… our point of view is being respected and it’s also allowing me to see another point of view that makes it easier to not be angry or upset by something.” Another reported, “it made me think of things differently. Like it was possible that there was another way to think about it.” Many of them noted specific parts of the application that they liked/found helpful (e.g., the dings when they got answers right, the fact you could pause the treatment and return later, the progress section). Regarding weaknesses of the application, one participant reported that at times he had perceived the treatment sessions suggested that he was wrong to feel anger. One person found the nightly diary unhelpful and another did not understand its purpose. One participant stated that they did not understand the rationale of the treatment, and this was the same participant who had only completed five of the sessions. One participant found the treatment repetitive. Two participants noted that they did not agree with the benign interpretation of the scenarios that were about inattention (e.g., when the benign interpretation was related to a person not paying attention) because “inattention can be deadly” (especially in a combat scenario). Lastly, a few participants noted some programing errors (e.g., sometimes the “dings” stopped working during a treatment session).

Participants had several valuable suggestions for the application. Three of the six participants reported that they would have preferred to complete the treatment daily, rather than five times a week, noting that they prefer doing daily practice consistently. So, additional scenarios were developed and reviewed by our PTSD experts to develop the content for eight more treatment sessions, for a total of 28 sessions over four weeks. Based on feedback from participants who appeared to misunderstand the rationale of the treatment (including the participant who perceived that we were saying it was wrong to feel anger), we clarified the treatment rationale and normalized anger:

“This treatment changes thinking habits that are hard to control. It is normal to feel anger when you have been wronged, but some people are more likely to feel wronged in many situations. They may be more likely to think that others have harmed them on purpose, even in situations where this is not necessarily the case. They interpret ambiguous information as anger-provoking. This habit is so automatic that it is very difficult to “catch” or change on purpose. This treatment will help you to consider other ways to interpret ambiguous situations. With practice, new habits can become more automatic. The sessions are very simple, but are helping you to change the way you think about things that happen to you. Like strengthening your muscles, it takes regular training. As you repeat it, it will become easier, and you will improve.”

Some additional questions were added to the nightly diary based on suggestions from participants expressing that they would like this activity to take longer and they would like to have the option to write in answers rather than only rating Likert-type questions. We added questions about how happy, content, and sad they had felt during the day (on a scale of 0–10), pain level during the day (scale of 0–10), and how helpful they had found the application that day (scale of 0–10). We also added a question about whether the treatment sessions had made them think or feel differently about anything that had happened that day. If participants endorse this question, they are then asked to briefly explain what happened and how they think the treatment affected their thoughts or feelings. These questions were added in order to help reinforce the treatment and the rationale for changing their interpretations. Additional pop-up notifications were added to the application based on participants’ suggestions (e.g., adding one at the end of the last session that says, “Wonderful job! You have completed all 28 treatment sessions. We hope that this treatment has been helpful to you. If you would like to continue, you can go back and repeat the sessions.” Also, based on suggestions from a participant, the progress graph was edited to make it easier to read (e.g., the axes of the graph were clearly labeled). The treatment modifications can be found in Table 2.

Overall, participants reported that they found the content of the treatment sessions relatable and understandable. Many reported that the My Progress section was motivating and helpful. Some of them liked being able to schedule reminders, whereas some found it better to do this on their personal phone. Many of the participants appreciated the pop-up messages and felt that they helped encourage them to engage with the application.

Discussion

The current study used a successive cohort design to iteratively refine a mobile anger treatment for veterans with PTSD and problematic anger. Results indicated that participants found the application helpful and easy to use, and were satisfied with the treatment. Adherence to the treatment schedule was excellent, with over 90% of the participants completing all 20 treatment sessions over four weeks1. In fact, participants reported that they would like to do more treatment, so the application was modified to include 28 treatment sessions for daily use.

Overall, participants reported that their hostile interpretation bias and anger moved in a positive direction from pre- to post-treatment; however, given the small sample size, statistical comparisons were not completed. Effect sizes were large for changes in hostile interpretations, medium-small for changes in problematic anger, and small for changes in benign interpretations, though these estimates are likely unstable due to the low sample size. Qualitative data from the post-treatment interview supported the direction of the changes. Participants reported that the treatment had helped them think and feel differently about everyday situations. Others reported that they had found it helpful to consider other peoples’ perspectives. These observations are promising, considering changing anger cognitions and increasing perspective taking are two important goals of cognitive behavioral treatments for anger (Day et al., 2008).

In addition to identifying the participants’ perceived strengths of the application, we learned about the application weaknesses. Some were easy to remedy (e.g., glitches in the technology) and some required more significant modifications to the application (e.g., revising the treatment rationale or nightly diary). Participants also identified some aspects of the application as weaknesses that cannot modified. For example, some participants expressed irritation by the repetitive aspect of the treatment; however, repetition is a necessary element to the treatment in order to modify interpretation biases. It is important, however, to highlight that some participants identified that the repetition was a strength of the treatment. It is evident that there are individual differences in treatment preferences and dislikes, and what works for some individuals may not work for others. Additionally, some participants noted that they did not like when the resolution of the scenario was related to a person not paying attention because they noted that inattention could be dangerous. While this is true, these interpretations (not paying attention vs. hostile intent) are likely to provoke different emotional reactions, or intensity of reactions; therefore, we kept these scenarios in the treatment.

When discussing weaknesses of the application, participants were encouraged to share suggestions for improvement. Many of the modifications that were made to the application were based on these suggestions from the participants. For example, one participant had reported that he had sometimes felt an urge to rush through the sessions by pressing random buttons rather than reading each scenario. He suggested that we add a “freezing point” to the treatment if the application detects that someone is attempting to bypass the treatment. Participants also suggested the wording of notifications to add to the application. Many of the suggestions from participants in the first cohort were cited as strengths in the second cohort (e.g., the “ding”, notifications). This study demonstrates the successful engagement of veterans as “subject matter experts” in the development and refinement of an intervention. By eliciting feedback from the veterans for whom this treatment was designed, we were able to develop an app that was tailored to their needs and improve the function and utility of the app.

This study has several limitations. First, due to a lack of a control group, it is not possible to rule out a potential placebo effect. Second, we did not have an adequate sample size to conduct significance testing on the pre- to post-treatment outcomes. Future studies should compare outcomes with the MARI application to an active control condition using larger samples. Third, the extent to which treatment had a clinically significant impact on anger and hostile interpretation bias is unclear. Additional research is needed to determine the clinical importance of these reductions and to define a clinically significant change in anger. Fourth, it is possible that the qualitative interviews were affected by bias, and that participants were hesitant to discuss treatment weaknesses. However, this limitation may be tempered by the fact that participants were informed that the purpose of this interview was to improve and modify the application based on participant feedback, so they were primed and encouraged to identify weaknesses. Despite these limitations, our data suggest that the MARI application is a feasible and acceptable treatment for veterans with PTSD and is associated with reductions in hostile interpretation bias and anger. Given these findings, a larger randomized clinical trial of MARI for veterans with PTSD is warranted.

Acknowledgements

We would like to thank Mariah Evans and Taylor Brandt for their help writing treatment scenarios. We would also like to thank Drs. Katherine Cunningham, Eric Dedert, Tatyana Kholodkov, Stefanie LoSavio, Elizabeth Van Voorhees, and Sarah Wilson for providing feedback and ratings on the treatment scenarios, based on their expertise working with this population.

Dr. Dillon was supported by a Career Development Award (IK2RX002965) from the Rehabilitation Research and Development Service of Department of Veterans Affairs Office of Research Development (VA ORD). Dr. Beckham was supported by a Senior Research Career Scientist award (1K6BX003777) from the Clinical Sciences Research & Development Service (CSR&D) of VA ORD. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the VA or the United States government or any of the institutions with which the authors are affiliated.

Footnotes

A provisional patent application (63/255,381, October 13, 2021) has been filed for the MARI application.

1

This adherence rate is for participants who returned the study equipment and does not include the two participants who were lost to contact at the onset of the COVID-19 pandemic.

References

  1. Attkisson CC, & Greenfield TK (2004). The UCSF Client Satisfaction Scales: I. The Client Satisfaction Questionnaire-8. In Maruish ME (Ed.), The use of psychological testing for treatment planning and outcomes assessment: Instruments for adults (pp. 799–811). Lawrence Erlbaum Associates Publishers. [Google Scholar]
  2. Cougle JR, Summers BJ, Allan NP, Dillon KH, Smith HL, Okey SA, & Harvey AM (2017). Hostile interpretation training for individuals with alcohol use disorder and elevated trait anger: a controlled trial of a web-based intervention. Behaviour Research and Therapy, 99, 57–66. [DOI] [PubMed] [Google Scholar]
  3. Crawford EF, Elbogen EB, Wagner HR, Kudler H, Calhoun PS, Brancu M, & Straits‐Troster KA (2015). Surveying treatment preferences in US Iraq–Afghanistan veterans with PTSD symptoms: A step toward veteran‐centered care. Journal of Traumatic Stress, 28(2), 118–126. [DOI] [PubMed] [Google Scholar]
  4. Day A, Howells K, Mohr P, Schall E, & Gerace A (2008). The development of CBT programmes for anger: The role of interventions to promote perspective-taking skills. Behavioural and Cognitive Psychotherapy, 36, 299–312. [Google Scholar]
  5. Dillon KH, Allan NP, Cougle JR, & Fincham FD (2016). Measuring hostile interpretation bias: the WSAP-hostility scale. Assessment, 23(6), 707–719. [DOI] [PubMed] [Google Scholar]
  6. Dillon KH, Glenn JJ, Dennis PA, LoSavio ST, Cassiello-Robbins C, Gromatsky MA, … Kimbrel NA (2021). Anger precedes and predicts nonsuicidal self-injury in veterans: Findings from an ecological momentary assessment study. J Psychiatr Res, 135, 47–51. doi: 10.1016/j.jpsychires.2021.01.011 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Dillon KH, Medenblik AM, Mosher TM, Elbogen EB, Morland LA, Beckham JC . (2020). Using interpretation bias modification to reduce anger among veterans with posttraumatic stress disorder: a pilot study. Journal of Traumatic Stress, 33, 857–863. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Doran N, Brikmanis K, Pittman J, Floto E, & Afari N (2017). Predictors of Suicide Risk in Iraq and Afghanistan Veterans. Military Behavioral Health, 1–11. [Google Scholar]
  9. Epstein EE, McCrady BS, Morgan TJ, Cook SM, Kugler G, & Ziedonis D (2007). The successive cohort design: A model for developing new behavioral therapies for drug use disorders, and application to behavioral couple treatment. Addictive Disorders & Their Treatment, 6(1), 1–19. [Google Scholar]
  10. First M, Williams J, Karg R, & Spitzer R (2015). Structured clinical interview for DSM-5—Research version (SCID-5 for DSM-5, research version; SCID-5-RV). Arlington, VA: American Psychiatric Association. [Google Scholar]
  11. Forbes D, Alkemade N, Mitchell D, Elhai JD, McHugh T, Bates G, … & Lewis V (2014). Utility of the Dimensions of Anger Reactions–5 (DAR‐5) scale as a brief anger measure. Depression and Anxiety, 31(2), 166–173. [DOI] [PubMed] [Google Scholar]
  12. Frueh BC, Henning KR, Pellegrin KL, & Chobot K (1997). Relationship between scores on anger measures and PTSD symptomatology, employment, and compensation-seeking status in combat veterans. Journal of Clinical Psychology, 53(8), 871–878. [DOI] [PubMed] [Google Scholar]
  13. Galovski TE, & Lyons JA (2004). Psychological sequelae of combat violence: A review of the impact of PTSD on the veteran’s family and possible interventions. Aggression and Violent Behavior, 9(5), 477–501. [Google Scholar]
  14. Glassman LH, Mackintosh MA, Wells SY, Wickramasinghe I, Walter KH, & Morland LA (2020). Predictors of quality of life following cognitive processing therapy among women and men with post-traumatic stress disorder. Military Medicine, 185(5–6), e579–e585. [DOI] [PubMed] [Google Scholar]
  15. Hamilton A Qualitative methods in rapid turn-around health services research. VA HSR&D National Cyberseminar Series: Spotlight on Women’s Health 2013. [Google Scholar]
  16. Hawkins KA, & Cougle JR (2013). Effects of interpretation training on hostile attribution bias and reactivity to interpersonal insult. Behavior Therapy, 44(3), 479–488. [DOI] [PubMed] [Google Scholar]
  17. Jääskeläinen R (2010). Think-aloud protocol. Handbook of Translation Studies, 1, 371–374. [Google Scholar]
  18. Jaspers MW (2009). A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. International Journal of Medical Informatics, 78(5), 340–353. [DOI] [PubMed] [Google Scholar]
  19. Larsen DL, Attkisson CC, Hargreaves WA, & Nguyen TD (1979). Assessment of client/patient satisfaction: Development of a general scale. Evaluation and Program Planning, 2(3), 197–207. [DOI] [PubMed] [Google Scholar]
  20. Lee AH, & DiGiuseppe R (2018). Anger and aggression treatments: a review of meta-analyses. Current Opinion in Psychology, 19, 65–74. [DOI] [PubMed] [Google Scholar]
  21. Lewis ZH, Swartz MC, & Lyons EJ (2016). What’s the point?: a review of reward systems implemented in gamification interventions. Games for Health Journal, 5, 93–99. [DOI] [PubMed] [Google Scholar]
  22. Miles SR, Dillon KH, Jacoby VM, Hale WJ, Dondanville KA, Wachen JS, … & STRONG STAR Consortium. (2020). Changes in anger and aggression after treatment for PTSD in active duty military. Journal of Clinical Psychology, 76(3), 493–507. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Morland LA, Greene CJ, Rosen CS, Foy D, Reilly P, Shore J, … & Frueh BC (2010). Telemedicine for anger management therapy in a rural population of combat veterans with posttraumatic stress disorder: a randomized noninferiority trial. The Journal of Clinical Psychiatry, 71(7), 20772. [DOI] [PubMed] [Google Scholar]
  24. Novaco RW (1975). Dimensions of Anger Reactions. Irvine, CA: University of California. [Google Scholar]
  25. Novaco RW, & Chemtob CM (2015). Violence associated with combat-related posttraumatic stress disorder: The importance of anger. Psychological Trauma: Theory, Research, Practice, and Policy, 7(5), 485–492. [DOI] [PubMed] [Google Scholar]
  26. Novaco RW, Swanson RD, Gonzalez OI, Gahm GA, & Reger MD (2012). Anger and postcombat mental health: Validation of a brief anger measure with US soldiers postdeployed from Iraq and Afghanistan. Psychological Assessment, 24(3), 661–675. [DOI] [PubMed] [Google Scholar]
  27. Osório FL, Loureiro SR, Hallak JEC, Machado‐de‐Sousa JP, Ushirohira JM, Baes CV, … & Crippa JAS (2019). Clinical validity and intrarater and test–retest reliability of the Structured Clinical Interview for DSM‐5–Clinician Version (SCID‐5‐CV). Psychiatry and Clinical Neurosciences, 73(12), 754–760. [DOI] [PubMed] [Google Scholar]
  28. Resick PA, Galovski TE, Uhlmansiek MOB, Scher CD, Clum GA, & Young-Xu Y (2008). A randomized clinical trial to dismantle components of cognitive processing therapy for posttraumatic stress disorder in female victims of interpersonal violence. Journal of Consulting and Clinical Psychology, 76(2), 243. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Sayer NA, Noorbaloochi S, Frazier P, Carlson K, Gravely A, & Murdoch M (2010). Reintegration problems and treatment interests among Iraq and Afghanistan combat veterans receiving VA medical care. Psychiatric Services, 61(6), 589–597. [DOI] [PubMed] [Google Scholar]
  30. Schnurr PP, & Lunney CA (2015). Differential effects of prolonged exposure on posttraumatic stress disorder symptoms in female veterans. Journal of Consulting and Clinical Psychology, 83(6), 1154. [DOI] [PubMed] [Google Scholar]
  31. Schnurr PP, & Lunney CA (2019). Residual symptoms following prolonged exposure and present-centered therapy for PTSD in female veterans and soldiers. Depress Anxiety, 36(2), 162–169. doi: 10.1002/da.22871 [DOI] [PubMed] [Google Scholar]
  32. Shea MT, Stout RL, Reddy MK, Sevin E, Presseau C, Lambert J, & Cameron A (2022). Treatment of anger problems in previously deployed post‐911 veterans: A randomized controlled trial. Depression and Anxiety, 39(4), 274–285. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Smith HL, Dillon KH, & Cougle JR (2018). Modification of hostile interpretation bias in depression: A randomized controlled trial. Behavior Therapy. [DOI] [PubMed] [Google Scholar]
  34. Van Voorhees EE, Dillon KH, Wilson SM, Dennis PA, Neal LC, Medenblik AM, … & Beckham JC (2021). A comparison of group anger management treatments for combat veterans with PTSD: results from a quasi-experimental trial. Journal of Interpersonal Violence, 36(19–20), NP10276–NP10300. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Weathers F, Blake D, Schnurr P, Kaloupek D, Marx B, & Keane T (2013). The Clinician-Administered PTSD Scale for DSM-5 (CAPS-5). Boston, MA: National Center for PTSD. [Google Scholar]
  36. Weathers FW, Bovin MJ, Lee DJ, Sloan DM, Schnurr PP, Kaloupek DG, … & Marx BP (2018). The Clinician-Administered PTSD Scale for DSM–5 (CAPS-5): Development and initial psychometric evaluation in military veterans. Psychological Assessment, 30(3), 383. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Wilkowski BM, & Robinson MD (2008). The cognitive basis of trait anger and reactive aggression: An integrative analysis. Personality and Social Psychology Review, 12, 3–21. [DOI] [PubMed] [Google Scholar]
  38. Wilkowski BM, & Robinson MD (2010). The anatomy of anger: An integrative cognitive model of trait anger and reactive aggression. Journal of Personality, 78(1), 9–38. [DOI] [PubMed] [Google Scholar]
  39. Wilks CR, Morland LA, Dillon KH, Mackintosh MA, Blakey SM, Wagner HR, VA Mid-Atlantic MIRECC Workgroup, & Elbogen EB (2019). Anger, social support, and suicide risk in U.S. military veterans. Journal of Psychiatric Research, 109, 139–144. [DOI] [PubMed] [Google Scholar]

RESOURCES