Development and Refinement of Educational Materials to Help Older Veterans Use VA Mental Health Mobile Apps

Abstract

The U.S. Department of Veterans Affairs (VA) in collaboration with the Department of Defense has created mobile applications (apps) that target military Veteran-specific mental health challenges. With over half of Veterans being older than 65, it is essential to support these individuals’ access to and use of these apps. One critical barrier to older adults using apps is that they may not be aware of mental health apps and often need assistance learning to use their devices. To address these gaps in knowledge, we designed and evaluated patient education materials teaching how to download apps and the basics of mobile device use. The materials also included step-by-step guides for three VA mobile apps: Mindfulness Coach, Mood Coach, and PTSD Coach. Guided by user-centered design and feedback from providers and older Veterans, the materials were developed and refined. Six local technology and geriatric content experts provided initial feedback. Next, six older Veterans (M = 78.5 years; 50% owned smartphones) formally evaluated the materials with a majority (83.3%) electing to ‘recommend the materials to others.’ Lastly, 12 providers provided feedback on the materials and 79% rated the materials as helpful. Providers viewed the materials as especially useful for patients who are unable to return to clinic. Overall, providers and Veterans found the materials easy to understand and valuable for novice users. Findings suggest the use of user-centered design principles and iterative evaluations to create patient technology education materials are vital to increase the use of mental health mobile apps among older Veterans.

The U.S. Department of Veterans Affairs (VA) in collaboration with the Department of Defense created a suite of free-to-use mobile applications (apps) that target mental health challenges facing military Veterans (Owen et al., 2018). Of these apps, eight focus on self-management of mental health concerns and deliver evidence-informed tools to the hands of smartphone users. As documented in systematic reviews, these apps have a growing body of research support in primarily young adult to middle aged samples including evidence of feasibility, acceptability, and preliminary evidence of efficacy in improving psychiatric symptoms, coping self-efficacy, and psychosocial functioning (Gould et al., 2019; Owen et al., 2018). Furthermore, surveys and interview studies have demonstrated that Veterans of all ages are interested in mobile apps for mental health (Connolly et al., 2018; Erbes et al., 2014; Gould et al., 2020; Zapata et al., 2018). Specifically, as many as 76% of Veterans who were, on average, in their mid-fifties have shown interest in using mobile apps to help manage mental health problems (Lipschitz et al., 2019).

Even though there is high interest in integrating mobile apps into Veteran’s mental healthcare, there is still a “digital divide” between those that have access to mobile devices (i.e., smartphones and tablets) and those who do not. Access may be a function of finances or limited knowledge about mobile devices or the availability of mental health-focused mobile apps. Additionally, residing in rural areas was related to lower interest in mobile apps among Veterans (Connolly et al., 2018). Although the rates of smartphone use are lowest among the oldest age groups, the percentage of U.S. citizens aged 65 or older who own a smartphone has steadily increased from 11% in 2011 to 53% in 2019 (Pew Research Center, 2017; 2019). Among 50 to 64 year-olds, 79% own a smartphone (Pew Research Center, 2019). Mobile device ownership among Veteran samples with mean ages ranging from the 50s to 60s, is estimated at 79.9% and 76.6%, respectively (Gould et al., 2020; Lipschitz et al., 2019). These findings indicate that the digital divide in terms of access to mobile devices continues to shrink with time.

Older adults, in particular, may not use mental healthcare technology due to a lack of knowledge about various aspects of the technology (Kuerbis et al., 2017; Wildenbos, Peute & Jaspers, 2018). The first knowledge-based barrier facing many older adults is an unfamiliarity with commonly-used symbols on mobile devices, technical terms, or the process of downloading apps. Second, lack of awareness that mobile apps for managing mental health exist precludes use of these resources. Relatedly, only 10% of Veterans owning a mobile device had ever used a mobile app for mental illness (Lipschitz et al., 2019). Third, sensory difficulties may inhibit interactions with mobile devices beyond basic functions such as calling and answering text messages. Although mobile devices incorporate accessibility features to address some sensory-related issues (e.g., increased font size, compatibility with Bluetooth-enabled hearing aids), many users are not aware of these features. These knowledge, awareness, and sensory-related barriers likely extend beyond older users to individuals of all ages with physical or cognitive disabilities, sensory impairment, or with limited English skills. It is important to consider how support can be provided to help these individuals access potentially helpful VA mobile apps.

The purpose of this project was to address knowledge-based barriers through the development of education materials to (1) address these gaps in mobile device knowledge; and (2) support older Veteran’s use of the mental health apps developed by the U.S. Departments of Veterans Affairs National Center for PTSD in collaboration with the Department of Defense National Center for Telehealth and Technology (now called Connected Health). The materials focused on three apps (PTSD Coach, Mindfulness Coach, and Mood Coach) that contain tools and information that target common mental health issues faced by older adults in primary care settings and could be used to promote well-being. We developed and evaluated the patient education materials through an iterative process that utilized user-centered design (UCD) principles (Brown, 2008; U.S. Department of Health and Human Services [HHS], 2006; IDEO, 2019; Norman & Draper, 1986; Ritter, et al., 2014) and mixed methods evaluations with multiple stakeholders to evaluate and improve the materials. UCD principles emphasize data collection methods that assist in answering person-centered questions such as: who will be using these products, when will they be most helpful, and how will they impact daily life. Using a UCD approach, we were able to design education materials to bridge the gap between generic mobile app systems and the specific needs of the older Veteran population we aim to better serve.

Methods

Design

Older adults are a small but important section of two main UCD subdivisions: User Experience (UX) and User Interface (UI) design. UX focuses on designing for every aspect of the user’s interaction with a product or process regardless of delivery modality. UI design describes the presentation, feel, and physical response of the product or process to the user (Zaphiris, Ghiawadwala, & Mughal, 2005; Norman & Draper, 1986). To systematically and accurately address older Veterans’ perspective on technology, we formed our own UCD approach by adapting several components of UX and UI methods used with more general populations (HHS, 2006; Ritter et al., 2014; Nielson, 2012; Norman & Draper, 1986). We then combined this UCD approach with information about using technology with older adults (Fisk et al., 2009; HHS, 2006) and with Baumel’s (2018) proposed steps in evaluating e-health products prior to implementation. Our team conducted a formal UCD evaluation process that involved an iterative cycle of: (1) analyzing our population’s need, context of use, and feedback; (2) designing and elaborating upon prototypes based on said analysis; and (3) testing for continued feedback. This cyclical approach required the materials to be continuously evaluated, underscoring the relevant design and usability heuristics held by older Veterans and VA providers (Nielson, 1994; Rubin, & Chisnell, 2008). Figure 1 depicts our overall UCD process in material development and refinement. The Stanford University School of Medicine Institutional Review Board reviewed this quality improvement/program development project and determined that it does not meet the definition of human subjects research as defined in federal regulations 45 CFR 46.102 or 21 CFR 50.3.

Figure 1.

Overview of User-Centered Design Process for Material Development and Refinement.

Material Development and Team-Based Review

Based on a thorough review of extant literature and our previous research with older Veteran technology needs (AUTHOR CITATION), we identified four areas in which older Veterans most often require further assistance: (1) ways to download apps; (2) basic information about mobile devices; (3) information about common features that older adults may need to help with sensory issues; and (4) step-by-step overviews of the mobile apps. This led to the development of (1) How to Download Guides; (2) Mobile Device Information Guides; and (3) Step-by-Step App Guides for Mindfulness Coach, PTSD Coach, and Mood Coach. Mindfulness Coach helps users learn about mindfulness and engage in guided practice. PTSD Coach provides education about PTSD and tools (e.g., self-guided activities) to manage common PTSD symptoms; many of these tools are also beneficial for managing general distress, including depressed mood and anxiety. Mood Coach facilitates behavioral activation through development of goals and activity schedules. More information about these apps, and others developed by VA, is available through the VA’s Mobile Health website (https://www.mobile.va.gov/). Separate materials were created for Android and iOS (Apple) devices. Of the app guides, PTSD Coach and Mindfulness Coach operate the same regardless of system. Mood Coach is only compatible with iOS devices. Specific information to assist older/novice users was included in the Mobile Device Information Guides, notably: increasing font size on one’s screen, establishing a Bluetooth connection if using Bluetooth-enabled hearing aids, and defining commonly-used terms and symbols on iOS and Android device systems.

After developing prototypes (lead material developer: INITIALS), our other team members (2 psychologists [INITIALS, INITIALS], 1 psychology intern [INITIALS]) reviewed and refined the documents with the developer, who was trained in aging and human factors. Each evaluator was asked to consider the documents’ information, consistency of language, and readability while reading through and making handwritten edits on our preliminary document versions (See Appendix A for specific questions asked of evaluators). Initial design feedback was discussed during a 1-hour meeting where our team came to several decisions regarding the formatting (e.g., bolding of important features noted in materials) and alterations to the language used in the patient-facing materials. Changes aligned with UCD principles were made following the meeting and the revised materials were used in the next phase of testing with local stakeholders.

Informal Needs Assessment

During the material development process, two members of the review team (INITIALS, INITIALS) conducted an informal needs assessment to ascertain what mental health providers desire in patient education materials targeting mobile apps. An informal group conversation was conducted with five mental health clinic providers (3 psychiatrists, 1 psychologist, 1 social worker). Notes were taken during the conversation and were reviewed with the development team.

Material Evaluation Procedures

Local Stakeholder Evaluation.

Following the development process and needs assessment, we sought feedback from local stakeholders who were experts related to using mobile apps or technology, or were geriatric primary care interdisciplinary team members. We invited 10 local VA staff to review the materials and complete a brief (2-page) survey about the patient education materials (See Supplemental Materials). A cover letter included a description of the typical use of materials, which explained that the materials were developed for “older Veterans experiencing mild to moderate anxiety, depressive, or PTSD symptoms who are interested in using a VA mobile app to improve well-being (e.g., to have less stress related to medical conditions).” A caveat that the materials are not targeted towards those who exhibit dementia, suicide ideation, psychosis, or active substance use was noted. The description of the typical use of materials stated that the materials would be used alongside a provider who “would meet with the Veteran in person to select an app to help meet the Veteran’s well-being goal.”

The stakeholders reviewed materials and were invited to submit a survey electronically or via interoffice mail. On the survey, respondents rated the extent to which “the materials would help an older Veteran use a VA mental health app” on a 5-point scale from strongly disagree to strongly agree. For each type of patient education material, three open-ended questions asked about what they liked, concerns they may have (e.g., content, readability and design), and suggestions for next iteration. Evaluators were asked to indicate whether they “would use this item to teach an older Veteran about a VA mental health app” using the 5-point scale ranging from strongly disagree to strongly agree. Lastly, they were asked open-ended questions to inquire about any other information that should be given to older Veterans and/or information to be included in our provider manual. The findings are referred to as round 1 provider feedback in the results.

Nationally-Dispersed Provider Survey.

After addressing the round 1 provider feedback, we subsequently sought feedback from providers who work with older Veterans in VA mental health or geriatrics settings across the nation. The team reached out to 27 VA providers identified through professional networks and through identified providers recommending colleagues as reviewers.

Providers were asked to answer questions about their clinical setting, the percentage of their patients aged 60+, and whether they have ever recommended a mobile app to an older Veteran (See Supplemental Materials). Then, the questions asked of the key stakeholders (round 1) were repeated. The question asking for suggestions was not included because the saturation had been reached in round 1 and respondents often generated suggestions alongside their feedback. We asked providers to respond to the following statement using a 5-point scale ranging from strongly disagree to strongly agree: “Having these materials would help me use mobile apps with older Veterans.” Providers also responded to two open-ended questions inquiring about the types of situations in which the materials would be used and any anticipated barriers in using the materials in one’s practice. The findings are referred to as round 2 provider feedback in the results.

Veteran Evaluation.

Concurrent with gathering feedback from the second round of providers, we conducted interviews with six Veterans who were over 60 years old. Veterans were invited to participate in the interviews (1) via inclusion of the project description in a newsletter sent to past research participants interested in additional opportunities or (2) through a local VA geriatric primary care clinic. We aimed to limit the interviews to one hour and thus focused the material review on the How to Download Guides, Mobile Device Information Guides,¹ and two of the three Step-by-Step App Guides. We rotated which app guides were reviewed to ensure that all were discussed by at least one Veteran. It is important to note that Veterans did not need to own a smartphone to participate.

Prior to the interview, we obtained information about Veteran age, smartphone or tablet ownership, and the type of device owned (Android/Apple) by phone. Mobile device owners were encouraged to bring their devices to the in-person interviews. Additionally, two interviews were conducted by phone with materials mailed in advance. The interview began with a general question to ascertain comfort in using mobile apps followed by a question about what apps they currently use. Following the review of each item, we asked whether the item would help when using a mobile app (e.g., “Does this guide help you understand how to use Mindfulness Coach?”). Lastly, Veterans were asked about recommended changes for the guide(s) and whether they would recommend the materials to others).

Veteran and Family Advisory Council Review.

The finalized materials were submitted to the Veteran and Family Advisory Council for feedback during one of their monthly meetings. The Veteran and Family Advisory Council consists of Veterans who receive services within the VA Health Care System and family members of Veterans with some level of involvement in the Veteran’s healthcare. No formal evaluation was conducted within this step beyond notes on aspects of the guides to refine. These notes were taken by the first author during an in-person meeting.

Data Analysis

Analysis consisted of summarizing the quantitative ratings and participant characteristics using frequencies and descriptive statistics. Qualitative feedback was transcribed from detailed field notes during interviews (Veteran Evaluations) or used from verbatim survey responses (Provider Evaluations). Feedback was reviewed iteratively as it was received using a team-based approach to revise the materials as described above. A minimum of two raters reviewed the feedback, which generally fell into two categories: (1) what raters liked and (2) issues/concerns about the materials. Decisions about what to change were made by rater consensus. When the raters concurred, the suggested changes were implemented, and the revised materials were used with the next interviewees or evaluation steps. When the raters did not agree about changes, they sought guidance from the development team or waited for additional feedback to be gathered from the provider evaluators, Veteran evaluators, or from the Veterans and Family Advisory Council.

We applied a team-based thematic analysis approach by two authors (INITIALS, INITIALS) to examine the feedback from the open-ended questions in round 2 of provider feedback about the perceived benefits of materials, settings in which materials would be used, and anticipated barriers to use. Consistent with a rapid qualitative approach (Beebe, 2003), no formal coding was conducted. The two reviewers read through data independently, then discussed, and categorized into themes/subthemes, and presented the findings to the team.

Results

Informal Needs Assessment

The informal needs assessment findings indicated that mental health providers wanted stand-alone patient education materials that would provide clear and detailed information and resources to patients. Ideally, these materials would result in reduced time needed to explain to patients how to use the materials and the mobile devices. Many providers expressed they may not have time to explore every feature of mobile apps prior to recommending them. More specific concerns regarding accessibility issues were raised, including older Veterans’ not already owning a mobile device and limited accessibility features on some older phone models. Some providers also discussed general barriers to introducing apps in sessions, such as the weak Wi-Fi signal in an outpatient clinic, making it difficult for patients to download apps in session. While some suggestions focused on clinic-related barriers and were beyond the scope of the education materials, when possible, the team revised patient education materials based on the providers’ feedback.

Feedback on Materials

Provider Perspective.

Six local stakeholders responded to requests for round 1 of provider feedback (response rate = 60%). After revisions were made in response to this feedback, round 2 of feedback was elicited from providers. This second round of feedback came from 12 primarily geriatric mental health providers (44% response rate). Respondents in round 2 consisted of providers in geriatric primary care, community living centers (nursing homes), home-based primary care, rehabilitation settings, mental health clinics, research, and administration settings. The majority of the providers’ patients were older (M = 93.5%, SD = 8.8%). Sixty-six percent of the respondents had recommended an app to a patient. Table 1 describes the feedback from each round of review and whether we made edits to the materials based on the providers’ suggestions. Edits were made unless the team needed more feedback to clarify the suggestion.

Table 1.

Overview of User-Centered Design and Iterative Revision of Materials

Category of Design Issue	Provider Round 1		Provider Round 2
	Concerns	Edits	Concerns	Edits
Specific steps needed or information suggested	How to Download: Need “how to connect to Wi-Fi or use cellular data for download” in first step.	+	How to Download: Some patients may need a cue to use the keyboard to search for app in step 3.	−
	App Guides: Need to define mindfulness. Explain that these products have training paired with them.	+	App Guides: Include instructions on using the “back button” to get to the previous screen(s). PTSD Coach guide mentions this under “things to remember,” but other guides do not.	+
	All: Include mobilementalhealth@va.gov email in addition to mobile helpdesk phone number	+
Clarifying information needed (e.g., descriptions of information, explanation of steps)	How to Download: Make more explicit how Bluetooth is used for accessibility (hearing aid). Improve specific description of steps.	+
	Mobile Ref.: “definitions seem a bit techy.” Edits recommended for definition of cellular plan, data plan, agreeing to EULA, how data are collected (FAQs).	+	Mobile Ref.: Improve readability, use 8th grade level or below. Glossary defines “broad concepts that may or may not be useful for folks who have already gotten as far as getting a cell phone.”	−, Seek Vet. feedback
User/Device Fit	All: Users would have to have good cognitive function. Apps may be better used with iPad as screen is bigger.	+, p	All: Veterans with cognitive support may need additional assistance, some may still benefit from 1:1 assistance (n = 5), particular with downloads	−, p
Layout and copy editing	How to Download: Some Android screenshots have small print. Need for more copy editing and consistency in ordering of steps.	Seek Vet. feedback	How to Download: Improve layout, add white [space]. Consider booklet vs. accordion folded pamphlet (n = 2).	+
	Mobile Ref.: Lost color, pop, engagement.	-	Mobile Ref.: Looks like a user manual.	-
	Only mention App Store in iOS and Play Store in Android. Include pictures of symbols.	+
	App Guides: Spacing is tight on some pages. Needs a table of contents. A lot of content and pictures which may be overwhelming. “Simplify, but may take away from effectiveness of guide.” Need for consistency with copy and terms (post-traumatic vs. posttraumatic). Simplify language.	+	App Guides: Very “busy” pages. Appreciate screen shots in guides, but the font size on the screen shots for the Mindfulness Coach Android version is small. Readability could be improved.	+

Notes. + = change made, − = left as is, p = add to provider manual. Mobile Device pages not ordered correctly with some round 2 booklets.

We found the subsequent round of reviewers often deemed the revisions made in the previous round to be helpful (see Table 1). For example, after round 1 of feedback on the How to Download Guides, the providers in round 2 mentioned liking several things, which were edited in round 1, such as the “ability to link Bluetooth to hearing aids,” the layout and formatting, and the inclusion of a “phone number for support.” As noted in Table 1, a few providers had concerns about the size of the How to Download Guides, which was designed to be the same size as a typical mobile device. Providers in round 2 appreciated the revisions despite the size of the guide: “[it] contains very useful information that is presented in a very user-friendly manner…it is small, but fonts and screenshots are very easy to see.” For the Mobile Device Information Guide, the revisions in response to round 1 feedback yielded positive comments, including descriptions of the guide as “basic,” “straightforward,” “a good reference for folks who need it,” and “captures all the basics.” The feedback on the Step-by-Step App Guides was extremely positive as well. In the second round of feedback, one provider described their impression of the guides as follows:

I love the structure and content of these guides. It is nicely laid out. I also appreciate the overview of the apps on the first page. The screen shots are a nice touch and will be helpful to older adults using the guides.

In addition to the positive responses on the patient education materials in the second round of feedback, some providers emphasized the need for having person support to help older Veterans use mobile apps:

Older adults will require orientation to where icons and buttons are located on the phone. Some [of the apps] have several screens and [an] icon may be on the second screen. Older adults may need prompting/cueing to have tech reference open and next to the instruction pamphlet. I would reinforce this in [a] face-to-face session.

In summary, the qualitative feedback showed that the iterative changes likely improved the guides.

Quantitative ratings demonstrated strong overall usefulness of the materials. Following round 1, 67% (n = 4) providers agreed that the materials were helpful overall and 33% (n = 2) strongly agreed with the statement. Then in round 2, 58% (n = 7) strongly agreed, 17% (n = 2) agreed, and 17% (n = 2) indicated neutral ratings regarding the overall helpfulness of the materials. Although the overall distributions did not differ significantly in the chi-square analysis, the percentage of individuals strongly agreeing that the materials were helpful substantially increased from 33% to 58% over the two rounds. Table 2 displays ratings regarding the specific usefulness for each guide.

Table 2.

Quantitative Ratings & Qualitative Feedback from Providers

	Quantitative Ratings		Positive Qualitative Feedback Regarding Materials
	Round 1 N=6	Round 2 N=12
How to Download
Strongly Disagree	1 (17%)	1 (8%)	“Large font. Colors which help differentiate content. Simple. Contains an appropriate number of steps to effectively download an app for a novel user. Has contact numbers if assistance is needed. I like the content of both sides (downloading app and changing font).”
Disagree	0 (0%)	1 (8%)
Neutral	0 (0%)	1 (8%)
Agree	2 (33%)	4 (33%)
Strongly Agree	3 (50%)	6 (50%)
Mobile Device Information			“A good reference for folks who need it. Captures all the basics.”
Guide	0 (0%)	1 (8%)
Strongly Disagree	0 (0%)	1 (8%)
Disagree	0 (0%)	1 (8%)	“Nice break down of main terms and icons. FAQs are great.”
Neutral	4 (67%)	5 (42%)
Agree	2 (33%)	5 (42%)	“Like the booklet format and large sized symbols. Like the clear definitions in the glossary. Like the high contrast.”
Strongly Agree
Step-by-Step App Guides			“Appropriate amount of content on each page – likely not to overwhelm someone. I like the nature of the content (i.e., education regarding the use of the app, how to navigate it, and who to call locally and nationally if there are problems). I like the size of the small booklet; it would be easy to transport and keep confidential. It’s great to have written and visual step-by-step instructions.”
Strongly Disagree	0 (0%)	1 (8%)
Disagree	0 (0%)	1 (8%)
Neutral	0 (0%)	1 (8%)
Agree	3 (50%)	3 (25%)
Strongly Agree	3 (50%)	7 (58%)

Notes. For round 2 of provider feedback, one respondent omitted the overall material rating, resulting in an n of 11 for that item. The distributions of ratings did not differ between rounds of feedback.

Veteran Perspective.

Six Veterans provided feedback on the materials. The Veteran interviewees had a mean age of 78.5 (SD = 10.6) years, 75% were men, and 50% owned a mobile device. Regarding level of education, one (17%) Veteran had an associate’s degree, three (50%) had bachelor’s degrees, and two (33%) had post-baccalaureate/advanced training. On a scale from 1 to 4, the mean comfort with mobile apps was a 2.3. Notably, three Veterans did not own a smartphone. Of the three smartphone users, one had never used an app for health or mental health. The remaining two had used apps previously, but had not used any VA-developed apps. Overall, 80% (4 of 5) and 83.3% (5 of 6) of interviewees found the How to Download and Mobile Device Information Guides to be helpful for themselves or for “others who are not experienced with computers.” One Veteran elected not to review the How to Download Guide because the individual already knew how to download apps.

Qualitative feedback revealed some confusion when using the guide(s) and recommended content changes. On the How to Download Guide, interviewees provided suggestions about improving the description of the accessibility features of one’s device, clarifying the description of where the apps are found once downloaded, and explaining how to find the Play store on Android devices. Meanwhile, the positive feedback on the Mobile Device Information Guide centered on the provision of a glossary for symbols:

No one has ever stopped to explain what these symbols/terms mean. They assume everyone knows. I'm not particularly savvy, so this is very helpful. So far, I'm feeling very positive about it. Info is usually verbal, no one provides instructions on paper. It makes it easy to go back and review. The terms are broken down into understandable pieces.

One interviewee did not find either the How to Download Guide or the Mobile Device Information Guide to be helpful. She noted that as a non-smartphone user:

I don't know what to look at or look for. All [are] very foreign to me. What is an app? What is Bluetooth? What does Wi-Fi mean?…I learn by doing, so just looking at these materials is not helpful.

Notably, the other two non-smartphone owners did find the materials helpful and one interview also echoed the need to learn by doing: “This is helpful, but I just need to get one to learn more. You can't learn skating by reading about it; you have to go on ice.”

Of those reviewing the Step-by-Step App Guides, four interviewees reviewed the PTSD Coach guide, six reviewed the Mindfulness Coach guides², and one reviewed the Mood Coach guide. All guides were reviewed by at least one non-smartphone owner. All found the PTSD Coach guide to be understandable and helpful, but one participant mentioned that the app itself was “too overwhelming.” The individual explained that “it would be helpful to review this app while not experiencing PTSD symptoms so it becomes easier to use when you really are experiencing increased PTSD symptoms.” The individual also noted a preference for “in-person guidance on how to use this app." Another individual expressed concern that the name of the app limited the audience:

The name of the app makes it feel like it is exclusive to PTSD patients. It narrows the number of people who can benefit from this app. [It] may be helpful to include who can benefit at the front page. [For example] ‘to help cope with stress.’

Fifty percent of individuals reported that the Mindfulness Coach materials were understandable regardless of version (e.g., Android or iOS). One critical point made by Veterans was the need for a definition of mindfulness within the step-by-step users guides despite the app containing this information. The individual who found the PTSD Coach app to be overwhelming, very much liked the Mindfulness Coach app guide (Android version):

This app is a lot less "threatening" or "intimidating" compared to the PTSD coach. This brochure clearly describes the features and its quick access to tools. I like it.

The Mood Coach guide was only reviewed by one Veteran. This individual found the guide to be understandable and did not recommend any changes.

Overall, five of six (83.3%) Veterans would recommend these materials to others and the remaining participant would recommend the materials to others once his recommended changes were implemented.

Veteran and Family Advisory Council Review.

Constructive feedback from the Council focused on improving readability of the text and refining the use of titles and headers throughout. Notably, at this step the suggestion to rename the Technology Reference to the Mobile Device Information Guide was made. Positive feedback was consistent with provider and Veteran feedback and is not repeated herein. After this review, the final reading level of the materials as measured with the Flesch-Kincaid Grade level was 5.77 (SD = .42, range = 5.1 to 6.3).

Feedback on Material Utilization

Context of Using the Materials.

The round 2 providers were asked questions about the types of situations in which they would foresee using the materials. Providers described a range of settings, from outpatient settings (group or individual) to short-stay rehabilitation units at the VA Community Living Centers in which they would use the materials with patients or caregivers. One provider described their anticipated use as:

When conducting psychotherapy to assist with at-home skills; to help older adult patients learn more about their smartphones which can help with social connectedness and pleasant event planning/monitoring mood. [Providers] could also use [the materials] in group therapy settings for at-home practice.

Providers viewed the materials as (1) fitting in their continuum of care from initial assessment to termination of psychotherapy, (2) facilitating self-management for those not interested in mental health services, and (3) complementing ongoing psychotherapy treatment. Providers stated the materials would assist Veterans in learning about the apps during a face-to-face visit. One individual mentioned that these materials may be particularly helpful for those Veterans who are “unlikely to commit to outpatient psychotherapy” as the materials would point those individuals to tangible resources that the Veteran could use on their own. Others mentioned that the apps may help older Veterans develop self-management skills and supplement face-to-face visits with consistent at-home practice:

To help veterans learn/be able to download and use apps would supplement in-office treatment or act as a standalone intervention for veterans who are unable or not interested in returning to clinic.

Providers also noted that having the materials may be helpful when the provider “cannot walk them [patients] through the steps in the moment.”

Barriers to Using the Materials.

When asked to identify barriers providers may encounter with older Veterans using the materials, three barriers were identified: cellular/internet access, smartphone access, and privacy and security. Providers emphasized that limited cellular/internet access either in their office setting or the patient’s home would hinder the patient’s ability to download the apps and for the provider to use the materials with the patient in real time in the clinic. For instance, one respondent indicated: “The biggest barrier in my current setting is that cellular access here is terrible. I tried to download the PTSD Coach app in my office and cannot access my App Store here.”

The second barrier of smartphone access touches on the difficulty older adults may have in accessing smartphones, either due to lack of knowledge about smartphones or the sometimes-overwhelming financial burden of owning a smartphone and an accompanying data plan. Related to both issues of access, one provider noted dislike of apps and explained most novice users would need “a lot of human tech support,” which should consist of “at least two in-person sessions to teach the older adult how to use the app, and a very rapid tech support response.” This provider’s viewpoint highlights the importance of person support, particularly for novice smartphone users.

The third barrier of privacy and security encompasses the concern(s) older adults may have regarding their privacy and security when using smartphones. If the privacy and security information is not explained properly to the patients, providers suggested that privacy-related fears of using smartphones and apps may increase. This could possibly deter patients away from wanting to not only download the apps but also use the educational materials. One provider wrote that “I foresee that some Veterans might be resistant to using apps due to concerns of privacy or fear of technology.” Overall, the qualitative findings suggest the providers feel they will be able to use the educational materials with their patients, yet several concrete access-related barriers remain.

Discussion

Utilizing a UCD method, we developed a set of patient education materials to assist Veterans in using VA mental health mobile apps. The use of mixed methods and iterative revisions following feedback from numerous stakeholders allowed our team to focus on users’ direct responses rather than assumptions based on external aging literature. Through this process, we used mixed methods to establish that our materials prioritized user-control, aesthetic consistency, and fit with the needs of the older Veteran users and their VA providers.

Overall, the helpfulness of the materials was high as evidenced by the quantitative ratings by the various evaluation participants. Through the iterative revision process, the materials were perceived to be helpful by a larger proportion of providers with each revision, though the change in proportion was not statistically significant. The Veteran interviewees also provided strong endorsements of the materials as being beneficial to potential users. Collecting detailed qualitative information allowed us to better recognize unclear sections of the materials. Through addressing these suggestions, we were able to fully incorporate user perspectives into the ultimate design of the guides. Overall, these materials seem to be perceived to be useful with the caveat that they may be particularly useful for geriatric outpatient, primary care, mental health clinic (psychiatry outpatient) type settings.

The qualitative feedback received suggests that these materials have the potential to reduce the knowledge gap in the use of mobile devices, which is a critical barrier impeding older adults’ use of mobile health (mHealth) interventions (Wildenbos et al., 2018). Using both text-based explanations and screenshots, we conveyed important information about the basics of mobile devices including how to download apps, information about symbols used on mobile devices, and definitions of commonly used terms such as ‘app’ and ‘data.’ Educating older adults on this basic information is particularly important as recent survey research demonstrated that older adults (65+ years) performed more poorly than younger adults (18-29 years old) on a digital knowledge quiz (Pew Research Foundation, 2019). Furthermore, the guides shared information about features that can address age-related sensory changes, which may help older users address usability difficulties stemming from default font size for smartphones to hearing-related issues. The three app user guides conveyed concrete information about the purpose of each app, app features, and app contents. Through the availability of these mobile app guides, it is possible that more individuals may become aware of these free public health resources provided by the VA.

Providers noted numerous ways in which they envisioned employing the materials in their practices, yet they also noted barriers to the implementation of the materials and some limitations of the materials. First, a key barrier noted was that some patients may have limited access to mobile devices and some clinics lack guest Wi-Fi networks or have poor cellular network reception. Second, providers highlighted the need for guidance from a live person, whether it be a provider, family member, spouse, or caregiver. This finding fits with older Veterans’ expressed desire to have a provider as a navigator to help them identify useful self-management tools (Gould et al., 2020). Learning occurs with the help of social support, such as family members, or through trial and error when exploring a device (Tsai et al., 2017). Third, providers desired additional information about security and privacy issues beyond our materials’ provision of simple information, such as information about how to turn off anonymous data sharing with the VA developers of the apps. This information has been incorporated in our manual developed to guide provider-based support of the apps (not described here, but available from authors upon request). The manual introduces the mobile apps and patient education materials with suggestions about how to incorporate these materials clinically.

The findings of our evaluation must be considered in the context of several limitations. First, we had a small sample size, particularly with regard to the Veteran interviews. Second, to decrease potential burden, Veterans were asked to provide feedback on a subset of the Step-by-Step App Guides, which led in turn to minimal feedback on the Mood Coach Guide in particular. Third, we did not ask providers to estimate how much time they had spent with the materials, nor did we collect information about provider comfort level with using mobile devices or apps, which in turn could influence our findings. Fourth, a broader limitation to providing patient education materials for technologies is that as new updates to mobile operating systems, mobile devices, and apps are released, the materials must be updated and refined to reflect the ever-changing technology landscape. Despite these limitations, our evaluation methods have several strengths including obtaining feedback from multiple different types of stakeholders (i.e., end users of materials, secondary stakeholders, and experts) and through various methods including surveys and interview methods.

The next steps for material development are to consider adding additional information and symbols to the glossary such as the difference between the email icon and text message icon. In order to fully evaluate the usefulness of the materials, we plan to implement them in our own clinical program alongside the provider manual. This clinical program will utilize personal coaching to assist in (1) identifying the most beneficial app for users according to presenting problem and symptoms, (2) creating tailored treatment plans, and (3) monitoring and support during the implementation of treatment plans. We also are currently disseminating the finalized materials nationally through announcements at national webinars, newsletter articles, VA internal SharePoint, and by word of mouth. Our next steps in evaluating the materials include seeking feedback on how providers are using the materials in their own clinical settings. Additionally, in response to the COVID-19 pandemic and recommendations for social distancing (CDC, 2020), we recently adapted our materials to an internet-delivered, printable format. These versions aim to help teach older Veterans the basics of mobile devices with a focus on staying connected through technology (i.e., texting, video calls, VA Video Connect with providers) during times of widespread electronic dissemination of information and care. These revised handouts are available on the VA Geriatrics and Extended Care website (https://www.va.gov/GERIATRICS/news/Stay_Connected.asp).

More research on the clinical use of mobile apps is needed in order to establish empirically-based guidelines for providers interested in utilizing mobile-app assisted interventions in various clinical settings. Future studies may consider identifying factors that predict treatment outcomes in mHealth interventions such as mobile proficiency, socioeconomic status, cognition, symptom severity, and present distress. Resulting insights will better assist providers in making empirically-guided decisions of which mobile app to recommend, to whom, and under what circumstances.

In summary, the present study demonstrates support for the application of UCD methods to the development of patient education materials to address knowledge-related barriers to using mobile apps by older users. The findings regarding the perceived helpfulness of the symbols and terms glossary, clear processes outlined on how to download apps, and step-by-step user guides with screenshots could be applied when developing educational materials to support older adults’ use of mHealth interventions and other technologies. Importantly, our effort to close the digital divide increases older Veterans’ access to important health information, maximizing the potential of healthcare technology.