Qualitative Examination of Participants' Reactions to an App‐Based Military Cultural Competence Training for Helping Professionals

ABSTRACT

There is a need to bolster military cultural competence among human service helping professionals by equipping them with foundational knowledge and evidence‐based tools to serve military families. On the basis of this need, a mobile application was designed to provide military cultural competence training by delivering brief summaries of recent scholarly publications on military culture and family life. This qualitative study utilized semistructured interviews to examine 36 helping professionals' reactions to the 8‐week app‐based training. Guided by the Kirkpatrick Training Evaluation Model, a directed qualitative content analysis identified five themes: intentional and active engagement in the training, satisfaction with the military cultural competence training app, enhanced access and accessibility to research, relevance of the content, and affirmation of existing knowledge and established priorities. Results suggest that mobile apps may be an acceptable modality for delivering continuing education and are well‐suited for helping professionals with limited time for continuing education.

1. Introduction

Military families are situated within a unique cultural context, centered on values of collectivism, duty, and sacrifice (Atuel and Castro 2018; Mancini et al. 2020; Meyer et al. 2016) as well as shared experiences, such as family separations and relocations (Mancini et al. 2020; Meyer et al. 2016). Military families are known for their adaptability and resilience (O'Neal et al. 2018), yet exposure to persistent stressors (e.g., deployment, financial and social hardships of relocating, and community instability) can place families and individual members at an elevated risk for poorer functioning (Ribeiro et al. 2023; Sherman et al. 2023).

Human service helping professionals, such as therapists, social workers, and family life educators, are well positioned to support the needs of military families. However, helping professionals who lack an understanding of military culture and life may undermine the provider–client relationship and the effectiveness of treatment and prescribed interventions (Meyer et al. 2016; Nedegaard and Zwilling 2017). By equipping helping professionals with foundational knowledge about military families (Atuel and Castro 2018), these providers can provide culturally competent practices that best promote healthy outcomes for military‐affiliated families (Isserman and Martin 2021).

This study qualitatively examined helping professionals' reactions to a mobile‐app training on military cultural competence using the Kirkpatrick Training Evaluation Model (Kirkpatrick and Kirkpatrick 2016), which posits that when participants find a training satisfactory, engaging, and relevant, the resulting favorable reactions foster learning and enduring behavior change. In this context, training may refer to “any type of modality in which individuals gain knowledge or skills to do their jobs more effectively” (Kirkpatrick and Kirkpatrick 2016, 20). Thus, evaluating feedback from participants is a vital first step in establishing training impact. Furthermore, given that reactions to military cultural competence trainings may differ based on participants' background and training (Nedegaard and Zwilling 2017), the current study explored variations in participant reactions based on initial levels of military cultural competence.

1.1. Military Culture at a Glance

Although the military functions as a government institution, the community within this institution possesses its own unique cultural tenets, such as camaraderie, loyalty, duty, stoicism, conformity, and sacrifice (Atuel and Castro 2018; Mancini et al. 2020; Meyer et al. 2016). Furthermore, the military operates under a hierarchical chain of command that dictates interactions among ranks, with many values codified into military law and reinforced through shared experiences (Atuel and Castro 2018). Despite these strengths, help‐seeking for mental health and family issues is often stigmatized, leading to underutilization of services due to career concerns and mistrust of providers (Hom et al. 2017; Weiss and Coll 2011). Although there has been some reduction in stigma and increased use of mental health resources since the wars in Afghanistan and Iraq, service utilization remains low (Quartana et al. 2014; Hom et al. 2017). Enhancing military cultural competence among helping professionals is useful to build trust between providers and clients, which may improve service utilization (Johnson et al. 2018).

1.2. Military Cultural Competence in Human and Family Service Provision

In the context of service provision, cultural competence reflects a helping professional's attitudes toward, beliefs about, and knowledge of a given culture, and their skills to serve and advocate on behalf of a group in an effective and sensitive manner (Sue 2001). On the basis of this broad definition of cultural competence, Atuel and Castro (2018) defined military cultural competence as being comprised of a helping professional's awareness of their attitudes toward the military and its members, knowledge of the social structure and norms of the military, and skills to assess and treat Service members, Veterans, and their families.

Some Service members and Veterans prefer seeking services from helping professionals with past military experience given a greater potential for shared understanding (Johnson et al. 2018); yet it is also the case that civilian providers—both within and outside the military system—can effectively build rapport with and treat military‐affiliated clients when they are knowledgeable of military culture and service‐related experiences (Atuel and Castro 2018; Clary et al. 2023; Weiss and Coll 2011). Helping professionals who are knowledgeable about military culture also report greater comfort when working with military‐affiliated clients (Isserman and Martin 2021), which may foster better provider–client rapport. Prior research has also shown that military cultural competence is primarily grounded in general military knowledge (Cameron 2023). Therefore, even a basic understanding of military culture, family life, and common service experiences can meaningfully enhance a provider's competence. Importantly, however, military cultural competence appears to be lacking among helping professionals, particularly those serving in civilian communities. For instance, among 522 mental health care professionals who participated in a national survey, only 19.2% self‐reported high military cultural competence (Tanielian et al. 2014). This lack of awareness of military cultural norms coupled with the importance of military cultural competence for promoting better client outcomes, indicates a clear need for military cultural education among helping professionals.

1.3. Evidence‐Based Continuing Education for Military Cultural Competence

Numerous military cultural competence trainings have been developed and implemented for helping professionals, especially those who provide behavioral health services. Participants who engage in military cultural competence trainings tend to demonstrate increases in some aspects of military cultural competence (Butler et al. 2024; Geraci et al. 2024; Isserman and Martin 2021; Nedegaard and Zwilling 2017), underscoring the notion that military cultural competence is modifiable and may be enhanced through training. Furthermore, initial evidence suggests that military cultural competence trainings may be beneficial for helping professionals with preexisting knowledge of military culture (Nedegaard and Zwilling 2017), suggesting that continuing education may be useful regardless of baseline military cultural competence. Interestingly, broader cultural competence trainings have demonstrated greater improvements among participants with less prior multicultural competence (Sung and Park 2021). Thus, the role of prior competence remains unclear regarding the feasibility, acceptability, and effectiveness of military cultural competence trainings.

Existing military cultural competence trainings offer education to helping professionals through a number of delivery modalities. Most trainings are delivered via in‐person workshops (Geraci et al. 2024; Morris et al. 2022; Nedegaard and Zwilling 2017), courses targeted toward clinicians with postgraduate training (Isserman and Martin 2021), or semester‐long, university‐based courses for clinicians‐in‐training (Butler et al. 2024). These learning modalities are often time‐ and cost‐intensive for participants, a barrier that may prohibit enrollment. For more convenient access to educational resources, helping professionals may choose to access virtual training courses. Such courses are available from various sources, such as PsychArmor (2024), the Uniformed Services University (n.d.), and OneOp (2024). Encouragingly, online training modalities do not appear to compromise the effectiveness of cultural trainings for helping professionals (e.g., Rivera et al. 2023).

One commonality among many existing military cultural competence trainings (excluding university coursework) is curriculum implementation at a high quantity over a short timeframe. That is, in‐person trainings are typically delivered over a few days (Geraci et al. 2024; Isserman and Martin 2021; Nedegaard and Zwilling 2017). Online trainings may also consist of brief webinars or training modules as well. However, little is known about whether military cultural competence trainings may be similarly successful when the training duration is extended, such that smaller doses of information are provided systematically over time.

1.4. Mobile Apps as an Emerging Modality of Delivering Continuing Education

The COVID‐19 pandemic highlighted limitations of in‐person continuing education trainings, which increased the need to offer and evaluate remote, mobile learning opportunities (Gottlieb et al. 2020). Mobile learning refers to learning that is facilitated by mobile devices, such as mobile phones, smartphones, tablets, or applications (henceforth called apps) (Kukulska‐Hulme 2010). Mobile learning is an increasingly popular means of continuing education across many professions, as it reduces logistical barriers related to travel and cost, increases availability of educational materials, and provides flexibility for self‐paced learning (Curran et al. 2019; Khanna and Kendall 2015). An additional asset of mobile‐based continuing education is that asynchronous online trainings increases the likelihood of a more standardized and consistent learning experience for participants (Khanna and Kendall 2015).

Although some virtual military cultural competence trainings are available for continuing education, to date, mobile‐app trainings on military cultural competence are not available. Mobile apps appear to be a convenient and appealing means of delivering trainings, but the effectiveness of these tools as a continuing education platform for helping professionals generally remains unclear (Rosen et al. 2017). With regard to enhancing cultural competence in particular, initial evidence shows promising effects of mobile‐app trainings (Sung and Park 2021). Thus, an investigation is required to understand helping professionals' perspectives on this delivery modality and whether trainings delivered through mobile apps may achieve their desired effects.

In pursuit of offering convenient, timely, research‐informed continuing education to enhance military cultural competence, the REACH for Military Cultural Competence mobile‐app training was designed based on best practices in adult education and learning (Wozniak 2020) to deliver brief summaries of recent scholarly publications about military culture and family life. The intention was that, by introducing helping professionals to research and providing practical implications based on research, helping professionals would be better equipped to serve military‐connected clients (i.e., the content was relevant to their work and immediately applicable). Further, delivering research at user‐requested times over an extended period of time and in smaller quantities was expected to make the information more palatable to helping professionals as they navigated busy schedules (i.e., creating flexibility in learning and bolstering autonomy).

1.5. Kirkpatrick's Training Evaluation Model (2016)

The assessment of this mobile military cultural competence training is guided by the Kirkpatrick Training Evaluation Model (Kirkpatrick and Kirkpatrick 2016). This hierarchical model identifies four key outcomes to use in assessing a training's performance: (1) Reaction, (2) Learning, (3) Behavior, and (4) Results. The Reaction level encompasses participants' satisfaction with, engagement in, and perceived relevance of the training. The next level, Learning, reflects the knowledge and skills participants gained or enhanced through the training. Behavior includes participants' behavioral intent and engagement in behaviors targeted by the training. Results represent the long‐term benefits that develop because of the training.

The current study assessed participant reactions to the mobile‐app training and explored whether these reactions differ based on helping professionals' baseline levels of military cultural competence. Given that positive reactions are the necessary precursor for achieving higher‐level outcomes, this study is a foundational evaluative starting point for assessing training outcomes (Kirkpatrick and Kirkpatrick 2016; Nawaz et al. 2022). That is, a participant's perception of and engagement in the training influences their capacity to learn new knowledge, which has downstream effects for changes in their professional behaviors and achievement of the training program's goals. Thus, if participants perceive the current mobile‐app training to be satisfying, engaging, and relevant, it indicates a positive step toward the achievement of training goals related to enhanced military cultural competence as well as implementing evidence‐based care among military clients.

1.6. Current Study

The current study employed a directed qualitative content analysis design (Hsieh and Shannon 2005) to address the following research questions:

Research Question 1 (RQ1)

What were helping professionals' reactions to the REACH for Military Cultural Competence mobile‐app training as a continuing education resource?

Research Question 2 (RQ2)

Did participants' reactions to the mobile‐app training vary based on baseline military cultural competence?

2. Methods

2.1. Author Positionality

The primary author and secondary coder for this study are doctoral students with 3–5 years of experience in summarizing and evaluating military family research. The primary author was raised in a military family and views family resilience through a systems orientation, grounded in the interconnectedness of the family, community, and society. The secondary coder has a background in public administration and nonprofit management, which informs her perspective that integrated systems of care are a salient leverage point for providing sufficient support for military‐connected families. Although the other authors were responsible for the development, implementation, and evaluation of the app, neither the primary author nor the secondary coder participated in the development or implementation of the app. To account for how these positions may influence analyses, each engaged in reflexive memo writing and conferencing during the coding process. The primary author also maintained a reflexive journal to document decision‐making and reflect upon how her position influenced those decisions.

2.2. Intervention and Evaluation Study Procedures

The research team conducted a randomized controlled trial to evaluate the REACH for Military Cultural Competence training delivered via mobile app. Target populations for trial participants included helping professionals (e.g., counselors, social workers, therapists, psychologists, and family life educators) in the United States. Helping professionals were recruited via professional organizations' email listservs, social media pages for helping professionals, and word‐of‐mouth. Eligible participants were required to be actively providing direct services to clients. Participants were not required to currently serve military‐affiliated clients at the time of the study.

Upon providing informed consent, all participants completed a baseline survey (i.e., before receiving any intervention) using Qualtrics software. Participants were then randomly assigned to the initial intervention group or the waitlist, control group, who received the treatment 8 weeks later, thus termed the delayed intervention group, at a 2:1 ratio, respectively. This allocation ratio was selected to facilitate recruitment, account for potentially higher dropout rates in the delayed intervention group, and optimize statistical power for quantitative analysis (Avins 1998; Lim and In 2019). Participants received a $20 Amazon gift card after completing each survey, up to $60 total. Participants who completed the training were also eligible to receive four continuing education credits for several professional organizations (i.e., UT Austin Steve Hicks School of Social Work and the National Council on Family Relations).

During their assigned intervention period, participants received a total of 16 in‐app research summaries over 8 weeks (two summaries per week). As part of the training, all participants received four research summaries specific to military cultural competence, while the remaining 12 research summaries were curated based on participants' selection of three military family life topics of interest (i.e., adolescents, caregivers, children, couples, deployment, family violence, parenting, service‐related trauma, or Veterans). Each research summary described the key findings, methodology, and strengths and limitations of the original study. In addition, each summary noted implications of the findings for military families, practitioners, and policymakers. Each study was also scored on its scientific rigor, novelty, and communicativeness. Additional app features included “Did you know?” facts (i.e., brief informational snippets designed to increase users' basic knowledge of military culture), a bookmark feature and archives tab to save content of interest, and clickable links to external resources. If a participant fell behind on reviewing their assigned summaries in a given week, the research team would contact the participant to provide encouragement and inquire about any technological difficulties that may have impeded access to the summaries.

Purposive sampling was employed to recruit a subsample of participants for semistructured interviews to provide insight regarding their experience with the app and training content upon completing the final survey (i.e., 8 weeks after completing the training). We prioritized participants who completed the training earlier in the study period to allow for their responses to inform needed implementation revisions, although participants from the initial and delayed intervention groups were included. Interviewees were also recruited with the intent of creating a diverse sample based on individual characteristics (e.g., gender, student or professional status, and years of experience in the field). Participants were offered an additional $20 gift card for their participation in an interview. Study procedures were approved by the Auburn University Institutional Review Board (20‐462 EX 2009).

2.3. Sample

A total of 258 helping professionals consented to participate in the randomized controlled trial of the mobile app (162 initial intervention participants; 96 delayed intervention participants). Attrition from baseline through the follow‐up period was 17.9% (i.e., 12 from the initial intervention group [7.4%] and 34 [35.4%] from the delayed intervention group). Participants were interviewed until theoretical saturation was reached (Roy et al. 2015). Thirty‐six participants who completed that training participated in the semistructured interview. Interview participants were 34.5 years old, on average (range 22–65 years old), and most identified as female (75.0%) and non‐Hispanic white (61.1%). Most participants' highest education obtained was a master's degree (66.7%), and more than half (52.8%) were currently pursuing further education. Half (50.0%) had a personal connection to the military, and 58.3% were currently serving military families. Three‐quarters (75.0%) of participants had at least some experience serving military‐affiliated families. More detailed information regarding participant characteristics can be found in Table 1.

Table 1.

Participant characteristics.

Pseudonym	Sex	Age	Race/ethnicity	Education	Student status	Experience serving SMVF	Personal military connection	MCC mean score
Cameron	Female	31	Asian non‐Hispanic	Masters	Nonstudent	Current	Related to multiple SMVs	3.75
Joy	Female	48	Hispanic	Masters	Masters	Current	Veteran	3
Jordan	Non‐Binary	23	White non‐Hispanic	Bachelors	Masters	Never	Child of an SMV	2.75
Amari	Female	25	White non‐Hispanic	Bachelors	Masters	Never	None	2.5
Tracy	Female	28	Black and White non‐Hispanic	Masters	Nonstudent	Current	None	3
Elliot	Female	34	White non‐Hispanic	Masters	Nonstudent	Current	Spouse of an SMV	4
Drew	Female	26	White non‐Hispanic	Bachelors	Masters	Never	Friend of an SMV	2.5
Bailey	Female	26	White non‐Hispanic	Masters	Nonstudent	Previous	None	2.25
Olivia	Female	40	White non‐Hispanic	Masters	Nonstudent	Current	None	2.5
Malik	Female	42	Black non‐Hispanic	Masters	Doctoral	Never	None	2
Parker	Male	42	Pacific Islander non‐Hispanic	Bachelors	Masters	Current	Veteran	3.5
Riley	Female	65	White non‐Hispanic	Masters	Nonstudent	Current	Related to multiple SMVs	3.5
Lucas	Male	26	White non‐Hispanic	Masters	Doctoral	Current	None	3.25
Shayla	Female	35	White non‐Hispanic	Masters	Nonstudent	Current	Spouse of an SMV	1.25
Laurel	Female	40	White non‐Hispanic	Masters	Nonstudent	Never	None	2
Karen	Female	41	White non‐Hispanic	Associates	Bachelors	Never	None	1
Leila	Female	28	White Hispanic	Masters	Nonstudent	Current	None	3.25
Sebastian	Male	43	White non‐Hispanic	Masters	Masters	Current	Veteran	3
Elijah	Male	29	White non‐Hispanic	Masters	Doctoral	Current	None	2.75
Mark	Male	64	White non‐Hispanic	Masters	Nonstudent	Current	None	4
Karina	Female	35	Black non‐Hispanic	Masters	Nonstudent	Previous	None	2.5
Clara	Female	29	White non‐Hispanic	Masters	Masters	Current	Veteran and related to multiple SMVs	3.75
Charlie	Male	30	White non‐Hispanic	Masters	Nonstudent	Current	Service member	3
Harper	Female	58	White non‐Hispanic	Masters	Masters	Never	Veteran	2.75
Zoe	Female	38	Black non‐Hispanic	Doctorate	Nonstudent	Current	Related to multiple SMVs	2.5
Lily	Female	38	White non‐Hispanic	Masters	Masters	Current	None	2
Casey	Female	26	White non‐Hispanic	Bachelors	Masters	Never	Spouse of an SMV	3.5
Aiden	Female	27	Mestiza Hispanic	Bachelors	Masters	Never	None	2.75
Idris	Female	23	Asian non‐Hispanic	Bachelors	Masters	Previous	Child of an SMV	2.5
Reese	Male	22	Hispanic	Bachelors	Masters	Current	None	2.5
Jaccari	Female	28	Black non‐Hispanic	Masters	Nonstudent	Current	None	1.75
Jaylin	Female	33	White non‐Hispanic	Bachelors	Masters	Never	Grandchild of a veteran	2
Mia	Female	41	American Indian/Alaskan Native non‐Hispanic	Masters	Nonstudent	Current	Veteran	3
Evelyn	Female	49	Asian non‐Hispanic	Masters	Nonstudent	Never	None	3
Carter	Male	27	White non‐Hispanic	Bachelors	Masters	Current	Child of an SMV	2.75
Skylar	Female	40	White non‐Hispanic	Masters	Nonstudent	Current	Spouse of an SMV	4

Notes: Participant names are pseudonyms.

Abbreviations: MCC, military cultural competence; SMV, service members or veterans; SMVF, service members, veterans, or military families.

2.4. Data Collection

2.4.1. Semistructured Interviews

Semistructured interviews were conducted via Zoom by trained research team members and lasted an average of 32 min (range 19–64 min). The research team, consisting of experts in evaluation and military family science, collaboratively devised the interview protocol (see Supporting Information: Appendix A). In accordance with the theoretical model, questions were selected to assess key aspects of participants' reactions to the training, such as the relevance of the training to their work, their experience using the mobile app, and their overall satisfaction with the training (Kirkpatrick and Kirkpatrick 2016). Additional questions captured participants' roles as helping professionals, work with military‐affiliated clients, history of consuming research, and takeaways from the training. Interviews were recorded and transcribed verbatim for analysis.

2.4.2. Measures of Participant Characteristics

In the baseline surveys, participants provided demographic information, such as their gender, age, race, ethnicity, and household income. Participants also reported their connections to military life by indicating whether they were a current Service member, Veteran or retiree, military family member, and/or helping professional who has or is currently working with military families. Participants selected all that applied.

To assess military cultural competence at baseline, participants responded to four items from the Military Cultural Knowledge and Skills scale, which was adapted from the Adapted School Psychology Multicultural Competence Scale (Malone et al. 2016) and subsequently validated (Lucier‐Greer et al. 2025). Sample statements include “I can make relevant recommendations to Service members, Veterans, and their families,” and “I am knowledgeable of evidence‐based intervention strategies used with Service members, Veterans, and their families.” Participants rated their agreement on a scale from 1 (strongly disagree) to 4 (strongly agree). Scores were averaged, such that higher scores indicated greater military cultural competence (M = 2.78, SD = 0.73). The average baseline score in the current sample was slightly higher than that of the overall trial sample (N = 258, M = 2.72, SD = 0.68). The items demonstrated good reliability (α = 0.89).

2.5. Data Analysis

Helping professionals' interviews were analyzed via directed qualitative content analysis (Hsieh and Shannon 2005) in Dedoose (SocioCultural Research Consultants LLC 2024). The first author began analysis by immersing herself in the data, reading through all transcripts. Next, the codebook was developed in accordance with Kirkpatrick's Training Evaluation Model (Kirkpatrick and Kirkpatrick 2016) with a focus on participants' reactions. On the basis of Kirkpatrick and Kirkpatrick's (2016) operationalization of reactions, deductive codes were created to identify participants' satisfaction with, engagement in, and perceived relevance of the mobile app. Inductive codes were also developed to capture additional elements of participant reactions beyond the three previously established components of satisfaction, engagement, and relevance. Qualitative data outside the scope of the current study (e.g., comments that captured higher‐order aspects of the evaluation model, experiences outside the training) were not coded for this study.

Once the codebook was developed, the first author trained the secondary coder. Next, the first author and secondary coder independently coded transcripts, conferencing frequently to review analytic memos and check reliability. Cohen's kappa was calculated to determine intercoder reliability at the code level (Cohen 1960). Through these conferences, the codebook was finalized. The final codebook is presented in Supporting Information S1. Fourteen transcripts were double‐coded, at which point consistent reliability (κ > 0.60; Landis and Koch 1977) was achieved, and each coder began independently coding the remaining interviews (i.e., 11 each). They continued to conference frequently to discuss analytic memos and ensure fidelity to the codebook.

Once all data were coded, thematic analysis began. The first author separated the excerpts into their respective code groups. Within each code, the first author reviewed all excerpts to reimmerse herself in the data. Then, she began to organize excerpts into categories based on common patterns, engaging in constant comparison to relate each data point to others within the code group. In an iterative process, categories were formed based on the core idea in each excerpt. During the constant comparison process, frequencies were recorded to identify patterns in reactions based on participants' baseline military cultural competence mean scores (i.e., within each category, consistency and relation to the average were observed). Initial categories were then repeatedly condensed as commonalities were identified, and patterns in military cultural competence levels were reexamined. Data were subsequently grouped into themes and subthemes. Any unique patterns in reactions within the themes or subthemes regarding participants' military cultural competence were explained in the descriptions. Themes were then presented to the secondary coder and the research team for review. The derived themes were revised until the research team reached agreement.

3. Results

Five primary themes emerged from the data. Themes and subthemes are detailed below and reviewed in Table 2. Unless otherwise noted, themes and subthemes did not appear to differ between participants based on their baseline military cultural competence. The names associated with participant quotes are pseudonyms to ensure participants remain anonymous.

Table 2.

Themes representing participant reactions to the REACH for Military Cultural Competence app.

Theme	Subtheme	Exemplar quotes
1. Intentional and active engagement in the training	N/A	“I felt like all of [the research summary] was, like, engaging. I was interested in all of it. It was like I wasn't finding myself distracted or anything as I was reading through it, and I really was trying to retain everything that I was reading.”
2. Satisfaction with the military cultural competence training app	Overall satisfaction with the training experience and contenta	“I just think it's a great app and I'm really thankful and fortunate to have been a part of this study …”
	Met expectations for the app and learning experiencea , b	“… my main expectation was the hope that [the military cultural competence app] would be an excellent resource, again, for someone who's new into providing services to the population, or a civilian provider who's interested in learning more to be able to best offer services and really understand perspectives of military families and spouses. And I really felt like the app did meet that.”
	Sufficient research overviews, yet more information and implications desireda	“I think one of the biggest strengths, was … the fact that you guys were able to cram so much information into a summary, but it didn't feel, like, overwhelming or too much or scattered. It was the right amount, it was what we needed to know, and it just took a few minutes to read.” “… the implications I always felt were, more often than not, were just really blasé. So it's like, ‘make sure to address depression with your clients’ and it's like, well shit yeah. I mean if you work in mental health and you're not addressing depression, like, you shouldn't have to be told to do that, you know what I'm saying?”
	User‐friendliness with opportunities for refinementa	“I thought [the military cultural competence app] was like an awesome, like, a very user friendly, very colorful app, so I don't know if there's a lot that I would recommend there to change in terms of the interface.” “… technically, it had some problems.”
3. Enhanced access and accessibility to researcha	Overcoming barriers to accessing evidence‐based informationb	“I felt like the app was way more, like—I want to say cost‐effective—but like with my time. It was like I did more with my time in comparison to just reading these articles that weren't broken down and organized as well as they were in the app. So that really helped me, and it's just so easy.” “… I really like to be able to have [research] right at your fingertips, and know it was a legitimate, you know, source and legitimate research.”
	Making research accessible and comprehensible	“But I love the way that it put every article into something that I could digest a lot easier, made it easy to look through.”
4. Relevance of the content	Varying applicability of research to helping professionals' current clinical practice	“… I would say it's very applicable to the work I do, the population I work with, and all of that.” “… it's still going to be a little while before I'm working with Soldiers in this capacity. Because a lot of the information that was coming out, it's not necessarily stuff that I need right now [in] my current job. It was more information that I'm going to need in the future.”
	App content bridged connections to personal experiencesa	“… the resources and the summaries that are in the app are useful and applicable to, I think, a person's personal life.”
5. Affirmation of existing knowledge and established prioritiesa , b	N/A	“It's nice just to hear that things you are doing, you're doing it the right way, you know.”

Note: The organization of themes and subthemes aligns with the codebook. All participant names listed are pseudonyms to protect participant identities.

^aIndicates that a theme or subtheme was formed inductively. All other themes/subthemes are deductive (i.e., informed directly by the Kirkpatrick model).

^b Indicates a theme varied based on participants' baseline average military cultural competence scores.

3.1. Theme 1: Intentional and Active Engagement in the Training

Thirty‐two of the 36 participants spoke about how they engaged with the military cultural competence app (e.g., content consumption, content reflection, usage of app features). Several participants (n = 10) explicitly mentioned reading all of the summaries delivered to them through the app in a timely manner, although two participants mentioned falling behind on their assignments. Some participants (n = 7) mentioned paying close attention to the details within the summaries, whereas others (n = 5) consumed the content more selectively by skimming summaries to look for the content they found most interesting. A few participants (n = 6) demonstrated high levels of reflective intentionality in reviewing summaries. For instance, Lily shared their process: “With some of [the summaries], I felt like ‘Okay, let me think about this, let me process and reflect on how it can help me and my job, and then go back and read it again.’”

Participants also demonstrated engagement through their utilization of app features and seeking technical support, although some app features were more popular than others. For example, one‐third of participants (n = 12) utilized the bookmark button to refer back to summaries, whereas clickable links to external resources were rarely utilized (n = 7). Some participants mentioned not being aware of certain app features (e.g., bookmark button, archives tab, clickable links to external resources), which may explain the varied use of these features. When technical issues arose, some but not all participants (n = 5) used the technical support features of the app to help resolve issues.

3.2. Theme 2: Varied Satisfaction With the Military Cultural Competence Training App

A central element of reactions in the Kirkpatrick model (Kirkpatrick and Kirkpatrick 2016) is participants' satisfaction with the training. In this evaluation, elements of satisfaction emerged as (1) varied overall satisfaction with the learning experience and content, (2) met expectations for the app and learning experience, (3) sufficient research overviews, yet insufficient direct implications for practice, and (4) user‐friendliness with opportunities for refinement.

3.2.1. Varied Overall Satisfaction With the Training Experience and Content

Nearly all participants (n = 33) shared their overall of satisfaction with the training experience and content. The training experience was described positively by many (n = 17) participants. Parker shared, “Like this app is really unique. So, you know, I have to say you guys are doing an amazing job.” When noting specific elements of the training experience they were satisfied with, participants commented on the responsiveness of the study team when issues arose (n = 5), personalization through topic selection (n = 3), and the study incentives, which included continuing education credits and e‐gift cards (n = 3).

Many participants (n = 30) also expressed satisfaction with the content in the summaries and were described as informative and useful (n = 18). Participants also identified specific content they appreciated, including the “Did you know” facts (n = 8), research summaries that they regarded as especially memorable (n = 7), and specific sections of the research summaries, such as the implications (n = 8) and brief summary (n = 4). Those who were less satisfied commonly expressed concerns about the quality of the research being summarized (e.g., highlighting limitations of some of the studies that were summarized) (n = 5). Other participants (n = 4) criticized the usefulness or clarity of specific summary sections, including the research team's assessment of each study, implications, and methods.

3.2.2. Met Expectations for the App and Learning Experience

Most participants (n = 31) referenced their expectations in some manner, with a few of these participants (n = 6) stating that they entered the training with no expectations. Of those who did have expectations, 77% of participants (n = 24) reported that their expectations for the training were met or exceeded. Some indicated that their expectations centered on what they anticipated they would learn through their exposure to research (n = 8). For instance, Karen said “Well, I was just hoping that I'd learned more about military life and about different issues that people in the military faced, and those expectations were met.”

A small number of participants (n = 7; 23% of those who discussed expectations) described ways in which their expectations were not fully met. Most unmet expectations focused on the app functionality and/or user experience (n = 4). For example, Joy remarked that they expected access to the full research articles through the app, not just the summaries. Other participants reported that the description of the app‐based training did not align with the training experience, as they expected to receive more explicit education on military culture (n = 2). Participants who reported unmet expectations tended to be more confident in their own military cultural competence prior to participating in the training (i.e., range 2.75–4.0 on a scale from 1 to 4). Notably, five of the seven participants who reported an unmet expectation also indicated that at least one of their expectations was fulfilled.

3.2.3. Sufficient Research Overviews, Yet More Information and Implications Desired

Thirty‐three participants shared a range of opinions regarding whether the research summaries provided enough information during the training. For instance, about half of the participants (n = 16) felt the summaries provided an appropriate level of detail in a succinct format. “… I want the nuts and bolts and how I can use something, and I think [the research summary] did a really good job of bringing that with the app,” said Zoe. However, other participants (n = 12) expressed that the summaries were lacking in important information, such as study details or clarity regarding how the scores assigned to each article were derived. Some participants were also looking for a greater scope of research, as some participants (n = 13) felt that key topics (e.g., discrimination) or populations (e.g., sexual minority Service members, aging adults) were not explored. Not many participants commented on the quantity of content delivered; the few that did (n = 4) voiced a desire for access to even more content. Regarding research implications, a few participants (n = 5) expressed their satisfaction with the level of detail in the implications, but more (n = 13) perceived that the implications missed the mark and were regarded as “common sense” or not specific enough to clinical practice to be useful.

3.2.4. User‐Friendliness With Opportunities for Refinement

Almost all participants (n = 35) commented on the user experience of the app with 89% (n = 31) describing the app as user‐friendly in nature, intuitive to navigate, and/or visually appealing. Some regarded the visual appearance as bland (n = 4) and others disliked the organization and found the app difficult to navigate (n = 4). Although some participants (n = 10) expressed appreciation for specific technological features of the app, such as the notifications and clickable links, over half mentioned encountering at least one technological error (n = 22; 63%), typically related to the notifications (n = 16), which appeared repeatedly or not at all. Notably, six of the 22 participants who encountered technical issues also appreciated other technical aspects of the app. For instance, in reflecting on the user experience, Jaylin shared “I thought [the app] was really good. It was clean. It was readable. The font was a good size. I felt like it was easy to use, easy to understand how to work it, and it wasn't complicated at all. It was a really good design.” However, Jaylin also experienced technical issues; in particular, the notifications for new article assignments did not work reliably on their mobile device. In acknowledging limitations to the technical functioning of the app, several participants (n = 8) suggested revisions to improve the user experience, such as adding sections or functions to make more information readily available.

3.3. Theme 3: Enhanced Access and Accessibility to Research

Although not a component of reactions identified by the Kirkpatrick model, participants of this military cultural competence training commonly commented on their improved access to research. Participants explained that engagement in the training helped them overcome barriers to accessing evidence‐based information and comprehend research more readily.

3.3.1. Overcoming Barriers (i.e., Time and Cost) to Accessing Evidence‐Based Information

Many participants (n = 24) shared that engagement in the training reduced burdens commonly associated with accessing research, especially the time it takes to read research (n = 16) and the resources needed to retrieve research (n = 4). Commonly, the need for institutional access or having to pay fees to access academic journals was cited as a concern. Related to both time and cost, Clara said “… five minutes is far better than the five hours I probably would have taken to try and find the article, if I found it at all …. And on top that, half of the time, you have to pay for them, and so clinicians can't even get access to the research to inform their practice ….” Participants (n = 5) with higher military cultural competence scores at baseline (2.75–4) also noted that the portability of a mobile app was especially useful.

3.3.2. Making Research Accessible and Comprehensible

All 36 participants addressed the comprehensibility of the research summaries, and two‐thirds of the participants (n = 24; 67%) found some aspect of the summaries easy to understand. The comprehensibility of content was commonly attributed to using accessible vocabulary that did not contain scientific or military jargon. Joy described, “… the style of writing felt like it was trying not to be so academic, like it was trying to be a little bit more casual—which is great, I think that's definitely easier to read.” However, a few participants (n = 3) described the language as difficult to understand.

Participants (n = 11) also appreciated the organization of the summaries to facilitate comprehension. For example, Charlie expressed appreciation for the bullet points in the summaries, as they helped to break up the text visually and make it easier to read through the content. The fact that summaries were short and to the point was also appreciated by many (n = 15). Most sections of the research summaries were deemed easy to comprehend. Some challenges were experienced with regard to understanding the studies' methodology sections and the research team's assessment of each study (n = 2).

3.4. Theme 4: Relevance of the Content

Another theme that emerged was the relevance of the mobile app to participants' clinical practice as well as their personal experiences.

3.4.1. Varying Applicability of Research to Helping Professionals' Current Clinical Practice

Almost all participants (n = 34) discussed the degree to which the content of the research summaries was applicable to their work as a helping professional. Most participants (n = 25) were able to identify content that related to their work in serving both military‐connected and civilian clients. When reflecting on the topics covered by the research summaries, Sebastian remarked “… there are just certain topics or subjects where it's like ‘oh, I work with that population, so this is very relevant to me.’” However, several participants noted that at least some content lacked applicability to their current professional role (n = 17) or current client base (n = 5). For instance, Zoe works with a very young child of a Service member, but felt the content was not applicable because the age of children reflected in the research was much older than their client. A quarter of participants who discussed the applicability of the training (n = 9; 26%) stated that the content was not relevant to their current professional role; yet, some of those participants (n = 7) believed that the information would be relevant in the future as they intended to work with military‐connected clients.

3.4.2. App Content Bridged Connections to Personal Experiences

Many participants (n = 20) spoke about the extent to which they felt the content of the app related to their personal lives, with almost all participants (n = 19; 95%) reporting that the content was personally relevant. Some (n = 12) explicitly noted that the content was reflective of their experiences as a Service member, Veteran, or military spouse. For instance, Clara described that content from one summary was so pertinent to their lived experiences that they chose to share it with their significant other to help them better understand Clara's needs. Others (n = 9) spoke about how the content related to more indirect connections they had to the military, such as extended family members or friends. Five participants, including some who had direct connections to the military, found that some content was not relevant to their personal lives, typically because the content did not reflect their current family or military status. “… we don't have any children, so the [research summaries] about families and children really don't personally apply,” said Skylar. Only one participant mentioned that none of the content was personally relevant, citing a lack of any relationship with someone who was military‐connected.

3.5. Theme 5: Affirmation of Existing Knowledge and Established Priorities

Although not a formal component of the Kirkpatrick model, a related theme of feelings of affirmation emerged. About a third of participants (n = 13; 36%) described how the content of the research summaries and the “Did you know?” facts validated their knowledge and experiences. For instance, when reflecting on their experience reading summaries, Drew recalled thinking “Oh, this is a dynamic in couples that I already knew and now it's reaffirmed in another setting, you know, in like military families.” A few participants (n = 5) felt validated because the research presented centered on subjects or populations that the participants believed to be deserving of attention.

Notably, a few participants (n = 5) mentioned that some content contradicted their professional knowledge of military families, with some even noting hesitancy to believe the new information. Of note, these participants who noted contradictions to their own knowledge tended to be more confident in their military cultural competence (i.e., scores 2.75–4.0). However, each participant who reported contradictions to their knowledge also shared that at least some content was validating for them.

4. Discussion

The Kirkpatrick Training Evaluation Model (Kirkpatrick and Kirkpatrick 2016) posits that participants' reactions to a training program have implications for the achievement of higher‐level goals (i.e., learning, behavior, and results). Drawing from this model, the current qualitative study examined helping professionals' reactions to a mobile‐app‐based training to enhance military cultural competence. Furthermore, emerging differences in reactions based on participants' baseline military cultural competence were observed to explore whether reactions to a cultural competence training vary for those with greater exposure to a given group or culture. These findings, as well as implications for clinicians, are further explored below.

4.1. Acceptability, Engagement, and Relevance of the Training

Kirkpatrick and Kirkpatrick (2016) conceptualized reactions to a training as participants' favor for the content and experience, their engagement during the training, and their perception of the relevance of the content to their work. Each of these facets of reactions was identified in interviews with participants who completed the REACH for Military Cultural Competence training. Overall, participants reported favorable reactions for the overall learning experience, met expectations, satisfaction with the dosage of information, and satisfaction with the app user experience. However, opportunities to improve the training were also identified, such as ensuring technological issues with the app notifications are resolved, providing more tangible implications, and implementing clearer marketing for the training so participants know more precisely what to expect (in this case, exposure primarily to research findings).

Most commonly, participants indicated high levels of engagement with the app and training materials, conveyed through their reading of the content delivered through the app and utilizing app features. For those participants who were less engaged, some indicated that time constraints related to heavy caseloads or their own educational commitments served as a barrier to participation in the training. This finding mirrors prior research, in which helping professionals have reported time as a deterrent for participating in continuing education (e.g., Stewart et al. 2012).

Regarding relevance, most participants found at least some content from the REACH for Military Cultural Competence training relevant to their role as a helping professional and/or foresaw how it would be helpful in their future work with Service members, Veterans, and their families. As suggested by best practices in adult education (Wozniak 2020), the personalization of training content for participants by allowing them to select topics of interest appeared to be useful in ensuring the content was applicable to their interests and role, particularly for those already serving military clients.

4.2. Emergent Reactions to the REACH App

In an expansion upon Kirkpatrick and Kirkpatrick's (2016) conceptualization of reactions, inductive analysis revealed additional aspects of participants' reactions to the military cultural competence app. These included participants' perceptions of enhanced access to and accessibility of research in the training, connections that emerged between the training content and their personal lives, and feelings of validation from the training content.

Many participants commented on how the training promoted accessibility to research, including literal access and ease of understanding and applying the information. Helping professionals are commonly trained in evidence‐based practice, and thus have prior exposure to research, which several participants referenced in their interviews. However, as many participants reflected, barriers, such as fees for accessing scholarly publications, jargon, or a lack of portability, make research inaccessible. A major strength of this continuing education opportunity voiced by participants was that research was made convenient and comprehensible; this type of reaction speaks to initial evidence of the value and acceptability of mobile apps as a tool to overcome accessibility barriers for helping professionals to receive continuing education through a convenient modality.

Although the Kirkpatrick Training Evaluation Model (Kirkpatrick and Kirkpatrick 2016) emphasizes the importance of the applicability of training content to participants' professional roles, participants in the current study also reflected on the applicability to their personal lives. Of note, half of the participants in this sample were Service members, Veterans, or military family members themselves, which may explain why so many participants found the content relatable. Although participants finding personal takeaways is not commonly a goal of continuing education, these findings highlight the opportunity for helping professionals to think about how the knowledge and skills they develop during continuing education may be applied in a variety of contexts across numerous roles.

It appeared that whether a participant's beliefs are affirmed or contradicted by a training program may be consequential for training outcomes. Participants who felt validated that their knowledge, practice, or priorities were supported by research may be more likely to share the information with colleagues and apply their knowledge and skills when working with military‐connected clients. Thus, the finding that most participants felt affirmed may serve as emerging evidence that participants may go on to provide military culturally sensitive services to their military‐connected clients. A notable few participants stated that some of the research findings were at odds with their prior knowledge or experiences. This contradiction may serve as a barrier to learning (e.g., not trusting the training), but because these same participants found other research summaries to be useful and credible, it may also represent an opportunity for participants to reconsider what is “known” and facilitate learning. These findings may reflect confirmation bias among participants, whereby participants were likely to accept information that aligned with their existing beliefs and discounted contradictory evidence (Nickerson 1998). Accordingly, researchers designing future training interventions should incorporate deliberate strategies, such as structured reflection prompts and exposure to diverse perspectives, to mitigate confirmation bias and foster adaptive learning (Whitt et al. 2023). An alternative explanation for these findings is that follow‐up interviews were conducted after the conclusion of the training, so for participants to still recall specific content they were wrestling with may convey careful reflection and sustained impact.

4.3. The Impact of Prior Military Cultural Competence on Reactions

Prior military cultural competence tends to be consequential for participant outcomes in military cultural trainings (Nedegaard and Zwilling 2017). Accordingly, the second goal of this study was to examine whether participants' reactions to this military cultural competence training differed based on their military cultural competence at baseline. In general, helping professionals' reactions to the app‐based training did not vary by baseline military cultural competence, with only a few notable exceptions. Prior knowledge seemed to shape participants' expectations for the training and influenced whether they felt their existing knowledge and experience were affirmed. Some participants with more military cultural competence at the start of the training voiced expectations to receive more detailed information to supplement their prior knowledge and thus expressed that some expectations were not met. Further, prior knowledge is a necessary function of being able to identify some information as contradictory to expectations; thus, it makes sense that participants with average or above‐average military cultural competence would be those who raised this concern at higher rates. These findings suggest that while the REACH for Military Cultural Competence training garnered mostly positive reactions, participants with greater initial confidence in their military cultural competence may have differing expectations and scrutiny for the content presented.

At the same time, helping professionals with greater military cultural competence at baseline also shared praise for the app. In particular, these participants appreciated the portability of the app. While it is unclear why participants with greater military cultural competence would be particularly attuned to the opportunity to have information readily accessible via a mobile app, it may be the case that some factors unaccounted for in the current analysis (e.g., comfort with technology and frequency of work‐related travel) may add further context to this pattern.

4.4. Implications for Mobile‐App‐Based Trainings for Helping Professionals

The themes identified in the current study speak to participants' reactions to the REACH for Military Cultural Competence training, and they also underscore takeaways for other training opportunities for adult learners, including but not limited to helping professionals. On the basis of participants' engagement and satisfaction with this training and consistent with prior research (Curran et al. 2019), it appears that mobile apps may be an acceptable modality for delivering continuing education. However, the key to this acceptability is a user‐friendly app experience. Consideration should be taken in app development, with support from computer science professionals to ensure a smooth user experience with minimal technical issues and responsive technical support for users.

Regarding dosage, participants appeared to be generally satisfied with systematically receiving smaller amounts of information over an extended period of time. This approach to delivering information may be especially well‐suited for busy helping professionals with limited time for continuing education. Other continuing educational trainings for helping professionals may consider the value of implementing this type of extended delivery format. Our findings also have implications for graduate programs training helping professionals; rather than attempting to equip students with large quantities of information at one time on specific subjects (e.g., military cultural competence), these programs are likely best positioned to provide trainees a solid foundation in their specific scope of practice coupled with research literacy, so that, as professionals, they will be equipped to continue consuming and utilizing evidence‐based information in their practice.

4.5. Limitations and Future Directions

The findings of this study should be interpreted in the context of its limitations. Due to the evaluation being focused on a military‐specific training, the findings may not transfer to other continuing education platforms for helping professionals. Although a number of precautions were implemented, it is also possible that participants' responses during interviews were subject to social desirability and thus may skew more positively. Additionally, this study examined participants' reactions to the training to provide foundational support for the propensity of participants to achieve learning and change their behavior; however, the current study did not specifically examine these training outcomes. Future studies are warranted to examine the achievement of higher‐level outcomes within the Kirkpatrick Training Evaluation Model (Kirkpatrick and Kirkpatrick 2016) (i.e., learning, behavior, and results) with both quantitative and qualitative evidence. Lastly, our thematic analysis included a thorough comparison of themes across participants based on their baseline military cultural competence. Complementary quantitative analyses would deepen our understanding of meaningful differences in reactions based on baseline military cultural competence.

5. Conclusion

The findings from qualitative evaluations of the REACH for Military Cultural Competence training appears to be an acceptable, mobile‐app‐based training for helping professionals. Per the Kirkpatrick Training Evaluation Model, participants' generally positive reactions to the training provide foundational support for the potential for participants to achieve learning and behavior change (Kirkpatrick and Kirkpatrick 2016). In addition to identifying themes of training satisfaction, relevance, and engagement, related themes of enhancing access and accessibility to research and affirming existing knowledge also emerged. Furthermore, participants generally viewed the training as useful regardless of their prior competence, suggesting it could be broadly acceptable among helping professionals. Although the REACH for Military Cultural Competence app is currently unavailable, the findings of this study highlight the value of mobile‐application‐based and research‐informed professional development among helping professionals who serve military‐connected clients. Further, the findings offer preliminary support for extending mobile‐application‐based efforts to enhance cultural competence for helping professionals who serve other populations with unique challenges. Readers interested in reading the research summaries delivered through the app may access summaries through the Military REACH website (www.militaryreach.org).

Supporting information

Military_Cultural_Competence_App_Supplemental.