Abstract
Access to effective, replicable services is critical to reduce known mental health disparities for sexual and gender minority or LGBTQ+ veterans (lesbian, gay, bisexual, transgender, queer, questioning, and related identities). This paper examines the impact of a manualized 10-week health education group, called PRIDE in All Who Served on veteran patient experience, protective factors (e.g., identity acceptance), and mental health outcomes (e.g., suicide risk) at 10 Department of Veterans Affairs (VA) facilities. Implementation facilitation strategies (e.g., consultation, staff training) supported adoption at new sites and initial facilitators and barriers are described. Forty-four veterans (M = 47.21 years old) completed outcome surveys before and after the group. Significant improvement in acceptance concerns, identity uncertainty, community involvement, and likelihood of future suicide attempts were observed; other changes in mental health symptoms were not replicated in this sample (e.g., depression, anxiety). Open-ended veteran feedback reflected improved social support and engagement and increased self-understanding as the most frequent themes. At the facility-level, Healthcare Equality Index scores (a Human Rights Campaign measure of affirmative care climate) improved from 30% to 90% achieving top-performer/leader status from pre- to post-implementation. Manualized approaches, like PRIDE in All Who Served, that are based on established minority stress models and can be spread for use with diverse LGBTQ+ veterans (e.g., age, race, gender identity, sexual orientation, rurality, housing) are needed. The PRIDE in All Who Served program is an increasingly available resource to VA clinicians advocating for greater health equity within a national healthcare setting.
Keywords: Veterans, gender identity, sexual orientation, health education, access, health disparity
Sexual and gender minority populations are at increased risk for a variety of mental and physical health issues – many of which are exacerbated among those with a history of military service. Elevated anxiety, depressive symptoms, suicidality, interpersonal trauma exposure, and alcohol and substance use disorders are consistently reported by sexual and gender minority persons (Jackson et al., 2016; Matarazzo et al., 2014; Potter & Patterson, 2019; Valentine & Shipherd, 2018). Sexual and gender minority veterans, or LGBTQ+ veterans (lesbian, gay, bisexual, transgender and related identities), face increased health disparities compared to their heterosexual/cisgender veteran peers. Depression, alcohol abuse, and post-traumatic stress disorder (PTSD) are greater for LGBTQ+ veterans relative to non-minority veterans (Cochran et al., 2013); and when compared to cisgender veterans, veterans diagnosed with gender identity disorder (as it was named in the previous diagnostic manual, APA, 2000) are twenty times more likely to experience suicide-related events (Blosnich et al., 2013).
To begin addressing identified health disparities, Veterans Health Administration (VHA) leadership has created policy directives and LGBTQ+ Veteran Care Coordinator (VCC) collateral positions at all VHA facilities to influence organizational climate and promote access to care (Valentine et al., 2019). These efforts are reflected in the substantial number of Department of Veterans Affairs (VA) facilities (e.g., 61%, of 105 participating VAs in 2020) that have achieved Leader Status on the Healthcare Equality Index (HEI), a measure of inclusion and equity for sexual and gender minority staff and patients. Yet, despite this notable progress, many resources for frontline providers (i.e., evidence-based manualized interventions, LGBTQ+ note templates) are in the early stages of development and vary widely at the local level (e.g., Duan-Porter et al., 2017; Ramirez et al., 2013). A critical next step for improving outcomes is to increase access to services that are both effective and replicable (Fortney et al., 2014).
PRIDE in All Who Served: A Health Education Group for LGBTQ+ Veterans
PRIDE in All Who Served (Lange, 2018) was developed at the Hampton VA Medical Center in collaboration with LGBTQ+ veterans using human-centered design principles and an iterative feedback process (Lange et al, 2020). Developed as a 10-week health education group, each session focuses on a different content domain designed to provide psychoeducation and opportunities for discussion, as well as empower attendees with information needed to effectively navigate available resources (i.e., continuums of identity, coming out process, identity models, military culture, VA culture, affirmative care and whole health, sexual health, healthy and safe relationships, LGBTQ+ families, and community resources). The standardized topic list and session structure of the health education group differs from traditional support group models, which are not typically focused on structured content and tend to have a common goal of coping with an identified stressor or sudden life change (e.g., grief, cancer diagnosis; Seebohm, et al., 2013). The PRIDE in All Who Served program goes beyond peer-supported coping and strives toward health literacy and empowerment of the individual to proactively engage in health-related services. A small pilot evaluation informed by minority stress frameworks (Meyer, 2013; Testa et al., 2015) was conducted at two VA medical centers with support of the VHA Innovation Ecosystem Spark-Seed-Spread investment program (Vega & Kizer, 2020). Results supported the feasibility of the approach and preliminary areas of impact (i.e., Cohen’s d range +/− 0.40 to 1.59) on identity-related resilience and protective factors (i.e., identity affirmation, community involvement, problem-focused coping), mental health symptoms (i.e., depression/ anxiety, suicidal ideation), and veteran willingness to access care within the VA system (i.e., satisfaction with VA services, perception of staff competence). Delivery support tools (i.e., treatment manual, patient handouts, recruitment materials, weekly facilitator consultation calls, staff training materials, site visit launch) were also compiled to enable group adoption at the second VA site (Lange et al., 2020).
Spreading PRIDE in All Who Served
With feasibility demonstrated, implementation strategies defined, and significant interest from frontline clinical providers, the PRIDE in All Who Served team secured additional support through the VHA Innovation Ecosystem to spread the PRIDE in All Who Served health education group to 7 new VHA facilities in one year. Funding was used to support the group developer’s time in order to provide implementation facilitation (see Figure 1). Facilitation is an evidence-based implementation strategy that includes interactive problem-solving and a supportive relationship to support the adoption, implementation, and sustainment of an innovation or practice at a new site (Goodrich, Miake-Lye, Braganza, Wawrin, & Kilbourne, 2020; Powell, et a., 2015). Facilitation was necessary because PRIDE in All Who Served: a) was reportedly among the first LGBTQ+ veteran groups at most of the new sites, and b) is a “complex” innovation (Goodrich et al., 2021) that required coordination of multiple disciplines and staff across levels for successful adoption (e.g., billing and workload credit, privacy, safety, development of new documentation practices). This paper describes evaluation of the PRIDE quality improvement initiative and indicators that may impact implementation at new adopting facilities. Figure 1 depicts the conceptual model for the evaluation which was informed by minority stress models (Meyer, 2013; Testa et al., 2015), the health equity implementation framework (Woodward et al., 2019), and the healthcare access literature (e.g., Fortney et al., 2011). Specifically, this paper addresses two aims:
- To examine the impact of PRIDE group participation and facilitation on veteran, staff, and organization-level outcomes at 10 participating VA facilities, including:
- Veteran patient experience (e.g., satisfaction, access to affirmative care), veteran protective factors (e.g., identity acceptance, resilience, coping), and veteran mental health outcomes (e.g., symptoms of anxiety, depression, and suicide risk);
- Staff training evaluations (e.g., health literacy, intent to use material, awareness); and
- Facility affirmative care climate (i.e., Healthcare Equality Index scores).
To identify initial facilitators and barriers that may affect adoption of PRIDE at new facilities, to inform implementation at additional sites in the future, if indicated.
Figure 1. PRIDE in All Who Served Intervention, Implementation, and Impact Model.

Note. Ŧ = Implementation facilitation is the overarching implementation strategy, with four areas of discrete support (Powell et al., 2015) provided to address provider, cultural, & system-level barriers. ǂ = Actual and perceived access as defined by Fortney et al., 2011, and internal resources adapted from Minority Stress Models by Meyer, 2013, and Testa et al. 2015. ȡ = Primary outcomes depicted here are mental health focused to align with VHA’s prioritization of suicide prevention -- a known area of disparity for LGBTQ+ Veterans; however, the PRIDE health education group aims to impact broader health care utilization and physical health outcomes as well.
Method
Participants
Ten VA facilities across seven states (Alabama, California, Colorado, North Carolina, Ohio, Virginia, & Wisconsin) in geographically diverse areas (i.e., rural to urban) participated in the PRIDE Spread program between October 2018 and September 2019. Two sites (the developing site and a pilot site) were implementing PRIDE In All Who Served before October 2018, seven sites enrolled in the Spread program and completed the required reporting elements, and a 10th site enrolled after the initial funding decision in February of 2019. Participating sites represented the full range on the 5-point VA Facility Complexity Model (i.e., ratings based on total patient population, available clinical services, and education and research programs; Committee on Facilities Staffing Requirements for Veterans Health Administration, 2019). Sites were identified primarily through word-of-mouth recruitment – for example, VA LGBTQ+ VCCs who heard about the group from the program developer (TL, an LGBTQ+ VCC) or internal quality improvement calls that highlighted findings from the initial clinical pilot Lange et al., 2020). Each VA facility was required to commit at least one staff group facilitator to participate (i.e., groups led solely by trainees were discouraged to ensure sustainability beyond year one). Facilitators were licensed independent practitioners, often in the LGBTQ+ VCC role (53%), and represented a range of disciplines with experience facilitating health education or behavioral health groups (i.e., psychologist, social worker, nurse practitioner, and clinical pharmacist). Veteran participant demographics are described in Tables 1 and 2.
Table 1.
Impact of the PRIDE in All Who Served Program on LGBTQ+ Veterans in Groups at Risk of Inequities across VHA facilities.
| VHA Site (State) | Recent Service Members | Pre 9/11 | OEF/OIF | Women | Native Am. | Black / African Am. | Homeless | Rural | SMI | Total Veteran Contacts | Total Staff Trained |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 (VA) | 13 | 56 | 67 | 85 | 3 | 28 | 11 | 0 | 33 | 122 | 83 |
| 2 (AL) | 0 | 1 | 6 | 0 | 0 | 2 | 4 | 0 | 3 | 8 | 75 |
| 3 (MO) | 4 | 24 | 9 | 12 | 2 | 10 | 1 | 1 | 21 | 30 | 213 |
| 4 (WI) | 0 | 8 | 8 | 5 | 0 | 0 | 2 | 8 | 11 | 16 | 44 |
| 5 (NC) | 9 | 23 | 50 | 36 | 0 | 19 | 3 | 3 | 49 | 61 | 134 |
| 6 (AL) | 3 | 24 | 10 | 16 | 0 | 12 | 0 | 1 | 15 | 30 | 83 |
| 7 (CA) | - | 9 | 5 | 5 | 0 | 1 | - | - | - | 14 | 109 |
| 8 (NC) | 0 | 8 | 6 | 7 | 0 | 6 | 0 | 1 | 11 | 14 | 29 |
| 9 (CO) | 0 | 5 | 6 | 4 | 0 | 0 | 0 | 2 | 1 | 10 | 27 |
| 10 (OH) | - | - | - | - | - | - | - | - | - | 5 | - |
| Totals | 29 | 158 | 167 | 170 | 5 | 78 | 21 | 16 | 144 | 310 | 797 |
Note. VHA Site 1 is the development site. Characteristics reported reflect key demographic groups of Veterans tracked by the Office of Rural Health, one of the program sponsors. The 10th site in Ohio was not part of the original Spread grant application and did not submit this data that was required quarterly by the funder. Pre 9/11 refers to Veterans who served in conflicts prior to September 11, 2001. OEF/ OIF = Veterans who served during conflicts in Iraq and Afghanistan in Operation Iraqi Freedom and Operation Enduring Freedom. Women is inclusively defined and encompasses all non-male Veterans; recent service members are Veterans who are within 5 years of exiting the military. Homeless veterans are veterans who reported housing instability and SMI includes psychotic disorders, bipolar disorder, treatment resistant mood disorders, and post-traumatic stress disorders.
Table 2.
Sample characteristics of Veterans who completed outcome surveys during the PRIDE In all who Served LGBTQ+ health education group
| Demographics | Baseline n = 65 | Completers n = 44 |
|---|---|---|
| N (%) or M (SD) | N (%) or M (SD) | |
| Age | 48.64 (14.08) | 47.21 (13.53) |
| Race | ||
| White | 33 (50.8) | 20 (45.5) |
| Black | 21 (32.3) | 18 (40.9) |
| Something else | 3 (4.6) | 0 (0) |
| Bi- or Multi-Racial | 8 (12.3) | 6 (13.6) |
| Ethnicity | ||
| Non-Hispanic | 62 (95.4) | 42 (95.5) |
| Hispanic/Latinx | 0 (0) | 0 (0) |
| Missing | 2 (3.1) | 2 (4.5) |
| Sexual Orientation | ||
| Gay | 20 (30.8) | 14 (31.8) |
| Lesbian | 11 (16.9) | 7 (15.9) |
| Queer | 3 (4.6) | 3 (6.8) |
| Straight | 3 (4.6) | 2 (2.5) |
| Questioning | 2 (3.1) | 2 (4.5) |
| Pansexual | 4 (6.2) | 0 (0) |
| Bisexual | 5 (7.7) | 3 (6.8) |
| Don’t Know | 3 (4.6) | 2 (4.5) |
| Something else | 11 (16.9) | 8 (18.2) |
| Multiple Identities | 2 (3.1) | 2 (4.5) |
| Missing | 1 (1.5) | 1 (2.3) |
| Gender | ||
| Male | 23 (35.4) | 16 (36.4) |
| Female | 21 (32.3) | 12 (27.3) |
| Female-to-Male | 6 (9.2) | 4 (9.1) |
| Male-to-Female | 10 (15.4) | 7 (15.9) |
| GenderQueer | 1 (1.5) | 1 (2.3) |
| Multiple Identities | 2 (3.1) | 2 (4.5) |
| Missing | 2 (3.1) | 2 (4.5) |
| Sex | ||
| Male | 38 (58.5) | 25 (56.8) |
| Female | 26 (40.0) | 18 (40.9) |
| Missing | 1 (1.5) | 1 (2.3) |
| Military Branch | ||
| Army | 39 (60.0) | 28 (63.6) |
| Air Force | 9 (13.8) | 5 (11.4) |
| Navy | 8 (12.3) | 4 (9.1) |
| Marines | 7 (10.8) | 6 (13.6) |
| Coast Guard | 2 (3.1) | 1 (2.3) |
Note. M = Mean; SD = Standard Deviation
Baseline indicates Veterans that completed the assessment during Session 1. Completers indicates Veterans that completed assessments at both time points, during Session 1 and 10. Data was collected at all 10 participating sites.
PRIDE Group Implementation
After hearing about the group and expressing interest in offering the PRIDE group at their VA (i.e., through word-of-mouth/organic spread), internal facilitators received individual consultation with the PRIDE group developer (TL) and were provided with a Site User Agreement to obtain a formal commitment for participating in the Spread program (e.g., leadership support, protected time to offer the 1-hour program each week, consultation attendance expectations, collection of veteran outcome measures and feedback). Facilitators were mailed a Welcome Package that included a bound copy of the PRIDE manual and related start-up materials (e.g., VA-specific rainbow lanyards, rainbow VA office magnets consistent with Safe Zones in University settings, (e.g., Wolowic et al., 2017). Next, group facilitators were encouraged to attend a weekly 1-hour learning collaborative call offered twice per week to accommodate schedules. The focus of the calls was to offer an opportunity for ongoing consultation, problem-solving, and support through a collaborative relationship with the group developer (PRIDE in All Who Served Expert) and peers at other facilities who were navigating system-logistics to launch the group for the first time. Content of the facilitator calls covered veteran screening procedures, group recruitment and referral strategies, billing and workload-capture recommendations, documentation best practices in the electronic health record (e.g., electronic consults, note titles), and other strategies to ensure fidelity and sustainment. The implementation strategies used are depicted in Figure 1 and described in terms accepted within implementation science best practices, defined elsewhere (i.e., see Expert Recommendations for Implementing Change, ERIC project, Powell et al., 2015).
Program Evaluation Overview
The program evaluation had several components that were coordinated and collected by a single site (Tuscaloosa). First, the number of staff and veterans impacted by the Spread program were collected each quarter from group facilitators using a standardized template. Veteran identification in key demographic priority groups were also recorded (see Table 1). Second, program-specific veteran outcomes were collected using paper surveys during the first and last sessions of the group (Session 1 and 10) and - where possible - included measures commonly used in VA measurement-based care. Given the potentially sensitive nature of the information collected, surveys were completed anonymously with pre-treatment responses matched to post-treatment using a code created by the veteran (e.g., Damrosch, 1986; Garvey Wilson et al., 2010). Third, staff training evaluations and semi-structured site visit field notes were collected by the program developer (TL) during on-site launch visits at seven of the participating sites. Three of the ten sites did not have a site visit field note and/or contribute staff training evaluation data: Site 1 (Virginia) was the development site, Site 2 (Alabama) was in the second year of program implementation during the Spread year, and Site 10 (Ohio) joined the program after the initial funding request when site visit resources were allocated. Finally, Healthcare Equality Index (HEI) scores were collected during consultation calls, site visits, or from the publicly available Human Rights Campaign (HRC) website, when not otherwise provided. The coordinating site’s Institutional Review Board (IRB) reviewed the evaluation plan before data collection and determined the program to be non-research quality improvement activity consistent with the Innovation Network program goals (ORD, January 9, 2019; Vega & Kizer, 2020).
Measures
Veteran Patient Experience
Demographics.
A demographic questionnaire assessed age, race, ethnicity, sexual orientation, gender identity, sex assigned at birth, medical conditions, Veteran status, branch of service, and LGBTQ+ community involvement. Extensive lists of sexual and gender identity labels were provided to participants, as well as options to write in, and/or to check multiple identities should a participant desire to do so. As such, veterans could report unique self-label(s).
Veteran satisfaction and feedback.
Veteran perception of the PRIDE program was collected in two ways. First, veterans responded to a 2-item questionnaire (rated on a 1-7 scale) assessing perception of staff cultural competence and satisfaction with VA health services delivery. Next, veterans were asked for open-ended written feedback on the group process (e.g., session length, topics) and about the impact of the group on their lives (e.g., What impact, if any, has the LGBTQ+ group had on your life?).
Veteran Internal Resources
Identity-related acceptance.
Subscales of the Lesbian, Gay, and Bisexual Identity Scale (LGBIS; (Mohr & Kendra, 2011) were modified to reflect both sexual and gender minority identity (LGBTIS) for the following domains: acceptance concerns (i.e., fear of negative judgment/ rejection by others), concealment motivation (i.e., pressure to hide LGBTQ+ identity), identity uncertainty (i.e., vacillation in self-identifying as LGBTQ+), internalized homonegativity/transphobia (i.e., internalization of heterosexist/ cisgender beliefs/ stigma), and identity affirmation (i.e., extent one has resolved coming out/ embraced identity). Internal consistency values were acceptable: acceptance concerns (.76), concealment motivation (.74), identity uncertainty (.70), internalized homonegativity (.87), and identity affirmation (.87).
Resilience and Coping.
Subscales of the Coping Self-Efficacy Scale (CSE; Chesney et al., 2006) and the Lesbian, Gay, and Bisexual Positive Identity Measure (Riggle et al., 2014 modified to include gender identity, LGBT-PIM) measured resilience and coping. The two subscales of the CSE related to group content were: problem-focused coping and getting social support. The third is excluded as it focuses on more traditional cognitive therapy skills (i.e., stopping unpleasant thoughts and emotions) not addressed in the group. Internal consistencies were good in the present sample (i.e., problem-focused coping = .89 and getting social support = .80). Three of five LGBT-PIM subscales, which assessed positive self-appraisals and social aspects of sexual and gender minority identity, aligning with PRIDE were used: authenticity (i.e., comfort expressing LGBTQ+ identity in social situations), self-awareness (i.e., belief that one’s LGBTQ+ identity improves personal insight), and community involvement (i.e., involvement in and support from the LGBTQ+ community) (Riggle et al., 2014). Higher scores indicate more positive identity and internal consistency was acceptable in the present sample: self-awareness (.89), authenticity (.88), and community involvement (.90).
Veteran Mental Health Outcomes
Depression, anxiety, and suicide symptoms.
Symptoms of distress were assessed via three instruments: Patient Health Questionnaire-9 (PHQ-9; Kroenke et al., 2001), Generalized Anxiety Disorder–7 Scale (GAD-7; Spitzer et al., 2006), and the Suicidal Behaviors Questionnaire–Revised (SBQ-R; Osman et al., 2001). The PHQ-9 is a measure of depressive symptoms (Kroenke et al., 2001); internal consistency values was .89 in the present sample. It also provides clinical cut-score ranges of symptom severity to aid provider interpretation, as follows: minimal, mild, moderate, moderate-to-severe, and severe (i.e., ≥ 15 indicates moderate or higher severity, Kroenke et al., 2001). The GAD-7 assesses generalized anxiety symptoms (Spitzer et al., 2006); internal consistency was .91. The GAD-7 further provides clinical cut-score ranges of symptom severity as follows: minimal, mild, moderate, and severe (i.e., ≥ 10 indicates moderate or higher anxiety; Spitzer et al., 2006). The SBQ-R is a 4-item suicide risk screener that assesses aspects of suicide-related behavior (each rated on a different scale). A total score can be tabulated; however, as we were interested in pre-post change, the last SBQ-R item (i.e., “How likely is it that you will attempt suicide someday?” 0 = never to 6 = very likely) was used because it provides a future-oriented assessment of suicide attempt risk (e.g., Cunningham et al., 2020; Lund et al., 2019). Clinical cut-score frequencies for suicide risk are based on the total score (i.e., ≥ 7 indicates elevated risk, Osman et al., 2001).
Staff Training Outcomes
Site visits were a critical part of the care environment scan and staff training components of implementation facilitation (see Figure 1). A 1-hour staff training was designed to mirror content delivered to veterans in the PRIDE in All Who Served health education group and approved by the Human Rights Campaign (HRC) to meet HEI requirements. Adequate staff knowledge, awareness of LGBTQ+ health factors, and available services are necessary for building a coalition to sustain referrals, promote a local affirmative care environment, and increase the likelihood that healthcare staff will utilize their training and be prepared to engage with veterans after they attend the group. This conceptualization of staff training goals aligns with social-cognition literature on the social and individual factors that influence new learning and subsequent behavior (e.g., Burnette, Ramchand, & Ayer, 2015). A 17-item post-training evaluation was developed to assess knowledge within the training hour consistent with continuing education assessment in healthcare settings. Two demographic questions characterized staff attendees (i.e., health service discipline/field, and prior LGBTQ+ health training experience, yes/no). A nine-item true/false LGBTQ+ health literacy quiz was developed based on training content. A six-item Likert-scale assessed perceptions of the LGBTQ+ care and the training program itself, including: (1) comfort providing culturally competency LGBTQ+ veteran care; (2) awareness of LGBTQ+ VCC at VA facilities; (3) familiarity with VA policies impacting LGBTQ+ veteran care; (4) awareness of LGBTQ+ veterans seeking services at VA facilities; (5) intent to use chosen names/pronouns for transgender and gender non-confirming (TGNC) veterans; and (6) intent to use content from the training program.
Staff training and evaluation was provided in one of two ways: 1) by the PRIDE in All Who Served developer during facilitation site visits, or 2) by the newly trained group facilitators. Newly trained group facilitators were only asked to track the number of staff they trained, therefore, the post-training evaluations reflect trainings conducted directly by the PRIDE in All Who Served team.
Facility Affirmative Care Climate
The HEI is an annual facility-level application submitted to the HRC that assesses a healthcare facility’s LGBTQ-related policies and practices. Scores range from 0 to 100 with points awarded based on evidence of equity and inclusion of a facility’s LGBTQ staff and patients. The HRC identifies a score of 90 as a top performing facility, and a perfect score of 100 is awarded status as a “Leader in LGBTQ Healthcare Equality.” For the PRIDE in All Who Served Spread program, facility scores were examined in the year before, during, and after PRIDE group adoption and implementation supports.
Data Analysis
To address Aim 1, descriptive statistics are reported for veteran, staff, and facility-level outcomes. Paired-samples t-tests with Cohen’s d effect sizes were used to evaluate pre-post changes in veteran outcomes (Cohen, 1988; Cohen et al., 2013). Missing data on scale items ranged from 0-11.4% in the final sample of veterans with pre and post data. Multiple imputation (Enders, 2017) was used to replace missing data on scale items, where feasible. Analysis of Variance (ANOVA) and independent samples t-tests were used to examine differences in LGBTQ+ health literacy by staff discipline and prior training experience in LGBTQ+ health. Linear regression modeling was used to identify predictors of intent to use training content.
Qualitative summaries of LGBTQ+ veteran program and service feedback (Aim 1), and site visit field note data (Aim 2) were analyzed by members of the team with qualitative methodology experience (MH, RC). A basic approach to identifying qualitative themes was employed following principles in the public health literature (e.g., CDC, 2018). Manual analysis was performed by following a stepwise approach of: (1) organizing data, (2) discussing and defining themes, (3) classifying themes. For veteran feedback, open-ended responses were sorted into themes, exemplar quotes were identified to reflect themes, and organized into a table for consensus review. Similarly, site visit field notes were examined using excel, with the Health Equity Implementation Framework (Woodward et al., 2019) as an organizational structure for sorting barriers and facilitators by site before looking for themes across sites by domain. This framework identifies healthcare system factors by setting: inner local (clinic or unit or ward), inner organizational (local facility), and outer healthcare systems (national VHA system). It also describes societal influences that may impact equity within the healthcare setting that are particularly relevant to patient access and care (e.g., sociopolitical influences, community resources).
Results
PRIDE in All Who Served Participants – Veteran and Staff Impacts
The total number of unique veteran and staff contacts during the implementation year are described by site in Table 1. Group facilitators reported a total of 310 LGBTQ+ veterans were served across the implementation sites (site 10 did not track detailed data since they were not required to report quarterly to the sponsor). More than half (56%) were women, inclusive of sex assigned at birth and gender identity; 27% were Black/African American or Native American; 47% had a serious mental illness; and veterans were evenly distributed between more recent (e.g., post-9/11 conflicts) and previous eras (e.g., Vietnam). This number includes veterans who participated in the group and veterans who were referred or screened for the group but did not participate due to scheduling conflicts, receiving care in a residential inpatient setting and thus not available to complete a 10-week cycle, veterans who received session content individually but did not participate in a group setting, and those who were screened but referred for other services to best meet their needs (e.g., veterans who required a higher level of care than could be provided in a health education group). A total of 797 VA staff members received PRIDE-related training during the implementation year; either directly from the PRIDE group developer (TL) during site launch visits (n = 145, 18%) or as training provided by the local group facilitator, often using PRIDE-related materials (n = 652, 82%). Staff post-training evaluations were only collected during the site visits.
PRIDE in All Who Served – Veteran Group Outcomes (Aim 1a)
A total of 65 LGBTQ+ veterans across sites elected to complete the pre-treatment assessment during Session 1. However, only 44 (68%) also completed a post-evaluation assessment during Session 10 (a 32% attrition rate in the anonymous evaluation measures). It is unknown how many veterans attended the group and opted out of the voluntary survey packet. Table 2 contains a summary of PRIDE group attendee demographics, as well as the pre-post evaluation sub-samples. No notable demographic differences were observed, suggesting no clear pattern in attrition for the evaluation.
Veteran Patient Experience
Veteran responses (n = 44) to an open-ended question about the impact of the PRIDE group on their lives revealed an overwhelmingly positive response. Table 3 depicts themes and verbatim quotes extracted from the hand-written responses. The most frequent response described a sense of social support and engagement (n = 20, 45%); improved self-understanding and identity (n = 7, 16%), reporting that they felt better with communication and openness (n = 7, 16%), positive impacts on well-being and confidence (n = 5, 11%), and improved understanding of LGBTQ+ healthcare (n = 1, 2%) were also spontaneously described to this open-ended question. Despite positive qualitative feedback and satisfaction with the group, quantitative changes on indicators of perceived access and satisfaction did not change from pre- to post-group (Table 4) in the current sample as they did in the initial two-site pilot (e.g., Lange et al., 2020).
Table 3.
Patient Experience / Narrative Thematic Feedback Quotes
| Theme (Number of responses) | Sample Quotes |
|---|---|
| Positive impact on social support and connectedness (n = 20) | “Always a great feeling to know I’m not alone—Identification is priceless.” – 55-year-old Black gay woman |
| “I was able to meet [other LGBTQ+ people] for the first time.” – 38-year-old White transgender woman | |
| “Being a part of the LGBT group [helped] give me the confidence to be a part of other groups.” – 36-year-old Black transgender man | |
| Improved self-understanding and identity (n = 7) | “This group has helped me to come out and be myself.” – 37-year-old Black gay man |
|
| |
| Better with communication and openness (n = 7) | “I feel I can openly discuss my needs.” – 68-year-old White transgender man |
| “[I have] new ways to express my health care goals.” – 44-year-old White queer woman | |
| “[I am] able to relate to my partner and associate better [with] my health care provider.” – 52-year-old Black gay man | |
|
| |
| Positive impact on well-being, confidence (n = 5) | “It made me feel at peace.” – 57-year-old Biracial lesbian “Made me more comfortable with myself [and] allowed an outlet for me to voice my fears and pains.” – 35-year-old Black asexual/bisexual woman |
| Improved understanding of LGBTQ+ healthcare (n = 1) | “[I have a] better understanding of advances in affirmative care for LGBT [veterans]. That made me more comfortable.” – 33-year-old Black lesbian |
Note. Participant descriptions reflect self-identification of age, race, sexual orientation, and gender identity. Data was collected at all 10 participating sites.
Table 4.
Summary Statistics for LGBTQ+ Veteran Pre-Post Patient Experience and Clinical Impact
| Outcome | Pre-Program M(SD) | Post-Program M(SD) | T(df) | p-value | Cohen’s d |
|---|---|---|---|---|---|
| Perceived Access / Satisfaction | |||||
| VA Staff Cult. Competence1 | 4.98(1.47) | 5.11(1.80) | 0.45(40) | .65 | 0.07 |
| Satisfaction w/ VA Services1 | 5.25(1.51) | 5.25(1.78) | −0.02(42) | .98 | 0.00 |
|
| |||||
| Internal Resources | |||||
| Acceptance Concerns2 | 11.52(3.92) | 10.41(3.78) | −2.79(43) | .008 | −0.41 |
| Concealment Motivation2 | 12.82(3.73) | 12.52(3.69) | −0.68(43) | .50 | −0.08 |
| Identity Uncertainty2 | 8.17(3.86) | 7.03(3.47) | −2.13(43) | .04 | −0.30 |
| Internalized Homonegativity2 | 7.79(4.89) | 7.19(4.16) | −1.43(43) | .16 | −0.20 |
| Self-Awareness3 | 25.20 (6.64) | 26.11 (5.53) | 1.49 (43) | .14 | 0.21 |
| Authenticity3 | 26.74 (6.94) | 27.40 (5.76) | 0.89 (43) | .38 | 0.13 |
| Community Involvement3 | 20.56(7.29) | 21.86(7.65) | 1.98(43) | .05 | 0.31 |
| Problem-focused Coping4 | 37.18(13.02) | 40.08(11.30) | 1.64(43) | .11 | 0.23 |
| Getting Social Support4 | 14.96(8.38) | 16.24(8.67) | 0.96(43) | .34 | 0.15 |
|
| |||||
| Mental Health Symptoms | |||||
| Depression5 | 10.64(7.04) | 9.31(6.18) | −1.00(31) | .33 | −0.28 |
| Anxiety6 | 8.56(6.32) | 7.40(5.24) | −1.97(31) | .06 | −0.25 |
|
| |||||
| Suicide Risk | |||||
| Suicide Attempt Likelihood7 | 1.40(1.44) | 1.06(1.37) | −2.35(31) | .03 | −0.31 |
Note.
VAMC derived question;
LGBTIS subscale (adapted from Mohr & Kendra, 2011),
LGBT-PIM subscale (Riggle et al., 2014),
Coping Self-Efficacy subscale (Chesney et al., 2006),
PHQ-9 total (Kroenke et al., 2001),
GAD-7 total (Spitzer et al., 2006),
SBQ-R item 4 (Osman et al., 2001), Bold font denotes significance at p − .05.
df = degrees of freedom; Effect size interpretation = small (+/− 0.2), moderate (+/− 0.5) (Cohen, 1988); VA = Veterans Affairs Medical Center; Effect size directions specified so negative values denote pre-post program reductions. Data was collected at all 10 participating sites.
Veteran Internal Resources
Identity.
Pre-post changes in two of the identity-related domains were significant (Table 4). A reduction in acceptance concerns and a reduction in identity uncertainty both reflected small to moderate effect sizes. Other identity domains were not significant in this relatively small sample.
Resilience.
A significant increase in community involvement was observed. No other significant or meaningful pre-post changes in identity resilience or coping beliefs were detected.
Veteran Primary Outcomes / Mental Health Impacts
Depression.
PHQ-9 scores decreased by 1-point from pre- to post-assessment (see Table 4), a small effect that did not reach significance. Analyses examining changes in symptom severity revealed a small reduction of veterans falling in the moderate and severe categories from pre- to post-program assessment. Specifically, pre-program depression symptom severity frequencies were: minimal (n = 12, 27.3%), mild (n = 8, 18.2%), moderate (n = 11, 25.0%), moderate-to-severe (n = 8, 18.2%), and severe (n = 5, 11.4%). Post-program symptom severity frequencies were: minimal (n = 12, 27.3%), mild (n = 11, 25.0%), moderate (n = 10, 22.7%), moderate-to-severe (n = 8, 18.2%), and severe (n = 3, 6.8%).
Anxiety.
A 1-point reduction was also observed from pre- to post-assessment on the GAD-7 (see Table 4), a small Cohen’s d effect size at p = .06, which may not achieve clinical significance. A similar pattern of small reduction of veterans falling in the severe category from pre- to post-program assessment was observed in symptom severity cut-off scores, such that pre-program anxiety symptom severity frequencies were: minimal (n = 15, 34.1%), mild (n = 11, 25.0%), moderate (n = 8, 18.2%), and severe (n = 10, 22.7%). Post-program anxiety symptom severity frequencies were: minimal (n = 15, 34.1%), mild (n = 12, 27.3%), moderate (n = 13, 29.5%), and severe (n = 4, 9.1%).
Suicide Risk.
Suicide attempt likelihood was significantly reduced during the 10-week group (Table 4). Categorical risk also reflected a reduction after the group, with 72.7% (n = 32) of veterans at elevated suicide risk pre-group and 56.8% (n = 25) at elevated risk at post-assessment, reflecting a small effect size.
Staff Training Outcomes (Aim 1b)
A total of 100 of the 145 VA staff (69% participation rate) who attended a PRIDE in All Who Served training provided evaluation information. The majority (n = 64, 64.0%) of staff reported prior LGBTQ+ health training. Discipline was collapsed into the following categories: mental health (n = 46, 46%), nursing (n = 22, 22%), other health discipline (n = 29, 29%), and declined (n = 3, 3%).
Staff averaged 92.2% correct (SD = 11.1%) responses to health literacy items on a 100-point scale. Differences by discipline were observed in LGBTQ+ health literacy, F(2, 94) = 3.57, p = .03, with mental health professionals (M = 95.4%, SD = 7.2%) scoring higher on LGBTQ+ health literacy than those in nursing professions (M = 90.4%, SD = 8.6%, p = .02, Cohen’s d = 0.63) or other staff (M = 91.2%, SD = 10.0%, p = .04, Cohen’s d = 0.48) on LSD post-hoc analyses. Intent to use the training content was not associated with LGBTQ+ health literacy scores (r = −.01, p = .93), but was associated with awareness of LGBTQ+ veteran services (r = .36, p < .001) and awareness that LGBTQ+ veterans are seeking services at their facility (r = .30, p = .002). Knowledge of available LGBTQ+ veteran services was also associated with the perception that LGBTQ+ veterans are seeking services (r = .28, p = .005). LGBTQ+ health literacy, positive awareness of LGBTQ+ veteran services, and perceived likelihood of LGBTQ+ veterans seeking services were examined for collective effects on intent to use training content. The set of attitudes and knowledge accounted for significant and small-to-moderate variance in intent to use training content, F(3, 96) = 6.79, p < .001, Adj. R2 = .15. Positive awareness of LGBTQ+ veteran services demonstrated a significant moderate positive effect on intent to use training content (B = 0.14, SE = 0.05, t = 3.09, p = .003, ηp2 = .09).
Facility Affirmative Care Climate (Aim 1c)
Figure 2 summarizes HEI scores, an indicator of facility climate, for all ten implementation sites; facilities are described here at the state-level rather than city level for privacy reasons. Site 1 (development site in Virginia) began PRIDE in All Who Served in 2016; Site 2 began in 2018. All other sites began implementation in 2019 during the Spread program. HEI scores were expected to improve during implementation and in the year after the implementation phase. The Missouri site did not participate in 2020; Wisconsin, California, and Colorado sites did not participate in 2019. All sites except for Missouri, which did not submit, and Colorado which was a Top Performer, received a score of 100, making them Leaders in Healthcare Equality. Sites in North Carolina and Alabama increased their scores by 25 and 30 points respectively in the year after initial implementation of PRIDE in All Who Served. The three sites that did not report during the implementation year (2019), increased their scores from 2018 to 2020 by 70, 30, and 40 points.
Figure 2. Healthcare Equality Index (HEI) status as a proxy for access to affirmative care services before, during, and after the PRIDE in All who Served program at 10 VA Medical Centers.

Note. Annual facility HEI applications are scored by the Human Rights Campaign at the end of each year. HEI status (Top Performer, HEI Leader, Participant) and scores are publicly available. Leader/Top Performer indicates HEI scores ranging from 80 to 100. Participant indicates HEI scores below 80. Missing indicates that the facility did not submit an HEI application to the HRC that year. Leadership turnover is a common reason for missing a cycle.
Implementation Facilitators & Barriers (Aim 2)
Most of the field notes collected during site-visits (i.e., environmental readiness scan, staff training, building coalitions on the ground, problem-solving) focused on the provider, organizational, and local health system contexts. Observations were organized into 17 discrete facilitators, and 21 barriers (see Table 5). As early adopter sites that requested involvement in the PRIDE Spread program, many of the PRIDE group leaders were noted to be enthusiastic champions with expertise in affirmative care who were facing barriers associated with their role (e.g., protecting time or managing turnover in the LGBTQ+ VCC position). The presence or absence of LGBTQ+ visibility (i.e., services, providers, leadership support), coordination of care (i.e., disjointed campuses or services that exist in isolation) also emerged as themes at the hospital and local clinic levels. The influence of specific individuals within the system were also noted (e.g., an unsupportive supervisor, an endocrinologist who refused to provide transgender affirmative services, delays in communications/recruitment due to problematic gate keepers). Sociopolitical factors outside of the hospital context were largely described as a barrier, with only one exception, and themes ranged from lack of available resources in the community due to rural geography to more explicit indications of bias and negative influences (e.g., race-related tension, religious norms, military culture, and local community that was perceived to be “not LGBTQ+ friendly”).
Table 5.
Implementation Facilitators and Barriers by Health Equity Implementation Framework Domain
| Domain | Facilitator Themes | Barrier Themes |
|---|---|---|
| Group Facilitator Factors: | 1. Veteran comfort with provider; 2. Enthusiastic / committed provider / facilitator; 3. Staff expertise in affirmative care |
1. LGBTQ+ Veteran Care Coordinators – without dedicated time for role; 2. LGBTQ+ VCC turnover (interest in leaving the role, new/unfamiliar with the role/HEI) |
| Organizational Level: | 1. Diversity / visibility of LGBTQ+ staff (out in the workplace); 2. Leadership that prioritizes LGBTQ+ programming/training; 3. Visible symbols of affirmative care (e.g., lanyards); 4. Affirmative care integrated across services; 5. Support of LGBTQ+ employees (EEO, HR) | 1. Leadership turnover; 2. Facility-level communication barriers (e.g., accessing public affairs officers); 3. Organizational structure of LGBTQ+ VCC role under discipline rather than direct report to leadership; 4. Split campus – geographic complexity of meeting needs; 5. Staff diversity does not translate to Veteran experience; 6. Low visibility of services / difficult to identify & navigate |
| Local Level (e.g., clinic): | 1. Location of the group outside of mental health; 2. Transgender specialty care established / visible; 3. Trainee involvement; 4. Investment of multiple disciplines (chaplain, infectious disease, etc.); 5. Protected time for LGBTQ+ Veteran Care Coordinator Role / access to resources | 1. Lack of supervisor support for LGBTQ+ VCC role; 2. No materials in waiting rooms, common areas; 3. Visibility / services limited to mental health; 4. Endocrinologist full case load / unwilling to provide services for transgender Veterans; 5. Poor coordination of existing services / low admin. support. |
| Sociopolitical Influences & Community Visibility, & Resources: | 1. LGBTQ+ visibility in the local community (e.g., neighborhood, spaces, Pride Events) | 1. Local community “not LGBTQ friendly”; 2. Rural location with few / no community spaces / resources; 3. Veteran population less inclusive of LGBTQ+ peers than VA staff; 4. Racial tension persists in community; 5. Strong religious norms make coming out difficult. |
Note. Data was collected from 7 of the participating sites that received a site visit during the 1-year Spread Evaluation.
Discussion
Healthcare disparities for sexual and gender minority people cannot effectively be addressed until interventions are adopted into clinical practice settings (Perry & Elwy, 2021). The VA Innovation Ecosystem offers a path for frontline innovative practices, developed in collaboration with “end users” (e.g., LGBTQ+ veterans) and stakeholders (e.g., LGBTQ+ VCCs), to be piloted and spread to other VA facilities to meet urgent clinical needs (Kilbourne et al., 2019; Vega & Kizer, 2020). The goals of this paper were to: 1) examine impacts of the PRIDE in All Who Served Health Education program – a VHA Innovation Spread Investment – on veteran, staff, and system-level outcomes, as well as, to 2) describe conditions that may facilitate or slow implementation and uptake in a new facility. The results of this evaluation showed that the PRIDE group may be a viable resource for frontline providers, like LGBTQ+ VCCs, who are often faced with the impossible task of meeting the needs of a diverse group of individuals with a range of identities in a time-limited, 1-hour per week format.
Several positive veteran impacts (Aim 1a) were observed across both open-ended and quantitative outcome measures in this small sample. Quotes reflecting a sense of increased social connection (e.g., “I’m not alone”) were the most frequently observed theme (Table 3), a finding that may align with the significant change on the resilience measure of community involvement (i.e., subscale of the LGBT-PIM in Table 4). Reduction in identity-related stress such as concerns about being accepted and uncertainty about their identity were also observed and aligned with themes identified in the open-ended questions (i.e., improved self-understanding and identity, enhanced self-confidence). Spontaneous reports about improved ability to communicate needs, relationship to mental health provider, and LGBTQ+ healthcare in general were less frequent and quantitative measures of perception of the VHA were not significantly changed in this sample. In terms of mental health impacts, one of the most promising findings (and a replication of the previous pilot) is the reduction in suicide risk – both in terms of those who are at elevated risk and those who indicate a future likelihood of attempting in the future. This reduction is particularly noteworthy given the focus on health education – and corresponding emphasis that this is not a mental health or psychotherapy group. Aligning the group organizationally with other health behavior groups (e.g., smoking cessation, weight loss) rather than traditional mental health groups (e.g., trauma-focused groups for posttraumatic stress disorder) communicates to both the veterans and providers that LGBTQ+ status is not a mental health diagnosis. Though generalizability beyond this small sample is limited at this time, a future direction could be to examine positive changes in community involvement and reductions in minority stress as an explanatory pathway consistent with minority stress models.
Staff-focused measures (Aim 1b) – although limited in scope – do reflect the utility of implementation strategies used by the external facilitator/ PRIDE group developer (TL) during trainings and site visits. With increased attention on affirmative care principles, inclusivity in healthcare, and competencies in LGBTQ+ health for clinical providers and trainees (e.g., Hardacker et al., 2014; Keuroghlian et al., 2017; Valentine et al., 2019), more provider-focused educational resources are available than ever before. Yet more than one third of staff trainees reported no prior LGBTQ+ health training – making the post-training health literacy scores of 90-95% across disciplines notable. Importantly, the staff who were more aware of LGBTQ+ veterans accessing care were also the ones who were most likely to use the content in the future (i.e., a moderate positive effect size), consistent with social-cognition-informed training models (e.g., Burnette, et al., 2015). Given the critical role that staff serve in setting an affirmative tone in healthcare settings, as referral sources, and sometimes as gatekeepers, it is promising that a brief 45-60 minute classroom training was well-received.
Finally, preliminary facility-level impacts (Aim 1c) were observed over a 3-year period (pre-implementation, during implementation, and post-implementation) for detectable patterns at the system level. Although causality cannot be determined in the current quality-improvement design – particularly for organizational-level outcomes like the HEI – positive movement was observed in the 10 implementing sites. It is unclear how much facility scores fluctuate from year-to-year without system-supports or active implementation facilitation/consultation. Decreases in scores and/or missed application cycles have been observed in about one-third of VA facilities who participated in the past 3-years, perhaps indicating staffing instability (e.g., turnover in staff positions) or lack of coordination in the submission process across the many departments that must contribute (e.g., Equal Employment Opportunity, Human Resources, patient care providers, outreach teams). Future evaluation work could randomly select facilities matched by complexity and geographic location to compare HEI scores at sites that receive affirmative care-focused facilitation like that offered with the PRIDE in All Who Serve Group to sites without such supports. Though empirical studies examining the relationship between HEI scores and patient care quality are scant, early evidence suggests that HEI scores are positively correlated with other care quality indicators (e.g., nursing competencies; Blackwell et al., 2020). Furthermore policy revisions made at the national level or local facility leadership open to adopting more LGBTQ+ services may be contributing to the positive impacts observed. More work is needed to determine what role – if any – programs like this one with systemic facilitation elements (e.g., Figure 1) can consistently have on facility-level measures.
Implementation Context: A Theory-Informed Understanding of Barriers and Facilitators
This work builds on established minority stress models (Meyer, 2013; Testa et al., 2015) and more recent applications of implementation science frameworks into health disparities research (Woodward et al., 2019). To our knowledge, this project is among the first to apply evidence-based implementation strategies (e.g., implementation facilitation, Kirchner et al., September 2019; Kirchner et al., 2014) to a healthcare innovation for sexual and gender minority (SGM) individuals - either within or outside of VHA settings. Though Woodward and colleagues (2019) Health Equity Implementation Framework and interview guide addresses the impacts of racial discrimination on barriers to care, more work is needed to harness the growing field of implementation science to the benefit of SGM individuals broadly and LGBTQ+ veterans in particular (Perry & Elwy, 2021). In our initial observations about the conditions that may foster growth or impede it (Aim 2), we are reminded that addressing disparities is a local issue. Clinic-level and facility-level factors were more frequently described as facilitators and barriers than provider or sociopolitical influences (Table 5). Themes identified for the group facilitators, organizational, and local level (Table 5) were just as likely to characterize facilitators as barriers, but the sociopolitical and community influences were described more often as negative than supportive. Instances of discrimination and lack of leadership support were salient reminders of the barriers that LGBTQ+ veterans face in accessing care (e.g., endocrinologist who would not provide affirmative services for transgender individuals, see Table 5).
Contributions and Future Directions
This paper contributes to the literature and clinical practice with SGM individuals in several notable ways. First, group facilitators at all 10 sites were able to use the PRIDE treatment manual and implementation facilitation supports to meet the needs of diverse LGBTQ+ veterans at their facilities (Table 1). Sixty-five LGBTQ+ veterans were willing to complete the survey measures at least once to contribute to the quality improvement initiative and the majority completed it a second time. Like the initial pilot at two sites (Lange et al., 2020), the results of this evaluation demonstrate that identity-related patient outcomes (e.g., acceptance concerns, identity uncertainty, community involvement) can show some progress after a brief health education group. It also replicated the significant finding of the prior pilot on reduction in suicide attempt likelihood – an outcome of critical importance to veterans and the VHA healthcare system alike. A frequent critique of existing LGBTQ+-focused studies has been an overreliance on convenience samples from LGBTQ+ health centers that are predominantly white, wealthy, and educated. This sample, in contrast, is majority female, racially diverse, includes individuals with housing instability (e.g., see Table 2), and was recruited from a large national healthcare setting rather than an LGBTQ-focused health clinic.
As a quality improvement initiative and not a research study, this evaluation was limited in ways that could impact the interpretation of findings. First, surveys were completed by a relatively small number of individuals (Table 2) compared to those that staff facilitators reported were contacted (Table 1). There was not a way for the evaluation team to independently examine attendance at the groups (e.g., through electronical health records data). Unfortunately, data are not available on the reasons for the discrepancy in numbers served and numbers with outcome measures (i.e., how many were screened out before attending a group, how many received the content but were not offered the surveys, how many attended but declined outcome measures). Second, all veteran data were collected by the frontline clinicians offering the PRIDE group (not independent researchers) and the measures were limited in length to reduce burden on group facilitators and patient participants. Third, a comparison condition was not readily available within usual care at implementing sites (for either veteran-level outcomes or facility-level outcomes) – a problem that is further complicated by challenges identifying and documenting LGBTQ+ status in electronic health record systems. Many VA facilities offer a monthly unstructured support group; however, since internal quality improvement was the focus of this evaluation, outreach and comparison to other services was beyond the scope of the current program. Fourth, the evaluation focused on the first 1-2 groups provided at a new site during the narrow implementation window (i.e., approximately 6-months). Therefore, sustainability of the group over time and the duration of clinical impacts observed in attendees (e.g., do reductions in suicide risk continue at 3 or 12 months after the group?) are areas that need further assessment in the future. Systematic assessment of fidelity to the intervention (i.e., beyond self-report and expectations on clinical consultation calls) would also increase confidence as the group is implemented with new facilitators and at additional sites in the future. As such, development of fidelity assessment tools and strategies warrants attention in the future. Finally, these data were collected for in-person groups before the current COVID-19 pandemic so additional work is needed to understand the impact of adaptations for telehealth on delivery, implementation, and outcome indicators.
Sustainability of the PRIDE in All Who Served groups was emphasized from the initial implementation (Figure 1) to require minimal time of the practitioner after the initial start-up phase (i.e., 1 hour of clinical service per week). At the time of this paper, 90% of the facilities (9 of 10 sites) reported meeting the sustainment benchmark of offering two or more group cycles and several have maintained engagement in a monthly PRIDE in All Who Served community of practice call. Additional opportunities exist to understand the contextual conditions (i.e., implementation facilitators and barriers) in which PRIDE groups grow over time or are diminished as a result of factors like the ones described in the field notes. This will become even more important as currently available VA patient care services expand (e.g., pre-exposure prophylaxis for HIV prevention, assessments of readiness for hormone therapy, voice modification therapies) and newly developed ones gather additional evidence (Lange, 2020) over the next decade.
Summary and Conclusions
New opportunities to elevate veteran’s voices and the clinicians advocating for them are increasingly possible, yet significant work remains (Atkins et al., 2014). The PRIDE in All Who Served health education group could serve as a model for progress for other agencies and outreach efforts that interact with active service members and veterans who identify as sexual and/or gender minorities. Establishing a shared language, acknowledging the role of military culture on identity and healthcare access, and empowering individuals to connect with others who share their experience are powerful goals that can be applied across University, military, and community Vet Center settings.
Impact Statement:
LGBTQ+ veterans are at risk for health disparities that can be addressed, in part, by improving access to services in an affirming environment. PRIDE in All Who Served is the first standardized health education group for veterans that improves identity-related acceptance, community involvement, engagement and social support, and reduces suicide risk. Tailored health education groups like this one could become an essential tool for providers focused on reducing health disparities.
Acknowledgements:
This work was supported by a FY19 Spread Investment (Lange & Hilgeman, Co-Leads) from the VHA Innovation Network in collaboration with April Jones, Innovation Specialist; Allison Amrhein, Director of Operations; and Brynn Cole, Director of Programming. We deeply appreciate the commitment and ongoing partnership with Pride Group facilitators at our 10 early adopting sites (alphabetically): Vincent Intoccia, PsyD; Lori Hall, Psy.D.; Fallon Trent, Psy.D. Shanyn Aysta-Isaac, Psy.D.; Caitlin Singletary, Ph.D. and B. Charmaine Mosier; Liz Davis Goldman, Ph.D., Jamie F. Klenke, Psy.D., and Kait Portz, Ph.D.; Sierra Phillips and Shannon Wilder; G. Channing Harris, Ph.D. and Cassandra Nelson PMHNP-BC; and Ranya Garcia Pharm.D. and Sommer Feliciano LVN; and Lisa Hayes, MSW, LISW-S. Finally, we appreciate the support of leadership at the Hampton VAMC, where Dr. Lange was employed at the time this work was conducted; leadership at the Tuscaloosa VAMC: John F. Merkle – Medical Center Director, Amir Farooqi – Associate Director (former); VISN6 – Shanekia Williams-Johnson; and for the input of Drs. Michael R. Kauth and Jillian C. Shipherd – VHA LGBTQ+ Health Program Office Co-Directors.
Footnotes
Disclaimer: The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.
References
- American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., Text Revision). Washington, D.C. 10.1176/appi.books.9780890425596 [DOI] [Google Scholar]
- Centers for Disease Control and Prevention (CDC). (2018). Brief 19: Analyzing Qualitative Data for Evaluation.https://www.cdc.gov/healthyyouth/evaluation/pdf/brief19.pdf
- Atkins D, Kilbourne A, & Lipson L (2014). Health equity research in the Veterans Health Administration: we’ve come far but aren’t there yet. American Journal of Public Health, 104(Suppl 4), S525–526. 10.2105/ajph.2014.302216 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Blackwell CW, Diaz-Cruz A, & Yan X (2020). Equality and quality: The relationship between Magnet® status and healthcare organizational commitment to lesbian, gay, bisexual, and transgender equality. Journal of Social Service Research, 46(5), 705–712. [Google Scholar]
- Blosnich JR, Brown GR, Shipherd JC, Kauth M, Piegari RI, & Bossarte RM (2013). Prevalence of gender identity disorder and suicide risk among transgender veterans utilizing veterans health administration care. American Journal of Public Health, 103(10), e27–32. 10.2105/aiph.2013.301507 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Burnette C, Ramchand R, & Ayer L (2015). Gatekeeper training for suicide prevention: A theoretical model and review of the empirical literature. Rand health quarterly, 5(1), 16. [PMC free article] [PubMed] [Google Scholar]
- Chesney MA, Neilands TB, Chambers DB, Taylor JM, & Folkman S (2006). A validity and reliability study of the coping self-efficacy scale. Brittish Journal of Health Psychology, 11(3), 421–437. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cochran BN, Balsam K, Flentje A, Malte CA, & Simpson T (2013). Mental health characteristics of sexual minority veterans. Jounral of Homosexuality, 60(2-3), 419–435. 10.1080/00918369.2013.744932 [DOI] [PubMed] [Google Scholar]
- Cohen J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge. 10.4324/9780203771587 [DOI] [Google Scholar]
- Cohen J, Cohen P, West SG, & Aiken LS (2013). Applied multiple regression/correlation analysis for the behavioral sciences. New York, NY: Routledge. 10.4324/9780203774441 [DOI] [Google Scholar]
- Committee on Facilities Staffing Requirements for Veterans Health Administration (2019). Facilities Staffing Requirements for the Veterans Health Administration-Resource Planning and Methodology for the Future. National Academies of Sciences, Engineering, and Medicine; Division of Behavioral and Social Sciences and Education; Board on Human-Systems Integration; Division on Engineering and Physical Sciences; Board on Infrastructure and the Constructed Environment. Washington (DC): National Academies Press (US); Dec 24, 2019. Available from: https://www.ncbi.nlm.nih.gov/books/NBK555781/ doi: 10.17226/25454 [DOI] [PubMed] [Google Scholar]
- Cunningham CA, Cramer RJ, Cacace S, Franks M, & Desmarais SL (2020). The coping self-efficacy scale: Psychometric properties in an outpatient sample of active duty military personnel. Military Psychology, 32, 261–272. 10.1080/08995605.2020.1730683 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Damrosch SP (1986). Ensuring anonymity by use of subject-generated identification codes. Res Nurs Health, 9(1), 61–63. 10.1002/nur.4770090110 [DOI] [PubMed] [Google Scholar]
- Department of Veterans Affairs Office of Research & Development. (2019, January 9). Program Guide 1200.21: VHA operations activities that may constitute research. https://www.research.va.gov/resources/policies/ProgramGuide-1200-21-VHA-Operations-Activities.pdf
- Duan-Porter W, Martinson BC, Taylor B, Falde S, Gnan K, Greer N, MacDonald R, McKenzie L, Rosebush C, & Wilt TJ (2017). VA Evidence-based Synthesis Program Reports. Evidence review: Social determinants of health for veterans. Department of Veterans Affairs (US). https://www.hsrd.research.va.gov/publications/esp/socialdeterminants-REPORT.pdf [PubMed] [Google Scholar]
- Enders CK (2017). Multiple imputation as a flexible tool for missing data handling in clinical research. Behaviour Research and Therapy, 98, 4–18. 10.1016/j.brat.2016.11.008 [DOI] [PubMed] [Google Scholar]
- Fortney JC, Burgess JF Jr., Bosworth HB, Booth BM, & Kaboli PJ (2011). A re-conceptualization of access for 21st century healthcare. Journal of General Internal Medicine, 26 (Suppl 2), 639–647. 10.1007/s11606-011-1806-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Garvey Wilson AL, Hoge CW, McGurk D, Thomas JL, Clark JC, & Castro CA (2010). Application of a new method for linking anonymous survey data in a population of soldiers returning from Iraq. Annals of Epidemiology, 20(12), 931–938. 10.1016/j.annepidem.2010.08.008 [DOI] [PubMed] [Google Scholar]
- Goodrich DE, Miake-Lye I, Braganza MZ,, Wawrin N, Kilbourne, & A.M. (2020). Quality Enhancement Research Initiative. QUERI roadmap for implementation and quality improvement. United States Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development Health Services Research and Development. Available at https://www.queri.research.va.gov/tools/roadmap/ [PubMed] [Google Scholar]
- Hardacker CT, Rubinstein B, Hotton A, & Houlberg M (2014). Adding silver to the rainbow: the development of the nurses’ health education about LGBT elders (HEALE) cultural competency curriculum. Journal of Nursing Management, 22(2), 257–266. 10.1111/jonm.12125 [DOI] [PubMed] [Google Scholar]
- Jackson CL, Agénor M, Johnson DA, Austin SB, & Kawachi I (2016). Sexual orientation identity disparities in health behaviors, outcomes, and services use among men and women in the United States: a cross-sectional study. BMC Public Health, 16(1), 807. 10.1186/s12889-016-3467-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Keuroghlian AS, Ard KL, & Makadon HJ (2017). Advancing health equity for lesbian, gay, bisexual and transgender (LGBT) people through sexual health education and LGBT-affirming health care environments. Sexual Health, 14(1), 119–122. 10.1071/sh16145 [DOI] [PubMed] [Google Scholar]
- Kilbourne AM, Braganza MZ, Bowersox NW, Goodrich DE, Miake-Lye I, Floyd N, Garrido MM, Frakt AB, Bever CT, Vega R, & Ramoni R (2019). Research lifecycle to increase the substantial real-world impact of research: Accelerating innovations to application. Medical Care, 57(Suppl 10), S206–S212. 10.1097/MLR.0000000000001146 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Ritchie MJ, Dollar KM, Miller CJ, Smith JL, Oliver KA, Kim B, Connolly SL, Woodward E, Ochoa-Olmos T, Day S, Lindsay JA, Kirchner JE. Using implementation facilitation to improve healthcare (Version 3). Veterans Health Administration, Behavioral Health Quality Enhancement Research Initiative (QUERI), 2020. https://www.queri.research.va.gov/tools/implementation/Facilitation-Manual.pdf [Google Scholar]
- Kirchner JE, Ritchie MJ, Pitcock JA, Parker LE, Curran GM, & Fortney JC (2014). Outcomes of a partnered facilitation strategy to implement primary care-mental health. J Gen Intern Med, 29 Suppl 4, 904–912. 10.1007/s11606-014-3027-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kroenke K, Spitzer RL, & Williams JB (2001). The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med, 16(9), 606–613. 10.1046/M525-1497.2001.016009606.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lange TM (2018). PRIDE In All Who Served: A Health Education Group for LGBT Veterans. In: Developed with support from the VHA Innovation Network Spark-Seed-Spread Program (FY18, FY19) and Leadership at the Hampton VA Medical Center. [Google Scholar]
- Lange TM (2020). Trans-affirmative narrative exposure therapy (TA-NET): A therapeutic approach for targeting minority stress, internalized stigma, and trauma reactions among gender diverse adults. Practice Innovations, 5(3), 230–245. 10.1037/pri0000126 [DOI] [Google Scholar]
- Lange TM, Hilgeman MM, Portz KJ, Intoccia VA, & Cramer RJ (2020). Pride in all Who Served: Development, Feasibility, and Initial Efficacy of a Health Education Group For LGBT Veterans. J Trauma Dissociation, 21(4), 484–504. 10.1080/15299732.2020.1770147 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lund EM, Nadorff MR, Galbraith K, & Thomas KB (2019). Comparing the internal consistency, overall scores, and response patterns on the Suicidal Behavior Questionnaire–Revised in people with and without disabilities. Rehabilitation Counseling Bulletin, 62(2), 108–120. 1-3, 422-427. [Google Scholar]
- Matarazzo BB, Barnes SM, Pease JL, Russell LM, Hanson JE, Soberay KA, & Gutierrez PM (2014). Suicide risk among lesbian, gay, bisexual, and transgender military personnel and veterans: what does the literature tell us? Suicide Life Threat Behav, 44(2), 200–217. 10.1111/sltb.12073 [DOI] [PubMed] [Google Scholar]
- Meyer IH (2013). Prejudice, social stress, and mental health in lesbian, gay, and bisexual populations: Conceptual issues and research evidence. Psychology of Sexual Orientation and Gender Diversity, 1(S), 3–26. 10.1037/2329-0382.1.S.3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Mohr JJ, & Kendra MS (2011). Revision and extension of a multidimensional measure of sexual minority identity: the Lesbian, Gay, and Bisexual Identity Scale. Journal of Counseling Psychology, 58(2), 234–245. 10.1037/a0022858 [DOI] [PubMed] [Google Scholar]
- Osman A, Bagge CL, Gutierrez PM, Konick LC, Kopper BA, & Barrios FX (2001). The Suicidal Behaviors Questionnaire-Revised (SBQ-R): validation with clinical and nonclinical samples. Assessment, 8(4), 443–454. 10.1177/107319110100800409 [DOI] [PubMed] [Google Scholar]
- Perry NS, & Elwy AR (2021). The role of implementation science in reducing sexual and gender minority mental health disparities. LGBT Health, 8(3), 169–172. 10.1089/lgbt.2020.0379 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Potter EC, & Patterson CJ (2019). Health-related quality of life among lesbian, gay, and bisexual adults: The burden of health disparities in 2016 Behavioral Risk Factor Surveillance System Data. LGBT Health, 6(7), 357–369. 10.1089/lgbt.2019.0013 [DOI] [PubMed] [Google Scholar]
- Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Proctor EK, & Kirchner JE (2015). A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implement Sci, 10, 21. 10.1186/s13012-015-0209-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Ramirez MH, Rogers SJ, Johnson HL, Banks J, Seay WP, Tinsley BL, & Grant AW (2013). If we ask, what they might tell: clinical assessment lessons from LGBT military personnel post-DADT. Journal of Homosexuality, 60(2-3), 401–418. 10.1080/00918369.2013.744931 [DOI] [PubMed] [Google Scholar]
- Riggle ED, Mohr JJ, Rostosky SS, Fingerhut AW, & Balsam KF (2014). A multifactor lesbian, gay, and bisexual positive identity measure (LGB-PIM). Psychology of Sexual Orientation and Gender Diversity, 1(4), 398. [Google Scholar]
- Seebohm P, Chaudhary S, Boyce M, Elkan R, Avis M, & Munn-Giddings C (2013). The contribution of self-help/mutual aid groups to mental well-being. Health and Social Care in the Community, 21(4), 391–401. [DOI] [PubMed] [Google Scholar]
- Spitzer RL, Kroenke K, Williams JB, & Löwe B (2006). A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine, 166(10), 1092–1097. 10.1001/archinte.166.10.1092 [DOI] [PubMed] [Google Scholar]
- Testa RJ, Habarth J, Peta J, Balsam K, & Bockting W (2015). Development of the gender minority stress and resilience measure. Psychology of Sexual Orientation and Gender Diversity, 2(1), 65–77. 10.1037/sgd0000081 [DOI] [Google Scholar]
- Valentine SE, & Shipherd JC (2018). A systematic review of social stress and mental health among transgender and gender non-conforming people in the United States. Clinical Psychology Review, 66, 24–38. 10.1016/j.cpr.2018.03.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Valentine SE, Shipherd JC, Smith AM, & Kauth MR (2019). Improving affirming care for sexual and gender minority veterans. Psychological Services. 10.1037/ser0000378 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Vega RJ, & Kizer KW (2020). VHA’s Innovation Ecosystem: Operationalizing Innovation in Health Care. New England Journal of Medicine Catalyst, 1(6). 10.1056/CAT.20.0263 [DOI] [Google Scholar]
- Wolowic JM, Heston LV, Saewyc EM, Porta C, & Eisenberg ME (2017). Chasing the rainbow: lesbian, gay, bisexual, transgender and queer youth and pride semiotics. Culture, Health, & Sexuality, 19(5), 557–571. 10.1080/13691058.2016.1251613 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Woodward EN, Matthieu MM, Uchendu US, Rogal S, & Kirchner JE (2019). The health equity implementation framework: proposal and preliminary study of hepatitis C virus treatment. Implementation Science, 14(1), 26. 10.1186/s13012-019-0861-y [DOI] [PMC free article] [PubMed] [Google Scholar]
