Abstract
Background
The US Department of Veterans Affairs designated education officer (DEO) is a unique facility-based leadership role responsible for training of > 40 health professions in cooperation with affiliated academic institutions.
Methods
We conducted mixed methods analyses of data from a DEO needs assessment. Quantitative analysis identified differences between DEOs who are physicians and DEOs who are other professions on role characteristics and self-perceived task effectiveness. Qualitative analysis using rapid analysis procedures was applied to open-ended responses on facilitators and barriers.
Results
Responses were received from 127 DEOs (96% response rate). About 80% were physicians. There were no statistically significant differences between physician and other professional DEOs self-ratings for general tasks. For profession-specific tasks, physician DEOs were significantly less confident than other professional DEOs in working with associated health (P < .001–.01) and nurse training programs (P < .001–.03). DEOs identified multiple facilitators that assist their individual effectiveness (eg, training, mentorship, communication) and common barriers (eg, not enough staff).
Conclusions
Our findings are supportive of individuals from various health disciplines serving in the DEO role with responsibilities that span multiple health profession training programs. Future quantitative and qualitative work should include additional measures of individual and organizational characteristics, and actual measures of educational effectiveness.
The US Department of Veterans Affairs (VA) operates the largest integrated health care system in the United States, providing physical and mental health care to more than 9 million veterans enrolled each year through a national system of inpatient, outpatient, and long-term care settings.1 As 1 of 4 statutory missions, the VA conducts the largest training effort for health professionals in cooperation with affiliated academic institutions. From 2016 through 2020, an average of 123,000 trainees from various professions received training at the VA.2 Physician residents comprised the largest trainee group (37%), followed by associated health students and residents (20%), and nursing professionals (21%).2 In VA, associated health professions include all health care disciplines other than allopathic and osteopathic medicine, dentistry, and nursing. The associated health professions encompass about 40 specialties, including audiology, dietetics, physical and occupational therapy, optometry, pharmacy, podiatry, psychology, and social work.
The VA also trains a smaller number of advanced fellows to address specialties important to the nation and veterans health that are not sufficiently addressed by standard accredited professional training.3 The VA Advanced Fellowship programs include 22 postresidency, postdoctoral, and postmasters fellowships to physicians and dentists, and associated health professions, including psychologists, social workers, and pharmacists.3 From 2015 to 2019, 57 to 61% of medical school students reported having a VA clinical training experience during medical school.4 Of current VA employees, 20% of registered nurses, 64% of physicians, 73% of podiatrists and optometrists, and 81% of psychologists reported VA training prior to employment.5
Health professions education is led by the designated education officer (DEO) at each VA facility.6 Also known as the associate chief of staff for education (ACOS/E), the DEO is a leadership position that is accountable to local VA facility executive leadership as well as the national Office of Academic Affiliations (OAA), which directs all VA health professions training across the US.6 At most VA facilities, the DEO oversees clinical training and education reporting directly to the facility chief of staff. At the same time, the ACOS/E is accountable to the OAA to ensure adherence with national education directives and policy. The DEO oversees trainee programs through collaboration with training program directors, faculty, academic affiliates, and accreditation agencies across > 40 health professions.
The DEO is expected to possess expertise in leadership attributes identified by the US Office of Personnel Management as essential to build a federal corporate culture that drives results, serves customers, and builds successful teams and coalitions within and outside the VA.7 These leadership attributes include leading change, leading people, driving results, business acumen, and building coalitions.7 They are operationalized by OAA as 4 domains of expertise required to lead education across multiple professions, including: (1) creating and sustaining an organizational work environment that supports learning, discovery, and continuous improvement; (2) aligning and managing fiscal, human, and capital resources to meet organizational learning needs; (3) driving learning and performance results to impact organizational success; and (4) leading change and transformation through positioning and implementing innovative learning and education strategies (Table 1).6
TABLE 1.
Designated Education Officer Domains of Expertise and Task Examples
| Domains | Roles | Task Examples |
|---|---|---|
| Enabling learning | Creates/sustains organizational work environment to support learning, discovery, and continuous improvement | Applies policies & procedures related to VA trainees, eg, evaluation Participates on key educational committees at the affiliate Maintains compliance with accreditation agency standards Provides guidance on legal issues including communication with Office of General Counsel |
| Aligning resources | Manages fiscal, human, and capital resources to meet organizational learning needs | Negotiates with educational institutions for a trainee mix that will support specific programs and veteran services Designs/implements innovative trainee education solutions to meet trainee and organizational needs Provides oversight of education office staff for health professions education administration |
| Driving results | Provides educational oversight to achieve learning and performance results | Provides oversight to ensure trainees have proper clinical supervision in place and monitored accordingly to required guidelines Monitors patient safety by reviewing risk events involving trainees Uses trainee satisfaction survey data to monitor effectiveness of trainee education programs and as a quality improvement tool |
| Leading change | Leads organizational change/transformation through positioning and implementing learning/educational strategies | Promotes diversity initiatives working with academic affiliates and actively seeks out minority-facing institutions, eg, historically black colleges and universities, Hispanic-serving Institutions Develops interprofessional learning activities to promote high-quality team-based education and patient care Develops resources to promote culture of trainee and supervisor/faculty well-being Reviews educational literature and/or publishes in peer-reviewed journals and other publications |
In this article we describe the VA DEO leadership role and the tasks required to lead education across multiple professions within the VA health care system. Given the broad scope of leading educational programs across multiple clinical professions and the interprofessional backgrounds of DEOs across the VA, we evaluated DEO self-perceived effectiveness to impact educational decisions and behavior by professional discipline. Our evaluation question is: Are different professional education and practice backgrounds functionally capable of providing leadership over all education of health professions training programs? Finally, we describe DEOs perceptions of facilitators and barriers to performing their DEO role within the VA.
METHODS
We conducted a mixed methods analysis of data collected by OAA to assess DEO needs within a multiprofessional clinical learning environment. The needs assessment was conducted by an OAA evaluator (NH) with input on instrument development and data analysis from OAA leadership (KS, MB). This evaluation is categorized as an operations activity based on VA Handbook 1200 where information generated is used for business operations and quality improvement.8 The overall project was subject to administrative rather than institutional review board oversight.
A needs assessment tool was developed based on the OAA domains of expertise.6 Prior to its administration, the tool was piloted with 8 DEOs in the field and the survey shortened based on their feedback. DEOs were asked about individual professional characteristics (eg, clinical profession, academic appointment, type of health professions training programs at the VA site) and their self-perceived effectiveness in impacting educational decisions and behaviors on general and profession-specific tasks within each of the 4 domains of expertise on a 5-point Likert scale (1, not effective; 5, very effective).6,9 The needs assessment also included an open-ended question asking respondents to comment on any issues they felt important to understanding DEO role effectiveness.
The needs assessment was administered online via SurveyMonkey to 132 DEOs via email in September and October 2019. The DEOs represented 148 of 160 VA facilities with health professions education; 14 DEOs covered > 1 VA facility, and 12 positions were vacant. Email reminders were sent to nonresponders after 1 week. At 2 weeks, nonresponders received telephone reminders and personalized follow-up emails from OAA staff. The response rate at the end of 3 weeks was 96%.
Data Analysis
Mixed methods analyses included quantitative analyses to identify differences in general and profession-specific self-ratings of effectiveness in influencing educational decisions and behaviors by DEO profession, and qualitative analyses to further understand DEO’s perceptions of facilitators and barriers to DEO task effectiveness.10,11 Quantitative analyses included descriptive statistics for all variables followed by nonparametric tests including χ2 and Mann-Whitney U tests to assess differences between physician and other professional DEOs in descriptive characteristics and self-perceived effectiveness on general and profession-specific tasks. Quantitative analyses were conducted using SPSS software, version 26. Qualitative analyses consisted of rapid assessment procedures to identify facilitators and barriers to DEO effectiveness by profession using Atlas.ti version 8, which involved reviewing responses to the open-ended question and assigning each response to predetermined categories based on the organizational level it applied to (eg, individual DEO, VA facility, or external to the organization).12,13 Responses within categories were then summarized to identify the main themes.
RESULTS
Completed surveys were received from 127 respondents representing 139 VA facilities. Eighty percent were physicians and 20% were other professionals, including psychologists, pharmacists, dentists, dieticians, nurses, and nonclinicians. There were no statistically significant differences between physician and other professional DEOs in the percent working full time or length of time spent working in the position. About one-third of the sample had been in the position for < 2 years, one-third had been in the position for 2 to < 5 years, and one-third had been in the role for ≥ 5 years. Eighty percent reported having a faculty appointment with an academic affiliate. While 92% of physician DEOs had a faculty appointment, only 40% of other professional DEOs did (P < .001). Most faculty appointments for both groups were with a school of medicine. More physician DEOs than other professionals had training programs at their site for physicians (P = .003) and dentists (P < .001), but there were no statistically significant differences for having associated health, nursing, or advanced fellowship training programs at their sites. Across all DEOs, 98% reported training programs at their site for associated health professions, 95% for physician training, 93% for nursing training, 59% for dental training, and 48% for advanced fellowships.
Self-Perceived Effectiveness
There were no statistically significant differences between physician and other professional DEOs on self-perceived effectiveness in impacting educational decisions or behaviors for general tasks applicable across professions (Table 2). This result held even after controlling for length of time in the position and whether the DEO had an academic appointment. Generally, both groups reported being effective on tasks in the enabling learning domain, including applying policies and procedures related to trainees who rotate through the VA and maintaining adherence with accreditation agency standards across health professions. Mean score ranges for both physician and other professional DEOs reported moderate effectiveness in aligning resources effectiveness questions (2.45–3.72 vs 2.75–3.76), driving results questions (3.02–3.60 vs 3.39–3.48), and leading change questions (3.12–3.50 vs 3.42–3.80).
TABLE 2.
Designated Education Officer Self-Perceived Effectiveness Ratings on General Tasks Applicable Across Health Professions Training Programs
| Tasksa | Physicians, Mean (SD) | Other Professionals, Mean (SD) | P value |
|---|---|---|---|
| Enabling learning | |||
| Applies policies and procedures related to trainees | 4.25 (0.74) | 4.19 (0.80) | .73 |
| Maintains compliance with accreditation agency standards, | 4.11 (0.88) | 4.32 (0.69) | .27 |
| Participates on key educational committees at the affiliate academic institution | 4.05 (1.07) | 3.91 (1.15) | .59 |
| Provides guidance on legal issues involving trainees | 3.81 (0.98) | 3.95 (0.81) | .55 |
|
| |||
| Aligning resources | |||
| Provides oversight of education office staff on administration functions | 3.72 (1.09) | 3.76 (1.09) | .86 |
| Leverages facility funding for indirect educational expenses | 2.45 (1.36) | 2.75 (1.45) | .34 |
|
| |||
| Driving results | |||
| Monitors patient safety by reviewing risk events involving trainees | 3.60 (1.14) | 3.39 (1.22) | .44 |
| Uses Office of Academic Affiliations trainee satisfaction survey data as a quality improvement tool to monitor the effectiveness of trainee education programs | 3.02 (1.14) | 3.48 (1.36) | .09 |
|
| |||
| Leading change | |||
| Develops resources to promote a culture of trainee and supervisor/faculty well-being | 3.50 (1.09) | 3.80 (1.04) | .22 |
| Promotes diversity initiatives with academic affiliates and seeks minority-facing institutions | 3.14 (1.18) | 3.42 (1.14) | .32 |
| Develops interprofessional learning activities to promote high-quality team-based education and patient care | 3.12 (1.22) | 3.60 (0.91) | .07 |
Task effectiveness was rated on a 5-point Likert scale; 1, not effective; 5, very effective.
For profession-specific tasks, effectiveness ratings between the 2 groups were generally not statistically significant for medical, dental, and advanced fellowship training programs (Table 3). There was a pattern of statistically significant differences between physician and other professional DEOs for associated health and nursing training programs on tasks across the 4 domains of expertise with physicians having lower mean ratings compared with other professionals. Generally, physician DEOs had higher task effectiveness when compared with other professionals for medical training programs, and other professionals had higher task effectiveness ratings than did physicians for associated health or nursing training programs.
TABLE 3.
DEO Self-Perceived Effectiveness Ratings on Profession-Specific Tasks
| Tasksa | Physicians, Mean (SD) | Other Professionals, Mean (SD) | P value |
|---|---|---|---|
| Enabling learning | |||
|
| |||
| Establishes/maintains education infrastructure | |||
| Associated health | 3.64 (1.13) | 4.38 (0.80) | < .01 |
| Medical | 4.31 (0.75) | 4.09 (0.92) | .25 |
| Dental | 3.71 (1.19) | 3.91 (1.38) | .62 |
| Nursing | 3.12 (1.25) | 4.09 (1.04) | < .01 |
| Advanced fellowships | 3.64 (1.19) | 3.57 (1.09) | .85 |
|
| |||
| Completes educational reports | |||
| Associated health | 3.65 (0.69) | 3.65 (0.69) | .63 |
| Medical | 3.76 (0.57) | 3.64 (0.58) | .36 |
| Dental | 3.73 (0.63) | 3.88 (0.35) | .52 |
| Nursing | 3.46 (0.86) | 3.64 (0.86) | .37 |
| Advanced fellowships | 3.68 (0.70) | 3.45 (0.82) | .35 |
|
| |||
| Aligning resources | |||
|
| |||
| Negotiates/works with multiple educational institutions to ensure opportunities for all parties | |||
| Associated health | 3.51 (1.14) | 4.20 (0.87) | < .01 |
| Medical | 4.15 (0.95) | 3.81 (1.17) | .17 |
| Dental | 3.74 (1.02) | 3.88 (1.25) | .73 |
| Nursing | 3.28 (1.34) | 3.96 (1.19) | .03 |
| Advanced fellowships | 3.29 (1.29) | 3.60 (1.27) | .50 |
|
| |||
| Oversees educational activity record keeping | |||
| Associated health | 3.44 (1.31) | 4.24 (0.96) | .01 |
| Medical | 4.05 (0.96) | 4.00 (0.79) | .84 |
| Dental | 4.08 (1.04) | 4.00 (1.00) | .83 |
| Nursing | 3.08 (1.43) | 4.24 (0.90) | < .01 |
| Advanced fellowships | 3.49 (1.36) | 3.40 (1.08) | .86 |
|
| |||
| Driving results | |||
|
| |||
| Oversight of trainee supervision consistent with VA policy and accreditation requirements | |||
| Associated health | 3.78 (1.11) | 4.08 (0.91) | .22 |
| Medical | 4.30 (0.79) | 4.00 (0.93) | .13 |
| Dental | 4.00 (1.05) | 3.60 (1.08) | .27 |
| Nursing | 3.36 (1.25) | 4.04 (1.07) | .02 |
| Advanced fellowships | 3.74 (1.32) | 3.23 (1.09) | .21 |
|
| |||
| Leading change | |||
|
| |||
| Works to establish the optimal clinical learning and working environment | |||
| Associated health | 3.44 (1.08) | 4.52 (0.77) | < .01 |
| Medical | 4.05 (0.99) | 4.35 (0.89) | .19 |
| Dental | 3.82 (1.03) | 4.20 (0.92) | .28 |
| Nursing | 3.10 (1.22) | 4.21 (0.98) | < .01 |
| Advanced fellowships | 3.57 (1.19) | 3.92 (1.24) | .37 |
|
| |||
| Promotes opportunities for formal faculty development to enhance teaching skills | |||
| Associated health | 2.96 (1.22) | 3.71 (1.30) | .01 |
| Medical | 3.37 (1.24) | 3.58 (1.28) | .46 |
| Dental | 3.13 (1.22) | 3.33 (1.41) | .64 |
| Nursing | 2.80 (1.24) | 3.62 (1.36) | .01 |
| Advanced fellowships | 2.98 (1.26) | 3.18 (1.25) | .63 |
Abbreviations: DEO, designated education officer; VA, US Department of Veterans Affairs.
Task effectiveness was rated on a 5-point Likert scale; 1, not effective; 5, very effective.
Facilitators and Barriers
Seventy responses related to facilitators and barriers to DEO effectiveness were received (59 from physicians and 11 from other professionals). Most responses were categorized as individual level facilitators or barriers (53% for physician and 64% for other professionals). Only 3% of comments were categorized as external to the organization (all made by physicians). The themes were similar for both groups and were aggregated in Table 4. Facilitators included continuing education, having a mentor who works at a similar type of facility, maintaining balance and time management when working with different training programs, learning to work and develop relationships with training program directors, developing an overall picture of each type of health professions training program, holding regular meetings with all health training programs and academic affiliates, having a formal education service line with budget and staffing, facility executive leadership who are knowledgeable of the education mission and DEO role, having a national oversight body, and the DEO’s relationships with academic affiliates.
TABLE 4.
DEO-Reported Facilitators and Barriers to Role Effectiveness
| Target Levels | Facilitators | Barriers |
|---|---|---|
| Individual/DEO | Continuing leadership education for professional development Having a mentor who works at a similar type of facility Adequate protected time for DEO responsibilities Maintain balance and time management when working with different training programs Learn to work and develop relationships with training program directors of each profession Develop an overall picture of each type of health professions training program Hold regular (quarterly) meetings with all health professions education training programs Holding regular (quarterly) meetings with academic affiliates |
Assigned multiple roles Focus on regulation, monitoring leaves no time for development of new programs, strategic planning initiatives Lack of education budget Lack of staff |
| Organization | Formal education service line with budget and dedicated staff Hospital executive leadership who are knowledgeable of education mission and DEO role |
Difficulty getting organization’s core services to engage with trainees Siloed education leadership |
| External to Organization | National oversight body to guide organizational education Relationship with academic affiliates, eg, faculty appointments |
Abbreviation: DEO, designated education officer.
Barriers to role effectiveness at the individual DEO level included assignment of multiple roles and a focus on regulation and monitoring with little time for development of new programs and strategic planning. The organizational level barriers included difficulty getting core services to engage with health professions trainees and siloed education leadership.
DISCUSSION
DEOs oversee multiple health professions training programs within local facilities. The DEO is accountable to local VA facility leadership and a national education office to lead local health professions education at local facilities and integrate these educational activities across the national VA system.
The VA DEO role is similar to the Accreditation Council for Graduate Medical Education designated institutional official (DIO) except that the VA DEO provides oversight of > 40 health professions training programs.14,15 The VA DEO, therefore, has broader oversight than the DIO role that focuses only on graduate physician education. Similar to the DIO, the VA DEO role initially emphasized the enabling learning and aligning resources domains to provide oversight and administration of health professions training programs. Over time, both roles have expanded to include defining and ensuring healthy clinical learning environments, aligning educational resources and training with the institutional mission, workforce, and societal needs, and creating continuous educational improvement models.6,16,17 To accomplish these expanded goals, both the DEO and the DIO work closely with other educational leaders at the academic affiliate and the VA facility. As health professions education advances, there will be increased emphasis placed on delivering educational programs to improve clinical practice and health care outcomes.18–20
Our findings that DEO profession did not influence self-ratings of effectiveness to influence educational decisions or behaviors on general tasks applicable across health professions suggest that education and practice background are not factors influencing self-ratings. Nor were self-ratings influenced by other factors. Since the DEO is a senior leadership position, candidates for the position already may possess managerial and leadership skills. In our sample, several individuals commented that they had prior education leadership positions, eg, training program director or had years of experience working in the VA. Similarly, having an academic appointment may not be important for the performance of general administrative tasks. However, an academic appointment may be important for effective performance of educational tasks, such as clinical teaching, didactic training, and curriculum development, which were not measured in this study.
The finding of differences in self-ratings between physicians and other professionals on profession-specific tasks for associated health and nursing suggests that physicians may require additional curriculum to enhance their knowledge in managing other professional educational programs. For nursing specifically, this finding could also reflect substantial input from the lead nurse executive in the facility. DEOs also identified practical ways to facilitate their work with multiple health professions that could immediately be put into practice, including developing relationships and enhancing communication with training program directors, faculty, and academic affiliates of each profession.
Taken together, the quantitative and qualitative findings indicate that despite differences in professional backgrounds, DEOs have high self-ratings of their own effectiveness to influence educational decisions and behaviors on general tasks they are expected to accomplish. There are some profession-specific tasks where professional background does influence self-perceived effectiveness, ie, physicians have higher self-ratings on physician-specific tasks and other professionals have higher self-ratings on associated health or nursing tasks. These perceived differences may be mitigated by increasing facilitators and decreasing barriers identified for the individual DEO, within the organization, and external to the organization.
Limitations
Our findings should be interpreted with the following limitations in mind. The self-report nature of the data opens the possibility of self-report bias or Dunning-Kruger effects where effectiveness ratings could have been overestimated by respondents.21 Although respondents were assured of their anonymity and that results would only be reported in the aggregate, there is potential for providing more positive responses on a needs assessment administered by the national education program office. We recommend further work be conducted to validate the needs assessment tool against other data collection methods, such as actual outcomes of educational effectiveness. Our study did not incorporate measures of educational effectiveness to determine whether self-perceived DEO effectiveness is translated to better trainee or learning outcomes. Before this can happen, educational policymakers must identify the most important facility-level learning outcomes. Since the DEO is a facility level educational administrator, learning effectiveness must be defined at the facility level. The qualitative findings could also be expanded through the application of more detailed qualitative methods, such as indepth interviews. The tasks rated by DEOs were based on OAA’s current definition of the DEO role.6 As the field advances, DEO tasks will also evolve.22–24
CONCLUSIONS
The DEO is a senior educational leadership role that oversees all health professions training in the VA. Our findings are supportive of individuals from various health disciplines serving in the VA DEO role with responsibilities that span multiple health profession training programs. We recommend further work to validate the instrument used in this study, as well as the application of qualitative methods like indepth interviews to further our understanding of the DEO role.
Footnotes
Author disclosures
The authors report no actual or potential conflicts of interest or outside sources of funding with regard to this article.
Disclaimer
The opinions expressed herein are those of the authors and do not necessarily reflect those of Federal Practitioner, Frontline Medical Communications Inc., the US Government, or any of its agencies.
Ethics and consent
This evaluation was determined to be an operations activity based on VA Handbook 1200.21 and was subject to administrative rather than institutional review board oversight.
References
- 1.US Department of Veterans Affairs. Veterans Health Administration. [Accessed May 6, 2022]. Updated April 18, 2022. https://www.va.gov/health/aboutvha.asp .
- 2.US Department of Veterans Affairs, Veterans Health Administration. [Accessed May 6, 2022];Office of Academic Affiliations. Health professions education: academic Year 2019–2020. Published 2020. https://www.va.gov/OAA/docs/OAA_Statistics_2020.pdf . [Google Scholar]
- 3.US Department of Veterans Affairs, Veterans Health Administration, Office of Academic Affiliations. [Accessed May 6, 2022];Advanced Fellowships and Professional Development. Updated November 26, 2021. https://www.va.gov/oaa/advancedfellowships/advanced-fellowships.asp . [Google Scholar]
- 4.Association of American Medical Colleges. Medical school graduation questionnaire 2019 all schools summary report. [Accessed May 6, 2022]. Published July 2019. https://www.aamc.org/system/files/2019-08/2019-gq-all-schools-summary-report.pdf .
- 5.US Department of Veterans Affairs, National Center for Organization Development. [Accessed May 6, 2022];VA all employee survey. Published 2019. https://www.va.gov/NCOD/VAworkforcesurveys.asp . [Google Scholar]
- 6.US Department of Veterans Affairs, Veterans Health Administration Office of Academic Affiliations. Education leaders in the VA: the role of the designated education officer (DEO) [Accessed May 6, 2022]. Published December 2019. https://www.va.gov/OAA/docs/DEO_Learning_Leader_2019.pdf .
- 7.US Office of Personnel Management. [Accessed May 6, 2022];Policy, data oversight: guide to senior executive service qualifications. Published 2010. https://www.opm.gov/policy-data-oversight/senior-executive-service/executive-core-qualifications/ [Google Scholar]
- 8.US Department of Veterans Affairs, Office of Research and Development. [Accessed May 6, 2022];Program guide: 1200, 21 VHA operations activities that may constitute research. Published January 9, 2019. https://www.research.va.gov/resources/policies/ProgramGuide-1200-21-VHA-Operations-Activities.pdf . [Google Scholar]
- 9.Riesenberg LA, Rosenbaum PF, Stick SL. Competencies, essential training, and resources viewed by designated institutional officials as important to the position in graduate medical education [published correction appears in Acad Med 2006; Dec: 81(12):1025] Acad Med. 2006;81(5):426–431. doi: 10.1097/01.ACM.0000222279.28824.f5. [DOI] [PubMed] [Google Scholar]
- 10.Palinkas LA, Mendon SJ, Hamilton AB. Innovations in Mixed Methods Evaluations. Annu Rev Public Health. 2019;40:423–442. doi: 10.1146/annurev-publhealth-040218-044215. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Tashakkori A, Creswell JW. Exploring the nature of research questions in mixed methods research. J Mix Methods Res. 2007;1(3):207–211. doi: 10.1177/1558689807302814. [DOI] [Google Scholar]
- 12.Averill JB. Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qual Health Res. 2002;12(6):855–866. doi: 10.1177/104973230201200611. [DOI] [PubMed] [Google Scholar]
- 13.Hamilton AB, Finley EP. Qualitative methods in implementation research: An introduction. Psychiatry Res. 2019;280:112516. doi: 10.1016/j.psychres.2019.112516. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Bellini L, Hartmann D, Opas L. Beyond must: supporting the evolving role of the designated institutional official. J Grad Med Educ. 2010;2(2):147–150. doi: 10.4300/JGME-D-10-00073.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Riesenberg LA, Rosenbaum P, Stick SL. Characteristics, roles, and responsibilities of the Designated Institutional Official (DIO) position in graduate medical education [published correction appears in Acad Med. 2006 Dec 81(12):1025] [published correction appears in Acad Med. 2006 Mar 81(3):274] Acad Med. 2006;81(1):8–19. doi: 10.1097/00001888-200601000-00005. [DOI] [PubMed] [Google Scholar]
- 16.Group on Resident Affairs Core Competency Task Force. Institutional GME leadership competencies. 2015. [Accessed May 6, 2022]. https://www.aamc.org/system/files/c/2/441248-institutionalgmeleadershipcompetencies.pdf .
- 17.Weiss KB, Bagian JP, Nasca TJ. The clinical learning environment: the foundation of graduate medical education. JAMA. 2013;309(16):1687–1688. doi: 10.1001/jama.2013.1931. [DOI] [PubMed] [Google Scholar]
- 18.Beliveau ME, Warnes CA, Harrington RA, et al. Organizational change, leadership, and the transformation of continuing professional development: lessons learned from the American College of Cardiology. J Contin Educ Health Prof. 2015;35(3):201–210. doi: 10.1002/chp.21301. [DOI] [PubMed] [Google Scholar]
- 19.World Health Organization. Framework for Action on Interprofessional Education and Collaborative Practice. [Accessed May 10, 2022]. Published September 1, 2020. https://www.who.int/publications/i/item/framework-for-action-on-interprofessional-education-collaborative-practice .
- 20.Weiss K, Passiment M, Riordan L, Wagner R for the National Collaborative for Improving the Clinical Learning Environment IP-CLE Report Work Group. [Accessed May 6, 2022];Achieving the optimal interprofessional clinical learning environment: proceedings from an NCICLE symposium. doi: 10.33385/NCICLE.0002. Published January 18, 2019. [DOI] [Google Scholar]
- 21.Althubaiti A. Information bias in health research: definition, pitfalls, and adjustment methods. J Multidiscip Healthc. 2016;9:211–217. doi: 10.2147/JMDH.S104807. Published 2016 May 4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Gilman SC, Chokshi DA, Bowen JL, Rugen KW, Cox M. Connecting the dots: interprofessional health education and delivery system redesign at the Veterans Health Administration. Acad Med. 2014;89(8):1113–1116. doi: 10.1097/ACM.0000000000000312. [DOI] [PubMed] [Google Scholar]
- 23.Health Professions Accreditors Collaborative. [Accessed May 6, 2022];Guidance on developing quality interprofessional education for the health professions. Published February 1, 2019. https://healthprofessionsaccreditors.org/wp-content/uploads/2019/02/HPACGuidance02-01-19.pdf . [Google Scholar]
- 24.Watts BV, Paull DE, Williams LC, Neily J, Hemphill RR, Brannen JL. Department of Veterans Affairs Chief Resident in Quality and Patient Safety Program: a model to spread change. Am J Med Qual. 2016;31(6):598–600. doi: 10.1177/1062860616643403. [DOI] [PubMed] [Google Scholar]