Qualitative Findings of a Nominal Group Process to Identify Critical Factors in New LIC Implementation

Abstract

OBJECTIVES

Medical schools considering longitudinal integrated clerkships (LICs) have access to literature that provides recommendations for planning, implementation, and sustainability. However, LIC development and implementation remain notoriously challenging. University of Utah’s LIC development process was informed by the documented experiences of long-established LIC programs. A literature gap was identified pertaining to more recently implemented LICs. The aim of this study was to explore the experiences of faculty in the early stages of LIC development.

METHODS

Thirteen representatives from eight LICs implemented after 2015 participated in 2 Zoom focus groups (5 participated in the first and the other 8 participated in the second). Participants were asked questions to assess key supports, barriers, and recommendations. Following the focus groups, participants were asked to rank the responses based on their level of importance.

RESULTS

Highest ranked supports included stakeholder and partner involvement; a dedicated coordinator or team; and strong, committed leadership. Highest ranked barriers included difficulty recruiting preceptors and clinical sites; underestimation of the amount of work required to coordinate the LIC; and challenges in providing the needed faculty development. Top recommendations for new LICs included investing in the needs of clinical partners; staffing or assigning a dedicated coordinator early in the development and implementation process; and frequent communication with all stakeholders.

CONCLUSION

Despite variation among the types of new LICs represented, there was consensus among participants on the importance of key supports, barriers, and recommendations. Knowledge of these factors can help new schools plan and allocate resources during their LIC development process. Participants found the focus group process and follow-up discussions useful and have formed an ongoing workgroup which meets quarterly.

Introduction

While undergraduate medical education varies across schools, third-year medical students typically move out of the classroom and begin an intensive clinical education. These students are often assigned to traditional block rotations (TBRs) in which they engage in a series of brief (eg, 4-8 weeks) clinical placements within different disciplines (eg, pediatrics, family medicine, obstetrics/gynecology, surgery, internal medicine, psychiatry). Students are immersed sequentially in each discipline and work with clinical faculty within each specialty before moving on to the next rotation. An alternative to the TBR model is the longitudinal integrated clerkship (LIC) model. In an LIC, students engage with multiple specialties simultaneously, for less time per specialty in any given week, but over a longer period of time (the full length of the LIC). LICs allow for an educational experience that is grounded in longitudinal relationships with preceptors, patients, peers, places, and pedagogy.

In the half-century since LICs began emerging as alternatives to TBRs in clinical medical education, a great deal of literature has been published on this model. The model's types and program logistics have been described,^1–10 along with considerable benefits. Benefits for students include closer relationships with preceptors^11–17 and higher quality of feedback^{3,11,14,15,18–20}; professional identity development^{11,12,21–23}; and academic outcomes, clinical experiences, skill, and confidence.^{2,13,15–17,19,24–33} LIC students are more likely to enter primary care^24,34–37 or rural practice^{15,23,37–39}; and demonstrate greater patient-centeredness.^{15,19–21,40,41} Benefits have also been shown for patient satisfaction and perceived patient health benefits^42–46; as well as preceptor satisfaction.^{8,14,27,47,48} Additionally, recent review articles have compiled recommendations for planning, implementing, and sustaining LICs.^7,10,49–52

LICs represent a significant departure from the TBR model in which most faculty and clinicians completed their education. Additionally, the training and logistics within most existing clerkships have been designed around the TBR model. Accordingly, the initiation of a new LIC remains a complex and difficult process that bears continuing investigation. Moreover, much of the literature focuses on important factors for ongoing program and student success rather than those factors critical in early-stage LIC implementation. In their 2021 literature review, Hense et al noted that although they originally wanted to focus on barriers and facilitators of LIC development in the early implementation phase, most studies they reviewed did not indicate where they were in the development process. ⁵⁰ Bartlett et al's 2020 review intentionally gave more weight to longer-running programs, as these have demonstrated sustainability. ⁷ Additionally, many of the studies in recent reviews were published more than 10 years ago.^1,7,10,49,50 Brown et al (2019) note a “relative dearth of recent literature” on factors enabling LIC success. ⁴⁹ Given the rapidly evolving landscape of undergraduate medical education, including the accessibility of guidance from literature and faculty from well-established LIC programs, we undertook a qualitative study of the factors affecting the early phases of LIC development and implementation.

Previous Findings

The literature on LIC implementation includes factors relevant to the initial development of LICs as well as ongoing LIC success. During the early development stages, the literature indicates that medical schools need to determine their goals and objectives in choosing the LIC model, and based on these, make informed decisions regarding the type of LIC to be adopted (eg, amalgamative, blended, or comprehensive ¹ ), setting and duration of the LIC, and percentage of students who will participate in the LIC.^7,10,49 Other factors that should be considered at the outset include how the LIC will integrate with the broader curriculum, including online resources^7,10,51,53; how the LIC will recruit and support students, mentors, and clinical sites^4,7,10,50,54; and how assessment, evaluation, and quality assurance will be managed.^7,10,49–51 Significant time and resources must be allocated to the development process.^{7,49,50,53,54}

The same considerations that are important for the ongoing success and sustainability of LICs seem likely to be important for successful development and initiation. These include recruiting and preparing engaged students^7,26,49,55 and enthusiastic preceptors^{7,49,52,54,55}; eliciting and responding to evaluation data and stakeholder feedback^7,49,51,53; securing strong support from school of medicine and clinic leadership^{2,3,7,10,49,53,55}; encouraging student sense of belonging at clinical sites^7,26,49,55; facilitating use of the Electronic Health Record system and notifications to enhance continuity with patients^19,49,51; engaging in cross-disciplinary collaboration^3,49; providing professional development and other support to preceptors^{2,4,7,10,17,49,51,53,54}; and ensuring the program provides mutually beneficial outcomes to both academic and clinical partners.^2,7,50 Barriers to ongoing success and sustainability include failure of faculty to adapt to a different style of teaching^7,49,50,54; student isolation in remote placements^7,17,49,50; and conflict or poor match between student and preceptor.^10,49 Recruitment of new faculty and clinics not accustomed to teaching can present challenges^3,4,8 and recruitment of faculty and clinics that already work with traditional block students can present a different set of challenges.^4,10,49 Costs associated with LICs may be daunting, both in terms of added expenses for students placed in remote locations and costs of faculty time for developing and running the LIC. Research suggests that these expenses may be recovered through eventual benefits to the teaching clinics and their larger communities.^{10,49–51,56–59}

While the aforementioned works shed light on previously reported factors related to LIC development, faculty at the Spencer Fox Eccles School of Medicine at the University of Utah were interested in exploring the experiences of programs that had initiated LICs in the last 5 years in hopes of identifying the most important facilitators, most challenging barriers, and top recommendations from a group of early-stage LICs. An ongoing understanding of these factors is needed to inform evidence-based LIC development, particularly as this model continues to be an increasingly popular approach to undergraduate medical education worldwide.

Methods

Participant recruitment

Faculty and staff of LICs were eligible to participate if they were implementing an LIC in the United States that was initiated less than 5 years ago or was still in the development stage. There were no other inclusion or exclusion criteria. To identify LICs established within the last 5 years, we first downloaded the listing of all LIC programs provided on the website of the Consortium of LICs (CLICs). ⁶⁰ We identified 45 LIC programs located in the United States. If the LIC initiation year was not included in the program listing, we reviewed program websites to determine their start date or anticipated start date. (If the program website did not provide the initiation year, the program was removed from consideration.)

Ten LICs established in 2016 or later were identified, and leadership from nine of those programs were able to be reached via email or phone and invited to participate in a 1.5-h focus group. Five of these nine LICs participated. Additionally, the University of Utah participated in an ad hoc group of 10 medical schools awarded Value-Based Medical Student Education Training Program grants from the Health Resources and Services Administration (HRSA) in 2020. Seven of these grantees (including the University of Utah) were developing LICs and were also invited to participate. Three of these seven LICs participated. Thus, staff from 16 LICs were invited and 8 LICs participated (13 individual participants). Reasons for nonparticipation included lack of time and being too early in the development process to have information to contribute. Figure 1 illustrates our recruitment process. Table 1 shows that participating schools varied in terms of their LIC length, focus, and percentage of students participating.

Figure 1.

Consolidated criteria for reporting qualitative research (COREQ) diagram.

Table 1.

Participating new LICs.

Participating LICS	LIC initiation year	Length in weeks	AY21-22 cohort size	LIC focus	Percentage of students in LIC
MedStar Franklin Square Medical Center/Georgetown University	2016	22	12	Urban underserved	6%
Elson S. Floyd COM, WA State University	2018	46	80	General	100%
Sidney Kimmel Medical College at Thomas Jefferson University	2018	48	7	General	2%
TX Christian University and University of North TX Health Science Center SOM	2019	40	60	Urban, suburban	100%
University of MO SOM	2021	50	4	Rural	3%
Spencer Fox Eccles SOM at the University of UT	2021	26	3	Urban underserved in pilot; expanding to multiple types	2% in pilot Transitioning to 100%
University of Houston COM	2022	44	30	Community-based urban, suburban	100%
University of OK School of Community medicine	2025	48	N/A	Tribal, rural, and medically underserved	100%

LIC, longitudinal integrated clerkships.

Data Collection

Two 1.5-h focus groups were held over Zoom in June 2021. Five participants attended the first session, and the remaining eight participants attended the second. Both sessions were recorded. The three questions posed to the members of each focus group were:

1. What have been key supports or key actions which have supported or led to your successful LIC launch?

2. What significant barriers have been encountered in your LIC implementation process?

3. What recommendations do you have for those seeking to start an LIC (which would be helpful in successful implementation)?

We used a modified nominal group process (see Figure 2).^61,62 Participants were emailed the questions beforehand, and asked to consider their responses prior to the focus group. During the focus group, answers to each question were solicited from participants in a round-robin fashion. A moderator summarized each answer on a shared screen and asked for clarification as needed. During the focus groups, SG led the discussion and SKE summarized participant answers. LLB was a participant in the first focus group as the LIC Director of the University of Utah Spencer Fox Eccles School of Medicine. Once participants had no more answers to share for a given question, they were given time to review the written summaries and select the 2 answers that they felt were most important. Once everyone made their selections, they shared their choices with the rest of the group. (In the first focus group, participants shared their selections privately with the moderator using the chat feature, and the moderator summarized the results back to the group. During the second focus group, the chat feature was inadvertently disabled, so participants shared their selections out loud to the full group.)

Figure 2.

Modified nominal group process.

Due to time constraints, after each initial vote, we did not have the customary discussion and final vote. Instead, following the second focus group, the authors compiled all of the responses that received at least one vote across the 2 groups. Participants were asked via email to rank order what they felt were the top 5 most important answers to each question. All 13 focus group participants provided their final rankings.

Data Analysis

Because the researchers summarized the focus group responses during the focus group and the participants provided clarification as needed, we were confident that the written summaries accurately captured the participants’ sentiments. However, because participants had prepared their answers in advance, in many cases there were similar answers provided by multiple respondents, both within and across the 2 focus groups. Therefore, prior to sending the responses back to the participants for their final ranking, SKE and SG combined similar answers from both focus groups into concise statements to prevent redundancies from diluting the ranking process. Table 2 provides an example of multiple statements combined into a single item for ranking. Once all 13 respondents had submitted their rankings, the percentages of participants who ranked each answer in their top 5 were calculated and graphed.

Table 2.

Example of combined statements.

Statements recorded during the focus group	Combined statement for ranking
Steering committee that included all different levels of stakeholders; met regularly, everyone is kept abreast as changes take place in the development process	Stakeholder/key partner involvement—includes buy-in, communication, collaboration, and agreement with the direction of the LIC; includes a multidisciplinary approach that involves all levels (from students to leadership) including both the university and the health system.
Buy in of key partners—ability to have all the key partners bought in and on board
Multidisciplinary representation in the LIC meetings; buy-in
Support from the university and from the medical facility. Support from both arenas is critical
Time for collaboration, facetime for problem-solving and building collaboratively; clerkship directors and regional deans; need for a shared mental model for the LIC.
Willing partners: from the preceptor level all the way to systems level and SOM leadership. Key players must buy in.

LIC, longitudinal integrated clerkships.

Despite a small sample size, our methodology, and the content of the participants’ answers gave us confidence that we were able to reach saturation of the most important supports, barriers, and recommendations for new LIC implementation as experienced by these participants. The nominal group method encourages participants to spend time prior to the focus group thinking about their answers. Additionally, the round-robin format ensures that all participants have an opportunity to provide their answers. We did not end the discussion of any question until all participants indicated they had provided all their answers. Most importantly, there were redundancies among the answers both within and between the groups that indicated we had reached data saturation. ⁶³

Ethical Considerations

This study was determined to be exempt by the Institutional Review Board of the University of Utah (IRB_ 00141272) in 2021. Participants were informed of the study purpose, questions, and procedure prior to the focus groups. Verbal consent was obtained at the beginning of both focus groups with the approval of our IRB because the study presented no more than minimal risk of harm to subjects and involved no procedures for which written consent is normally required outside of the research context.

Results

Key Supports for Successful Launch of an LIC

The first question discussed in the focus group addressed key supports or actions that supported the successful launch of the LIC. The first group (5 participants) provided 15 key supports, and the second group (8 participants) provided 23 key supports. The vote tallies taken during the Zoom group for this question in both groups revealed diversity in opinions about the most important key supports, with the majority of key supports receiving only one vote. The final key supports are provided in Figure 3, along with the frequency with which respondents ranked each factor in their top 5. Table 3 provides additional information about each factor as well as additional supports that were not ranked as top factors.

Figure 3.

Percentage of respondents who ranked each key support in their top 5.

Table 3.

Details about key support.

Key supports	Details
Stakeholder and partner involvement	Willing partners are needed from all subspecialties, university faculty and administration, community clinical staff, and the student body.
	Frequent communication is needed to ensure all stakeholders understand and support the structure, purpose, and benefits of the LIC model.
	Dedicated time needs to be built into the process to develop relationships and resolve problems collaboratively.
	Roles and responsibilities of partners need to be clearly defined.
	“I don't think this would be possible if we hadn't had partnership from all of those aspects, all of those different key players.” (FG1, P1)
Dedicated coordinator or team	Developing the LIC takes more time than expected. The coordinator or coordination team needs “a significant amount of dedicated time for developing curriculum and relationships with [partners.]” (FG1, P4)
	Ideally, the coordinator or coordination team should be in place early in the planning process.
Leadership	LIC Directors need experience, knowledge, passion, commitment, and dedication. The Director needs to have or be able to develop relationships that allow them to facilitate connections with the community and clinicians.
	School of Medicine Leadership needs to provide support and champion the LIC, and effectively address challenges that arise.
	“It's having buy-in from your top leadership, and it really is that it's unwavering like the thing that our school is committed to is this LIC above a lot of other priorities and that has certainly helped our success.” (FG2, P4)
Clerkship directors identified prior to launch	Clerkship directors for each of the included specialties should be identified as early in the process as possible. Roles and responsibilities should be defined before the LIC launch.
	“Really having clerkship director roles identified […] as well as you know, clearly defining what their responsibilities are and how they each work with the staff has been really helpful. To just have that already set before the launch of LIC.” (FG2, P3)
Consultation with external experts	Consultation with experts who have experience directing or developing a similar type of LIC can provide valuable insight into strategies and challenges.
	“One of the things that really helped us a lot, at least getting started, was consultation with outside experts […] a number of times to give us guidance and give us the overall contours and help us be aware of what the challenges are going to be and how we do this or that and so that's been invaluable for us in the in the planning process, and I think in getting successfully approved to now implement the LIC.” (FG2, P2)
Assessment and evaluation systems developed early	Assessment and evaluation systems should be developed with careful forethought ahead of the LIC launch so that problems can be identified early and adjustments can be made before problems get too big.
	“Go ahead and build out your evaluative feedback process, again, for multiple stakeholders. You need feedback from students, from faculty, from your clerkship directors to allow for that flexibility and rapid cycle continuous quality improvement. You're going to need to be able to make changes, but they need to be driven by data, so you need that system in place really before you get started.” (FG2, P4)
Relationships between primary care and subspecialty providers	Relationships between primary care providers (PCPs) and specialty physicians can help students follow patients longitudinally and ease scheduling challenges.
	“Relationships already established between our primary care physicians and some of our specialty physicians eased the transition [to LIC] for them.” (FG2, P5)
Student champions	Students committed to the LIC model can help their classmates see the value and embrace the new model.
	Student champions can also serve as near-peer mentors to new LIC students. LIC students in inaugural classes may benefit from connections with LIC alumni at other medical schools.
	“When we finally had students who themselves had decided they were all-in and love the model, we had student champions, which just helped the rest of their classmates really embrace and see the LIC experience a little differently.” (FG1, P2)
Additional supports not rated in the top 5	Clerkship directors experienced in TBRs can help ensure the LICs meet the same competencies.
	Clinical coaches can support students and alert the team to student needs.
	A creative educational technology team can help keep curriculum delivery integrated.
	Grant funding for the coordinator, the development process, and evaluation can be instrumental in getting the LIC off the ground.
	Dedicated recruiters can help with the process of hiring preceptors and getting buy-in from clinical stakeholders.
	Directors with a history at the institution can leverage personal relationships and institutional knowledge to explain the new model and gain needed approvals.
	Careful selection of students can help ensure a successful LIC launch.
	Software and strong tech support can help with scheduling and evaluation.
	The flexibility and independence of the LIC can be leveraged to meet clerkship requirements.
	Electives such as emergency medicine can be offered if students will be getting the experiences in the course of the LIC.
	The relationship with on-site medical coordinators is important and can be facilitated by careful selection of clinical sites and frequent meetings. Likewise, working with a single medical director can help avoid the need to please too many people.
	The extension school may be able to facilitate relationships with rural communities.

LIC, longitudinal integrated clerkships.

Significant Barriers Encountered During LIC Implementation

The second question discussed in the focus group addressed significant barriers to the successful launch of the LIC. The first group provided 17 barriers, and the second group provided 22 barriers. The vote tallies taken during the first group showed diversity in opinions on the most important barriers. The vote tallies in the second group showed more convergence in opinions, with all respondents voting for preceptor and clinical site recruitment and 5 of the 7 participants voting for underestimating coordination work. This convergence was also seen in the online rankings of the responses. Figure 4 provides the responses that received at least one vote, and the percentage of respondents who ranked each factor in their top 5 after the focus groups. Table 4 provides additional information about each factor as well as additional barriers that were not top factors.

Figure 4.

Percentage of respondents who ranked each barrier in their top 5.

Table 4.

Details about barriers.

Barriers	Details
Preceptor and clinical site recruitment	Clinical sites may be inundated with requests for preceptors and may already have a high concentration of learners.
	Funding for preceptors is an ongoing concern. Clinicians, especially those operating on the relative value unit (RVU) model, may fear financial costs of precepting. Medical schools that do not offer preceptor compensation may be competing with schools that do.
	Preceptors who trained in the TBR model may not understand the LIC model and may find it daunting.
	It can be challenging to find preceptors for all subspecialties of the LIC within a community site.
	The COVID-19 pandemic was a substantial barrier as some health systems withdrew precepting support due to patient overload or social distancing needs. Recruitment of new preceptors was also stymied by system overload and the inability to meet face-to-face with administration at potential sites.
	“I think one of our biggest challenges is in identifying our preceptor network with our community partners - you know, having preceptors motivated to supervise students, because of the complexity experienced with the pressures for clinical revenue within their healthcare systems.” (FG2, P2)
	“For us, recruitment has been challenging as well, and what keeps coming up is funding for these very busy preceptors who are RVU-based [relative value unit]. […] They're not professors or assistant professors here at the School of Medicine, so they don't have to teach and they're doing it out of the goodness of their heart for the most part.” (FG1, P1)
Underestimating coordination work	New LICs underestimated the need for coordinator time, commitment, and allocation of resources within each of the clerkships.
	Clinical sites underestimated the need for a dedicated coordinator.
	New LICs specifically noted that adding LIC responsibilities to existing positions was not effective.
	Underestimation of work and time led some sites to take on too much responsibility.
	“The healthcare system that we're working with in our rural community greatly underestimated the need for a dedicated person to coordinate on their behalf. What they did was take a person who already had a full-time job and just shoved all the responsibilities onto them, and that has been a nightmare for all of us.” (FG2, P6)
Faculty need professional development	New community faculty may not have experience precepting.
	Experienced preceptors may need support to shift their teaching strategies from the TBR model to the LIC model.
	Logistical and communication issues can make professional development and support for community-based faculty difficult to provide effectively.
	“I think one of the hardest things for us … as you know, the distributed model can be great, but it also has been challenging with a lot of our partners really never having had a student before or not being familiar with medical education. That's definitely been a challenge that's needed additional faculty development.” (FG2, P4)
Logistics and structural misalignment	The medical education system and the health systems may have different priorities. Institutional policies reflecting these priorities can create a misalignment in the structure of the two systems and cause challenges in LIC implementation.
	The medical school must coordinate with all participating health systems, each of which may have different policies and requirements. Likewise, each health system may have to learn the policies and procedures of multiple medical schools.
	Legal requirements (affiliation agreements) require a lot of work in a short amount of time. University organizational processes that conflict with external partner processes can cause challenges to finalizing formal agreements, and result in implementation delays.
	“It is difficult to develop academic programs in non-academic spaces.” (FG1, P4)
Resistance to change	LICs can be confusing to stakeholders who are unfamiliar with the model. New programs reported needing to explain the model and its benefits repeatedly.
	Preceptors may not initially understand the difference between learning in the classroom (TBR model) and learning in the workplace (LIC model).
	“Something that we have struggled with is letting the preceptors know that change isn't bad. We know this is a brand-new model and it's scary - it's scary for everybody. And it comes with a new system, a new software system, it comes all these different things, but it's not necessarily a bad thing […] So facing that but encouraging change.” (FG2, P7)
Funding	Funding was a barrier for the new LICs, particularly finding funding for enough LIC staff, including clerkship and site directors to properly run the program, as well as funding for clinical sites.
	“Obtaining funding. For people that are valuable to the implementation of the LIC such as clerkship directors or site directors. That would be very helpful in maintaining relationships and faculty development.” (FG1, P4)
Additional barriers not rated in the top 5	Core principles (eg, “continuity”) were perceived differently by different stakeholders.
	Initial scheduling looked very different across sites.•
	Developing an assessment model that tracked longitudinally was difficult.
	Preceptor drop-out (due to administration or RVU pressures) can create chaos.
	Students may be resistant to the inconvenience of rural placements. The right balance between commute times and educational outcomes must be considered.
	Integrating didactics for TBR and LIC students created challenges.
	Fees for student licenses for electronic health records may cause clinics to limit student participation.
	It may be difficult to obtain effective technology to manage the complex LIC model.
	The health system administration did not understand what it meant to teach in an LIC and sometimes blocked payment of preceptor stipends.
	It was challenging to systematize all of the hospital-based logistics, such as credentialing, onboarding, and training.

LIC, longitudinal integrated clerkships.

Recommendations for Starting an LIC

The final focus group question addressed recommendations for the successful launch of the LIC. The first group provided 14 recommendations, and the second group provided 21 recommendations. As with the first question, the vote tallies taken by both groups showed diversity in opinions about the most important recommendations. Figure 5 provides the combined recommendations that received at least one vote, and the percentage of respondents who ranked each factor in their top five after the focus groups. Table 5 provides additional information about each factor as well as recommendations that were not top factors.

Figure 5.

Percentage of respondents who ranked each recommendation in their top 5.

Table 5.

Details about recommendations.

Recommendations	Details
Invest in needs of clinical partners	Establish clinical partners and get commitments early in the development process.
	Invest extensively in addressing needs of clinical partners, taking care to understand their contexts and involving their leadership early.
	Plan enough time to provide professional development for faculty; communicate regularly with preceptors before they take their first student. “Start faculty development early, even before you get the LIC off the ground.” (FG2, P1)
	Make the time and effort to really understand the clinical sites you will be making requests from. For example, determine whether it is appropriate to initiate the conversation with faculty or leadership first. “Success may depend on going about the process properly, which depends on what their definition of properly is.” (FG1, P1)
Hire a qualified coordinator early	Have a coordinator who can do everything that needs to be done (planning, implementation, and networking) and have that person involved from the very beginning (in the planning phase).
	Allocate dedicated time for LIC responsibilities.
	“Be cautious about thinking that because the LIC is only a few students, it can be added on to an existing job, or that it's just another rotation. It is a lot of work and people with dedicated responsibility for the LIC are needed.” (FG2, P1)
Frequent communication with all stakeholders	Communication must be clear, frequent, open, and consistent to avoid miscommunication and misunderstanding.
	Make sure all stakeholder voices are heard.
	Meet frequently with the initial student cohort so they can accurately champion the program.
	Have frequent meetings with leadership to ensure their ongoing support and commitment.
	“Over-communicate to all levels of everyone involved in the planning process, both internal and external stakeholders; make sure everyone is on the same page at all times.” (FG2, P3)
	“Exercise the art of persuasion with people who may not understand LICs and may be afraid of change.” (FG2, P2)
Consult other LICs	Research other LICs, and consult with the ones that match your desired outcomes and structure.
	“It was such a departure from the traditional way the curriculum has been delivered…so I even consulted with an outside institution's admissions and records officer to find out how they did all this…and that was all very helpful.” (FG2, P2)
Focus on the student experience	Keep a focus on the students; their experience will ultimately determine the success of the LIC.
	“Don’t forget to put yourself in the shoes of the students. Think about the user, your students, and what their experience will be like; always be thinking of students.” (FG2, P4)
Start with desired outcomes	Determine what you want to accomplish with the LIC at the very beginning of the development process.
	“Before you start, define what the non-negotiable outcomes are for your LIC, and then figure out how that matches your community and resources. There are lots of ways to [run an LIC]; determine what will work for your community.” (FG2, P4)
Make the LIC curriculum equitable (not equivalent) to TBR 7	Develop an equitable (not equivalent) curriculum that mirrors the TBR. Accept that the LIC and TBR experiences will not be equivalent.
	“We had to make sure that the equity was there in the amount of education the student was getting with the LIC versus traditional blocks.” (FG2, P5)
Monitor and address issues immediately	If the LIC has enough students, have dashboards in place to help monitor the metrics of the learning environment so that problems can be identified and addressed early (before becoming larger problems.)
	Develop your evaluative feedback system before starting the LIC to allow for flexibility and data-driven continuous quality improvement.
	“Have a program of evaluation in place before starting; this saves a lot of trouble later on.” (FG1, P2)
Choose the right LIC for your stakeholders	Choose the right model for your specific context and goals.
	Identify a clear mission and rationale for your LIC, especially one that can stand out as unique.
	“Do what is right for the context of your school and elevates the mission of both the school and the community health care organization.” (FG2, P1)
Additional recommendations not rated in the top 5	Build in enough time to identify your community, faculty, and student champions.
	Build relationships with clinic managers or the process will fail even if preceptors are engaged.
	Discuss and prioritize the specific dimensions of continuity that the LIC should support (eg, students and preceptors, patients, peers, and place).
	Maintain flexibility during the planning phase. “There's nothing like flying a plane, while building it.” (FG1, P3)
	Account for driving distance so that it does not detract from clinical education for students in community placements.
	Create a detailed timeline for both planning and implementation.
	Conduct multiple trial runs with all technology. Use an accessible electronic resource to track process and progress.
	Make physician identity development a focus of the LIC.
	Start well in advance and include 2-3 months beyond the time you think you will need.

LIC, longitudinal integrated clerkships.

Discussion

Overall, our qualitative research indicates that the challenges described in earlier studies continue to apply, and lead to a conclusion that was woven throughout the study regarding facilitators, barriers, and recommendations: The early stage of LIC development takes a significant amount of time, generally more than anticipated, and needs to be resourced accordingly. Moreover, having those resources and staff in place early in the process was strongly recommended.

While there was general consensus around the key supports, barriers, and recommendations, there was divergence in participants’ votes regarding the most important factors. The greatest convergence was found among barriers. All of the participants selected preceptor and clinical site recruitment and all but one selected underestimating the amount of time and effort needed to develop and launch an LIC as a top barrier. These two barriers are not unrelated, considering the time and effort participants described as necessary for the preceptor recruitment process. The minor divergences in importance ratings represent the unique context of each school of medicine. Despite dissimilarities in the top two rankings of supports and recommendations collected during the focus group, participants agreed that the factors discussed were relevant to their programs. During the focus group, it was clear that participants could relate to all of the key supports, barriers, and recommendations.

Across the three categories, the majority of the factors listed as most important were related to human capital. As the LIC is fundamentally a model of learning through relationships,^26,53 it makes sense that relationships with stakeholders form the backbone of the early development process. Within this overarching topic of relationships were contributing factors of good relationships: frequent communication, investment of time, and reciprocity in terms of needs and priorities. Participants also noted the importance of repetition of a consistent message and updates as changes occur to avoid misunderstandings. Despite the increasing popularity of the LIC model, a lack of understanding of the model persists, even among medical school faculty, but especially among clinical partners.

Limitations

This study limited participation to new LICs located in the United States, in order to increase the likelihood of finding consensus. While there are likely common issues for new LICs regardless of location, medical schools considering LICs in other geographical areas may have different barriers, facilitators, and recommendations based on their specific local and cultural contexts. Additionally, while the current study included LICs with a range of characteristics in terms of length, size, focus, and percentage of students in the LIC, we did not have enough participants to be able to draw conclusions about different subtypes of LICs.

Next Steps

Preliminary results were presented virtually and discussed at the CLICs 2021 Conference in Stellenbosch, South Africa. ⁶⁴ Subsequent efforts led to a collaboration of several participating medical schools on a Personally Arranged Learning Session (PeArL) on recruiting, retaining, and supporting preceptors in the early stages of LICs at the CLIC 2022 Conference in Limerick, Ireland, ⁶⁵ as well as a roundtable discussion on community preceptor recruitment for LICs at the Society of Teachers of Family Medicine (STFM) 2023 Conference on Medical Student Education in New Orleans. ⁶⁶

After the initial focus groups were held, we realized there was an opportunity for a continuing workgroup to problem solve and support one another in their early stage LIC implementation. We invited participants to continue to meet over Zoom on a quarterly basis. LIC staff from all eight schools have continued to participate, as well as a faculty member from another school who joined after hearing about our project at the 2021 CLIC conference. Discussion topics have included barriers and strategies for preceptor recruitment; how the LIC model compares to the TBR model in terms of the numbers of preceptors needed and the amount of time precepting requires of preceptors; administrative staff-to-student ratios needed for LICs; and how to engage preceptors in professional development and how much professional development to provide. Additional research is needed on the critical factors in early stage implementation of LICs, including expanding the geographical scope and comparing different subtypes of LICs.

Conclusions

The early stages of LICs continue to be challenging. While there was divergence in the participants’ preliminary votes regarding the most important supports, barriers, and recommendations, there was consensus regarding the final importance rankings of each of these factors. These findings demonstrate both the commonality of the issues in the LIC development process and the uniqueness of the specific needs and challenges in early-stage implementation within each LIC. Knowledge of these factors can help new schools allocate resources during their LIC development process. Participants found the focus group process and follow-up discussions useful and have formed an ongoing workgroup to meet quarterly.

Authors’ information

SG (MS, PhD, MPH) is an Adjunct Professor in Family and Preventive Medicine at the University of Utah Spencer Eccles Fox School of Medicine. He is also a Professor Emeritus at East Stroudsburg University of Pennsylvania. At the time of this study, he served as the lead program evaluator for the Health Resources and Services Administration (HRSA) grant-funded Value-Based Medical Student Education and Training Program. He has extensive experience with mixed methods program evaluation, including teaching research methods and statistics to Public Health graduate students.

SKE (PhD) is a Research Associate at the University of Utah Spencer Eccles Fox School of Medicine and the Evaluation Manager for the HRSA grant. She has extensive experience with mixed methods program evaluation.

LLB (MD) is a practicing pediatrician. At the time of the study, she was the LIC Director and clinical faculty at the University of Utah Spencer Eccles Fox School of Medicine.