Association of Family Medicine Milestone Ratings With Early-Career Patient Retention: A US National Insurance Claims Study

ABSTRACT

Background

US family medicine (FM) residency training emphasizes the development of patient-physician relationships, which are associated with patient satisfaction, improved health outcomes, cost containment, and retention. It is important to examine whether this emphasis translates into observable performance during physicians’ early years of independent practice.

Objective

To assess the association between residency Milestone ratings related to patient-physician relationships and patient retention in early clinical practice.

Methods

We analyzed data from 2375 FM residents across 435 programs who completed training in June 2016, linked to 416 249 patients in a commercial health insurance claims dataset. The dataset included patients seen for evaluation and management services and chronic disease visits within 2 years after resident graduation. Subject matter experts identified, a priori, Milestone subcompetencies relevant to patient-physician relationships and retention. A logistic regression model was used to evaluate the association between composite Milestone ratings and patient retention.

Results

After adjusting for each patient’s total visit count to any primary care physician, FM graduates with higher Milestone ratings than their program peers were associated with greater patient retention (OR, 1.19; 95% CI, 1.16-1.22). This association was stronger among graduates from programs with higher composite program-level Milestone performance (OR, 1.30; 95% CI, 1.26-1.35).

Conclusions

Higher ratings in patient-physician relationship subcompetencies were associated with increased patient retention among early-career FM physicians, with stronger associations in programs with higher program mean ratings.

Introduction

Continuity of care (CoC) is a key aspect of health care because it is associated with greater patient satisfaction,^1,2 improved health outcomes,³ and cost containment.^4-7 A major component of CoC is the ongoing relationship between patients and health care professionals, especially the long-term physician-patient relationship. This real-world patient need is reflected in family medicine (FM) residency training and in the FM Milestone subcompetencies used to assess trainee competence. Relevant subcompetencies include continuous monitoring of disease progression, collaborative treatment decision-making with patient input, and displaying humanism and cultural sensitivity to foster professional physician-patient relationships.⁸ These physician attributes have been positively correlated with patient satisfaction and retention,^9-14 whereas their absence has been associated with patients considering changing their physician.¹⁵

Despite their conceptual relevance, little is known about the extent to which Milestone ratings received during residency are associated with FM graduates’ real-world CoC outcomes during early practice. Understanding this relationship is important for evaluating how well FM residents’ competency development during training manifests in meaningful patient outcomes. Therefore, we evaluate the association between national Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings and patient retention for FM graduates during the initial years of practice.

KEY POINTS

What Is Known

Strong patient-physician relationships are believed to contribute to better continuity of care and patient retention. Residency Milestone assessments are used to evaluate trainees’ competency in these relationship-centered skills.

What Is New

This study linking the ACGME Milestone database for family medicine graduates to a commercial health insurance claims data set showed graduates with higher patient-physician relationship Milestone ratings than their peers had significantly greater patient retention in early practice.

Bottom Line

Training programs that foster and achieve higher performance in these competencies may help produce physicians who build more durable patient relationships.

Methods

Data Selection

We identified 3450 FM graduates who completed one of 456 US residency programs during the 2015-2016 academic year using the ACGME national database (Figure 1). Graduates were included if Milestone ratings were reported 6 months prior to graduation. Using national provider identifiers (NPIs), we identified FM graduates who appeared in a health insurance claims dataset collected by a Blue Cross Blue Shield (BCBS) plan in the United States. Graduates were excluded if Milestone ratings were missing or if they were not captured in the BCBS claims dataset.

Figure 1.

Physician-Patient Cohort Definition

Abbreviations: FM, family medicine; ACGME, Accreditation Council for Graduate Medical Education; BCBS, Blue Cross Blue Shield.

Selection of Eligible Patients

The BCBS network insures approximately 109 million Americans (approximately one-third of the US population) and contracts with about 90% of hospitals and physicians. Claims data include the physician’s NPI, Current Procedural Terminology (CPT), and International Classification of Diseases (ICD) codes, describing each rendered service.¹⁶ Outpatient visits for evaluation and management (E&M) or chronic disease care were selected for investigating patient retention, as these types of visits typically require regular follow-up appointments. Outpatient visits were identified using the category of service column in the BCBS claims dataset. E&M visits were defined using CPT codes 99201-99205, 99211-99215, 99241-99245, 99341-99345, 99347-99350, 99381-99387, 99391-99397, 99401-99404, 99411-99412, 99429, 99455-99456, and 99483. Chronic disease care visits were identified using ICD codes, as determined by a family physician with over 30 years of experience in clinical practice and medical education. The list of ICD codes is provided in the online supplementary data. Patients were included in the study only if their visits met one of the following criteria: (1) the visit was classified as outpatient in the category of service and had an E&M CPT code, or (2) the visit was classified as outpatient and was identified as chronic disease care based on the ICD code. This brought the final cohort to 2375 residents from 435 programs.

Selection of Milestone Subcompetencies

We contacted 3 current FM program directors (PDs) as subject matter experts, and 2 agreed to identify the Milestone subcompetencies relevant to patient retention. One was an osteopathic family physician with 13 years as PD in a community-based program; the other, an allopathic family physician with 8 years as PD at a large academic center, who also served as faculty and Clinical Competency Committee member. They were unaffiliated with this study. They independently rated each FM subcompetency on a 4-point scale: 0 “not,” 1 “not very,” 2 “somewhat,” and 3 “highly” for its relevance. Interrater agreement based on unweighted Cohen’s Kappa was 0.77 (95% CI, 0.59-0.95). Subcompetencies were excluded if either PD judged them as “not” or “not very” relevant. For the remaining subcompetencies, PD consensus was assessed using the average of both PDs’ judgment scores. Subcompetencies averaging 2.5 or higher were retained to allow for limited disagreement within the top 2 relevance ratings.

This process yielded the following subcompetencies, pertaining to medical knowledge, interpersonal qualities involving partnering with patients, cultural competency, and communication, except for those under domains of Systems-Based Practice and Practice-Based Learning and Improvement:

Medical Knowledge domain:

• Applies critical thinking skills in patient care

Patient Care domain:

• Cares for patients with chronic conditions

• Partners with the patient, family, and community to improve health through disease prevention and health promotion

• Partners with the patient to address issues of ongoing signs, symptoms, or health concerns that remain over time without clear diagnosis despite evaluation and treatment, in a patient-centered, cost-effective manner

Professionalism domain:

• Demonstrates humanism and cultural proficiency

Interpersonal and Communication Skills domain:

• Develops meaningful, therapeutic relationships with patients and families

• Communicates effectively with patients, families, and the public

The use of Milestones 1.0 and the reporting period we chose avoids introducing effects of the COVID-19 pandemic on assessment or on entry into unsupervised care. FM Milestones 2.0 took effect in July 2020, preventing us from observing a cohort trained under the new Milestones.

Dependent Variable

Patient retention for each FM graduate was defined as repeated visits by outpatients for E&M or chronic disease care between June 2016 and June 2018. We extracted an initial pool of eligible outpatients from the professional claims dataset, consisting of patients with 2 or more such visits with any primary care physician (PCP). An FM graduate was considered to have “retained” a patient if the patient visited the same graduate more than once during the study period.

Statistical Analysis

We used the Milestone ratings reported 6 months prior to training completion.^17,18 This reporting period was selected because specialty-wide ratings at the final reporting period tend to center around 4.0¹⁹ and show substantially less variability than preceding periods. Also, time remaining until graduation allows the opportunity to support trainees, if their Milestone ratings indicate challenges with competencies related to patient retention. We employed a generalized estimating equations regression model to account for the clustering of patient retention within each FM graduate-patient pair, using an exchangeable covariance matrix.

We coded patient retention as a dichotomized outcome variable: 1, if an FM graduate was visited by the same patient more than once, and 0, if the patient visited the graduate only once. Patient retention was regressed on (1) composite ratings, which were calculated by averaging the selected Milestone subcompetencies assigned to FM graduates during the 6 months prior to residency completion, and (2) each patient’s total number of visits for E&M or chronic disease care with any PCP during the study period. We included each patient’s total number of visits as a control variable to examine the adjusted relationship between FM graduate’s Milestone ratings and their ability to retain patients.

To facilitate interpretation, composite Milestone ratings were further decomposed into 2 components before linking to the BCBS data: the training program mean (program-level performance) and each FM graduate’s deviation from that mean (resident-level competency).²⁰ This approach addresses program-level variation observed in national Milestone raw ratings.^21-26 Such variation is attributable both to the collective competency levels of residents within each program (ie, construct-relevant variance) and to potential sources of construct-irrelevant variance, including leniency bias, variability in the quality of assessment tools, and different levels of program engagement in Milestones judgment.^21,23,26 To disentangle these overlapping sources of variability, the program mean was included in the regression model, using the linked outcome as a benchmark to evaluate whether program-level variation reflects construct-relevant variance.²³

We also examined the association between each graduate’s deviation from the program mean and patient retention. An interaction term between the program mean and graduate’s deviation was included to assess whether the association between individual-level competency and patient retention depends on program-level performance.

Results are presented as odds ratios (ORs) with 95% confidence intervals. Statistical significance was determined using 2-tailed P values (threshold, P<.05). All analyses were performed using SparkR version 3.5.0 (Apache Software Foundation).

This study was exempt from review by the American Institutes of Research as it uses administrative data. This study follows STROBE standards (Strengthening the Reporting of Observational studies in Epidemiology).

Results

Of the 416 249 outpatients, 244 096 (58.6%) were women and 172 153 (41.4%) were men The patients’ median age was 42 years, ranging from 0 to 82 (IQR, 28-54; Table 1). Overall, the number of patient visits to any PCPs ranged from 2 to 207, with a median of 4 (IQR, 3-7; Table 2).

Table 1.

Characteristics of Residents and Patients

Characteristic	n (%)
Patients, N=416 249
Gender
Women	244 096 (58.6)
Men	172 153 (41.4)
Age, median (IQR) [min, max]	42 (28-54) [0, 82]
Number of visits to any primary care physician(s), median (IQR) [min, max]	4 (3-7) [2, 207]
Family medicine physician, N=2375
Composite Milestone ratinga
Mean (SD)	3.63 (0.47)
Median (IQR) [min, max]	3.64 (3.36-3.93) [1.50, 5.00]
Physician-patient pairing, N=426 682
Patient’s repeated visit of 1	253 917 (59.5)
2	87 939 (20.6)
3-102	84 826 (19.9)

^aAverage of Milestone ratings on 7 subcompetencies identified as relevant to patient retention.

Abbreviations: IQR, interquartile range; SD, standard deviation.

Table 2.

Exposure Variables

Variable	N	Mean	SD	Min	25th Percentile	Median	75th Percentile	Max
Total visits made by the patient	416 249	5.65	5.22	2	3	4	7	207
Deviation of FM graduate’s Milestone ratinga from program Milestone mean	2375	0.01	0.31	-1.65	-0.15	0.02	0.19	1.40
Program-level Milestone mean	435	3.62	0.36	2.55	3.43	3.64	3.86	4.58

^aAverage of Milestone ratings on 7 subcompetencies identified as relevant to patient retention.

Abbreviation: FM, family medicine.

For the 2375 FM graduates, the composite Milestone rating has a mean (SD) of 3.63 (0.47), with a range of 1.50 to 5.00 (Table 1). Graduates from 435 (95.4%) of the original 456 programs were identified in the BCBS dataset (Table 2). The program-level mean Milestone ratings ranged from 2.55 to 4.58, with a mean (SD) of 3.62 (0.36). The individual deviation ratings ranged from -1.65 to 1.40, with a mean (SD) of 0.01 (0.31).

The 426 682 FM graduate-patient pairs involved 253 917 pairs (59.5%) where FM graduates were visited only once, while the remaining 172 765 pairs (40.5%) included FM graduates who were revisited by any given patient (Table 1).

Programs with higher mean ratings had greater odds of patient retention (OR, 1.09; 95% CI, 1.07-1.11; Table 3). Residents whose ratings were above their program’s mean also had greater odds of the outcome (OR, 1.19; 95% CI, 1.16-1.22). Beyond these effects, a significant interaction was observed between the program-level mean and individual deviations (OR, 1.19; 95% CI, 1.12-1.27). Specifically, for FM graduates from programs with lower program means, the association with patient retention was weaker, whereas this effect was stronger for those graduating from programs with higher program means. This interaction effect is illustrated in Figure 2 as the differences among the 3 curves.

Table 3.

Odds Ratios (ORs) for Family Medicine Graduates’ Patient Retention

Variable	OR (95% CI)	P value
Total visits to physician made by the patient	1.03 (1.02-1.03)	<.001
Individual deviation from the program mean (A)a	1.19 (1.16-1.22)	<.001
Program mean (B) (centered at 3.5)	1.09 (1.07-1.11)	<.001
Interaction between A and B	1.19 (1.12-1.27)	<.001

^aCalculated when program Milestone mean was set to 3.50.

Figure 2.

Composite Milestone Ratings and Probability of Patient Retention by Program Mean

Note: This graph illustrates the relationship between composite Milestone ratings and the probability of patient retention by program means of 3.50 (solid line), 3.75 (dashed line), and 4.00 (dotted line). For graduates of programs with a Milestone mean rating of 3.50, the odds ratio (OR) for patient retention was 1.19, indicating a 19% increase in odds for every 1-point increase from program peers (OR, 1.19; 95% CI, 1.16-1.22). Corresponding odds ratios for programs with mean ratings of 3.75 and 4.00 were 1.25 (95% CI, 1.21-1.28) and 1.30 (95% CI, 1.26-1.35), respectively. All probability estimates are based on a typical patient making 4 visits to any physician for evaluation and management services or chronic disease treatment.

For graduates of programs with a mean of 3.50, the OR for FM graduates’ likelihood of retaining patients was 1.19, meaning an 19% increase in the odds for every 1.0-point deviation from mean rating (OR, 1.19; 95% CI, 1.16-1.22; Table 3). The odds of increased patient retention were statistically significant for programs with other program mean ratings: for a program mean of 3.75 (OR, 1.25; 95% CI, 1.21-1.28), 4.00 (OR, 1.30; 95% CI, 1.26-1.35), and 4.25 (OR, 1.36; 95% CI, 1.30-1.43). An increase in patient visits with any PCP was significantly associated with a higher likelihood of patient retention (OR, 1.03; 95% CI, 1.02-1.03), suggesting that patients with more visits are likely to return to the same FM graduate. A sensitivity analysis excluding patients with more than 100 visits yielded results similar to the original analysis.

Discussion

This study examined the relationship between selected Milestone subcompetencies and patient retention among early-career FM graduates. Through expert consensus, we identified subcompetencies a priori relevant to FM graduates’ ability to retain patients for E&M or chronic care visits, captured in a national health insurance claims dataset as an important real-world outcome. The predicted probability differences associated with the program mean were modest, and the effects for the resident-level deviation scores were somewhat larger. These findings indicate a small but consistent association, suggesting that FM graduates with higher Milestone ratings on the selected subcompetencies appear to have slightly higher patient retention than their peers across programs. Consistent with previous studies,^10,12-14 our findings suggest that higher competency ratings in partnership with patients and their family, collaborative treatment decision-making, and cultural sensitivity during FM residency may be related to improved patient retention.

Limitations

The findings of the present study should be interpreted in the context of several limitations. First, our operationalization of patient retention serves as a proxy rather than a direct measure of patient-physician loyalty. Patients may appear to change their PCP in claims data for many other reasons, including relocation, changes in employer insurance contracts, seeking care at urgent or retail clinics, or receiving treatment within team-based primary care models where visits are distributed across different physicians. Particularly, team-based care models appropriately shift patients among physicians and other care team members, meaning that a patient may be well retained within a practice while not repeatedly seeing the same physician. Although we restricted the analysis to PCP visits to improve attribution, patients in large team-based practices may still rotate among multiple physicians for reasons unrelated to a physician’s competency. Thus, the current study may underestimate continuity for FM graduates in highly team-based environments. In other words, patient retention in this study may reflect a narrower definition—repeated visits to the same physician—rather than patient-physician loyalty within modern primary care systems.

Second, the prevalence of international medical graduates in the FM specialty²⁷ may contribute to discontinuity of care. Some physicians may relocate after completing their visa-waiver requirements, potentially interrupting continuity for their patients.

Third, the competitiveness of practice locations can influence patients’ choices and their ability to schedule with their preferred physician. These include factors such as geography, organization (eg, ownership type, size, role of receptionists, inter-organizational communication), disease-specific physician selection,^6,7 and the degree of competition among practices for patients in various market places.^28,29 Also, care continuity may depend on each FM graduate’s initial practice setting. Starting a practice de novo may offer different opportunities for forming a patient panel and building recurring visits compared to joining an existing group. We were unable to adjust for these variables because they were not sufficiently captured in the BCBS dataset.

Fourth, we included only FM graduates captured in a single commercial insurance claims dataset. Results may not generalize to other specialty graduates, individuals covered by different insurers (eg, Medicaid or Medicare), or those in different age groups. Our study’s patient population may be relatively healthier, and their reasons for remaining with the same physician could differ from those who need more frequent care.

Finally, we relied on a small number of experts to identify CPT or ICD codes for determining patients’ E&M or chronic disease care visits and to select subcompetencies relevant to retaining patient visits. Nonetheless, the selected subcompetencies are supported by extant literature.^30,31

Conclusions

FM trainees during the penultimate assessment period with higher Milestone ratings on expert consensus-selected subcompetencies were associated with a greater likelihood of patient retention. Programs and trainees with higher composite Milestone ratings were more likely to have patients return for E&M or chronic disease care visits.

Author Notes

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.

Disclosure: AI was not used in writing this article.

Editor’s Note

The online supplementary data contains the International Classification of Diseases code taxonomy for chronic disease case identification.