Using United States medicare program claims to identify reliable measures of low value services influenced by primary care physician decisions

Abstract

Background

Low-value services (LVS) are those that “… provide little to no benefit to patients….”¹ Among an existing set of 31 LVS are invasive procedures not typically performed by primary care physicians (PCPs), and other LVS that may be commonly provided by specialists. To facilitate research on the role of primary care physicians in low-value care, we sought to identify the reliability of two subsets of LVS measures that are more directly subject to PCPs’ decisions. We refer to these as primary care adjacent (“PCA”) LVS measures.

Methods

Retrospective observational cohort study to identify which of a 31-item LVS measure set were most commonly performed, ordered, or referred by PCPs and examine the reliability of subsets of PCA-LVS measures. Setting was primary care practices in the Comprehensive Primary Care Plus (CPC+) model evaluation. 5,741,061 Medicare fee-for-service (FFS) beneficiaries attributed to 11,910 practices and 40,981 physicians in 2018 were observed. Based on percentages of LVS commonly performed, ordered, or referred by PCPs, we defined a 4-item and an 11-item PCA-LVS measure. We compared the reliability of the 4-item, 11-item, and 31-item LVS measure sets using Spearman ρ (rho) rank-order correlation and the Intra-Class correlation coefficient (ICC). Limitations included that PCP performing, ordering, and referring data are not available in all claims. Claims-based analyses of diagnosis codes may misclassify some appropriate services as PCA-LVS.

Results

The 4-item PCA-LVS measure has a significantly higher mean reliability estimate than the original 31-item measure in identifying PCPs who provide low versus high rates of LVS. The 11-item PCA-LVS measure also had significantly higher reliability than the 31-item measure but lower reliability than the 4-item measure.

Conclusions

We identified two LVS subsets reflecting LVS events most influenced by PCPs. Further work will be required to ascertain how best to use this or related PCA-LVS measures to enhance the quality of primary care.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12913-026-14983-4.

Introduction

Low-value services (LVS) in health care provide little to no benefit to patients and have potential to cause patient harm or result in unnecessary costs [1]. Measurement of LVS may be helpful to identify and address examples of such low-value care. Schwartz and colleagues identified 31 services detectable in Medicare claims to examine variation in provision of LVS among Medicare beneficiaries [2, 3]. To address changes in diagnosis coding and clinical practice since 2009 not reflected in the work by Schwartz and colleagues, we previously updated the list of 31 claims-based LVS) and demonstrated the updated LVS measure reliably detected variations in care across primary care practice sites observed in the Comprehensive Primary Care Plus (CPC+) initiative [4, 5].

Many of these examples of low-value care are invasive procedures not typically performed by primary care physicians (PCPs), and other LVS within the competencies of PCPs may be commonly provided by specialists. Therefore, to identify PCP-relevant measures, Schwartz and colleagues selected a subset of the original 31 low-value care examples “whose use could be influenced substantially by a primary care physician’s treatment or referral decisions.”³ They reported similar variations among Medicare beneficiaries using measure subsets of 17, 12, and 8 low-value services. Baum and colleagues used Medicare claims analysis of the 31 LVS measures to quantify the share of Medicare spending directly attributable to PCPs through 2014 [6]. They found PCP ordering or performing claims accounted for a median of 8.3% of attributed patients’ costs for LVS. They also estimated PCP referrals accounted for a median of 15.4% of their patients’ costs for low-value care. Their data confirm that some LVS (e.g., pulmonary artery catheterization) were rarely performed by PCPs, while others (e.g., total or free T3 level testing for patients with hypothyroidism) were more commonly ordered or performed by PCPs than specialists.

While these studies confirmed some LVS were more commonly ordered or performed by PCPs, these studies used older LVS measures and did not assess the reliability of potential LVS measure subsets. To facilitate research on contemporary variations in low-value primary care, we sought to identify a subset of the updated LVS measures that are more directly influenced by current decisions of PCPs. To identify these “primary care-adjacent” services, we analyzed 2018 Medicare claims for beneficiaries observed during the CPC+ evaluation. Specifically, we sought to understand which of the updated 31-item LVS measure set were most commonly performed, ordered, or referred by PCPs and examined the reliability of two proposed subsets of primary care adjacent LVS measures.

Methods

Data source and CPC+ background

For this work, we used claims data associated with Medicare beneficiaries attributed to practices participating in the CPC+ Model, a Traditional Fee-for Service Medicare Alternative Payment Model launched by CMS on January 1, 2017. The CPC+ Model ran for five years in 26 regions of the country (additional details provided in Appendix A). To identify which of the 31 updated LVS measures (see Table 1) were more often performed or referred by PCPs (hereafter referred to as “primary-care adjacent” or PCA-LVS), we used fee-for-service (FFS) claims data for 5,741,061 Traditional Medicare beneficiaries attributed to 11,910 practices in 2018 that were either participating in CPC + or were chosen as comparison practices for the evaluation.

Table 1.

Measure descriptions, updated LVS measure set

Measure category, name	Description
CANCER SCREENING
Cancer screening for patients with CKD receiving dialysis	Screening for cancer of the breast, cervix, colon, or prostate for patients age 75 and over with CKD receiving dialysis services
Cervical cancer screening for women age 65 and over	Screening Papanicolaou test for women age 65 and over with no personal history of cancer or dysplasia noted in claim or in prior claims, and no diagnoses of other female genital cancers, abnormal Papanicolaou findings, or human papillomavirus positivity in prior claims
Colorectal cancer screening for adults over age 85	Colorectal cancer screening (colonoscopy, sigmoidoscopy, barium enema, or fecal occult blood testing) for patients age 86 or over with no history of colon cancer
PSA testing for men age 75 and over	PSA testing for patients age 75 and over with no history of prostate cancer
DIAGNOSTIC AND PREVENTIVE TESTING
Bone mineral density testing at frequent intervals	Bone mineral density test within two years of a prior bone mineral density test for patients with established osteoporosis diagnosis
Homocysteine testing for cardiovascular disease	Homocysteine testing with no diagnoses of folate or B12 deficiencies in the claim and no folate or B12 testing in prior claims
Hypercoagulability testing for patients with DVT	Lab tests for hypercoagulable states within 30 days following diagnosis of lower extremity DVT or pulmonary embolism; no prior evidence of recurrent thrombosis, defined by diagnosis of DVT or pulmonary embolism more than 90 days before the testing claim
PTH measurement for patients with stage 1–3 CKD	PTH measurement for patients with CKD and no dialysis services before PTH testing or within 30 days following testing, as well as no hypercalcemia diagnosis during the year
Total or free T3 level testing for patients with hypothyroidism	Total or free T3 measurement in a patient with hypothyroidism diagnosis during the year
1,25-dihydroxyvitamin D testing in the absence of hypercalcemia or decreased kidney function	Calcitriol testing for patients without hypercalcemia, secondary hyperparathyroidism of renal origin, or conditions related to non-PTH-mediated hypercalcemia noted in claim (sarcoidosis, tuberculosis, selected neoplasms), and without a history of CKD; no diagnosis of hypercalcemia in the past 30 days
EMG for low back paina	Individuals without diagnoses of leg pain or sciatica who receive EMG studies
PREOPERATIVE TESTING
Preoperative echocardiography	Echocardiogram not associated with inpatient or emergency care and occurring within 30 days before a low or intermediate risk noncardiothoracic surgical procedure
Preoperative PFT	PFT not associated with inpatient or emergency care and occurring within 30 days before a low or intermediate risk surgical procedure
Routine preoperative stress tests	Stress electrocardiogram, echocardiogram, nuclear medicine imaging, cardiac MRI, or CT angiography, not associated with inpatient or emergency care and occurring within 30 days before a low or intermediate risk surgical procedure
IMAGING
CT of the sinuses for uncomplicated acute rhinosinusitis	Maxillofacial CT study with a diagnosis of sinusitis and no complications of sinusitis, immune deficiencies, nasal polyps, or head/face trauma noted in claim and no sinusitis diagnosis between 30 and 365 days before imaging
Head imaging in the evaluation of syncope	CT or MR imaging of the head with a diagnosis of syncope and no diagnoses in claim warranting imaging
Head imaging for uncomplicated headache	Brain CT or MR imaging with non-post-traumatic, non-thunderclap headache diagnosis, and no diagnoses in claim warranting imaging
EEG for headaches	EEG with headache diagnosis in claim, and no epilepsy or convulsions noted in current or prior claims
Back imaging for patients with nonspecific low back pain	Back imaging with a diagnosis of lower back pain occurring within six weeks of initial back pain diagnosis and with no indication of radiculopathy or other diagnoses in claim warranting imaging
Screening for carotid artery disease in asymptomatic adults	Carotid imaging not associated with inpatient or emergency care for patients without a history of stroke or TIA, and without a diagnosis of stroke, TIA, or focal neurological symptoms in claim
Screening for carotid artery disease for syncope	Carotid imaging with syncope diagnosis for patients without a history of stroke or TIA, and without a diagnosis of stroke, TIA, or focal neurological symptoms in claim
Imaging for diagnosis of plantar fasciitis	Radiographic or MR imaging with diagnosis of plantar fasciitis occurring within two weeks of initial foot pain diagnosis
MRI for RAa	Individuals who receive an MRI of the peripheral joints following a diagnosis of RA
CARDIOVASCULAR TESTING AND PROCEDURES
Stress testing for stable coronary disease	Stress testing not associated with inpatient or emergency care for patients with an established diagnosis of acute myocardial infarction (6 months or more before testing)
Percutaneous coronary intervention with balloon angioplasty or stent placement for stable coronary disease	Coronary stent placement or balloon angioplasty, not associated with an ER visit, or patients with an established diagnosis of acute myocardial infarction (greater than or equal to 6 months before testing)
Renal artery angioplasty or stenting	Renal/visceral angioplasty or stent placement with a diagnosis of renal atherosclerosis or renovascular hypertension noted in procedure claim
Carotid endarterectomy for asymptomatic patients	Carotid endarterectomy, not associated with an emergency room visit, for female patients without a history of stroke or TIA and without stroke, TIA, or focal neurological symptoms noted in claim
Pulmonary artery catheterization in the ICU	Pulmonary artery catheterization for monitoring purposes during an inpatient stay that involved an ICU and nonsurgical MS-DRG; claim contains no diagnoses indicating pulmonary hypertension, cardiac tamponade, or preoperative assessment
OTHER INVASIVE PROCEDURES
Vertebroplasty or kyphoplasty for osteoporotic vertebral fractures	Vertebroplasty/kyphoplasty for vertebral fracture, with no bone cancers, myeloma, or hemangioma noted in procedure claim
Spinal injection for low back pain	Outpatient epidural, facet, or trigger point injections for lower back pain, excluding etanercept; no radiculopathy diagnoses in the claim
Laminectomy or spinal fusiona	Individuals without clear indications of radicular pain or of herniated disc who receive a laminectomy and/or spinal fusion

Source: Fleming et al. (2022)

a This is a new measure identified for use in the 2021 LVS measure set

CKD = chronic kidney disease; CT = computed tomography; DVT = deep vein thrombosis; EEG = electroencephalogram; EMG = electromyography; FFS = fee-for-service; ICU = intensive care unit; MRI= magnetic resonance imaging; PCI = percutaneous coronary intervention; PFT = pulmonary function testing; PSA = prostate-specific antigen; PTH = parathyroid hormone; RA = rheumatoid arthritis; T3 = triiodothyronine; TIA = transient ischemic attack

Primary care practices that participated in CPC+ could participate in either Track 1 or Track 2, with Track 2 offering more enhanced reimbursement but imposing advanced primary care practice requirements. Therefore the types of practices electing Tracks 1 and 2 were different, with comparison practices selected by Track with similar characteristics to the CPC+ practices before the start of the initiative (see Appendix A for more details regarding Track 1 and Track 2 practices).

Identifying PCA-LVS measures

We used a step-wise approach to identify subsets of PCA-LVS measures. First, building off the work of Baum and colleagues, we analyzed Medicare FFS claims to observe how often LVS were ordered/referred, or performed by PCPs [6]. As noted by Baum et al., PCP Performed services “indicates claims for which a beneficiary’s PCP was the performing physician on the Medicare claim; ” PCP Referred services reflect those where the PCP’s National Provider Identifier (NPI) was entered on the Medicare claim as the physician referring the patient for the service. The term “LVS events” denotes claims for LVS-associated HCPCS codes observed at the claim-line level among beneficiaries attributed to Track 1 or Track 2 practices. The programming code for the LVS measures searches Medicare FFS Carrier, Outpatient, and Inpatient claims files for relevant services (see Appendix A for more information about the programming code and our work updating it).

As CMS allows for documentation of clinician performing, ordering, or referring information as well as clinician specialty information in claims, we searched relevant claims variable fields to identify 2018 claims associated with possible LVS events [7, 8].

To identify PCPs, we utilized provider specialty codes in claim line items associated with LVS services. For LVS claims, we identified PCPs via the Line CMS Provider Specialty Code variable. The following codes indicate a PCP in our analyses: 01 (general practice physician), 08 (family practice physician), 11 (internal medicine physician), 37 (pediatric medicine physician), and 38 (geriatric medicine physician). Because it is often difficult to identify physician assistants and nurse practitioners in these settings who are chiefly collaborating with specialized physicians, we excluded claims linked to those types of clinicians. We then finalized our coding scheme by creating a binary indicator which identified whether a LVS event was linked to a PCP ordering, referring, or performing the relevant service as reflected in claims.

Once our coding scheme was in place, for each of the 31 LVS measures we calculated percentages of each service that were PCP-referred or were performed by a PCP, i.e., the percentage that could be categorized as a PCA-LVS event, such as a PCP performing bone mineral density testing within two years of a prior bone mineral density test for patients with established osteoporosis diagnosis. We calculated these percentages across all LVS measures, where denominators were set equal to all LVS in 2018, and the numerators were set to the number of events to which a PCP was linked. We then identified individual LVS measures for which PCPs performed or referred LVS at rates higher than the average such rate across all 31 LVS measures. We refer to groups of such measures as PCA-LVS measure subsets, which we identified based on the distribution of percentages of PCP involvement across LVS events. We report our results by CPC+ track (see Appendix A for more information about how we calculated LVS events).

Approach to testing reliability of PCA-LVS measures

Our second step involved conducting tests of measure reliability in detecting physician-specific performance, both for the overall LVS measure set (31 LVS measures) and for the PCA-LVS subsets we identified. To do this we conducted reliability analysis using the data from beneficiaries attributed to physicians at CPC + and comparison practices. Reliability is defined as the extent to which the measure scores can be replicated over repeat measurements [9]. Higher values of reliability at the physician level also indicate a greater ability to differentiate physicians from one another on that measure [10].

In our previous work, we used the split-half method in addition to the test-retest method to measure reliability [4]. Since the split-half method can, under some conditions, over- or under- estimate reliability because of chance variations in the measure results caused by sampling, we only assessed stability of the measure scores using the test-retest method, which involves comparing the measure scores for the same physicians within the same measurement period. Since we only had one year of data, we opted to use bootstrap resampling to generate independent samples of patients within the same physicians. By creating multiple repeated samples and then averaging the results, the bootstrap method cancels out the chance variations in the split-half design and avoids biased sampling, maintains the original sample size, and allows estimation of confidence intervals for the reliability estimates. We drew 2,000 independent samples of patient records from each physician, maintaining the same number of patients as in the original sample. These random samples from a given physician are assumed to reflect an independent set of re-measurement of the LVS rates for a physician. We then grouped these 2,000 samples into 1,000 pairs, with each pair consisting of two consecutive bootstrap draws. Then, we calculated the correlations between the LVS measure rates within each pair of samples. This process allowed us to generate a distribution of correlation coefficients, from which we estimated the average correlation and the 95% confidence interval in 1,000 paired samples.

We estimated measure reliability via Spearman ρ (rho) rank-order correlation and the intraclass correlation coefficient (ICC). The Spearman’s ρ measures statistical dependence of the physician’s score rank in sample A relative to their rank in sample B. The Spearman ρ ranges between − 1 and 1. The ICC, which ranges between 0 and 1, captures the degree of correlation and agreement between measurements and is represented as a ratio of the variance in the measure scores between physicians over the sum of the variances between and within physicians, where the variance in the measure scores within physicians was estimated via bootstrap resampling. The ICC is a measure of how similar the outcomes for the same physicians would be if they treated different samples of patients, relative to other physicians. Hence, the smaller the disagreement between the measure scores for each physician in different samples, the larger the ICC. Conceptually, calculating the ICC using bootstrap resampling allowed us to assess how similar the LVS scores for the same physicians would be in different samples of patients, relative to all other physicians. Since reliability can be affected by low numbers of patients for some physicians, we restricted the sample to only include physicians with at least 20 patients. For more information on the reliability methods, see Appendix B.

Results

Identifying PCA-LVS subsets

We report LVS events among patients attributed to CPC + and comparison practices in 2018 by CPC+ track. Table 2 reports the number of PCA-LVS events performed or referred by a PCP, the total number of low-value service events (all LVS events across the 31 LVS measures), and the percentage of LVS events influenced by the PCP (PCA Percentage) for each of the 31 LVS measures for beneficiaries attributed to Track 1 practices. Table 3 reports the same information for Track 2 practices. Across the 31 LVS measures, the rate of PCA-LVS events among Track 1 practices and beneficiaries ranged from 71% for colorectal cancer screening for adults over age 85 to 5% for pulmonary artery catheterization in the intensive care unit. On average 39% of the services received were PCA for beneficiaries attributed to Track 1 and Track 2 practices. In both tracks, 11 of the 31 LVS measures had PCA rates greater than the average PCA rates for LVS events in that track, and 4 measures were above the 90th percentile of the distribution of PCA rates. The following LVS measures represent the 4-item and 11-item PCA-LVS subsets we identified as the most commonly performed or referred by PCPs (the first four LVS measures listed comprise the 4-item PCA-LVS subset):

Table 2.

LVS events among CPC+ track 1 beneficiaries influenced by primary care physicians, 2018

Measure name	PCA-LVS Events	All LVS Events	PCA Percentage
Colorectal cancer screening for adults over age 85a, b	6,408	8,984	71.3%
PSA testing for men age 75 and over with no prostate cancer historya, b	102,511	153,726	66.7%
Total or free T3 testing for patients with hypothyroidisma, b	46,504	80,771	57.6%
1,25-dihydroxyvitamin D testing in the absence of hypercalcemia or decreased kidney functiona, b	6,901	12,644	54.6%
Homocysteine testing for cardiovascular diseasea	3,136	6,513	48.1%
Cancer screening for patients with CKD receiving dialysisa	586	1,290	45.4%
Bone mineral density testing at frequent intervalsa	5,813	13,160	44.2%
Screening for carotid artery disease for syncopea	5,995	13,963	42.9%
Preoperative PFTa	1,663	3,926	42.4%
Screening for carotid artery disease in asymptomatic adultsa	55,671	137,264	40.6%
Back imaging for patients with nonspecific low back paina	50,708	126,109	40.2%
All LVS	387,253	1,002,613	38.6%
Preoperative echocardiography	3,693	11,500	32.1%
EEG for headaches	321	1,084	29.7%
Imaging for diagnosis of plantar fasciitis	4,322	15,179	28.5%
Routine preoperative stress tests	2,234	8,185	27.3%
Hypercoagulability testing for patients with DVT	478	1,827	26.2%
PTH measurement for patients with stage 1–3 CKD	27,425	109,626	25.0%
CT of the sinuses for uncomplicated acute rhinosinusitis	2,574	10,435	24.7%
Head imaging in the evaluation of syncope	770	3,190	24.1%
EMG for low back pain	7,895	32,885	24.0%
Head imaging for uncomplicated headache	20,831	87,346	23.9%
Spinal injection for low back pain	136	571	23.8%
Renal artery angioplasty or stenting	13,185	56,144	23.5%
Vertebroplasty or kyphoplasty for osteoporotic vertebral fractures	406	1,819	22.3%
Carotid endarterectomy for asymptomatic patients	182	914	19.9%
Stress testing for stable coronary disease	2,959	16,123	18.4%
MRI for RA	1,664	9,774	17.0%
Cervical cancer screening for women age 65 and over with no history of cancer or dysplasia	11,833	72,919	16.2%
Laminectomy or spinal fusion	311	2,545	12.2%
Percutaneous coronary intervention with balloon angioplasty or stent placement for stable coronary disease	130	2,028	6.4%
Pulmonary artery catheterization in the ICU	9	171	5.1%

a This measure is part of the 4-item PCA measure subset

b This measure is part of the 11-item PCA measure subset

PCA = primary care adjacent; CKD = chronic kidney disease; CT = computed tomography; DVT = deep vein thrombosis; EEG = electroencephalogram; EMG = electromyography; FFS = fee-for-service; ICU = intensive care unit; MRI= magnetic resonance imaging; PCI = percutaneous coronary intervention; PFT = pulmonary function testing; PSA = prostate-specific antigen; PTH = parathyroid hormone; RA = rheumatoid arthritis; T3 = triiodothyronine; TIA = transient ischemic attack

Table 3.

LVS events among CPC+ track 2 beneficiaries influenced by primary care physicians, 2018

Measure name	PCA-LVS Events	All LVS Events	PCA Percentage
Colorectal cancer screening for adults over age 85a, b	4,812	6,939	69.4%
PSA testing for men age 75 and over with no prostate cancer historya, b	83,010	123,419	67.3%
Total or free T3 testing for patients with hypothyroidisma, b	38,740	65,707	59.0%
1,25-dihydroxyvitamin D testing in the absence of hypercalcemia or decreased kidney functiona, b	5,895	11,030	53.4%
Homocysteine testing for cardiovascular diseasea	2,377	5,095	46.7%
Bone mineral density testing at frequent intervalsa	4,701	10,359	45.4%
Screening for carotid artery disease for syncopea	5,376	12,236	43.9%
Preoperative PFTa	1,313	3,198	41.1%
Cancer screening for patients with CKD receiving dialysisa	408	995	41.0%
Screening for carotid artery disease in asymptomatic adultsa	45,719	111,562	41.0%
Back imaging for patients with nonspecific low back paina	43,642	107,144	40.7%
All LVS	324,521	832,916	39.0%
Preoperative echocardiography	3,042	9,421	32.3%
Hypercoagulability testing for patients with DVT	485	1,573	30.8%
Imaging for diagnosis of plantar fasciitis	3,857	12,585	30.6%
EEG for headaches	258	862	29.9%
PTH measurement for patients with stage 1–3 CKD	26,693	96,803	27.6%
Routine preoperative stress tests	1,829	6,739	27.2%
EMG for low back pain	693	2,635	26.3%
CT of the sinuses for uncomplicated acute rhinosinusitis	2,150	8,488	25.3%
Head imaging for uncomplicated headache	18,002	74,783	24.1%
Head imaging in the evaluation of syncope	6,652	28,053	23.7%
Renal artery angioplasty or stenting	113	484	23.3%
Spinal injection for low back pain	10,866	46,981	23.1%
Carotid endarterectomy for asymptomatic patients	164	797	20.5%
Vertebroplasty or kyphoplasty for osteoporotic vertebral fractures	299	1,475	20.3%
MRI for RA	1,445	8,218	17.6%
Stress testing for stable coronary disease	2,275	13,276	17.1%
Cervical cancer screening for women age 65 and over with no history of cancer or dysplasia	9,285	57,895	16.0%
Laminectomy or spinal fusion	287	2,200	13.1%
Percutaneous coronary intervention with balloon angioplasty or stent placement for stable coronary disease	130	1,805	7.2%
Pulmonary artery catheterization in the ICU	5	159	2.9%

a This measure is part of the 4-item PCA measure subset

b This measure is part of the 11-item PCA measure subset

PCA = primary care adjacent; CKD = chronic kidney disease; CT = computed tomography; DVT = deep vein thrombosis; EEG = electroencephalogram; EMG = electromyography; FFS = fee-for-service; ICU = intensive care unit; MRI= magnetic resonance imaging; PCI = percutaneous coronary intervention; PFT = pulmonary function testing; PSA = prostate-specific antigen; PTH = parathyroid hormone; RA = rheumatoid arthritis; T3 = triiodothyronine; TIA = transient ischemic attack

• Colorectal cancer screening for adults over age 85;

• Prostate-specific antigen (PSA) testing for men age 75 and over with no prostate cancer history;

• Total or free T3 testing for patients with hypothyroidism;

• 1,25-dihydroxyvitamin D testing in the absence of hypercalcemia or decreased kidney function;

• Homocysteine testing for cardiovascular disease;

• Cancer screening for patients with chronic kidney disease receiving dialysis;

• Bone mineral density testing at frequent intervals;

• Screening for carotid artery disease for syncope;

• Preoperative pulmonary function testing;

• Screening for carotid artery disease in asymptomatic adults; and.

• Back imaging for patients with nonspecific low back pain.

Reliability analysis

We assessed reliability of the 4-item and the 11-item PCA subsets of LVS measures, relative to the previously validated 31-item LVS measure. As we did in a previous paper, we conducted bootstrap resampling to assess reliability of the 4-item and 11-item PCA subsets, and the 31-item LVS measure set (results presented in Table 4) [4].

Table 4.

Distribution of the Spearman ρ statistic in the bootstrap samples among Tracks 1 & 2 CPC + and comparison physicians with a minimum of 20 attributed beneficiaries, 2018

Statistic	4-Item PCA-LVS Measure Subset		11-Item PCA-LVS Measure Subset		31-Item LVS Measure Set
	Track 1	Track2	Track 1	Track 2	Track 1	Track 2
Mean ρ	0.768	0.750	0.754	0.733	0.750	0.731
95% Confidence Interval for the mean	0.762–0.774	0.744–0.756	0 748 − 0.760	0.727–0.739	0.744–0.756	0.725–0.737
Standard deviation	0.003	0.003	0.003	0.003	0.003	0.003
Minimum ρ	0.759	0.740	0.742	0.722	0.738	0.719
2.5th percentile	0.762	0.744	0.748	0.727	0.744	0.724
Lower quartile	0.766	0.748	0.752	0.731	0.748	0.729
Median	0.768	0.751	0.754	0.733	0.750	0.731
Upper quartile	0.770	0.753	0.756	0.735	0.752	0.733
97.5th percentile	0.774	0.757	0.760	0.739	0.755	0.736
Maximum ρ	0.777	0.761	0.762	0.743	0.758	0.741

For the bootstrap resampling analysis, among both Track 1 physicians (N = 20,920) with at least 20 beneficiaries (N = 3,090,020) as well as Track 2 physicians (N = 20,061) with at least 20 beneficiaries (N = 2,884,399), the 4-item and 11-item PCA-LVS subsets had higher reliability relative to the 31-item LVS set, using the Spearman ρ (rho) rank-order correlation. The mean N of patients for clinicians with > = 20 patients was 148 patients among Track 1 physicians and 144 patients among Track 2 physicians.

Comparing results between the 4-item and 11-item PCA-LVS subsets, we found the 4-item subset to be more reliable, as evidenced by the higher correlations across the 1,000 pairs of bootstrap samples. Mean Spearman ρ correlation values for the 4-item subset across the 1,000 pairs of bootstrap samples were 0.768 (95% C.I.: 0.762–0.774) and 0.750 (95% C.I.: 0.762–0.774) among Track 1 and Track 2 physicians, respectively, compared to 0.754 (95% C.I.: 0.748–0.760) and 0.733 (95% C.I.: 0.727–0.739) for the 11-item subset and 0.750 (95% C.I.: 0.744–0.756) and 0.731 (95% C.I.: 0.725–0.737) for the full 31-item LVS set (Table 4).

The 4-item and 11-item PCA-LVS subsets also demonstrated slightly higher reliability than the 31-item LVS set according to the ICC statistics. Among Track 1 physicians mean ICCs were 0.842 (95% C.I.: 0.835–0.850), 0.834 (95% C.I. 0.828–0.841), and 0.803 (95% C.I. 0.796–0.809) for the 4-item and 11-item PCA-LVS subsets and 31-item LVS set, respectively (Table 5). Among Track 2 physicians, mean ICCs were 0.803 (95% C.I.: 0.791–0.815), 0.796 (95% C.I. 0.788–0.805), and 0.776 (95% C.I. 0.768–0.783) for the 4-item and 11-item PCA-LVS subsets and 31-item LVS set, respectively (Table 5). Following Portney and Watkins (2000), ICC values above 0.75 indicate good reliability [11]. According to the National Quality Forum guidelines, ICC coefficients above 0.6 indicate adequate test-retest reliability [12].

Table 5.

Distribution of the ICC statistic in the bootstrap samples

Statistic	4-Item PCA-LVS Measure Subset		11-Item PCA-LVS Measure Subset		31-Item LVS Measure Set
	Track 1	Track2	Track 1	Track 2	Track 1	Track 2
Mean ρ	0.842	0.803	0.834	0.796	0.803	0.776
95% Confidence Interval for the mean	0.834–0.850	0.791–0.815	0.828–0.840	0.788–0.804	0.797–0.809	0.768–0.784
Standard deviation	0.004	0.006	0.003	0.004	0.003	0.004
Minimum ρ	0.827	0.779	0.824	0.781	0.791	0.764
2.5th percentile	0.835	0.790	0.828	0.787	0.796	0.768
Lower quartile	0.840	0.800	0.832	0.794	0.801	0.773
Median	0.842	0.803	0.835	0.796	0.803	0.776
Upper quartile	0.845	0.807	0.837	0.799	0.805	0.778
97.5th percentile	0.850	0.814	0.841	0.804	0.809	0.783
Maximum ρ	0.855	0.820	0.844	0.809	0.814	0.787

While the mean reliability estimate (based on the Spearman ρ correlation) of the 11-item set was significantly higher than for the 31-item set, the magnitude of the difference between the 11- and 31-item sets as captured by the t-statistic and Cohen’s d effect size, was smaller than the magnitude of the differences between the 4- and 11-item sets (Table 6) [13]. Our results thus suggest that the 4-item set had significantly higher bootstrap reliability than either the 11- or 31-item set.

Table 6.

Differences in the mean reliability estimates (Spearman ρ) between 4, 11 and 31 item sets

Statistic	Track 1			Track 2
	4 vs. 31 item set	4 vs. 11 item set	11 vs. 31 item set	4 vs. 31 item set	4 vs. 11 item set	11 vs. 31 item set
t-statistic for the differences in the mean reliability estimates (Spearman ρ) across 1,000 pairs of bootstrap samples	134.165	104.350	19.814	131.618	126.711	14.907
Cohen’s d effect size for the differences in the mean reliability estimates (Spearman ρ) across 1,000 pairs of bootstrap samples	6.000	4.667	1.333	6.333	5.667	0.667

Notes: t-statistics were significant at p < 0.001; Cohen’s d > 0.8 indicates that the mean difference between the item sets (effect size = signal) is large compared to the variability in the rates within the item sets (noise)

Discussion

Using Medicare claims data, we identified two subsets of PCA-LVS reflecting low-value service events most influenced by PCPs. Our analyses demonstrate that the 4-item PCA-LVS measure has a significantly higher mean reliability estimate than the original 31-item measure (or the 11 item PCA-LVS measure) in identifying those PCPs who provide low versus high rates of low-value services (Tables 4, 5 and 6).

The four services included in the 4-item PCA-LVS measure (colorectal cancer screening for adults over age 85; PSA testing for men age 75 and over with no prostate cancer history; total or free T3 testing for patients with hypothyroidism; and 1,25-dihydroxyvitamin D testing in the absence of hypercalcemia or decreased kidney function) reflect examples of two different types of low-value primary care decisions: cancer screening and diagnostic testing. For each of the four low-value services, PCPs accounted for over 50% of the LVS events. Collectively, these 4 services constituted 16% of the LVS events experienced by CPC+ Medicare beneficiaries in 2018. For entities (e.g., payers, health systems, or individual primary care practices) considering initiatives that assess LVS provision by PCPs, the 4-item measure set is statistically more reliable than the 11-item or 31-item measure sets. Although the increase in reliability for the 4-item measure is modest, even small improvements in precision of estimates can often have significant implications for inferences about the quality of care [14, 15]. Reducing physician-specific error estimates decreases the likelihood of misinterpreting a physician’s performance due to noise and lowers the risk of misclassifying providers into incorrect performance categories. Additionally, enhanced reliability improves the ability to detect small changes in performance over time, facilitating more effective quality improvement efforts and better tracking of progress.

For each of these 4 PCA-LVS there are opportunities for some PCP practices to generate extra Medicare FFS revenue by providing these services. In some organizations the physicians (or their practice sites) may receive the revenue but in other circumstances (such as integrated delivery systems) another cost center within the larger organization may receive the revenue. Interestingly, recent research using the 31-item LVS measure found that PCPs working in health systems had lower 31-item LVS scores [5]. These significantly lower rates of LVS were concentrated among a small subset of the 31 services, including colorectal and prostate cancer screening, and T3 level testing for patients with hypothyroidism. While it is possible the lack of physician-level financial incentives for low-value testing might have contributed to this result, other work has found LVS also occur in settings lacking PCP FFS incentives for overuse [16]. Analyzing a larger set of low-value services, Ganguli and colleagues (2021) showed some health system characteristics were associated with higher rates of low-value care, including systems with a smaller proportion of primary care physicians and those serving areas with higher levels of health care spending [17]. In their analyses of a larger set of “primary care influenced” LVS measures, Schwartz and colleagues (2019) found “extensive physician-level variation not attributable to patient, organizational, or regional factors.”³ Further research using the 4-item PCA-LVS measure may prove useful in better understanding the physician incentives, practice settings, organizational affiliations, and patient and community factors that contribute most strongly to such low-value primary care. Such information may be informative to professional associations and provider organizations seeking to facilitate reductions in rate of PCP use of such low-value care.

Other potential applications of such PCA-LVS measures are in interventions such as PCP feedback or pay for performance. Various feedback programs aimed at PCPs have proven successful in reducing some examples of low-value care [18, 19]. Research has been more mixed regarding the benefits of pay for performance initiatives on low-value care [20]. While the simplicity and primary care salience of the 4-item PCA-LVS measure may facilitate its application, various policy researchers have noted the risks of PCP-directed pay-for-performance plans that focus on narrow areas of clinical care and the potential for sophisticated health care systems to manage such isolated metrics without addressing more patient-centered outcomes [21, 22].

Applying the 4- or 11-item PCA-LVS measures to track LVS provision at primary care practices may incentivize some primary care practices to continue to provide other LVS not included in these metrics. Limiting the number of measures tracked to those most pertinent to PCP decision making, however, would allow practices to more effectively target resources to ensure patients are receiving quality care. Alternatively, the 11-item measure is also a reliable measure of variations in PCP LVS and might highlight additional opportunities for PCPs to reduce low-value care. With either metric, as we found in previous work, initiatives to apply these measures in clinical settings would require monitoring and updating as clinical evidence, payer responses, and practice patterns change [4].

In conclusion, we have identified a 4-item measure of Primary Care Adjacent LVS which reliably identifies high LVS vs. low LVS PCPs. Further work will be required to ascertain how best to use this or related PCA-LVS measures to enhance the quality of primary care practice.

Limitations

Though we sought to reduce the number of appropriate services misclassified as “low-value” (using the methods of Schwartz and colleagues [2]), these limitations are difficult to avoid in a strictly claims-based analysis lacking the clinical nuance available from medical records. Conversely, it’s quite likely this approach also underestimates the number of LVS delivered. See Appendix A for more on the methods and the programming code.

As mentioned, our analyses were restricted to Medicare claims that contained referring and performing provider data as well as provider specialty information. In an analysis of 2018 and 2019 Medicare FFS claims, we found that 95% of Medicare FFS Carrier claims contained referring National Provider Identifier (NPI) and related specialty information while 70% of Carrier claims contained performing NPI and specialty information.

As with any analysis of Traditional Medicare beneficiaries, the low value services studied apply to an older age population with insurance benefits determined under Medicare regulations.

Despite these limitations, we have identified a 4-item measure of Primary Care Adjacent LVS which reliably identifies high LVS vs. low LVS PCPs. The environment and context for applying measurement to improve primary care practice has become even more complex since 2018 [23, 24]. Accordingly, further work will be required to ascertain how best to use this or related PCA-LVS measures to enhance the quality of primary care practices.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary Material 1 ^{(44.8KB, docx)}

Supplementary Material 2 ^{(45.2KB, docx)}

Supplementary Material 3 ^{(47.4KB, docx)}

Supplementary Material 4 ^{(45.7KB, docx)}

Author contributions

Chris Fleming led the writing of the manuscript and worked with the co-authors on the analyses therein. Dmitriy Poznyak led the reliability analyses presented in the manuscript. Rhea Powell provided guidance regarding research aims and interpretation of associated analyses, as well as provided editorial and technical guidance throughout the process of producing the manuscript. Claire Erba and Claire Burkhart carried out most programming tasks associated with our analyses. Eunhae Shin provided editorial guidance as well as led statistical analyses for the previous paper which aided the team in defining the current low value services measure set analyzed here. Rui Wang led the programming for the reliability analyses. Arkadipta Ghosh managed the project; provided guidance regarding research aims and associated interpretations and written summaries; and provided editorial and technical guidance throughout the process of producing the manuscript. Eugene Rich provided thought leadership, reviewed and helped update manuscript drafts, summarized findings, and provided guidance throughout the entire process of producing the manuscript.

Funding

Supported by the U.S. Department of Health and Human Services, Centers for Medicare & Medicaid Services, under contracts HHSM-500-2014-00034I/75FCMC19F0005 and HHSM-500-2014-00034I/HHSM-500-T0010.

Data availability

The Medicare claims data used for this study are not available to the public. The programming code used to develop the LVS measure sets used in the study is available to the public, and links to the code are the provided in the supplementary materials (appendices) to the study.

Declarations

Ethics approval and consent to participate (IRB approval)

The need for informed consent was waived per the New England Institutional Review Board (IRB), which granted the CPC+ evaluation an IRB exemption on the basis of the federal common rule (Sect.  45 CFR 46.101[b][5]), because the purpose of the study was to evaluate a public benefit program.

Consent for publication

Not applicable.

Authors’ information

Not applicable.

Disclaimer

The contents of this manuscript are solely the responsibility of the authors and do not necessarily represent the official views of the U.S. Department of Health and Human Services or any of its agencies.

Competing interests

The authors declare no competing interests.