Monitoring Prevalence, Treatment, and Control of Metabolic Conditions in New York City Adults Using 2013 Primary Care Electronic Health Records: A Surveillance Validation Study

Abstract

Introduction:

Electronic health records (EHRs) can potentially extend chronic disease surveillance, but few EHR-based initiatives tracking population-based metrics have been validated for accuracy. We designed a new EHR-based population health surveillance system for New York City (NYC) known as NYC Macroscope. This report is the third in a 3-part series describing the development and validation of that system. The first report describes governance and technical infrastructure underlying the NYC Macroscope. The second report describes validation methods and presents validation results for estimates of obesity, smoking, depression and influenza vaccination. In this third paper we present validation findings for metabolic indicators (hypertension, hyperlipidemia, diabetes).

Methods:

We compared EHR-based estimates to those from a gold standard surveillance source - the 2013–2014 NYC Health and Nutrition Examination Survey (NYC HANES) - overall and stratified by sex and age group, using the two one-sided test of equivalence and other validation criteria.

Results:

EHR-based hypertension prevalence estimates were highly concordant with NYC HANES estimates. Diabetes prevalence estimates were highly concordant when measuring diagnosed diabetes but less so when incorporating laboratory results. Hypercholesterolemia prevalence estimates were less concordant overall. Measures to assess treatment and control of the 3 metabolic conditions performed poorly.

Discussion:

While indicator performance was variable, findings here confirm that a carefully constructed EHR-based surveillance system can generate prevalence estimates comparable to those from gold-standard examination surveys for certain metabolic conditions such as hypertension and diabetes.

Conclusions:

Standardized EHR metrics have potential utility for surveillance at lower annual costs than surveys, especially as representativeness of contributing clinical practices to EHR-based surveillance systems increases.

Introduction

Since 2010, clinical medicine and public health have benefited from a rapid surge of research on the burden and management of chronic diseases using electronic health records (EHRs).1–12 EHRs are appealing because they can offer large sample sizes, timely information, and clinical data beyond that obtained from health surveys or administrative data.9,13,14 Primary care EHRs in particular can potentially extend chronic disease surveillance by assessing the burden of common cardiovascular risk factors and metabolic conditions such as hypertension, hyperlipidemia, and diabetes. They also have the potential to monitor treatment and control patterns for such conditions using nationally recognized standards.

Until recently, objectively measured population estimates of chronic disease burden, treatment, and control were obtained only from population-based examination surveys such as the National Health and Nutrition Examination Survey (NHANES) or rarely conducted local equivalents.15 As EHR networks expand to cover defined geographic jurisdictions or population subgroups, new initiatives are emerging to track population-based metrics using indicators developed specifically for EHR data. However, few of these systems have been validated for accuracy or reliability.16,17

Beginning in 2012, public health practitioners in New York City (NYC) designed a new EHR-based population health surveillance system known as NYC Macroscope, using a large distributed EHR network to monitor chronic health conditions, behavioral risk factors, and clinical preventive services among NYC adults. To guide the initiative, indicators and validation criteria were defined a priori before implementation.18–20 This report is the third in a three-part series describing the development and validation of the NYC Macroscope. The first report describes in detail the governance and technical infrastructure underlying the NYC Macroscope, design decisions made to maximize data quality, characteristics of the patient population sampled, completeness of data, and lessons learned in developing the system.19 The second report describes validation methods used to evaluate the validity and robustness of NYC Macroscope estimates and presents validation results for estimates of obesity, smoking, depression, and influenza vaccination.20 In this third paper we present validation findings for metabolic indicators-including an assessment of the comparability and accuracy of prevalence, treatment, and control measures for hypertension, diabetes, and high cholesterol derived from EHR network data-by comparing NYC Macroscope estimates to estimates generated from a gold standard surveillance source, the 2013–2014 NYC Health and Nutrition Examination Survey (NYC HANES) and by performing an exploratory chart review study. Our aim was to assess overall quality and measure-by-measure variation in validity in an emerging surveillance data source, given the current stage of EHR penetration and documentation quality in this network. Due to declining response rates and increasing costs of population-based surveys, alternate models for enhanced chronic-disease surveillance such as EHR-based systems are needed for local jurisdictions.

Methods

Study Design and Data Sources

This validation study used cross-sectional data from two sources designed to represent NYC adults ages 20 years and older in 2013–2014. The EHR surveillance data source is NYC Macroscope, developed by the NYC Department of Health and Mental Hygiene (DOHMH). The gold standard reference survey was the 2013–2014 NYC HANES, conducted by the City University of New York School of Public Health (CUNY SPH) and DOHMH.

NYC Macroscope is derived from a large, distributed EHR-data network that includes more than 700 ambulatory practices using eClinicalWorks EHR software, serving nearly 2 million individual patients. The network is part of a broader DOHMH initiative to help ambulatory practices in NYC adopt and use EHRs to increase delivery of preventive care services, address chronic disease risks, and improve disease management.21–23 These practices have agreed to share aggregated, de-identified data with DOHMH for public health and clinical quality improvement purposes.24 For NYC Macroscope, data were obtained from 392 primary care practices meeting specific inclusion and exclusion criteria. Of those, 356 practices had electronic lab interfaces to also submit lab measures. Detailed information regarding methodology and sample characteristics of NYC Macroscope have been described elsewhere.18–20

Briefly, information was limited to primary care providers; specialty providers were excluded to minimize double-counting patients visiting more than one practice in the network and to limit the influence of biased prevalence estimates from specialist provider offices. Data were also limited to providers with demonstrated competency in EHR documentation using completeness thresholds for specific fields. Contributing practices were located throughout NYC and concentrated in low-income neighborhoods. A total of 716,076 patients ages 20 and older visited one of these providers in 2013, representing approximately 15.2 percent of an estimated 4.7 million NYC adults ages 20 years and older receiving primary care in the past year.25

The 2013–2014 NYC HANES is a representative population-based survey of adults ages 20 years and older, based on household-based sampling, in-person interviews, a brief physical examination, and collection of fasting biological specimens.26 The final sample consisted of 1,524 NYC adults, 1,135 of whom reported having visited a provider for primary care in the past year (“in care”) and who had a valid NYC ZIP code. All estimates were limited to the population in care, adjusted for nonresponse, and age adjusted to the U.S. 2000 Standard Population.

Data from NYC Macroscope were weighted to the sex, age group (20–39, 40–59, 60–100 years of age), and neighborhood poverty distribution (percentage living in poverty per ZIP code:

<10 percent, 10 percent to 19 percent, 20 percent to 29 percent, ≥ 30 percent) of the adult NYC population in care,23 derived from NYC HANES data. NYC HANES data were weighted to the sex, age group, race and ethnicity, education, marital status, and borough per the American Community Survey 2013, as well as weighted to account for complex survey design and nonresponse per NYC HANES analytic guidelines.27 Relative to the unweighted distribution of the population in care, there were fewer NYC Macroscope patients from wealthier neighborhoods (P = 0.02, data not shown).

Measures

Table 1 presents definitions from NYC Macroscope and NYC HANES for prevalence of hypertension, hyperlipidemia, and diabetes, as well as for the proportion of diagnosed adults that were treated and adequately controlled. NYC Macroscope data were extracted from structured EHR fields using ICD-9 diagnosis codes, vital signs, laboratory data, and medication information, and were developed to mirror widely used survey-based surveillance definitions.18–19

Table 1.

New York City Macroscope and Reference Survey Indicator Definitions, 2013

DOMAIN	OUTCOME	NYC MACROSCOPE		NYC HANES SURVEY
		NUMERATOR	DENOMINATOR	NUMERATOR	DENOMINATOR
Hypertension (HTN)*	“Base” Prevalence — Diagnoses only	Ever had diagnosis of HTNa	All patients seen in 2013	Ever told had HTN	Adults in care (have seen a doctor in past year)
	“Augmented” Prevalence — Diagnosis, HTN medications, or blood pressure exam results	Last blood pressure (BP) measurement ≥140/90 in 2013, ever been diagnosed with HTN, or prescribed a HTN medication in 2013b	All patients seen in 2013	Ever told had HTN or BP at exam ≥140/90, among those with no diagnosis6	Adults in care (have seen a doctor in past year)
	“Survey Gold Standard” Prevalence — Diagnosis and on HTN medications, or blood pressure exam results	Last blood pressure (BP) measurement ≥140/90 or ever diagnosed with HTN and on HTN medication prescribed, past year	All patients seen in 2013	BP at exam ≥140/90 or ever told had HTN and currently self-reporting taking HTN medications	Adults in care (have seen a doctor in past year)
	Treatment among diagnosed	Any HTN medications6 prescribed in 2013 among ever diagnosed with HTN	All patients seen in 2013 ever diagnosed with HTN	Self-reported recall of HTN medications prescribed in the past year among ever told had HTN	In-care adults ever told they had HTN
	Control among diagnosed	Last BP measurement in 2013 <140/90 among ever diagnosed HTN	All patients seen in 2013 ever diagnosed with HTN	BP at survey exam <140/90 among ever told had HTN	In-care adults ever told they had HTN
Hyperlipidemia, age restricted to men aged 40 and older and to women aged 45 and older (CHOL)	“Base” Prevalence — Diagnoses only	Ever had diagnosis of high CHOLc	All patients seen in 2013	Ever told had high CHOL	Adults in care (have seen a doctor in past year)
	Augmented” Prevalence — Diagnosis, CHOL medications, or cholesterol lab results	Last total CHOL lab value in 2012 or 2013 ≥240, ever had diagnosis of high CHOL, or been prescribed a CHOL medication in 2013b	All patients seen in 2013	Ever told had high CHOL or total CHOL ≥240 at time of the exam among adults with no diagnosis6	Adults in care (have seen a doctor in past year)
	“Survey Gold Standard” Prevalence — Diagnosis and on CHOL medications, or cholesterol lab results	Last total CHOL lab value in 2012 or 2013 ≥240 or ever had diagnosis of high CHOL and on CHOL medications prescribed in 2013b	All patients seen in 2013	Total CHOL ≥240 at time of survey or ever told had high CHOL and self-reporting taking HTN medications	Adults in care (have seen a doctor in past year)
	Treatment among diagnosed	Any CHOL medicationsb prescribed in 2013 among ever diagnosed with high CHOL	All patients seen in 2013 ever diagnosed with high CHOL	Self-reported recall of CHOL medication prescribed in the past year among ever told had high CHOL	In-care adults ever told they had high CHOL
	Control among diagnosed	Last total CHOL lab value in 2012 or 2013 <240 among ever diagnosed with high CHOL	All patients seen in 2013 ever diagnosed with high CHOL	Total CHOL <240 at time of the exam among ever told had high CHOL	In-care adults ever told they had high CHOL
Diabetes mellitus (DM)	Prevalence — Diagnoses only	Ever had diagnosis of DMd	All patients seen in 2013	Ever told they had DM	Adults in care (have seen a doctor in past year)
	Prevalence — Diagnosis, DM medications, or Hemoglobin A1c lab results	Last hemoglobin A1c lab value in 2012 or 2013 ≥ 6.5, ever had diagnosis of DM, or been prescribed a DM medication in 2013b	All patients seen in 2013	Ever had diagnosis of DM or Hemoglobin A1c ≥ 6.5 at time of the exam among adults with no diagnosis6	Adults in care (have seen a doctor in past year)
	Treatment among diagnosed	Medications prescribed in 2013 among ever diagnosed with DM	All patients seen in 2013 ever diagnosed with DM	Currently taking medications among ever told had DM	In-care adults ever told they had DM
	Control among diagnosed	Last hemoglobin A1c lab value in 2012 or 2013 ≤ 9 among ever diagnosed with DM	All patients seen in 2013 ever diagnosed with DM	A1c ≤ 9 at time of the exam among ever told had DM	In-care adults ever told they had DM

Notes:

^*Measurements of hypertension are based on having either SBP:≥140 or DBP: ≥90.

^aDiagnoses assessed through end of 2013. ICD-9 codes to identify a diagnosis of hypertension: ‘362.11’, ‘401.0’, ‘401.1’, ‘401.9’, ‘402.00’, ‘402.01’, ‘402.1’, ‘402.10’, ‘402.11’, ‘402.9’, ‘402.90’, ‘402.91’, ‘403’, ‘403.0’, ‘403.00’, ‘403.1’, ‘403.10’, ‘403.9’, ‘403.90’, ‘404’, ‘404.0’, ‘404.00’, ‘404.01’, ‘404.1’, ‘404.10’, ‘404.11’, ‘404.9’, ‘404.90’, ‘404.91’, ‘437.2’.

^bSee Newton-Dame et al. 201619 for a complete list of medications used in each deinition.

^cDiagnoses assessed through end of 2013. ICD-9 codes to identify a diagnosis of hyperlipidemia: ‘272.0’, ‘272.1’, ‘272.2’, ‘272.3’, ‘272.4’, ‘272.7’, ‘272.8’, ‘272.9’

^dDiagnoses assessed through end of 2013. ICD-9 codes to identity a diagnosis of diabetes : ‘249.00’, ‘249.01’, ‘249.10’, ‘249.11’, ‘249.20’, ‘249.21’, ‘249.30’, ‘249.31’, ‘249.40’, ‘249.41’, ‘249.50’, ‘249.51’, ‘249.60’, ‘249.61’, ‘249.70’, ‘249.71’, ‘249.80’, ‘249.81’, ‘249.90’, ‘249.91’, ‘250’, ‘250.0’, ‘250.00’, ‘250.01’, ‘250.02’, ‘250.03’, ‘250.1’, ‘250.10’, ‘250.11’, ‘250.12’, ‘250.13’, ‘250.2’, ‘250.20’, ‘250.21’, ‘250.22’, ‘250.23’, ‘250.3’, ‘250.30’, ‘250.31’, ‘250.32’, ‘250.33’, ‘250.4’, ‘250.40’, ‘250.41’, ‘250.42’, ‘250.43’, ‘250.5’, ‘250.50’, ‘250.51’, ‘250.52’, ‘250.53’, ‘250.6’, ‘250.60’, ‘250.61’, ‘250.62’, ‘250.63’, ‘250.7’, ‘250.70’, ‘250.71’, ‘250.72’, ‘250.73’, ‘250.8’, ‘250.80’, ‘250.81’, ‘250.82’, ‘250.83’, ‘250.9’, ‘250.90’, ‘250.91’, ‘250.92’, ‘250.93’, ‘357.2’, ‘362.0’, ‘362.01’, ‘362.02’, ‘362.03’, ‘362.04’, ‘362.05’, ‘362.06’, ‘362.07’, ‘366.41’, ‘648.0’, ‘648.00’, ‘648.01’, ‘648.02’, ‘648.03’, ‘648.04’

Three different prevalence definitions were validated for hypertension and hyperlipidemia, and two prevalence definitions were validated for diabetes. In NYC Macroscope, the first prevalence definition (“base”) used EHR documentation of an ICD-9 diagnosis, whereas NYC HANES used a self-reported affirmative survey response of having ever been diagnosed with the condition by a health care provider, compared to a negative or “don’t know” response. The second definition (“augmented”) identified additional cases using information typically available in EHRs, including prescribed medications for the condition or the physical exam and lab data listed in structured fields (measured in 2013 for blood pressure readings, and measured in either 2012 and 2013 for hemoglobin A1c lab results and cholesterol lab results). These measures were compared with NYC HANES results that incorporated exam and lab values obtained during the survey examination among those reporting no diagnoses. For hypertension and hyperlipidemia, a third definition (“survey gold standard”) restricted diagnosed cases to those on medication for the condition in the past year, in accordance with gold standard surveillance definitions used with examination survey data.28

Hyperlipidemia estimates were age restricted to men ages 40 and older and women ages 45 and older, due to early poor performance of these indicators among younger age groups and to reflect recommendations to screen men ages 35 and older and women aged 45 and older once every five years.29

Other covariates obtained from the EHR and NYC HANES included sex and age (20–39, 40–64, 65 and older). Neighborhood poverty categories were based on the proportion of individuals with annual income below the federal poverty threshold in the patient’s or participant’s ZIP code of residence (< 10 percent, 10 percent to < 20 percent, 20 percent to < 30 percent, ≥30 percent).

Statistical Analysis

Defined indicators were translated into SQL queries and run for each age, sex, and poverty stratum for both numerators and denominators. Practice sample sizes varied for each measure due to random nonresponse to automated queries and capacity to submit lab measures. There were no patient-level missing data for base measures within practices returning data, as absence of a diagnosis was interpreted as “no diagnosis.” We compared population-level NYC Macroscope estimates with NYC HANES survey estimates overall and stratified by sex and age group. For overall comparisons, we used the two one-sided test of equivalence (TOST)30,31 with a +/−5 percentage point equivalence margin to obtain the probability that estimates were statistically equivalent and compared them to statistical differences measured by t-tests generated by the contrast statements. We also assessed whether NYC Macroscope indicators met additional validation criteria commonly used to validate surveillance systems.32–35 We assessed whether the NYC Macroscope-to-NYC HANES prevalence ratio was within the range of 0.85–1.15, and whether the absolute difference between NYC Macroscope and NYC HANES estimates was within five points. To assess whether goodness of fit varied by sex and age group, we examined whether NYC Macroscope and NYC HANES 95 percent confidence intervals overlapped for each of six strata; those that did not were identified as performing poorly. Spearman correlation coefficients of 0.80 or more met our a priori criteria for linearity across strata. All statistical analyses were performed in 2015 using SAS version 9.2 and SUDAAN version 11.0. Validation methodology is described in more detail in McVeigh et al.20

Criterion Validation of Individual-Level Concordance

As an exploratory analysis, we abstracted EHR data from 48 NYC HANES participants who received primary care from a practice contributing data to NYC Macroscope. Sensitivity and specificity were assessed.

Results

Prevalence

Population prevalence estimates for hypertension, hyperlipidemia, and diabetes generated from NYC Macroscope and NYC HANES are presented in Table 2a, with statistical tests of goodness of fit between the two data sources for each indicator. For base definitions, overall prevalence estimates of hypertension and diabetes were statistically equivalent between NYC Macroscope and NYC HANES. Approximately one-third of in-care NYC residents ages 20 years and older had a diagnosis of hypertension (32.3 percent in NYC Macroscope, 32.5 percent in NYC HANES), and roughly 1 in 8 had a diagnosis of diabetes (13.9 percent in NYC Macroscope, 12.6 percent in NYC HANES). Prevalence estimates for diagnosed hyperlipidemia were slightly higher in NYC Macroscope than in NYC HANES and were not statistically equivalent (49.3 percent in Macroscope, 46.9 percent in NYC HANES).

Table 2a.

Prevalence of Cardiovascular/Metabolic Conditions among New York City Adults in Care, Past Year (2013)

OUTCOME	# PRACTICES RESPONDING TO AUTOQUERY	2013	2013–2014	STATISTICALLY
		NYC MACROSCOPEa	NYC HANES	EQUIVALENT	DIFFERENT
		% (95% Cl)	% (95% Cl)	(TOSTb)	(T TEST)
HYPERTENSION PREVALENCE DEFINITIONS
Base - Diagnosis only	380	32.3 (32.2–32.4)	32.5 (29.4–35.7)	0.001	0.93
Augmented - Diagnosis, meds, or condition indicated during BP measurement	357	39.2 (39.1–39.3)	40.3 (37.3–43.5)	0.007	0.47
Survey Gold Standard - Diagnosis and on HTN meds, or HTN indicated during BP exam	360	33.7 (33.6–33.8)	35.5 (32.5–38.7)	0.02	0.25
HYPERLIPIDEMIA PREVALENCE DEFINITIONS
Base - Diagnosis only	388	49.3 (49.1–49.5)	46.9 (42.6–51.3)	0.12	0.29
Augmented - Diagnosis, meds, or condition indicated in labs	330	54.5 (54.4–54.7)	56.8 (52.3–61.2)	0.11	0.31
Survey Gold Standard - Diagnosis and on CHOL meds, or condition indicated in labs	295	34.0 (33.8–34.2)	41.0 (36.7–45.4)	0.81	0.002
DIABETES PREVALENCE DEFINITIONS
Base - Diagnosis only	383	13.9 (13.8–14.0)	12.6 (10.6–14.8)	<0.001	0.19
Augmented - Diagnosis, meds, or condition indicated in labs	330	15.3 (15.2–15.3)	17.8 (15.5–20.4)	0.03	0.04

For hypertension, the augmented prevalence estimates were higher than base estimates in both data sources and remained statistically equivalent to each other. For hyperlipidemia and diabetes, augmented prevalence estimates also increased in both data sources, but only the augmented diabetes prevalence estimate in NYC Macroscope was statistically equivalent to the augmented NYC HANES prevalence estimate. In NYC HANES the augmented diabetes estimate increased by about 40 percent by including cases identified through systematic laboratory screening of participants. While deemed similar enough to be equivalent (p=0.03 for TOST), the augmented diabetes estimate value was still significantly higher than the NYC Macroscope augmented estimate using medications and laboratory results entered into the EHR (17.8 percent versus 15.3 percent, p-0.04 for t-test). Discrepancies between the two data sources for augmented indicators were greatest for cholesterol.

For hypertension and hyperlipidemia, a third survey gold standard definition was constructed, restricting diagnoses to those on medications only. For hypertension, this prevalence estimate was substantially lower than the augmented estimate (decreasing from 39.2 percent to 33.7 percent in NYC Macroscope, and from 40.3 percent to 35.5 percent in NYC HANES), and estimates between the two data sources remained statistically equivalent. However, concordance between the two data sources for this third definition was poor for hyperlipidemia, with estimated prevalence significantly lower than in NYC Macroscope than NYC HANES (34.0 percent versus 41.0 percent). In a sensitivity analysis to explore the potential influence of diagnosed adults managing their cholesterol through behavioral change alone, we examined NYC HANES data and found that 38 percent of those with diagnosed high cholesterol reported behavioral modification only. This was less of an issue with hypertension, with only 12 percent using behavioral modification alone.

We then examined goodness of fit for prevalence estimates across six different age and sex strata (Tables 2b–2d). As with citywide estimates, substrata prevalence measures generally performed best for hypertension (see appendix for additional detail). Using abstracted EHR data for 48 NYC HANES participants receiving care from a NYC Macroscope practice, we generated preliminary sensitivity and specificity estimates for the different Macroscope prevalence indicators (see appendix for details regarding chart review findings). For hypertension and diabetes, sensitivity and specificity for the base definition was high; hypertension had a sensitivity of 1.0 and specificity of 1.0, and diabetes had a sensitivity of 1.0 and specificity of 0.95. Cholesterol was assessed in a subset of 26 records meeting the age restriction; sensitivity and specificity were 0.69 and 0.62, respectively. Using the augmented definition, performance declined slightly for hypertension (sensitivity 0.83, specificity 0.93), improved slightly for hyperlipidemia (sensitivity 0.77, specificity 0.60), and remained very high for diabetes (sensitivity 1.0, specificity 0.97). With the survey gold standard definition, performance was worse for hypertension (sensitivity 0.76, specificity 0.97), but most improved for cholesterol (sensitivity 0.90, specificity 0.77).

2b.

Comparability Metrics for Prevalence Estimates Between Two Data Sources Using Diagnosis Only (base definition)

METRIC	PREVALENCE RATIO	DIFFERENCE IN PREVALENCE	LINEARITY ACROSS STRATA	POORLY PERFORMING STRATA
EVALUATION CRITERIA	0.85–1.15	≥ 5	R2 ≥ 0.80	≥1 STRATA PPERFORMING POORLY	POORLY PERFORMING SUBPOPULATIONSa
Hypertension	1.00	−0.1	1.00	4	W and M 20–39 W and M 60 +
Hyperlipidemia	1.05	2.4	0.80	1	W 60 +
Diabetes	1.11	1.4	1.00	1	W 60 +

2d.

Comparability Metrics for Prevalence Estimates Between Two Data Sources Using Diagnosis, Meds, or Exam/Labs (survey gold standard definition)

METRIC	PREVALENCE RATIO	DIFFERENCE IN PREVALENCE	LINEARITY ACROSS STRATA	POORLY PERFORMING STRATA
EVALUATION CRITERIA	0.85–1.15	≥ 5	R2 ≥ 0.80	≥1 STRATA PERFORMING POORLY	POORLY PERFORMING SUBPOPULATIONSa
Hypertension	0.95	−1.8	0.94	2	M 40–59; W 60 +
Hyperlipidemia	0.83	−7.0	0.80	1	W 40–59

Treatment

We then examined the estimated proportion of adults with a diagnosed condition who were being treated with medications in the past year. Estimates from NYC Macroscope were not statistically equivalent to those from NYC HANES for any of the three metabolic conditions, and two (hypertension and diabetes) were evaluated to be statistically different using t-tests (Table 3a). For hypertension, 79.4 percent of adults with diagnosed hypertension were treated based on NYC Macroscope estimates versus 63.9 percent treated in NYC HANES. Differences were largely concentrated in the younger age group strata (20–39 years old, see appendix). For adults with diagnosed hyperlipidemia, estimated proportions being treated were nonequivalent but were within three percentage points of each other: 62.4 percent were being treated with medications in NYC Macroscope versus 59.8 percent in NYC HANES. The pattern observed for adults with diagnosed diabetes was reversed, with a significantly lower proportion with a medication prescription in NYC Macroscope than those reporting currently taking medication in NYC HANES (76.9 percent versus 91.0 percent) observed among all age groups (see appendix).

Table 3a.

Treatment of Diagnosed Cardiovascular/Metabolic Conditions among New York City Adults in Care, Past Year (2013)

OUTCOME	2013 NYC MACROSCOPEa	2013 NYC HANES	STATISTICALLY
			EQUIVALENT	DIFFERENT
	% (95% Cl)	% (95% Cl)	(TOSTb)	(T TEST)
Hypertension	79.4 (79.1–79.8)	63.9 (57.3–70.0)	1.00	<0.001
Hyperlipidemia	62.4 (62.2–62.7)	59.8 (53.0–66.2)	0.24	0.42
Diabetes	76.9 (76.4–77.4)	91.0 d (80.8–96.0)	0.99	<0.001

Notes:

^aWeighted to the NYC HANES distribution of the population in care.

^bEquivalent margin: ≥5.

^cEquivalent margin: ≥6.

^dEstimate should be interpreted with caution. The relative standard error (a measure of estimate precision) is greater than 30%, making the estimate potentially unreliable.

Control

For all three conditions, the estimated proportion of diagnosed adults achieving control targets were not statistically equivalent across the two data sources and were confirmed as statistically different for two conditions (Table 3b). Estimates of hypertension and hyperlipidemia control were higher in NYC Macroscope compared with NYC HANES. For adults with diagnosed diabetes, a lower proportion was controlled in NYC Macroscope than in NYC HANES.

Table 3b.

Control of Diagnosed Cardiovascular/Metabolic Conditions among New York City Adults in Care, Past Year (2013)

OUTCOME	2013 NYC MACROSCOPEa	2013 NYC HANES	STATISTICALLY
			EQUIVALENT	DIFFERENT
	% (95% Cl)	% (95% Cl)	(TOSTc)	(T TEST)
Hypertension	65.7 (65.3–66.0)	58.5 (51.1–65.6)	0.72	0.05
Hyperlipidemia	87.1 (86.9–87.3)	79.3 (73.2–84.3)	0.84	0.006
Diabetes	80.4 (79.9–80.9)	82.6d (68.2–91.3)	0.31	0.71

Notes:

^aWeighted to the NYC HANES distribution of the population in care.

^bEquivalent margin: ≥5.

^cEquivalent margin: ≥6.

^dEstimate should be interpreted with caution. The relative standard error (a measure of estimate precision) is greater than 30%, making the estimate potentially unreliable.

For each disease control indicator, a number of NYC Macroscope patients were missing examination or laboratory data, due either to certain practices lacking electronic lab interfaces or to providers not ordering labs. Missing data at the patient level was very low for hypertension (Table 4): only 1.9 percent of patients with a diagnosis of hypertension were missing a blood pressure reading. In contrast, 23.3 percent of patients with diagnosed hyperlipidemia were missing a cholesterol lab result and 26.6 percent of patients with diagnosed diabetes were missing an A1c lab result. Patient-level missingness was clustered within clinical practices; 15 percent and 18 percent of practices returning data for the cholesterol and diabetes indicators, respectively, were missing labs on more than 50 percent of patients.

Table 4.

Missing Data on Control of Diagnosed Metabolic Conditions among New York City Adults in Care, Past Year

OUTCOME	# OF PRACTICESa	# OF PATIENTS WITH A DIAGNOSIS	% PRACTICES MISSING INFORMATION (EXAM OR LAB) IN PAST YEAR ON >50% OF DIAGNOSED PATIENTS TO ASSESS CONTROL	% PATIENTS MISSING INFORMATION (EXAM OR LAB) AMONG DIAGNOSED TO ASSESS CONTROL
Hypertension	337	194782	2.1%	1.9%
Hyperlipidemia (men aged 40 or older and women aged 45 or older)	310	194059	14.8%	23.3%
Diabetes	320	100541	18.1%	26.6%

Note:

^aPractices that consist of patients with no diagnoses of select conditions are excluded.

Discussion

In this study, we compared population health surveillance measures derived from a large, distributed EHR network with similarly defined measures from a gold standard examination survey from the same urban setting. At the current stage of EHR penetration and documentation quality, we found measure-by-measure variation in validity, yet findings confirm that valid prevalence estimates for chronic diseases can be derived using primary care EHRs. All EHR-based hypertension prevalence estimates tested were highly concordant with NYC HANES estimates. Diabetes prevalence estimates were highly concordant when measuring diagnosed diabetes but less so when incorporating laboratory results, potentially reflecting risk-based screening for diabetes in clinical practices compared with universal testing in the survey. None of the hyperlipidemia prevalence estimates were concordant between NYC Macroscope and NYC HANES, possibly due to smaller NYC HANES sample sizes when applying indicator-specific age restrictions and an EHR data extraction window of 2 years when testing may be done less frequently (e.g., every 3–5 years). In addition, factors guiding diagnostic and treatment decisions such as low-density lipoprotein levels and 10-year cardiovascular disease risk are not captured in the case definitions. All measures to assess treatment and control of the three metabolic conditions performed poorly. While NYC Macroscope is one of the first geographic-based EHR surveillance systems to be mounted in the United States, it is important to note that NYC Macroscope is still in early stages of development. Increasing representativeness of contributing clinical practices to EHR-based surveillance systems, standardizing documentation, as well as improved access to historical laboratory data can improve completeness and accuracy of indicators over time.

Using any of the three hypertension prevalence definitions, we found that EHR estimates closely mirrored-within two percentage points-those obtained from NYC HANES. Our experience is consistent with successful efforts mounted in several other countries to determine hypertension prevalence using primary care EHR data.3,4,16,17,36 Indeed, our exploratory sensitivity findings of 83 percent to 100 percent for base and augmented hypertension prevalence definitions were comparable to or better than published sensitivity results from larger individual-level validation studies conducted in Canada (85 percent)16 and Sweden (83 percent).37 Our observations of age and gender subgroup variability using the diagnosis definition may result in part from tendencies among clinicians who do not document diagnoses of hypertension for younger adults based on onetime or infrequent elevated blood pressure readings,38,39 whereas having ever had a blood pressure reading of >140/90 might be recalled as hypertension diagnosis equally across all age groups. Indeed, we observed very high treatment rates among younger adults in NYC Macroscope, suggesting that diagnoses are only documented among younger adults when conditions are medically treated.

Our EHR-generated diabetes prevalence estimates were very similar to NYC HANES estimates when using a diagnosis-based definition, with good performance across most demographic strata, and high sensitivity and specificity per medical chart review. Studies using primary care EHR data to estimate diagnosed diabetes in countries with higher EHR penetration, such as Spain,4,36,40 Switzerland,3 the United Kingdom,17 and Canada,41 have found diabetes prevalence compares well with population-based surveys. We found statistically significant differences when the augmented definition was applied, with prevalence 2.5 percentage points higher in NYC HANES than in NYC Macroscope, most likely reflecting both statistical properties and screening differences. The large sample size of NYC Macroscope facilitates the ability to detect modest significant differences, and systematic A1c testing of all NYC HANES survey participants facilitates greater detection of undiagnosed diabetes. Both national and prior NYC-based examination surveys have found that nearly one-third of adults with diabetes are undiagnosed.42 Researchers elsewhere have also determined that more comprehensive definitions including laboratory readings and medications are preferable for EHR-based diabetes prevalence metrics.3,41 Our preliminary chart review identified very high sensitivity and specificity for both diabetes prevalence definitions, comparable to other studies.16,41

Our less successful efforts to validate prevalence metrics for hyperlipidemia were similar to experiences of researchers elsewhere.3,4,17 Estimates using diagnoses were not statistically equivalent, although estimate magnitudes were within two percentage points of each other, potentially reflecting limited statistical power in NYC HANES estimates due to age restrictions. Prevalence increased by more than five percentage points in both data sources when applying the augmented definition, potentially due to the large number of adults on cholesterol medication without formal diagnoses. It remains unclear whether this is an overestimate, reflecting liberal statin uses on persons whose cholesterol levels are inconsistent with hyperlipidemia,43–45 or a more accurate prevalence estimate that corrects for diagnosis underdocumentation. Interestingly, when the survey gold standard surveillance definition was applied (diagnoses restricted to those on meds only or labs), EHR-based prevalence dropped significantly, from 49 percent to 34 percent, and NYC HANES estimates decreased as well (47 percent to 41 percent) (Table 2a). The drop may be explained by the large proportion of hyperlipidemia patients (37 percent) using only physician-prescribed behavioral changes to control their illness, which suggests that restricting indicators to those on medication substantially underestimates burden and calls into question the utility of such a definition, even in surveys.

We found that treatment and control measures for the three metabolic conditions performed poorly. Statistical power for comparisons of these indicators was limited by small sample sizes of diagnosed adults in NYC HANES. Definitions were also more complex, requiring long medication lists or documentation of examination or lab results in standardized fields. Multiple queries to generate these estimates resulted in higher levels of practice nonresponse. We observed higher treatment and control levels in NYC Macroscope estimates for diagnosed hypertension and hyperlipidemia, but lower treatment and control levels for diabetes. We speculate that disproportionate use of specialists for treating diabetes compared with the other two conditions may be a result of differential missing diabetes information in primary care records. Few EHR-based surveillance systems with validated prevalence metrics have examined treatment and control metrics.4 Improvements in EHR coverage, provider representativeness, patient-level documentation, and management of these conditions in primary care settings will improve quality of treatment and control metrics in EHRs for both surveillance and quality-improvement purposes. More broadly, as the medical care landscape progresses from a fee-for-service to a capitated managed care environment, with standardized reporting incentives under Meaningful Use and other related programs, capacity to monitor disease management indicators should improve.

This study has a number of limitations. First, practices included in NYC Macroscope were not selected at random, and both provider behavior and patient populations may not be generalizable to all providers or to the total NYC in-care population, respectively. To address this, indicators were statistically weighted to the adult NYC population in care. Nonetheless, selection bias at the practice level and residual selection bias at the patient level may exist. Statistical weighting was also used to reduce survey nonresponse bias in NYC HANES-derived estimates. More broadly with respect to generalizability, we have demonstrated elsewhere that the in-care adult population in NYC, compared with the not-in-care population, is more likely to be older, female, non-Hispanic, and insured. They are also more likely to have diagnosed diabetes, hyperlipidemia, and hypertension than is the not-in-care population.25 While this system currently does not purport to represent adults not in care, the extent to which changes in health reform result in decreases in the proportion of the population not in care, this limitation may diminish.

Second, measures obtained from EHRs are more variable in quality than data obtained from standardized examination surveys due to the lack of standardized training on data collection and documentation across providers and clinical practices contributing to the surveillance system. For example, some practices have all data entered into standardized fields, while others retain some or all lab information on scanned PDFs or in free-text fields that are difficult to extract. Third, we used survey definitions to define prevalence of hypertension or hyperlipidemia in NYC HANES; while enhanced with actual blood pressure measurement information, these definitions are not consistent with clinical diagnoses that require measuring the condition on more than one patient visit, which may have contributed to some of the observed variability between sources. Fourth, while inclusion of exploratory chart abstraction improved synthesis of findings and comparison to other studies, the sample size was small, limiting inference. Finally, the current distributed data model used by NYC Macroscope limits our ability to perform small area (neighborhood) estimation, one of the envisioned promises of electronic health record data. To generate the estimated prevalence of one health outcome across NYC’s 59 Community Districts, for example, it would have required 2,784 queries, much larger than the standard 48 queries required for citywide estimates, limiting the logistic feasibility of routinely generating such measures. New efforts now underway will soon allow us to modify queries and permit within-query stratifications (as opposed to across-query) and generate small areas estimates more easily. EHR-based surveillance systems based on line-level data are not be subject to these computational limitations, but any small area estimation should carefully assess for sampling bias within subunits. We are currently exploring more efficient models for potentially generating neighborhood estimates within NYC. These same recent system upgrades will also make it possible to obtain estimates by race and ethnicity in the near future, capitalizing both on improvements in recording race and ethnicity that have occurred as a result of meaningful use criteria and on the technical ability to perform within strata queries .

The main strength of this paper was the use of two data sources representing the same geographic region, as well as careful coordination in the design of metrics between them to maximize comparability. In addition, inclusion criteria on EHR documentation thresholds improved the quality of NYC Macroscope estimates. To our knowledge, NYC Macroscope is the first multicondition EHR surveillance system in the United States designed to represent a geographic jurisdiction, with a large sample size and sizable penetration.

Overall, we found that a carefully constructed EHR-based surveillance system can generate prevalence estimates comparable to those from gold standard examination surveys for certain metabolic conditions, namely hypertension and diabetes. Standardized EHR metrics have potential utility for both surveillance and regional quality-improvement initiatives at lower annual costs than surveys. As NYC Macroscope and other EHR networks expand in coverage and improve documentation quality, accuracy of such metrics should increase. Monitoring validity and sharing lessons learned as networks mature can support data-driven initiatives to bridge primary care and population health.

2c.

Comparability Metrics for Prevalence Estimates Between Two Data Sources Using Diagnosis, Meds, or Exam/Labs (augmented definition)

METRIC	PREVALENCE RATIO	DIFFERENCE IN PREVALENCE	LINEARITY ACROSS STRATA	POORLY PERFORMING STRATA
EVALUATION CRITERIA	0.85–1.15	≥ 5	R2 ≥ 0.80	≥1 STRATA PERFORMING POORLY	POORLY PERFORMING SUBPOPULATIONSa
Hypertension	0.97	−1.1	0.94	2	W 20–39; W 60 +
Hyperlipidemia	0.96	−2.3	0.80	1	W 40–59
Diabetes	0.86	2.6	0.89	1	W 40–59