Development and Validation of the Agency for Healthcare Research and Quality Measures of Potentially Preventable Emergency Department (ED) Visits: The ED Prevention Quality Indicators for General Health Conditions

Abstract

Objective

To develop and validate rates of potentially preventable emergency department (ED) visits as indicators of community health.

Data Sources

Agency for Healthcare Research and Quality, Healthcare Cost and Utilization Project 2008–2010 State Inpatient Databases and State Emergency Department Databases.

Study Design

Empirical analyses and structured panel reviews.

Methods

Panels of 14–17 clinicians and end users evaluated a set of ED Prevention Quality Indicators (PQIs) using a Modified Delphi process. Empirical analyses included assessing variation in ED PQI rates across counties and sensitivity of those rates to county‐level poverty, uninsurance, and density of primary care physicians (PCPs).

Principal Findings

ED PQI rates varied widely across U.S. communities. Indicator rates were significantly associated with county‐level poverty, median income, Medicaid insurance, and levels of uninsurance. A few indicators were significantly associated with PCP density, with higher rates in areas with greater density. A clinical and an end‐user panel separately rated the indicators as having strong face validity for most uses evaluated.

Conclusions

The ED PQIs have undergone initial validation as indicators of community health with potential for use in public reporting, population health improvement, and research.

Hospital emergency departments (EDs) can be viewed as a window into aspects of local community health and the resources in place to meet community health needs (Institute of Medicine 2007). Local EDs are an important gateway into the health care system for many patients. As the rate of potentially preventable adult inpatient stays decreased 19 percent between 2005 and 2012, the rate of potentially preventable treat‐and‐release ED visits rose 11 percent during the same time period (Fingar et al. 2006). In this paper, we report the development and validation of the Agency for Healthcare Research and Quality (AHRQ) Emergency Department Prevention Quality Indicators (ED PQIs). The ED PQIs are measures that build on the concept of ambulatory care sensitive conditions (ACSC), or conditions for which hospitalization may have been avoidable through access to high‐quality community‐based care. Recently the concept has been expanded to potentially preventable ED visits (Johnson et al. 2012; Brownell et al. 2014; Galarraga, Mutter, and Pines 2015).

This study builds on previously developed AHRQ measure sets, namely the Prevention Quality Indicators (PQIs) that use readily available hospital administrative data to identify community rates of potentially preventable hospitalizations (Davies et al. 2011a; Agency for Healthcare Research and Quality 2014a). We sought to identify a set of ACSC‐like measures that capitalizes on the unique attributes of ED encounters, offers utility to communities working to improve the health of their populations, and aligns with widely available data sources. Ideal measures must capture conditions frequently seen in the ED setting that may reflect inadequate community health resources (e.g., physicians, home health care, dental care, etc.) or high disease burden or both. They must also vary enough across geographic areas to serve as meaningful points of comparison for the challenges faced by many community health care systems.

ED‐based measures provide an additional view of community health that complements inpatient‐based measures by capturing events that do not result in hospitalization. For example, some conditions, such as pediatric asthma, are increasingly being treated in the ED rather than through inpatient admission. For other conditions, such as otitis media and sinusitis, hospitalizations are rare but ED visits for these conditions quite common, and in many cases, they can be prevented through the provision of outpatient care in other settings. One advantage of hospital administrative ED data is the ability to capture information from all presenting patients regardless of condition or ability to pay (including the uninsured). Also, use of administrative data allows measurement of rates at the community level based on the patients’ county of residence, rather than ED location.

The ED PQIs reported in this paper represent a novel set of measures intended to reflect community health and resources.

Methods

Our multidisciplinary team consisting of health services researchers, emergency department physicians, community clinicians, and statisticians, under contract with AHRQ, developed the ED PQIs using a standard measure development process of evidence‐based indicator concept selection, specification of precise definitions, and validation through empirical analyses and structured panel review by external clinical experts and potential end users (McDonald 2009; Davies et al. 2011a).

Development

Indicator Selection

We reviewed the peer‐reviewed and gray literature, the National Quality Forum (NQF) endorsed standards (National Quality Forum 2011), the AHRQ National Quality Measures Clearinghouse (Agency for Healthcare Research and Quality 2014c), and relevant federal government reports from 2008 to 2013 to identify candidate indicators or concepts that could be adapted to community‐level indicators. We created a list of candidate indicators and indicator concepts by abstracting from these sources any QIs or study outcomes that demonstrated a link between ED encounters and aspects of access to care or community health.

To select indicators from the list of candidate indicators, we conducted a structured review with a convenience sample of 16 ED, behavioral health, and community health clinicians and researchers who first identified their 10 highest and 10 lowest priority indicators and then rated the indicators on the strength of their relationship to various aspects of community health. The results of the selection process were reviewed by a second group of 15 invited experts with expertise in medicine, behavioral health, community health, measurement and data, tasked with reviewing, evaluating, and providing feedback on various stages of the development project (Standing Workgroup). Six general health indicators were selected based on a high ranking by the clinical group and receiving confirmatory priority by the Standing Workgroup and AHRQ. The resulting candidate indicators reported here do not include another set of behavioral health indicators, as those underwent a separate process.

Indicator Specification

We developed precise definitions of the indicator numerators and denominators and software that can be used to calculate the indicators from hospital administrative data. Indicator definitions are specified with International Classification of Diseases, 9th Revision, Clinical Modification (ICD‐9‐CM) codes, which we identified through review of published algorithms, the AHRQ Clinical Classification System (CCS), American Hospital Association Coding Clinic, and ICD‐9‐CM Coding Manuals. Inclusion and exclusion criteria were suggested by the same clinical group that participated in indicator selection. These specifications were reviewed by an external validation panel described below, who offered further modifications. All indicators were developed using county‐level denominators.

Data

We used the 2008–2010 Healthcare Cost and Utilization Project (HCUP) State Inpatient Databases (SID) and State Emergency Department Databases (SEDD) data for 28 states (Arizona, California, Connecticut, Florida, Georgia, Hawaii, Illinois, Indiana, Iowa, Kansas, Kentucky, Maryland, Massachusetts, Minnesota, Missouri, Nebraska, Nevada, New Jersey, New York, North Carolina, Ohio, Rhode Island, South Carolina, South Dakota, Tennessee, Utah, Vermont, and Wisconsin). HCUP is the largest collection of longitudinal hospital care data in the United States, including all‐payer, encounter‐level administrative data (Agency for Healthcare Research and Quality 2014b). By combining ED records from the SEDD (discharge records for ED visits that do not result in hospitalization) and the SID (discharge records for hospital stays including those originating in the ED), we were able to capture all ED visits (regardless of whether the patient was hospitalized or treated and released). The final dataset contained 80.2 million ED visits from approximately 67 percent of the U.S. resident population. Some analyses required encrypted patient identifiers to link records across ED encounters, for example, to identify repeated ED visits. For these analyses, we used data from only the 11 states (California, Florida, Georgia, Hawaii, Massachusetts, Missouri, Nebraska, New York, South Carolina, Tennessee, and Utah) with reliable record linkage information. We used U.S. Census Bureau data from 2010 and the American Community Survey (ACS) 5‐year estimates for 2007–2011 to calculate U.S. population denominators, including stratification by age, sex, and county‐level poverty rates. We used data from the Area Health Resource File (AHRF) 2010 release to obtain county‐level rates of uninsurance and primary care and dental provider density.

Validation

Validation Panels

To assess face validity and usefulness of the indicators, we convened two additional national expert panels whose member selection and indicator review through a Nominal Group technique is detailed elsewhere (Davies et al. 2011b). In brief, 57 national and state professional and advocacy organizations were invited to nominate panelists. The first validation panel consisted of 14 ED‐ and community‐based physicians, dentists, nurses, and social workers (Clinical Validation Panel), while the second included 14 potential end users of the indicators, including public health administrators, researchers, and developers of community health report cards (End‐User Validation Panel). Panelists first reviewed indicator specifications and evidence summaries and then rated the indicators using a 16‐item (clinical panels) or 19‐item (end‐user panels) questionnaire (available upon request from authors). Panelists reviewed aggregated ratings summaries prior to a conference call during which they discussed their opinions about the indicators and then re‐rated them using the same questionnaire. We calculated appropriateness of the indicators for each of three purposes evaluated (public reporting, population health improvement, and research) using the RAND/UCLA Appropriateness method algorithm (Fitch et al. 2001).

Indicator Rates

We calculated observed and risk‐adjusted rates using data from 1,778 counties contained within the 2008–2010 HCUP SID and SEDD data. We used patients’ county of residence to assign ED encounters to counties and included visits to any hospital within patients’ state of residence. We excluded records with missing age, gender, or discharge diagnosis as well as those resulting in transfer from the ED. Transfers accounted for between 1 and 5 percent of cases (dependent on the indicator).

Two risk adjustment models were developed using data from the ACS: (1) age and sex distribution of residents within a county and (2) age and sex with poverty (percent of the population age 18–64 with income below the federal poverty line). We used indirect standardization to calculate the risk‐adjusted rate as the observed rate over the expected rate, multiplied by the reference population rate (i.e., the overall rate for the 2010 SEDD /SID datasets).

Empirical Analyses

We assessed the relationship of the ED PQI and three measures of poverty—the percent of households under the federal poverty line, median household income, and the percent of ED visits with a primary payer of Medicaid—using an over‐dispersed Poisson regression model with counties as random effects and controlling for county‐level age and sex distributions. We calculated the risk ratio and 95 percent confidence interval associated with residence in areas with greater levels of deprivation (90th percentile for federal poverty level; 10th percentile for median income; 90th percentile for Medicaid level) compared to areas with lower deprivation (10th percentile for federal poverty level; 90th percentile for median income; 10th percentile for Medicaid level).

We performed the same analysis to explore the relationship of the indicators to measures of access to care—uninsurance, density of primary care physicians (PCP) within counties, and for one indicator focused on dental care, density of dental providers. For these analyses, we used both risk adjustment models.

We conducted a sensitivity analysis using two additional models with alternative definitions of economic deprivation and access to care: (1) controlling for age, sex, and the percent of ED visits within the county where Medicaid is designated as the primary payer and (2) controlling for age, sex, and percent of residents that are uninsured.

Results

Table 1 presents brief descriptions of the ED PQIs resulting from the development process. Detailed indicator specifications are available in Appendix SA2. The first indicator focuses on visits for dental conditions that might be avoided in many cases with access to community‐based dental care, the next three on acute and chronic ambulatory care sensitive conditions, and the last on repeated use of the ED associated with back pain, a common and often chronic condition.

Table 1.

Description of ED PQIs

Indicator Title and Descriptiona	Panel Ratings: Overall Usefulness by Intended Use
	Intended Use	Clinical Panel Ratingb	End‐User Ratingb
ED Visits for Non‐Traumatic Dental Conditions ED visits with a first‐listed diagnosis of a nontraumatic dental condition, for individuals age 5 years and older.	Community‐level Public Reporting	Appropriate+	Appropriate+
	Population Health Improvement	Appropriate	Appropriate+
	Community Health‐Related Research	Appropriate	Appropriate+
ED Visits for Chronic Ambulatory Care Sensitive Conditions ED visits with a first‐listed diagnosis of a chronic ACSC (including asthma, COPD, heart failure, acute diabetic hyper‐ and hypoglycemic complications, chronic kidney disease), age 40 years and older. A first‐listed diagnosis of lower respiratory infection with a second‐listed diagnosis of COPD or asthma is also included.	Community‐level Public Reporting	Appropriate	Appropriate
	Population Health Improvement	Appropriate+	Appropriate+
	Community Health‐Related Research	Appropriate	Appropriate+
ED Visits for Acute Ambulatory Care Sensitive Conditions ED visits with a first‐listed diagnosis of selected acute ACSC without hospital admission (“treat‐and‐release” visits only), age 64 years and younger. It Includes allergic rhinitis, otitis media, influenza without pneumonia, viral syndrome, cellulitis and other local skin infections, uncomplicated urinary tract infections among females age 18–34 years or upper respiratory infections.	Community‐level Public Reporting	Appropriate+	Uncertain
	Population Health Improvement	Appropriate	Appropriate
	Community Health‐Related Research	Appropriate	Appropriate+
ED Visits for Asthma ED visits with a first‐listed diagnosis of asthma, age 5–39 years.	Community‐level Public Reporting	Appropriate+	Appropriate+
	Population Health Improvement	Appropriate+	Appropriate+
	Community Health‐Related Research	Appropriate+	Appropriate+
ED Visits for Back Pain Two or more ED visits in one calendar year with a first‐listed diagnosis of back pain, age 18 years and older. Patient identifiers required to link records across encounters	Community‐level Public Reporting	Uncertain	Appropriate
	Population Health Improvement	Appropriate	Uncertainc
	Community Health‐Related Research	Appropriate	Appropriate

^aDenominator for all indicators is per 100,000 population, except where noted. Full specifications are available in Appendix SA2.

^bAppropriate + denotes median rating of 7–9 with agreement (< 5 panelists rating <7); appropriate denotes median rating of 7–9 without agreement or disagreement; uncertain denotes median of 4–6 or any median with disagreement (>4 panelists rating in each of the extreme ranges, 1–3 and 7–9).

^cUncertain denotes disagreement.

ACSC, ambulatory care sensitive condition; COPD, chronic obstructive pulmonary disease; ED, emergency department.

Validation Panels’ Ratings of Appropriateness of Use

Clinical and end‐user panelists rated most indicators as appropriate for public reporting, population health improvement, and community health‐related research uses (Table 1). However, the end‐user panel rated the ED Visits for Acute ACSC indicator as having uncertain appropriateness for public reporting and the ED Visits for Back Pain indicator as having uncertain appropriateness for population health improvement. The ED Visits for Back Pain indicator was also rated as having uncertain appropriateness for public reporting by the clinical panel. No indicator received the inappropriate rating for any use.

ED PQI Rates

The ED PQIs showed wide variation in county‐level rates (Table 2), with standard deviations exceeding 50 percent for all of the indicators except the risk‐adjusted (with poverty) rate for ED Visits for Chronic ACSC. Observed rates were greatest for the acute and chronic indicators, in particular ED Visits for Acute ACSC Conditions (3,591.8 ED visits per 100,000 population).

Table 2.

Average (SD) County‐Level ED PQI Observed and Risk‐Adjusted Rates, 2010

Indicator	Observed Rate	Age and Sex Risk Adjustment	Age, Sex and Poverty Risk Adjustment
ED Visits for Non‐Traumatic Dental Conditions	594.3 (454.5)	644.2 (493.2)	612.0 (431.5)
ED Visits for Chronic Ambulatory Care Sensitive Conditions	2,095.7 (1,058.4)	2,045.4 (1,067.0)	1,897.2 (802.9)
ED Visits for Acute Ambulatory Care Sensitive Conditions	3,591.8 (2,276.2)	3,705.9 (2,348.2)	3,463.7 (1,864.4)
ED Visits for Asthma	536.3 (325.7)	535.0 (325.6)	513.6 (295.5)
ED Visits for Back Paina	169.3 (114.4)	172.8 (116.4)	156.5 (96.1)

Denominator for all indicators is per 100,000 population, except where noted. Demographic risk adjustment—age, sex distribution by county; demographic and poverty risk adjustment—age, sex, and percent of county population below the federal poverty level.

^aRates for this indicator are based on a subset of 11 states with reliable patient identifiers to allow data linkage.

ED, emergency department; SD, standard deviation; SEDD, state emergency department dataset; SID, state inpatient database.

Source: 2010 SEDD and SID for 28 states (see Methods); American Community Survey 5‐year estimates for 2007–2011

Relationship with Economic Deprivation

Areas with greater economic deprivation were associated with statistically significantly higher ED PQI rates for all indicators (Table 3). The effect of economic deprivation was particularly strong when measured by the percent of ED visits within a county with a primary payer of Medicaid, with rate ratios (RR) ranging from 1.86 to 2.79. For each indicator, effect sizes were smallest when examining associations with median income, but they remained moderate and statistically significant.

Table 3.

Effect of Economic Deprivation on County‐Level ED PQI Rates

Indicator	Rate Ratio (95% CI) Counties in Highest Decile of Economic Deprivation Compared to Lowest Decile of Economic Deprivationa
	Percent Below Poverty Line	Median Household Income	Percent Medicaid
ED Visits for Non‐Traumatic Dental Conditions	1.87 (1.70–2.05)	1.61 (1.48–1.74)	2.79 (2.54–3.06)
ED Visits for Chronic Ambulatory Care Sensitive Conditions	1.91 (1.78–2.04)	1.44 (1.35–1.53)	2.00 (1.86–2.15)
ED Visits for Acute Ambulatory Care Sensitive Conditions	2.05 (1.88–2.23)	1.59 (1.47–1.71)	2.65 (2.44–2.88)
ED Visits for Asthma	1.50 (1.39–1.62)	1.19 (1.11–1.27)	1.86 (1.72–2.01)
ED Visits for Back Pain	1.86 (1.67–2.10)	1.67 (1.54–1.81)	2.14 (1.89–2.43)

^aRate ratio and 95% confidence interval of areas with greater economic deprivation (90th percentile of percent county population below federal poverty line and percent of county ED visits with primary payer of Medicaid; 10th percentile of county median income) compared to areas with less economic deprivation (10th percentile of percent below federal poverty line and percent of ED visits with primary payer of Medicaid; 90th percentile of median income).

CI, confidence interval; ED, emergency department; SEDD, state emergency department dataset; SID, state inpatient database; RR, rate ratio.

Source: 2010 SEDD and SID for 28 states (see Methods); American Community Survey 5‐year estimates for 2007–2011.

Relationship with Access to Care

For all indicators, rates were moderately higher in areas with higher levels of uninsurance, the first of two aspects of access to care evaluated (Table 4), with rate ratios ranging from 1.29 (ED Visits for Asthma) to 1.57 (ED Visits for Acute ACSC).

Table 4.

Effect of Measures of Access to Care on ED PQI Rates

Indicator	Rate Ratio (95% CI) Counties in Highest Decile of Lack of Access to Care Compared to Lowest Decile of Lack of Access to Carea
	Percent Uninsured	PCP Density with Age, Sex Controlb	PCP Density, with Age, Sex, and Poverty Controlb
ED Visits for Non‐Traumatic Dental Conditions	1.36 (1.23–1.49)	1.02 (0.93–1.12)	0.92 (0.84–1.00)
ED Visits for Chronic Ambulatory Care Sensitive Conditions	1.42 (1.32–1.53)	1.05 (0.98–1.12)	0.94 (0.88–1.00)
ED Visits for Acute Ambulatory Care Sensitive Conditions	1.57 (1.44–1.72)	1.05 (0.97–1.14)	0.93 (0.86–1.01)
ED Visits for Asthma	1.29 (1.19–1.39)	0.88 (0.82–0.95)	0.82 (0.76–0.88)
ED Visits for Back Pain	1.32 (1.19–1.46)	1.13 (1.00–1.24)	0.98 (0.89–1.08)

^aRisk Ratio and 95% confidence interval of areas with less access to care (90th percentile of percent county population uninsured) compared to areas with greater access to care (10th percentile of percent county population uninsured).

^bPrimary care provider density per 100,000 county population, controlling for (1) county age and sex distribution and (2) county age and sex distribution in addition to county‐level percent of population below the federal poverty.

CI, confidence interval; ED, emergency department; PCP, primary care provider; RR, rate ratio.

Source: 2010 SEDD and SID for 28 states (see Methods); AHRF 2012–2013 release; American Community Survey 5‐year estimates for 2007–2011.

Few of the indicators had a statistically significant relationship with PCP density (Table 4), the second aspect of access to care evaluated. Only the ED Visits for Asthma was significantly associated with PCP density, demonstrating modestly lower rates in areas with lower PCP density. The strength and significance of this relationship changed little when poverty was included as a control variable in addition to age and sex. In the model controlling for poverty, several other indicators approached statistical significance, but effect sizes were modest, ranging from 0.92 (ED Visits for Non‐Traumatic Dental Conditions) to 0.94 (ED Visits for Chronic ACSC). The ED Visits for Back Pain indicator also approached statistical significance, in the opposite direction, when only age and sex were included as controls (RR 1.13, 95 percent confidence interval [CI] 1.00 to 1.24), but this relationship disappeared when adding county‐level poverty rates as a control (RR 0.98J, 95 percent CI 0.89–1.08). A sensitivity analysis controlling for percent uninsured and percent of ED visits with a primary payer of Medicaid instead of the percent of population at or under the federal poverty level showed very similar results (data not shown).

For the ED Visits for Non‐Traumatic Dental Conditions indicator, we also examined the relationship with dental provider density, finding a moderate statistically significant effect of higher ED PQI rates in areas with lower provider dental density (RR 1.30, 95 percent CI 1.19 to 1.43). However, the magnitude and significance of this effect diminished when controlling for poverty (RR 1.10, 95 percent CI 1.00 to 1.20). In the sensitivity analysis, controlling for county‐level rates of uninsurance and percent of ED visits with a primary payer of Medicaid instead of poverty rates yielded statistically significant rate ratios of 1.21 and 1.13, respectively.

Discussion

Based on two independent structured panel reviews (clinical and end‐user panels), these indicators have high face validity as metrics of community health. Most indicators were rated as appropriate for all three purposes evaluated: public reporting, population health improvement, and research. The ED PQI indicators can help inform resource allocation within communities; identify gaps in access to care, disparities, or other priorities for community‐level improvement; and evaluate improvement efforts or policy changes. They can highlight areas in need of further investigation and provide comparisons to the public. The ED visits captured by the indicators are frequent, with average county‐level observed rates ranging between 169 and 3,591 per 100,000 population. One critical feature of a useful population measure is variation by area or other delineating feature. These indicators display wide variation between counties in the sample. This variation remained after risk adjustment for county‐level poverty, suggesting that poverty alone did not explain the variation in rates evident across communities.

Consistent with previous research on ACSC, the indicators were strongly associated with measures of economic deprivation, with higher rates in areas with greater economic hardship. When examining the indicators’ relationship with aspects of access to care, we found that indicator rates were higher in areas with greater levels of uninsurance. However, the indicators’ relationship with the density of health care providers was less clear. Most indicators were not significantly associated with county‐level PCP density, the measure of provider availability used in our evaluation. ED Visits for Asthma had lower rates in areas with lower density of PCPs, countering our expectation that areas with fewer PCPs per population would be associated with increased use of the ED. In contrast, the ED Visits for Non‐Traumatic Dental Conditions indicator demonstrated significantly higher ED visit rate in areas with lower density of dental providers, but the effect was attenuated when controlling for poverty. Prior research has also shown mixed evidence about the relationship between provider density and ED visits (Richman et al. 2007; Mathison et al. 2013; Hunold et al. 2014).

Based on our initial evaluation, the ED PQIs appear to reflect levels of insurance coverage, and likely affordability of care within communities, but not necessarily availability of primary care, at least as measured through provider density. PCP density may not be a good indicator of access to care for areas with a large population covered by Medicaid when few doctors accept Medicaid insurance. Other factors, such as variation in Medicaid eligibility criteria across states, availability of after‐hours care, transportation, or health literacy, may be more important in determining how easily members of a community can access the care they need in a timely and convenient manner. A recent evaluation of ED Visits for Non‐traumatic Dental Conditions demonstrated this nuanced relationship between ED visits and provider density (Fingar et al. 2015).

The ED PQIs complement other publicly reported measures of community and national health, such as those found in the County Health Rankings (County Health Rankings & Roadmaps 2014), America's Health Rankings (UnitedHealth Foundation 2014), Dartmouth Atlas of Health Care (Dartmouth Institute for Health Policy and Clinical Practice), and the National Healthcare Quality and Disparities Report (Agency for Healthcare Research and Quality). Like the ED PQIs, the measures included in these resources are multifactorial, capturing aspects of communities that drive the need for care (e.g., social determinants of health, underlying disease burden, health behaviors), when and where individuals seek care, if at all (e.g., access to care, insurance coverage), and to some extent, the quality of care received in the community.

Familiarity with a community will enhance understanding of the indicator rates. For example, although lower rates are generally more favorable than higher rates, in some cases, higher rates are reflective of better access to care, such as in communities that choose to consolidate urgent care in EDs to achieve system efficiency. End‐user panelists highlighted this point particularly in the discussion about ED Visits for Acute ACSC. Knowledge of local systems of care and resources provides important context for interpreting indicator rates and patterns over time. For instance, awareness of a community health facility closure or changes in Medicaid coverage would be important to understand when interpreting changes in that community's ED PQI rates over time. Similarly, examining the county‐level rate of uninsurance, primary care provider density, or self‐reported access to care from the County Health Rankings (County Health Rankings & Roadmaps 2014) could help shed light on factors driving a high rate of ED visits for Chronic ACSC in a particular county. Coding practices may also vary between communities and be difficult to verify.

The ED PQIs were not developed to decide appropriateness of any particular ED visit, or types of ED visits. They do not provide information about the many different factors that influence individual decisions about where to seek care, including underlying need or perceived need for care, access to alternative sites of care, patient preferences, and patient beliefs about setting of care. The indicators could harm vulnerable populations if they are used to label certain ED visits as inappropriate without interventions to increase access to care outside the ED, improve population health, and support communities.

Limitations

The indicators were developed using county as the definition of community to facilitate use of available county‐level data on demographics and measures of poverty. However, the county may not always be the most useful denominator for understanding community health. For some applications, counties may be too geographically large or heterogeneous to identify subgroups that may have the highest rates and be most in need of interventions. Alternate definitions of community may be based on geographic areas such as neighborhoods or characteristics of residents within areas, such as Medicaid enrollees, race/ethnicity, or preferred language. Although the data used in development of the ED PQIs included a large number of varied states, the results may not be generalizable to all states.

Although the indicators have high face validity, more evaluation is necessary to further validate the indicators. First, we assessed the validity of the algorithms using analyses of administrative datasets to screen for patterns suggesting low data accuracy (data not shown). We considered specifically the adequacy of first‐listed diagnoses, permutations in coding practices, and underreporting of diagnoses. While ED data are subject to fewer coding guidelines than inpatient data, other studies have found that in most cases first‐listed diagnoses appear to capture the principal reason for the encounter, although symptom codes account for about 12 percent of cases (Senathirajah et al. 2011). However, chart review in a diverse group of settings would complement the empirical evaluations to assess more fully the adequacy of these algorithms to capture encounters of interest. Second, the study assessed convergent validity by examining the relationship between community characteristics related to social determinants of health and access to care (e.g., poverty, Medicaid, uninsurance, provider density) and the ED PQIs, but they did not investigate the relationship between the indicators and other aspects of community health for which county‐level data are not readily available, such as disease prevalence or barriers to healthy living. Finally, it is unknown whether ED PQI rates are truly actionable, as would be demonstrated by the successful reduction in rates following policy changes or community interventions to improve health and access to care.

Conclusion

The ED PQIs have face validity and have been tested empirically for identifying patterns in ED usage that reflect community‐level health, access to care, and supportive services. The indicators were validated using a multistage process that included both external review of the indicators by clinical and end‐user experts and empirical analyses to better understand the indicator performance for counties across the nation. As a window into community health, the ED PQIs provide a tool for community health improvement planning, public reporting, and research into community‐level factors associated with health. The indicators and associated software will be publicly available from AHRQ through their Quality Indicators program and can complement other available measures of community health. This program includes a user feedback loop to support continuous improvement of the indicators, in terms of both their technical features and their utility to those seeking to improve health.

Supporting information

Appendix SA1: Author Matrix.

Appendix SA2: General Health ED PQI Technical Specifications