Medical Home Characteristics and Asthma Control: A Prospective, Observational Cohort Study Protocol

Abstract

Background:

This paper describes the methods for an observational comparative effectiveness research study designed to test the association between practice-level medical home characteristics and asthma control in children and adults receiving care in safety-net primary care practices.

Methods:

This is a prospective, longitudinal cohort study, utilizing survey methodologies and secondary analysis of existing structured clinical, administrative, and claims data. The Scalable Architecture for Federated Translational Inquiries Network (SAFTINet) is a safety net-oriented, primary care practice-based research network, with federated databases containing electronic health record (EHR) and Medicaid claims data. Data from approximately 20,000 patients from 50 practices in four healthcare organizations will be included. Practice-level medical home characteristics will be correlated with patient-level asthma outcomes, controlling for potential confounding variables, using a clustered design. Linear and non-linear mixed models will be used for analysis. Study inception was July 1, 2012. A causal graph theory approach was used to guide covariate selection to control for bias and confounding.

Discussion:

Strengths of this design include a priori specification of hypotheses and methods, a large sample of patients with asthma cared for in safety-net practices, the study of real-world variations in the implementation of the medical home concept, and the innovative use of a combination of claims data, patient-reported data, clinical data from EHRs, and practice-level surveys. We address limitations in causal inference using theory, design and analysis.

Introduction

Asthma is one of the most common chronic illnesses in the United States. In 2009, the overall prevalence of asthma was 8.2 percent affecting approximately 24.6 million people (17.5 million adults and 7.1 million children ages 0–17 years).1 When managed appropriately, asthma is a controllable disease, such that many hospitalizations, missed school/work days, and deaths are preventable.2 Nevertheless, in 2009, an estimated 12.7 million people (8.7 million adults and 4.0 million children) experienced asthma exacerbations, 1.75 million had asthma-related emergency department (ED) visits and 456,000 had asthma-related hospitalizations.1 The evidence for medical management strategies to improve asthma control have been synthesized in national asthma guidelines3 and subsequent reviews4–6; the four key components of care central to asthma control are assessment and monitoring; asthma care education; control of environmental factors and comorbid conditions; and pharmacologic therapy.

The Patient-Centered Medical Home (PCMH) model, endorsed by many primary care professional organizations,7 may be an advantageous health care delivery system for asthma management. The medical home model is the focus of multiple reform efforts related to health care delivery, reimbursement, and primary care.8–10 The PCMH model facilitates chronic care by coordinating clinical staff and workflows to optimize patient management, monitoring and education and is associated with improved quality of care including for asthma management.11–14 However, there are remaining questions about the effects of efforts to implement PCMH on asthma outcomes.15–17

The Institute of Medicine designates as high priority comparative effectiveness research (CER) on health care delivery system characteristics such as the PCMH.18 Methods for such investigations are not well established, and randomized trials are infeasible given that it is unlikely that practices will agree to be randomized to the adoption of PCMH infrastructures and processes. However, widespread but varied implementation of the medical home model offers the opportunity to conduct observational research (as a “natural experiment”). In this paper, we present a prospective, observational cohort-study protocol designed to investigate the relationship between a practice’s medical home characteristics and asthma control in adults and children, among patients receiving care in federally qualified health centers and other safety net primary-care practices. The two central innovative aspects of this protocol are the development of an approach for CER on delivery system characteristics, and the creation of a data infrastructure that facilitates the participation of a large number of safety net practices in research. Following published design and reporting guidelines, the current paper therefore represents an a priori specification of the objectives and research design for this study.19,20

Hypotheses

The purpose of the prospective, observational cohort study described in this protocol is to estimate the effects of medical home characteristics on asthma control in adults and children. We hypothesize that greater practice-level medical home characteristics are associated with better asthma control, in terms of both patient-reported asthma control and asthma exacerbations.

Methods

Study Design

This is a prospective, longitudinal cohort study, utilizing survey methodologies and secondary analysis of existing structured clinical-, administrative-, and claims data (henceforth “electronic health data”). Practice-level medical home characteristics will be correlated with patient-level asthma outcomes, controlling for potential confounding variables, using a clustered design. Linear and nonlinear mixed models will be used for analysis. The study’s inception date was July 1, 2012. Institutional Review Board (IRB) approvals, including waivers for informed consent and Health Insurance Portability and Accountability Act (HIPAA) authorization, and data use agreements (DUAs) were obtained for this study.

Setting

The setting for this research is the Scalable Architecture for Federated Translational Inquiries Network (SAFTINet), an Agency for Healthcare Research and Quality (AHRQ)-funded safety net-oriented practice-based research network. Most participating practices are federally qualified health centers, and several are school-based health centers; all have either electronic health record (EHR) or other digital systems to store clinical data. There are approximately 50 participating practices from four health care organizations in Colorado and Tennessee; the practices care for approximately 200,000 patients, of whom an estimated 20,000 have asthma and about 40 percent are eligible for Medicaid. SAFTINet developed an information technology infrastructure designed to securely share electronic health data to support quality improvement and CER. The SAFTINet databases include existing administrative-, clinical-, Medicaid claims- and enrollment data, and patient-reported outcomes (PROs) data collected during routine clinical care; data are standardized to the Observational Medical Outcomes Partnership (OMOP) common data model (Version 4).21 The data are HIPAA-limited datasets, de-identified with the exception of service dates, birth dates for those <90, and ZIP codes. Data sharing partners maintain their own separate databases, which are connected via secure networks and can be queried via a secure web-based portal.

Data Sources

Electronic health data from the SAFTINet databases will be used to select patients for inclusion in the cohorts, to operationalize patient-level outcomes, covariates and descriptors, and to assign patients to practices. Data collected during routine clinical care between July 2010 and December 2013 will be included. Medical home characteristics were assessed using self-report, practice-level surveys completed by practice personnel.

Study Population

The study includes a cohort of children and adolescents with asthma and a cohort of adults with asthma. Asthma is a chronic condition punctuated with acute exacerbations; therefore, we elected to use a prevalent-diagnosis design, rather than an incident-diagnosis design. Using the data sources available, which do not include narrative documentation, an incident-diagnosis design would likely not be accurate, as fastidious clinical documentation is required to distinguish newly diagnosed asthma from prevalent asthma in new patients.

Patients eligible for inclusion in the cohorts will be identified based on demographic- and encounter data (dates and places of service and associated ICD-9 diagnosis codes). An asthma diagnosis will be defined as the presence of at least two unique encounters with diagnosis codes for asthma (ICD-9 = 493) occurring within an 18-month period but separated by at least 28 days. This requirement for two visits is to eliminate the single visit to the office or ED with a final diagnosis of “rule out asthma diagnosis,” which would be coded as 493 since there are no “rule out” coding modifiers.

To be included in the child cohort, patients must be between 2 and 17 years old at study inception. To be included in the adult cohort, patients must be men between the ages of 18 and 55 years or women between the ages of 18 and 60 years at study inception. Our age criteria were selected to increase the likelihood of an accurate asthma diagnosis, as asthma is often confused with chronic obstructive pulmonary disease in older adults22 and with serial respiratory viral infections in younger children.23 To be included in the analysis for a given outcome, patients must have had at least one encounter at a participating practice during the corresponding exposure period (Figure 1). To be included in analyses involving patient-reported asthma control, patients must have at least one recorded Asthma Control Test (ACT) value.

Figure 1.

Timing for Assessment of Exposure, Outcomes and Covariates

Exclusion criteria for both cohorts include diagnosis codes for other chronic lung diseases, including cystic fibrosis (277.0), chronic lung disease of prematurity (770.7, 765.21–765.26, chronic obstructive pulmonary disease (496), emphysema (492), chronic bronchitis (491), pulmonary fibrosis (503, 508.1, 515), or active tuberculosis (010, 011, 012, 018).

Variables and Measurement

The periods for measurement of study variables are depicted in Figure 1. We will assess patient-level asthma outcomes in two ways: patient-reported asthma control (ACT Value) and a measure of asthma exacerbations based on electronic health data. Both patient-reported asthma control and asthma exacerbations will be assessed repeatedly over time within patients during an 18-month outcome period (July 1, 2012–December 31, 2013). For each month during the outcome period, we will determine for each eligible patient whether or not an exacerbation occurred during that month. We will establish an index date for each outcome measurement: the index date for each ACT score will be the date on which the ACT was administered; the index date for each asthma exacerbation will be the last day of the calendar month in which that exacerbation occurred. For the ACT outcome analysis, we will measure exposure to medical home characteristics and covariate measures in the 24 months preceding the date of the ACT. For the asthma exacerbation outcome analysis, we will assess each patient-month for the presence or absence of an exacerbation. The exposure period assigned to each patient-month will be the 24-month period ending at the end of the patient-month during which exacerbations are assessed.

Medical home characteristics were measured via self-report survey at study inception. For each individual outcome measurement, the medical home exposure period is the 24 months preceding, and inclusive of, the index date. There is no single index date or exposure period. Patient exposure to a practice’s medical home characteristics will be assigned based on a patient’s health care utilization at a practice during a 24-month exposure period. We have chosen 24 months as the exposure period, as patients may seek care less often than once a year. As patients may visit more than one practice, for analytic purposes we will assign patients to the practice accessed most frequently during each exposure period, or, when treated an equal number of times at two or more practices, to the most recent practice. So that we may capture patients who access care only during an exacerbation, we include patients for whom exacerbation encounters are their only encounters within the exposure period. The anticipated time course to improved control after exposure to a medical home is likely to be short—a few weeks to a few months; therefore, we have not specified a time gap between the exposure and outcome measurement.

Data from the exposure period will also be used to assess several potential confounding variables. Prior asthma control and exacerbation rates are known predictors of the study’s outcome measures.24 Definitions, measures and data sources are summarized in Table 1.

Table 1:

Variables, Data Sources and Measurement

Variable(s)	Measure(s)	Data source
Medical Home Characteristics	Total score and domain scores	Practice Survey
Asthma Control	Asthma Control Test (ACT) score	Scalable Architecture for Federated Translational Inquiries Network (SAFTINet) database (patient-reported outcome—PRO)
	Exacerbation measure during the out-come period	SAFTINet database (clinical and claims)
Patient Demographics	Age, race/ethnicity, socioeconomic status (income level), urban/rural	SAFTINet database (administrative)
Prior exacerbations	Number of exacerbations during the exposure period	SAFTINet database (clinical and claims)
Asthma severity	Medication regimen indicative of intermittent or persistent (mild, moderate or severe) asthma	SAFTINet database (clinical)
Comorbidities	Presence of comorbid conditions known to exacerbate asthma symptoms (diagnosis codes)	SAFTINet database (clinical)
Prior ACT scores	ACT scores during exposure period	SAFTINet database (patient reported outcome—PRO)

Outcomes

Patient-Reported Asthma Control

The most recent asthma guidelines recommend the assessment of patient-reported asthma control as an indicator of both current symptom burden and risk of future exacerbations.4–6 Patient-reported asthma symptom control will be measured using the ACT25 and the Childhood ACT (C-ACT)26. The ACT is a copyrighted instrument of QualityMetric Incorporated; the C-ACT is a copyrighted instrument of GlaxoSmithKline. GlaxoSmithKline can grant access to both instruments. We received permission from GlaxoSmithKline to use the ACT and C-ACT for this study. The ACT and C-ACT (hereafter referred to together as the “ACT”) are widely used, valid and reliable tools for measuring patient-reported asthma symptoms control.27 The ACT has five items measured on a five-point scale, which are then summed to represent total control. ACT total scores range from 5 to 25; higher scores indicate better control. ACT scores can also be categorized as follows: In control (total score > 19), poorly controlled (total score 16–19), very poorly controlled (total score < 15). We will exclude from analyses ACT assessments that occur within 21 days of an asthma exacerbation.

Given the choice of several existing, validated patient-reported asthma measures, the SAFTINet clinical partners selected the ACT to assess asthma control because the ACT is available in relevant languages, is relatively brief and easy to administer, and is indicated in the asthma guidelines as an appropriate tool for identifying patients in need of intensified treatment. The clinical partners were allowed broad latitude for workflow implementation and data collection; the minimal criteria were that the ACT should be administered to all patients with asthma at least once a year and the ACT score should be documented in a structured data field so as to be included in the SAFTINet databases. The varied implementation plans included both point of care and telephone-based modes of ACT administration.

Asthma Exacerbations

An additional outcome is the presence of an asthma exacerbation for a given patient during each month of the outcome period. Evidence of asthma exacerbations will be inferred from utilization patterns and use of symptom control medications using encounter and prescription data from the SAFTINet database. For all patients, we will define asthma exacerbations based on the presence of any of the following criteria, using clinical data originating in the EHR or surrogate EHR system:

1. A prescription for oral (systemic) steroids prescribed within two days of the date of a practice visit that includes a diagnostic code for asthma, or

2. At least three outpatient visits occurring on separate days within a 14-day period that include a diagnostic code for asthma, or

3. Administration (not merely a prescription) of an inhaled beta-agonist medication (nebulizer or metered dose inhaler) at (on the same date as) an outpatient visit.

For the subset of patients (about 40 percent of the sample) covered by Medicaid, we will include an additional criterion for an asthma exacerbation, based on claims:

• 4. An asthma-related emergency department visit or hospitalization with asthma listed as the primary or secondary diagnosis.

Only one of the above criteria must be met to establish an exacerbation. We will count asthma exacerbations as separate events when at least 21 days elapse between the end of one event (or the date of an event occurring on one particular day) and the beginning of the next exacerbation-defining event. For each month in the 18-month outcome period, we will indicate whether or not an exacerbation occurred during that month. For exacerbations spanning two or more months (multiple criteria met over time, with less than 21 days between each criterion date), the exacerbation will only be counted in the first month.

Exposure Variables

Medical Home Characteristics

We assessed practice medical home characteristics via self-report surveys. To select a survey instrument suitable for measuring medical home characteristics in this study, we conducted a literature review, examined existing surveys, and sought recommendations from SAFTINet clinical partners and experts in the field. None of the existing options fully met the needs of the SAFTINet project, lacking both critical face validity and applicability to practices that were not in the process of seeking formal medical home recognition; therefore, a medical home characteristics survey was adapted for this study based on concepts and items from several sources.28–31 The resulting self-administered instrument, the SAFTINet Delivery of Coordinated Care Survey (DoCCS), was informed by our review of the literature, consultation with experts, and published surveys designed to assess progress toward becoming a medical home.28,31

The DoCCS includes nine sections organized by medical home domain, as shown in Table 2. The DoCCS assesses respondents’ perceptions of the extent to which their practice has adopted characteristics of these nine medical home domains. Each domain includes between 5 to 17 questions, with responses measured on a 1–5 scale, from “No/Almost Never” to “Almost Always.” For each practice, we will compute a mean score for each domain (continuous measure with a range of 1–5) and an overall score, a total of the mean scores for all nine domains (continuous measure with a range of 9–45).30 We plan to analyze the DoCCS score as a continuous measure, as well as to compare practices by DoCCS score quartile.

Table 2.

Medical Home Domains and Example Goals

Medical Home Domain	Example Goal
1. Personal Clinician and Sustained Partnership	Clearly link patients to a clinician and/or care team so both the patient and provider/care team recognize each other as partners in care.
2. Personal Clinician-Led, Team-Based Care	Team-based care led by clinician.
3. Coordinated and Integrated Care	Link patients with community resources to facilitate referrals and respond to social service needs. Provide coordinated care with specialists and other providers.
4. Patient/Family-Centered Care/Support Shared Decision-Making	Assess and respect patient and family values and expressed needs.
5. Quality Improvement and Safety	Establish and monitor metrics to evaluate improvement efforts and outcomes and to provide feedback.
6. Organized Care and Evidence-based Medicine	Use point of care reminders and other evidence-based protocols to provide optimal care.
7. Access	Provide scheduling options that are patient- and family-centered and accessible to all patients.
8. Engaged Leadership	Provide visible and sustained leadership, overall culture change, and specific strategies to improve quality and sustain and spread change.
9. Registries	Use patient tracking registries to monitor and inform clinical interventions for persons with specific health care needs.

Each participating practice was asked to identify three practice members (a lead clinician, such as a medical director; a practice manager; and a lead member of the nursing staff) to complete the survey on behalf of the practice. The initial DoCCS measurement was made in July–September 2012. Respondents had the option of completing the DoCCS on paper or through an online survey via RedCap, a secure, web-based application designed exclusively to support data capture for research studies.32 We plan to use only one set of responses per practice for the primary analyses. We will primarily use lead clinician responses, as this role is likely to have the most informed perception of how a practice operates. If no lead clinician was available to complete the survey, we will use practice manager, and then lead nurse, responses. A second DoCCS measurement was made, February–April, 2013, and will be used to assess the assumption of practice-level stability in DoCCS values during the study period.

Patient-Level Covariates

The patient-level covariates to be included in this analysis are patient demographics (age, race/ethnicity, income, urban versus rural residence) and other factors related to risk for poor asthma control (asthma severity, previous exacerbations, and certain comorbidities). For the child cohort comorbidities of interest, we selected allergic rhinitis (ICD-9 = 477), sinusitis (ICD-9 = 461, 473), and gastroesophageal reflux disease (ICD-9 = 530.81). For the adult cohort, we selected allergic rhinitis and sinusitis. For prior exacerbations, we will apply the same criteria used to define an asthma exacerbation outcome. Using prescription data in the SAFTINet database accrued during the exposure period, we will classify asthma severity as follows33:

1. Intermittent/exercise-induced asthma = only on short-acting beta-agonist (rescue medicines)

2. Mild persistent asthma = low-dose inhaled corticosteroid (ICS) such as fluticasone, or leukotriene receptor antagonist such as monteleukast

3. Moderate persistent asthma = moderate-dose ICS, or low-dose combination therapy such as fluticasone/salmeterol

4. Severe persistent asthma = high-dose ICS or medium- to high-dose combination therapy or omalizumab.

Practice-Level Covariates

Practice-level characteristics are collected through an online survey adapted from the Baseline Practice Survey34 and through community-level data from the Area Resource File35 and Index of Medical Underservice36 linked to the practice by the practice’s address. The practice-level characteristics survey was administered twice, concurrently with the two administrations of the DoCCS. The survey incorporates measures of the following domains: (1) patient demographics, including race/ethnicity, language, and payer; (2) organizational slack (size), including the number and type of providers and patients; and (3) pressure from payers, including membership in an accountable care organization (ACO), the presence of pay-for-performance pressures, and financial incentives to adopt medical home characteristics. The geographically linked variables assessed will include the proportion in the community with managed care insurance and with public insurance, the urban/rural designation, and the degree of medical underservice.35,36 We will assess practice level variables for their role as potential confounders, as described below. We will also consider changes in practice measures, such as changes in provider full-time equivalents and other resources, and the role of time-varying practice-level measures on the model.

Study Design Issues: Bias and Confounding

As with any nonexperimental study, several types of bias are possible, including selection bias, confounding, and misclassification bias. In this section, we discuss sources of bias that may influence internal validity and the ability to make causal inferences. We seek to address potential biases through analysis using multivariate models, sensitivity analysis, or through reasoning.

Selection bias

Selection bias pertains to nonrandom selection into the study population,37 and could be introduced by our selection criterion requiring 18 months between asthma diagnostic codes, as this likely selects a more severe asthma sample than if we also included patients with less frequent asthma-related encounters. We can use the data to test for this effect—to determine whether patients with more severe disease are likely to be included in the study, and include this assessment of selection bias in reporting our findings. Also, practices lower in medical home characteristics may lack mechanisms for routine data collection, and may disproportionately administer the ACT to patients who present for acute asthma problems. In this case, results would be biased in favor of practices with greater medical home characteristics. We have partially addressed this problem by excluding ACT assessments that occur within 21 days of an asthma exacerbation.

Confounding

Confounding is a problem of common causes, such that variables that influence asthma control also influence how patients get “selected” for exposure to a practice’s medical home characteristics. The factors that influence allocation of subjects into practices that have more robust medical-home characteristics may also be the factors related to patients’ greater medical complexity, and thus to their risk for poor outcomes (indication bias). Selection of covariates to control for confounding is described below. We also plan to use the data to determine baseline factors predictive of allocation to different exposure levels and address this statistically through inverse probability of exposure weights.

Misclassification bias

Misclassification of exposure to medical home characteristics may be due to measurement error. For example, a patient in poor asthma control at the time of an encounter at their exposure practice may have unsuccessfully attempted to access asthma care at a second practice; because the unsuccessful attempts were not recorded in the dataset, the first practice is misclassified as the patient’s exposure practice. Similarly, patients who switch practices during the study period may be misclassified by our protocol to assign them to the exposure characteristics of the practice at which they had more visits. Switching may have occurred for reasons related to either the practice’s medical home characteristics or the patient’s underlying disease severity, and thus may bias the time-varying treatment analysis. We plan to address this through a sensitivity analysis involving varying the protocol for exposure-assignment for the subgroup of patients who switch practices.

Misclassification of asthma outcomes may be due to limitations of the data sources. For patients lacking claims data, it is possible we will underidentify exacerbation events based on ED encounters.

Selection of Variables to Control Confounding

We approached the selection of variables to control confounding from a theoretical rather than an empirical (e.g., methods used in machine learning or in high-dimensional propensity score analysis) perspective. Consistent with causal graph theory,38 we developed a hypothesized causal structure for the relationship between medical home characteristics and asthma outcomes (Figure 2). First, we identified intermediate variables (see Figure 2)—those that may lie in the causal pathway between medical home characteristics and asthma outcomes—as adjusting for these variables would inappropriately attenuate an observed effect of exposure on outcomes. Second, we identified patient- and practice-level factors (see Figure 2) that may influence asthma control and level of medical home model concordance of the patient’s practice.

Figure 2.

Hypothesized Causal Structure for the Relationship between Medical Home Characteristics and Asthma Control

Patient-level predictors of exposure to a practice with medical home characteristics include demographics,39–47 health status including prior utilization39,45,48, neighborhood characteristics39,40 and comorbidity.45,47,49 Although there is limited evidence on how underserved patients select primary care providers, patients may self-select into a particular practice based on factors such as proximity, affordability, insurance coverage, provider reputation, and appointment availability during convenient hours.50–52 Where people choose to live, and thus which practices are in close proximity, may be related to certain demographic or health status factors.53

Patient-level characteristics are also known predictors of asthma control. Asthma control is related to patient demographics such as age, race, ethnicity, language, income and insurance status16,54–57, as well as intrinsic factors such as comorbidity, asthma severity and a history of poor control.58 After considering several diagnosis-based risk adjustment scores to adjust for overall comorbidity, based on expert opinion, we chose to adjust for specific comorbid conditions.45,49,59–61

Practice-level characteristics may also be predictive of medical home characteristics; for example, larger practices may have more resources for pursuing changes consistent with achieving greater concordance with the medical home model.62–67 Similarly, practice-level characteristics of the population the practice serves—such as community-level medical underservice68—may be associated with the practice’s medical home characteristics. We did not find studies indicating that practice-level characteristics (other than medical home characteristics) are predictive of asthma control (see dotted line in Figure 2), but will test potential practice-level factors for confounding prior to inclusion in multivariate models.

We narrowed the list to factors that are likely to be common causes of both medical home exposure and asthma control. We eliminated confounders that are medical home characteristics themselves (access and appointment availability), factors likely to have little variability in a safety net population (e.g., insurance coverage), and any unmeasured variables. As shown in Figure 1, we will consider patient demographics and factors related to risk for poor asthma control as potential confounders. We will empirically test that these factors are related to both medical home characteristics and asthma outcomes prior to inclusion in multivariate models.

Sample Size and Power Estimation

We used standard software for power- and sample-size estimation. We followed Murray for computing the variance inflation factor.69 In this clustered design, there are up to 55 practices that will be providing practice-level data, with at least 100 patients per practice. As intraclass correlations (ICC) for this study are unknown, we estimated detectable effect sizes based on a lower (ICC = .05) and a higher (ICC = .10) bound for ICC. For ICC = .05, the effective sample size would be 455. This would provide sufficient power to detect an effect size of Cohen’s d = .19. For ICC = .10, the effective sample size would be 248, allowing us to detect an effect size of d = .26. Both estimated effect sizes are in the “small” range, suggesting that this study is adequately powered to detect even small effects.

Statistical Methods

We will test our hypotheses separately for adults and children. Our analytic models must accommodate clustering both at the patient level (longitudinal), as asthma control will be measured repeatedly over time within patient, and at the practice level (patients clustered within practices); therefore, we will use mixed effects modeling.70 For the exacerbation outcome analysis, we will specify a nonlinear mixed effects model (logistic or ordinal regression). For the ACT outcome analysis, we will use a linear mixed effects model. The large number of patients (∼20,000) will likely ensure that parameter estimates and residuals will be normally distributed, and thus allow us to use linear models. This assumption will be confirmed prior to analysis.

The predictor variables are the total and domain medical home scores from the DoCCS. They will be included in the model as continuous variables. We will check the linearity of the relationship using a simple linear spline with a single knot at the mean of the medical home score. Prior to testing the primary hypotheses, we will test whether any of the suspected confounders are associated with (1) receiving care in practices that are higher versus lower in medical home characteristics or (2) asthma control. Only those factors associated with both medical home exposure and asthma control are likely to be confounders.

The models will be fit in four steps: (1) DoCCS score as the sole explanatory variable; (2) addition of variables identified as potential confounders (e.g., are correlated with both the DoCCS score and the outcome, and not in a proposed mediating pathway), (3) addition of select patient demographics as covariates; and (4) addition of select patient-level risk factors for poor asthma control as covariates. At each step, the parameter estimating the association of the outcome with DoCCS scores will be examined; if there is a large change in its value (e.g., parameter is sensitive to the addition of the covariates), we will attempt to identify the covariate driving the change. If more than 10 percent of the covariates values are missing, they will not be included in the models unless they have been identified as a confounder, in which case the missing values will be multiply imputed using a fully conditional specification (FCS). We will use Predictive Mean Matching for continuous variations and Logistic models for dichotomous variables. Models will include all variables measuring asthma control and medical home characteristics as variables that predict missingness of the confounder and have less than 10% missing values.71 If less than 10 percent of the values are missing, a simple imputation scheme will be developed, either using the most common value or the value that represents >50 percent of the units or a random selection (e.g., one imputation from a multiple imputation procedure). We take this strategy because small amounts of missing data are unlikely to affect results but will make sensitivity analysis difficult. Further details on the models are provided in the appendix.

Sensitivity Analyses

Where claims data are available, we will compare results when asthma exacerbations are defined both with and without the utilization data available from claims data: i.e., hospitalizations and emergency department visits. This would allow us to test whether the association between medical home characteristics and the number of asthma exacerbations varies based on the type of data available for exacerbation detection. For example, patients at a clinic with poor medical home characteristics may seek care for asthma exacerbations at emergency departments rather than at the clinic, and the availability of emergency department utilization data could reverse the direction of the association between medical home characteristics and the number of asthma exacerbations. We will also conduct sensitivity analyses using different exposure periods, such as varying the length of the exposure period (e.g., 24 months versus 18 months prior to the index date) as well as varying the inclusion or exclusion of encounters occurring on or within 30 days of the index date.

Expected Results

We anticipate finding variation among practices in the DoCCS total score, with greater variation for some domains over others. We hypothesize that greater practice-level medical-home total and domain scores are associated with better asthma control, in terms of both patient-reported asthma control and rates of asthma exacerbations.

Discussion

This study is one of the first to use existing electronic health data, augmented by patient-reported outcome (PRO) measures, to conduct comparative effectiveness research on health care delivery system characteristics. The most innovative aspect of this research protocol is the development of an approach to performing comparative effectiveness research (CER) on a practice-level, multifaceted, nonbinary health care delivery system variable: concordance with the medical home model. Developing this approach required new approaches to operationalizing the measure of medical home characteristics and to defining what it meant to be exposed to these characteristics in the context of the typical temporal treatment-response relationship for patients with asthma. A second innovative aspect is the setting for the research: real-world safety net practices and populations. Although prior asthma research has been conducted in underserved populations, the focus on a practice-level exposure variable required that we include a large number of practices. The innovative Scalable Architecture for Federated Translational Inquiries Network (SAFTINet) information technology infrastructure is key to involving safety net practices on this scale; without such infrastructure, few practices would have the resources to contribute data that could be utilized for CER.

Methodological and practical challenges addressed during the development and implementation of this protocol include the following: (1) identifying and implementing a measure of medical home characteristics pertinent to the study setting and research design that operationalizes exposure to multifaceted, health care delivery-system characteristics; (2) implementing PRO data collection in diverse health care organizations and clinical settings; (3) data collection and data quality assessment in a large network of geographically and technologically diverse health care organizations; (4) contingency planning for the heterogeneity of available and complete study data from multiple real-world primary and secondary sources; and (5) addressing multiple sources of confounding and bias.

Existing measures of medical home characteristics did not meet the needs of the current study, as they did not adequately define or operationalize the Patient-Centered Medical Home (PCMH) model for use as a CER tool, and most were designed as preparatory assessment to plan for PCMH recognition or assumed that the respondent was already engaged in a formal PCMH implementation initiative. Drawing on local research expertise in the PCMH model, SAFTINet researchers and clinical partners collaborated to adapt several existing instruments to create an instrument—the Delivery of Coordinated Care Survey (DoCCS)—better suited for both the clinical setting and the challenges of comparative effectiveness research modeling. Engaging both research expertise and clinical and organizational stakeholders was key to successful primary data collection for this measure.

SAFTINet researchers and clinical partners collaborated to select and implement a PRO measure for asthma control: the Asthma Control Test (ACT). To mitigate the variation in ACT data collection methods across practices and associated bias, the research team and clinical partners agreed on a set of minimum requirements for ACT data collection, beyond which collection methods were permitted to vary across practices. This engagement of clinical and research stakeholders—beginning early in the process and with regular opportunities for input—was key to partner buy-in for collecting valid PRO measure data. However, despite this close collaboration, there are inconsistencies in the timing and mode of ACT administration, as the participating organizations developed their own implementation plans that they felt would work within their own environments.

There are also limitations related to the secondary use of electronic health data. The SAFTINet research network encompasses many practices that are geographically and technologically diverse, resulting in varied approaches to data collection and use of electronic health records (EHRs); also, claims data are only available for patients during periods of active enrollment in Medicaid. We expect these facts will lead thereby to varied data availability, completeness, homogeneity and quality. To mitigate the issue of heterogeneity of data availability, this study protocol contains permutations for definition of inclusion criteria and outcomes based on type of data available, and planned sensitivity analyses to evaluate the impact of this heterogeneity. Although contingency planning for varied data availability added complexity to the analytic plan, having specified these approaches a priori enhances the methodological rigor of this study. Although beyond the scope of this paper, the SAFTINet research and technical technical teams have specified protocols for data quality assessment to both assess and mitigate variations in quality inherent in real-world data collection across diverse settings.

Finally, there are methodological challenges to be addressed in the conduct of rigorous observational CER. Regarding the multiple potential sources of bias and confounding, we addressed these challenges a priori during the development of the research protocol, by specifying a theoretical causal structure to facilitate identification of sources of bias, drawing on prior literature and experts in both asthma care delivery and the medical home model. As specified, we will adjust for those common cause variables available in the data-set. However, patients attending a safety net primary-care practice are unlikely to have multiple nearby options for obtaining primary health care services. Therefore, selecting a practice based on its medical home characteristics or status is unlikely, and therefore does not represent a major source of selection bias.

Conclusions and Next Steps

Practices vary in how they deliver care to patients, yet little is known about how these variations in health care delivery impact chronic disease control. A key strength of the SAFTINet infrastructure is that it is designed to measure these variations in delivery system characteristics. In the present study, we measure cross-sectional variation among SAFTINet clinics, but SAFTINet can also measure longitudinal changes related to internal change, such as quality improvement initiatives and research trials, and external change, such as that brought on by the changes in health care funding and policy.

The design of rigorous observational CER on health care delivery system characteristics in a real-world setting requires extensive stakeholder engagement starting early in the planning phase to allow sufficient time for consensus building and buy in. It also requires a team with diverse research and technological experience, able to access and quality-test data from diverse settings and adjust for real-world variation via advanced analytic techniques.

This protocol represents one of several designed by the SAFTINet CER team to explore the effects of health care delivery system characteristics, such as those concordant with the medical home model, on disease control for several primary care cohorts, including children and adults with asthma. The team’s initial work is observational in nature; with increased network participation and successful demonstration of preliminary findings, we expect to conduct more rigorous pragmatic trials and randomized controlled trials.

Appendix. Linear and Non-linear Mixed Effects Models

For the patient-level variables, given the i^th patient in the j^th practice at the t^th occasion

$$g\left\{E\left(Y_{ijt}\right)\right\}={\beta }_{0j}+{\beta }_{1j}\hspace{0.17em}\hspace{0.17em}{X}_{1ij}+\cdots +{\beta }_{nj}\hspace{0.17em}\hspace{0.17em}{X}_{nij}$$

where X_nij are patient-level covariates and Y_ijt are patient-level outcomes (ACT scores or asthma exacerbations).

For the practice-level models, in the j^th practice

$${\beta }_{0j}={\gamma }_{00}+{\gamma }_{01}\hspace{0.17em}\hspace{0.17em}{Z}_{1j}+\cdots +{\gamma }_{0n}\hspace{0.17em}\hspace{0.17em}{Z}_{mj}+{\delta }_{0j}$$

$${\beta }_{1j}={\gamma }_{10}+{\gamma }_{11}\hspace{0.17em}\hspace{0.17em}{Z}_{1j}+\cdots +{\gamma }_{1n}\hspace{0.17em}\hspace{0.17em}{Z}_{mj}+{\delta }_{1j}$$

where Z_mj are DoCCS total and domain scores.

The linear mixed model for ACT may be obtained through g{E(Y_ijt)} = E(Y_ijt), and the logistic mixed model for the presence of exacerbations may be obtained through $g\left\{E\left(Y_{ijt}\right)\right\}=\text{log}\left(\frac{P\left(Y_{ijt}=1\right)}{1-P\left(Y_{ijt}=1\right)}\right)$. Also note that for the ACT outcome, t represents the occasion on which the ACT was measured, which does not necessarily have to match for all individuals; however, for the exacerbation outcome, t corresponds to the month recorded for presence of an exacerbation. For the exacerbation outcome measure, we will model whether or not there was an exacerbation at the patient level in a given month.