Design, Implementation, and Evolution of the Medicaid Outcomes Distributed Research Network (MODRN)

Abstract

Background:

In the US, Medicaid covers over 80 million Americans. Comparing access, quality, and costs across Medicaid programs can provide policymakers with much-needed information. As each Medicaid agency collects its member data, multiple barriers prevent sharing Medicaid data between states. To address this gap, the Medicaid Outcomes Distributed Research Network (MODRN) developed a research network of states to conduct rapid multi-state analyses without sharing individual-level data across states.

Objective:

To describe goals, design, implementation, and evolution of MODRN to inform other research networks.

Methods:

MODRN implemented a distributed research network using a common data model, with each state analyzing its own data; developed standardized measure specifications and statistical software code to conduct analyses; and disseminated findings to state and federal Medicaid policymakers. Based on feedback on Medicaid agency priorities, MODRN first sought to inform Medicaid policy to improve opioid use disorder treatment, particularly medication treatment.

Results:

Since its 2017 inception, MODRN created 21 opioid use disorder quality measures in 13 states. MODRN modified its common data model over time to include additional elements. Initial barriers included harmonizing utilization data from Medicaid billing codes across states and adapting statistical methods to combine state-level results. The network demonstrated its utility and addressed barriers to conducting multi-state analyses of Medicaid administrative data.

Conclusions:

MODRN created a new, scalable, successful model for conducting policy research while complying with federal and state regulations to protect beneficiary health information. Platforms like MODRN may prove useful for emerging health challenges to facilitate evidence-based policymaking in Medicaid programs.

Introduction

In 2021, US state Medicaid programs provided health insurance coverage for approximately 80 million low-income Americans, nearly one-quarter of the US population.(1, 2) Each of the 56 Medicaid programs across US states, District of Columbia, and territories operate independently with unique administrative, operational, and evaluation activities. Substantial variation exists in most every aspect of Medicaid policy resulting from flexibility afforded states under federal law to administer Medicaid programs. Comparing access, quality, and costs across programs can provide much-needed information to policymakers on which Medicaid policies yield the best outcomes for beneficiaries and taxpayers.

Critical barriers exist to sharing individual-level data across state Medicaid programs,(3, 4) limiting opportunities to learn from state policy variation. Barriers include concerns about meeting federal Medicaid confidentiality standards for data sharing, lack of standardized data elements, and difficulties negotiating data use agreements between states.(3) National Medicaid data sets suffer a one-to-two-year lag, constraining their ability to be used to inform pressing policy decisions.

Responding to an urgent public health need to generate research on Medicaid, AcademyHealth collaborated with two existing state policy networks to develop and implement the Medicaid Outcomes Distributed Research Network (MODRN). MODRN aims to enable efficient, rigorous, person-level analyses of multiple states’ Medicaid data without sharing that person-level data between states, obviating the need for multiple data use agreements, and transferring protected health information. MODRN coordinates the efforts of public university research partners that provide analytic support to state Medicaid agencies and aims to facilitate learning among Medicaid agencies.

Distributed research networks use common data models to support centralized development and local execution of analytic programs.(5–7) Under MODRN, each state-university partnership adopted a Medicaid common data model, contributed to a common analytic plan, and conducted analyses locally using standardized code that a coordinating center (led by one of the participating university partners) developed. State-university partners provided each state’s results to the coordinating center, which further aggregated and analyzed state-level estimates.

Based on feedback from Medicaid agencies and their academic partners, MODRN focused first on the US opioid use disorder (OUD) epidemic, a chronic disease and the main driver of the leading cause of death among non-elderly adults,(8) with widening racial disparities(9) and increased incidence since the onset of the COVID-19 pandemic.(10, 11) With pilot funding from state Medicaid agencies and their university partners, and grant funding from National Institute of Drug Abuse, MODRN aimed to inform Medicaid policy to improve OUD treatment, particularly medication treatment of OUD (MOUD) and share findings with stakeholders. MODRN includes 13 participating states that capture approximately 22% of Medicaid enrollment nationally and had substantial variation in covered populations, delivery systems, and policy environments.

This article summarizes MODRN’s goals and development, describes facilitators and barriers encountered during implementation, and compares MODRN with other multi-state Medicaid data resources. Next, this article illustrates how MODRN sought to address the OUD epidemic and concludes by looking forward to how MODRN can expand its infrastructure to address other research and operational gaps among the US’s largest public insurance program. The goal is to inform development of other research networks, particularly those with an emphasis on policy.

Methods

This section describes MODRN’s design, governance and organization, the structure and content of its common data model, as well as establishing research priorities, early measure development, and its approaches to statistical analysis and dissemination of findings.

Partners

A collaboration between two state policy networks provided a backbone for MODRN. AcademyHealth, a national organization for health services researchers, policymakers, and health care practitioners and stakeholders, supports both networks. First, the State-University Partnership Learning Network (SUPLN) supports partnerships between state Medicaid agencies and university research partners in 27 states to promote use of evidence in policy and decision making and focuses on transforming Medicaid care delivery. SUPLN limits membership to state-university partnerships that commit representatives from a state agency and public university partner. Partnerships typically use broad master agreements rather than one-time, grant-funded research projects. Second, the Medicaid Medical Directors Network (MMDN) provides a knowledge exchange among 43 Medicaid programs to advise states’ Medicaid Directors on clinical policy and practice, including evidence-based care and services, assessment of healthcare quality, and delivery system redesign.

Drawing on members of both networks, MODRN initially included 9 states; and as of May 2021, 13 states participate in MODRN. States taking part in MODRN vary along program dimensions, including scope of individuals covered by Medicaid (e.g., whether and when eligibility expanded under the Affordable Care Act, income eligibility thresholds) (Table 1).

Table 1.

Characteristics of the Thirteen States Participating in the Medicaid Outcomes Distributed Research Network

	DE	KY	MA	MD	ME	MI	NC	OH	PA	TN	VA	WV	WI
Number of Medicaid Enrollees (Millions)1 (10/ 2020)	0.234	1.515	1.793	1.353	0.318	2.663	2.336	2.973	3.097	1.660	1.576	0.561	1.327
Percent of Medicaid Enrollees in Risk-Based MCOs2 (7/2019)	97.0%	91.0%	42.0%	85.7%	0%	76.5%	0.1%4	93.7%	89.3%	100%	98.0%	77.0%	78.3%
ACA Expansion	1/2014	1/2014	1/2014	1/2014	1/2019	4/2014	NA	1/2014	1/2015	NA	1/2019	1/2014	NA
Medicaid fee index3 (All Services, 2016)	1.40	0.98	1.12	1.35	0.85	0.90	1.05	0.85	0.93	NA	1.10	1.08	.80
Behavioral Health Carve-Out4 (7/2019)	Varies	No	No	Yes	NA	Yes	Yes5	No	Yes	No	Varies	No	No
Medication for OUD prior authorization (per state-specific Preferred Drug Lists)6	No, for B & N. Yes for M.	No, for B & N. Yes for M.	No	No, for B & N. Yes for M.	No, for N. Yes, for B & M	No, for B & N. Yes for M.	Yes	Yes, for B waived for some providers	Yes, for B. No for M & N.	Yes, for B & M. No for N.	Yes, for B waived in 2017 for some prov.	Yes, for M. No, for B & N.	No
IMD exclusion waiver7 (10/2019)	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
1115(a) SUD waiver status, dates (10/2019)8	Yes, since 7/2019	Yes, since 1/2018	Yes, since 10/2014	Yes, since 7/2016	Yes, since 12/2020	Yes, since 4/2019	Yes, since 10/2018	Yes, since 9/2019	Yes, since 6/2018	Yes, since 2/2016	Yes, since 4/1/2017	Yes, since 10/2017	Yes, since 10/2018

Notes: ACA is Affordable Care Act; IMD is Institutions for Mental Diseases; OUD is opioid use disorder; MCO is managed care organization; SUD is substance use disorder

^1.CMS https://www.medicaid.gov/medicaid/program-information/medicaid-and-chip-enrollment-data/enrollment-mbes/index.html

^2.CMS https://www.medicaid.gov/medicaid/managed-care/enrollment-report/index.html

^3.Kaiser Family Foundation. https://www.kff.org/statedata/ Medicaid fee index measures each state’s physician fees relative to national average.

^4.Some states ‘carve-out’ responsibility for managing behavioral health services from Medicaid managed care organization contracts and contract with separate behavioral health managed care organizations. Kaiser Family Foundation tracks state carve out decisions. https://files.kff.org/attachment/Tables-Report-A-View-from-the-States-Key-Medicaid-Policy-Changes

^5.North Carolina implemented managed care for physical health services in July 2021 and for behavioral health services in July 2022.

^6.B is buprenorphine, N is for injectable naltrexone, and M is for methadone. These medications approved to treat OUD are sometimes subject to utilization management controls such as requirements that prescribers obtain prior authorization prior to prescribing medications like buprenorphine.

^7.The 1965 amendments to the Social Security Act that created Medicaid and Medicare included a Medicaid Institutions for Mental Diseases (IMD) exclusion which prohibits the use of federalMedicaid financing for care provided to adult patients in mental health and substance use disorder residential treatment facilities that are larger than 16 beds. States can pursue federal waivers from this exclusion and the status of those waivers is tracked by the Kaiser Family Foundation. State Options for Medicaid Coverage of Inpatient Behavioral Health Services – Appendices – 9368 | KFF and the National Conference of State Legislatures. Medicaid 1115 Waivers by State (ncsl.org)

^8.Several states have taken advantage of opportunities under the authority of section 1115(a) of the Social Security Act (Act) to demonstrate and test flexibilities to improve the continuum of care for Medicaid beneficiaries with substance use disorders (SUDs) with approval from CMS. The table lists the dates when MODRN states received CMS approval to implement their 1115 SUD waiver programs which varied in scope by state.

Governance and Organization

MODRN’s organizational structure drew on lessons learned from existing distributed research networks, including the US Food and Drug Administration’s Sentinel Initiative(12) and the Patient-Centered Clinical Research Network.(13) It features four key components: 1) a Steering Committee, 2) a Data Coordinating Center, 3) a Methods Core, and 4) Members that analyze their states’ data (Figure 1).

Figure 1.

Medicaid Outcomes Distributed Research Network (MODRN) governance and structure

Notes: SUPLN is the State University Partnership Learning Network. UK is the University of Kentucky (KY), UD is the University of Delaware (DE), USM is the University of South Maine (ME), UMBC is the University of Maryland (MD), Baltimore County, UMass is the University of Massachusetts (MA), UM is the University of Michigan (MI), UNC is the University of North Carolina (NC), OSU is the Ohio (OH) State University, PITT is the University of Pittsburgh in Pennsylvania (PA), VAND is Vanderbilt University in Tennessee (TN), VCU is Virginia (VA) Commonwealth University, WVU is West Virginia (WV) University, UW is the University of Wisconsin (WI).

We developed a Steering Committee currently including 28 Medicaid Medical Directors and other state officials and 34 university partners in those states to guide MODRN efforts. It sets research priorities, provides analytic advice, shares state context, and disseminates findings to policymakers. University of Pittsburgh, in collaboration with AcademyHealth, serves as the Data Coordinating Center, which convenes regular Methods Core meetings, develops, and maintains the common data model, creates analysis plans, disseminates statistical code to each university partner for local execution, collects aggregate results, and conducts statistical analyses on aggregated results to produce global estimates. Composed of faculty and staff from university partners, the Methods Core advises the Data Coordinating Center on: 1) definition of Medicaid enrollee cohorts to include in measure denominators, 2) diagnosis, procedure, and National Drug Code values to include in measure specifications, 3) definitions of key covariates and population sub-groups, and 4) statistical analysis plans.

Members of the distributed research network perform (i.e., each of the 13 university partners) multiple tasks. University partners extract, transform, and load their data into the common data model. University partners then conduct analyses locally on their state’s Medicaid data and sends results to the data coordinating center for validation, aggregation, analysis of aggregated data, and stakeholder reporting. Prior to dissemination, the university partners share results with each state’s Medicaid agency for review.

Data Sources

University partners access Medicaid data through data use agreements, business associate agreements, memoranda of understanding, and/or under the auspices of master agreements or contracts with state Medicaid agencies.(14, 15) University partners receive current Medicaid program data, with a 3–12-month lag from service date to data receipt. Medicaid data include enrollment, claims (medical and pharmacy claims), and encounter data (when available) on all enrollees. Each university has a unique relationship with its state partners. Some predated MODRN whereas others developed more recently. Several university partners had other (non-MODRN) contractual relationships with state agencies to provide assistance on a wide range of Medicaid evaluation and analytic activities.

Initial Common Data Model Development, Structure, and Content

The Data Coordinating Center, with Methods Core input, developed a standardized common data model template. Initially, the Data Coordinating Center developed a survey tool to collect information from each state on structure, content, format, data quality, and completeness of its Medicaid data. It then created detailed instructions for and consulted with each state on converting Medicaid data to the common data model. The Data Coordinating Center guided each partner in a process to extract, transform, and load (12) their native Medicaid enrollment and claims files into the common data model format. Converted Medicaid data remain on each academic or state partner’s local servers, available for quick-turnaround analyses.

The initial common data model (‘version 1.0’) contained a relational database with five tables: 1) enrollment, 2) inpatient encounters, 3) outpatient encounters, 4) professional encounters; and 5) pharmacy encounters (Table 2). Enrollment files contained all enrollment episodes for an individual, along with demographic characteristics, including zip code of residence. A key derived variable identified five clinically- and policy-relevant eligibility groups: pregnant women, children, disabled adults, non-disabled adults, and Medicaid expansion adults. The encounter data tables contained elements such as claim number; claim line number (if applicable); diagnosis, procedure, and revenue codes; place of service; dates of service; admission and discharge dates for inpatient claims; and provider identifiers. Pharmacy encounter data contained claim number, claim line number (if applicable), prescription fill dates, National Drug Code, days and quantity supplied, and prescriber and pharmacy identifiers. All tables included a unique beneficiary identifier.

Table 2.

The Design and Evolution of the Medicaid Outcomes Distributed Research Network’s Common Data Model

Version 1.0			Updates in Version 2.0
Data file	Description	Key Data Elements
Enrollment	One record for every individual enrolled in Medicaid for at least one day during a given calendar year	Unique enrollee identifier, start and end dates of coverage, demographics (date of birth, gender, race/ethnicity, zip code), eligibility group,1 and date of death	Eligibility: newly included enrollees who are dually eligible for Medicare or in a partial benefit program Race/ethnicity: distinguished “other” category from missing Pregnancy indicator: added an indicator for pregnancy which was previously included as part of eligibility category. Also, method to identify pregnancy was updated to better estimate duration of pregnancy Vital statistics: cause, place, manner of death, etc. added if available
Inpatient encounters	Inpatient stay records for enrollees who had an encounter at an inpatient facility	Unique enrollee identifier, encounter identifier, start and end date for encounter (admission and discharge date for inpatient encounter), diagnosis, procedure code, performing and billing provider identifier, place or service, revenue code and modifier	Files combined: inpatient, outpatient, and professional encounter files combined into one file to reduce heterogeneity across states in how these encounter types were distinguished Data elements added: inpatient claims indicator, DRG, DRG type, paid amount, provider information
Outpatient encounters	Outpatient claim records submitted by institutional outpatient providers
Professional encounters	Professional claim records submitted by professional providers
Pharmacy encounters	Records of filled prescriptions	Unique enrollee identifier, encounter identifier, dispensed date, NDC, days and quantity of supply, prescribing provider identifier	Data elements added: paid amount, prescriber and pharmacy information.
Version 2.0
Monthly Enrollment File	Added monthly enrollment file to more accurately capture time-varying data across the calendar year. Contains one record for every month of enrollment for individuals enrolled in Medicaid for at least one day during a given calendar month	Unique enrollee identifier, month indicator, demographics (date of birth, gender, race/ethnicity, zip code), eligibility group, indicator for pregnancy, all specific to the month enrolled	N/A

Notes: DRG=diagnosis related group; NDC=national drug code; QMB=qualified Medicare beneficiary

^1.Eligibility groups in common data model 1.0 included: pregnant women, children, disabled adults, non-disabled adults (categorically eligible pre-Affordable Care Act), and Medicaid expansion adults. In common data model 2.0, eligibility categories evolved to include six broad groups with three subgroups under the ‘dual’ eligibility group: 1) partial benefit non-dual, 2) Dual (full benefit dual/partial benefit dual, QMB-only / partial benefit dual, non-QMB), 3) disabled, 4) children, 5) expansion adults, and 6) non-disabled adults/pregnancy eligibility. Clinical criteria including claims for deliveries and pregnancy-specific care using diagnosis and procedure codes identify pregnant women, rather than eligibility for Medicaid due to pregnancy. The common data model assigns enrollees with more than one eligibility category each year hierarchically to a single category based on the order in which groups appear above.

Establishing Priorities for Research

The Steering Committee and Methods Core focused first on OUD prevalence, its associated harms, the quality of care for OUD, and policies to improve systems of care. Medicaid covers 38% of adults with OUD and is the largest payer for OUD treatment nationwide.(16–18) Medicaid programs could play a vital role in measuring access and quality of care for OUD, but states remain limited in sharing timely, actionable Medicaid data. MODRN prioritized research on access to MOUD (i.e., buprenorphine, methadone, naltrexone), which reduces illicit opioid use, mortality, criminal activity, healthcare costs, and high-risk behaviors, (19–25) and improves patients’ quality of life.(26–29)

Adapting Statistical Analysis and Developing Strategies for Dissemination

To conduct descriptive analyses and hypothesis-testing, MODRN adapted methods applied in other health-related distributed research networks.(5, 30, 31) Statistical analysis goals included: 1) efficiency, i.e. minimizing each state’s labor to produce results, 2) ensuring data quality and validity of state-level and pooled estimates, 3) addressing heterogeneity within and among states, and 4) communicating findings to stakeholders. Regarding dissemination, MODRN developed a comprehensive plan for assessment of potential audiences for its findings, identified the need to generate multiple reporting formats and distribute these research products through academic and policy channels.

Results

The following sections discuss common data model implementation and evolution, OUD measures generated from it, use of a distributed research network to combine state-level common data model estimates in statistical analyses, and how the results translate into actionable evidence for state and federal policymakers.

Implementing a Common Data Model in a Distributed Research Network

Common Data Model Implementation.

The Data Coordinating Center selected data elements for the OUD project and those that would serve future MODRN projects and built the common data model. After loading Medicaid data into the common data model format, each university partner prepared a high-level data summary using descriptive characteristics of Medicaid enrollees. University partners confirmed consistency of distributions of characteristics based on findings from their prior analyses. Each partner shared details on how its internal eligibility codes mapped onto each common data model eligibility category. We assessed quality of transformed common data model data by having each state partner construct several initial MODRN OUD measures and compared results with prior work and with other MODRN states via the steering committee and methods core. We also compared the number of Medicaid enrollees in the MODRN common data model to CMS Medicaid enrollment reports.

Common Data Model Evolution.

After two years of using version 1.0, MODRN developed version 2.0 to allow for more flexibility and efficiency in analyses. Common data model 1.0 included only non-dual, full-benefit enrollees under age 65 whereas 2.0 included all Medicaid enrollees, regardless of their age and eligibility program, broadening possible populations and policy questions (Table 2). Common data model 2.0 accommodated comprehensive beneficiary, administrative, utilization, and provider data by adding data elements, including refined eligibility groups, amounts paid for services/procedures, diagnosis-related group, and provider type and specialty. It also added monthly enrollment information, a revised and expanded eligibility category scheme, pregnancy status, and fields for death information from linkage with vital statistics records in states that have authorization to do so. MODRN expects the common data model to further develop as the new policy questions emerge.

Unique features of MODRN.

MODRN generates a unique resource distinct from existing sources of Medicaid data including the new generation of CMS Medicaid data, the Transformed Medicaid Statistical Information System (T-MSIS) analytic files (TAF). CMS contractors generated the TAF using a data transformation system(32) that aggregated Medicaid claims data across states. However, data quality remains variable across items, fields, and states.(33)

MODRN’s direct access comes with advantages over alternative Medicaid data sources. Data are available with a shorter lag (6–12 months) compared to other Medicaid data sources, which have a one-to-two-year lag. MODRN can link Medicaid data to other state-level data, including vital records, corrections, child welfare, and other systems. An early exemplar includes linking vital statistics and Medicaid data to add cause of death fields into the common data model in select states. MODRN also facilitates information sharing between Medicaid agency officials on state-specific policy and practice that may explain between state-differences. MODRN partnerships with state policymakers facilitate rapid dissemination of study findings to officials who can make immediate policy changes.

Evolution of MODRN Measures

Despite a national focus on OUD, few validated measures of utilization and quality of treatment exist.(34) Several MODRN states monitored OUD prevalence, and a few had ‘opioid dashboards’, however, none had developed a comprehensive measure set to examine quality and outcomes of OUD treatment. The MODRN OUD project addressed this gap. MODRN’s organizational structure allowed for researchers and state partners to weigh in on feasibility and relevancy of measures. We conducted a scoping review for each measure, drawing on peer-reviewed literature and definitions from national stewards when measures existed.(34, 35) States and university partners provided information on how they defined outcomes in Medicaid data, including measures used in monitoring and evaluating their Section 1115 SUD Demonstration Waivers (36) (which states use to change SUD treatment coverage, payment, or delivery).

Two priorities informed measure selection. First, MODRN wanted to develop measures of evidence-based MOUD treatment to evaluate state policies. Second, MODRN aimed to create measures that reflected use across settings: the full continuum of OUD care, general medical/preventive care among those with OUD, and acute visits that might show OUD recurrence. MODRN identified 21 measures in 6 areas: 1) identification, initiation, and engagement in treatment, 2) MOUD including rates of treatment, duration, and concurrent use of counseling and monitoring services, 3) rates of follow-up after an emergency department visit, or residential treatment stay, and receipt of health care utilization during OUD treatment, 4) opioid and benzodiazepine prescribing, 5) acute care use and overdose outcomes, and 6) neonatal opioid withdrawal syndrome-related measures.

We added measures to reflect evolving state, policy, and research interests. For example, several states that introduced payment for residential treatment were interested in characteristics of those receiving residential care and patterns of followup care after a residential stay. Residential treatment measure development required significant collaboration across states, because of differences not only in how states define residential care, but in how to identify residential treatment in each state’s claims data. Similarly, during the COVID-19 pandemic, states collected a wide range of telehealth codes, and compared and defined codes for analyses on remote OUD treatment using the common data model, again requiring iterative expert input from each state partner.

Learning to Analyze Common Data Model Data and Combine Results in a Distributed Research Network

As each state generated data using standardized measures, MODRN adapted to maximize efficiency in conducting analyses, combining results from states to characterize heterogeneity and testing hypotheses (Figure 2). The Data Coordinating Center wrote SAS (SAS Institute Inc., Cary, NC) analytic code to generate OUD measures on its state’s common data model before distribution to partner states. Analysts in each state ensured successful execution of analysis code on their state’s common data model. Functions of analytic code included computation of summary statistics, fitting statistical models, and generating results. State analyses rendered SAS output into a spreadsheet to reduce transcription errors from manual data entry.

Figure 2.

Medicaid Outcomes Distributed Research Network (MODRN) data flows and analytic processes

Each state partner shared summarized state-level or subgroup-level (e.g., county, demographic group) statistical estimates with state partners for review and approval. University partners submitted aggregate results to the Data Coordinating Center, which reviewed results and identified issues with missing, inconsistent, or outlier results encountered by partner states. The Methods Core statistics subcommittee discussed how to resolve any issues, such as (1) changing variable definitions, (2) changing member inclusion criteria, (3) modification to statistical models and estimation approaches, and (4) adding sensitivity analyses.

The Data Coordinating Center developed analysis plans and analytic code in collaboration with Methods Core subcommittees organized around specific research questions. We aimed to produce standardized measurement across states where feasible and developed specifications in close consultation with each state. University partners participating in the Methods Core consulted with state Medicaid officials for guidance on measuring specific Medicaid-covered services to identify any idiosyncratic coding practices or relevant billing policies. In most analyses, we applied a single standardized measurement approach in all states. In some instances, measure specifications (e.g., combinations of procedure, modifier, or revenue codes) varied by state to produce valid measurement consistent with states’ policy. We performed sensitivity analyses in multiple states prior to finalizing analytic plans to assess potential misclassification.

After states returned estimates for review, the Data Coordinating Center reorganized and combined results to meet research goals according to a pre-approved statistical analysis plan. We performed multiple data quality checks. Before submitting results to the Data Coordinating Center, university partners reviewed results internally and obtained review and approval from their state Medicaid agencies. The Data Coordinating Center also performed data quality checks, which compared descriptive results and multi-variable estimates across states and within states over time to detect aberrant, outlier or incomplete values. When the Data Coordinating Center detected possible errors in analysis or reporting of results, it returned results to the state for re-analysis. The Data Coordinating Center used direct aggregation and meta-analysis approaches to combine state-level measures and compute global statistical quantities generated in MODRN, summing numerators and denominators to compute global proportions.

To compute global estimates from state-specific parameter estimates (log odds ratios, log hazard ratios, raw incidence rates, raw means, etc.) and to test heterogeneity of estimates across states, the Data Coordinating Center used random effects meta-analysis.(37) Expected heterogeneity occurred because of different Medicaid populations and policies, OUD prevalence, and treatment rates across states, as shown in Table 1. When generating global results, the Data Coordinating Center reported measures of heterogeneity across states. These included I-square (proportion of total variance due to between-state variance); Cochran’s Q (test of statistical significance of state-level variability); range of state-specific estimates; 90% prediction interval (estimated range of values within which the interested quantity would fall for 90% of the states); and Tau-square (between-state variance).

A key challenge centered on the best way to present heterogeneity across states. This decision depended on the particular research aims and observed directions and magnitudes of effects.(38) Prediction interval appeared as the most useful measure, with several advantages over other measures of heterogeneity: expression on its natural scale and not just as a proportion (a weakness of I²); it estimates between-state variability of true prevalence differences and prevalence ratios of state populations; and unlike the 95% confidence interval, it is relatively unaffected by the number of states included.(38) The Data Coordinating Center used the meta-analysis metafor package in R(39) to combine and report MODRN results. MODRN’s meta-analysis approach focused on the global population represented by the participating states.

Process for Approval of Papers and Proposals

MODRN developed a review and approval process of proposals for manuscripts and conducted a search for existing policies from other multi-center research groups to govern this process. The Multi-Ethnic Study of Atherosclerosis Publications & Presentations Policies(40) provided a base from which to develop policies governing publications and grants. We sought to: a) stimulate scientific presentations and papers from MODRN investigators; b) ensure and expedite reports to the scientific and policy communities; c) ensure accuracy and objectivity in MODRN research reports; d) ensure that all investigators have the opportunity for participation in study-wide MODRN papers; e) acknowledge the collective investment of participating members; f) facilitate communication with the MODRN Steering Committee on publications and presentations; g) prevent overlap of published material and duplication of analyses; and h) encourage use of MODRN analytic tools for ancillary studies.

MODRN developed ‘Publications, Presentations, Funding Proposal Policies’ and established a committee with membership from participating universities. The committee meets monthly to review, provide feedback, and approve proposals by MODRN members, using an abbreviated process for expedited reviews of ancillary studies (i.e., research studies extending and/or complementing the original grant-funded aims).

Translating Distributed Research Network Results into Actionable Evidence

MODRN’s primary audience includes state administrators and policy makers that oversee Medicaid programs, whereas university partners outside of SUPLN may target an academic audience.(41) An important MODRN development involved its dissemination process, which had to align with state and national stakeholder needs. We generated a bidirectional information flow to disseminate results to states. States provided policy context, feedback on results, and generating policy questions that MODRN needed to answer.

For example, recognizing the need to contextualize between-state variation in OUD treatment access, utilization, quality, and outcome measures observed across MODRN states, and to support evaluation of state policy changes, MODRN members conducted 27 key informant interviews in 9 states for 3–4 hours per state. The team created a robust policy inventory to capture OUD-related policies implemented by MODRN state Medicaid programs over a five-year period.(42) State input also occurs ad hoc. For example, amidst social unrest arising from police violence against Black/African American individuals and stark disparities in COVID-19 deaths, states demonstrated strong interest in understanding equity of MOUD care and changes in OUD treatment; MODRN adjusted its analytic plans to meet these immediate needs.

Discussion

MODRN represents an innovative distributed research network encompassing several key outcomes. MODRN supports standardized analyses by researchers with expertise in their states’ Medicaid policies and data systems, and trusted relationships with policymakers who can act on the findings. Already, state agency partners report using or intending to use MODRN analytic tools to train Medicaid analysts and for federal reporting requirements and using comparative state data to develop policy documents for Medicaid agency staff and state legislatures to drive coverage reforms.

Further, MODRN can address critical problems facing the US. In its first application, MODRN made substantial improvements in understanding OUD and MOUD access and quality. MODRN improved on earlier studies regarding geographic access to MOUD by focusing on Medicaid enrollees, direct measures of demand for OUD treatment, providers that deliver MOUD and accept Medicaid patients, granular measures of location for providers and patients, and measuring prescribing volume. MODRN has conducted the largest ever population-based study of over 1 million Medicaid enrollees with OUD to examine MOUD utilization(43) and reported on a sample of 1.6 million pregnancies and 1.3 million live births to examine the healthcare patterns of pregnant women and children affected by OUD.(44) Further, MODRN has presented its findings to a wide range of stakeholders at academic conferences, to federal agencies and workgroups, to state agencies, and for ongoing evaluations. These include evaluations of initiatives under the SUPPORT Act’s Section 1003 intended to increase the capacity of Medicaid providers to deliver SUD treatment services, and Section 1115 SUD Demonstration Waivers used by states to expand access to the continuum of care for SUD.(36, 45) Several states also used MODRN measures to assess changes in treatment for substance use disorders following the introduction of the COVID-19 Public Health Emergency.

Challenges and Limitations

Despite using a detailed, iterative process for identifying, creating, and modifying measures, the MODRN team encountered some persistent challenges. Limits of standardizing data can occur when participating state Medicaid agencies differ in billing codes they will accept and the guidance they give providers in which services to submit claims and how to submit them. Although others also need to address this issue when generating multi-state Medicaid data resources,(33) creating consistent definitions across states remains an imperfect science and researchers working with Medicaid data will face tradeoffs between standardization and validity of their measurement. The MODRN approach may minimize these limitations by using state teams (university researchers and state program officials) with a deep understanding of program differences.

We faced significant up front fixed costs to launch MODRN at the Data Coordinating Center and participating states. Flexibility in state and university support made it possible to demonstrate feasibility and generate preliminary data necessary to obtain extramural funding at substantially lower cost than existing distributed research networks.(46, 47) To scale up to include additional states or studies requires additional funding, yet no dedicated funding exists to support multi-state Medicaid research. Federal agencies, like the National Institutes of Health and Centers for Medicare and Medicaid Services, could support launching similar state health policy distributed research networks. Securing funds needed to maintain this effort remains challenging.

Finally, we learned important methodological lessons. Based on our distributed research network design, MODRN researchers could not query or use a web interface to request results. Our analyses required each state-university team to generate results using distributed SAS programs. Large variation in results across states posed challenges for aggregating data, while providing a unique opportunity to explore heterogeneity.

Opportunities and Future Directions

We expect future applications of MODRN’s organizational and analytic infrastructure. MODRN remains better positioned to conduct analyses of Medicaid managed care for at least two reasons, namely that the TAF redacts spending data on managed care claims and does not contain a flag indicating whether beneficiaries choose a managed care plan or are randomized to one (i.e., randomization flags). Such flags are useful for mitigating selection bias in comparisons of managed care plan performance or spending.

Although other data sources such as TAF could technically link to other individual-level data, MODRN has a greater feasibility of generating linkages to state resources such as vital statistics, corrections, juvenile justice, child welfare, food assistance, and housing. Such connections would expand the breadth of health outcomes that MODRN can examine, particularly those that occur outside of the healthcare setting. Incorporating social determinants of health would permit researchers to address vital public health research questions, such as patterns, trends, and disparities in pregnancy-associated morbidity and mortality and/or COVID-related diagnoses, vaccines, and outcomes at scale not currently available in the US. We could team with organizations to conduct qualitative research relevant to entities, such as the Medicaid and CHIP Payment and Access Commission, expand to more states, and conduct community-based participatory research. MODRN could incorporate Medicaid enrollee experiences beyond healthcare utilization through population-based surveys, as conducted in Ohio and Virginia.

Conclusion

Through a unique multi-state and multi-sector collaboration coalescing around an emergent opioid epidemic, our team created an innovative, productive, and useful resource to inform health systems and policy decisions state Medicaid agencies face. This paper characterized experiences developing MODRN, including methods employed, results generated, and challenges faced to inform developing future distributed research networks.