Follow-Up Assessment of Adherence to Methodological Standards in National Inpatient Sample Research

Abstract

This cohort study assesses changes in adherence to National Inpatient Sample (NIS) methodological standards among research studies since the original publication of the checklist of best practices.

Introduction

Large administrative datasets, such as the National Inpatient Sample (NIS), are key resources for studying care practices and outcomes. The NIS is a probabilistic sample of approximately 8 million all-payer claims drawn from the inpatient databases of 47 states.^1,2 In 2017, 85% of NIS-based studies had not adhered to at least 1 recommended analytic practice.³ Subsequently, the checklist based on that work became available on the Agency for Healthcare Research and Quality’s website as a set of methodological best practices.⁴ This study evaluated how nonadherence to NIS best practices among research studies has since changed.³

Methods

This cohort study used a random probability sample of studies using NIS standards during the 2 years following the original publication (July 1, 2018, to June 30, 2020) and 5 years after (July 1, 2021, to June 30, 2023). We replicated the design of the original study,³ sampling all studies published in journals with an impact factor (IF) of at least 10 and a random subset from journals with an IF less than 10, for a total initial sample of 100 studies. As the study was non–human participant research, it was exempt from review and informed consent per the Yale University Institutional Review Board. The study followed the STROBE reporting guideline.

We evaluated the studies on 7 core practices.³ All reviewers (L.S.D., A.F.A., C.C., S.M., B.B.) discussed the checklist and associated documentation.⁵ Two reviewers scored independently, with disagreements resolved with the senior author (R.K.).

We examined the proportion of studies not adhering to each required best practice during both periods and compared it with the baseline study (2015-2016). Given the random probabilistic sample of studies in each phase, we computed a survey-weighted estimate for the number and proportion of all NIS studies during these periods that were nonadherent to these practices. Analyses were performed using R, version 4.0.2 (R Foundation for Statistical Computing), and a 2-sided P < .05 was considered significant. Additional methodological details are summarized in the eMethods in Supplement 1.

Results

We analyzed 85 and 90 studies that used NIS as the primary dataset, representing an estimated 1474 and 1751 studies using NIS data over the 2 periods, respectively. Overall, 78 studies each in both periods did not adhere to at least 1 required practice, representing an estimated 94.9% (95% CI, 85.2%-99.0%) and 91.4% (95% CI, 79.7%-96.6%) of all NIS studies during the respective periods compared with an estimated 89.1% (95% CI, 81.3%-93.8%) in the baseline period (Figure). An estimated 19.8% (95% CI, 9.5%-34.0%) and 30.8% (95% CI, 17.9%-43.8%) had 3 or more instances of nonadherence, respectively, similar to baseline (24.9%; 95% CI, 16.5%-33.3%).

Figure. Estimated Weighted Percentage for Studies and Nonadherence to Required Research Practices With the National Inpatient Sample.

The baseline period was from July 1, 2015, to June 30, 2016; phase 1, from July 1, 2018, to June 30, 2020; and phase 2, from July 1, 2021, to June 30, 2023. Lines inside the boxes indicate the estimated percentage, and the upper and lower borders of the boxes indicate the 95% CI.

The most frequent issues were failure to account for complex survey design (62.2% [95% CI, 46.7%-76.0%] and 55.7% [95% CI, 41.6%-69.2%] in the 2 periods, respectively) and misidentification of hospitalizations as unique patients (51.7% [95% CI, 36.4%-67.0%] and 43.2% [95% CI, 29.7%-57.4%], respectively). Nonadherence was high in both high- and low-impact journals, although studies published in journals with an IF less than 10 had higher nonadherence with several practices, including misclassification of observations and omission of survey design (Table).

Table. Sample Frequency and Estimates for Studies Not Adhering to the 7 Individual Required Research Practices From the NIS.

Research practice and description	Analysis period
	Phase 1 (July 1, 2018, to June 2020)						Phase 2 (July 1, 2021, to June 30, 2023)
	Nonadherence, No. of studies/eligible studies in sample (%)			P valuea	Estimates of nonadherence for the universe of NIS studies (n = 1474)		Nonadherence, No. of studies/eligible studies in sample (%)			P valuea	Estimates of nonadherence for the universe of NIS studies (n = 1751)
	Overall	Journal IF					Overall	Journal IF
		<10	≥10		No. (95% CI)	% (95% CI)b		<10	≥10		No. (95% CI)	% (95% CI)b
Identifying observations as hospitalizations rather than unique patients	33/85 (38.8)	21/40 (52.5)	12/45 (26.7)	.03	762 (536-987)	51.7 (36.4-67.0)	30/90 (33.3)	21/48 (43.8)	9/42 (21.4)	.03	757 (520-1005)	43.2 (29.7-57.4)
Avoiding performing state-level analyses	0/85	0/40	0/45	NA	0	0	3/90 (3.3)	1/48 (2.1)	2/42 (4.8)	.48	38 (2-156)	2.2 (0.1-8.9)
Limiting hospital-level volume data to years after 1988-2011	6/83 (7.2)	2/39 (5.1)	4/44 (9.1)	.68	75 (13-216)	5.3 (0.9-15.0)c	0/87	0/48	0/42	NA	0	0
Avoiding estimating physician-level volume data	1/85 (1.2)	1/40 (2.5)	0/45	.47	36 (1-162)	2.4 (0.1-11.0)	0/90	0/48	0/42	NA	0	0
Avoiding use of nonspecific secondary diagnosis codes to infer in-hospital events	37/85 (43.5)	19/40 (47.5)	18/45 (40.0)	.52	697 (475-928)	47.3 (32.2-63.0)	25/90 (27.8)	15/48 (31.3)	10/42 (23.8)	.43	544 (334-788)	31.1 (19.1-45.0)
Accounting for complex survey design in the methods	49/85 (57.6)	25/40 (62.5)	24/45 (53.3)	.51	917 (688-1120)	62.2 (46.7-76.0)	41/90 (45.6)	27/48 (56.3)	14/42 (33.3)	.03	975 (728-1212)	55.7 (41.6-69.2)
Accounting for sampling redesign	33/66 (50)	18/29 (62.0)	15/37 (40.5)	.14	658 (464-837)	61.3 (43.2-78.0)d	31/45 (68.9)	18/24 (75.0)	13/21 (61.9)	.34	654 (481-782)	74.7 (54.9-89.3)d

Abbreviations: IF, impact factor; NA, not applicable; NIS, National Inpatient Sample.

^a P values are for comparison of studies in journals with an IF of less than 10 vs 10 or greater using Fisher exact test. P values are presented as nominal without adjustment for multiplicity and should be interpreted alongside the reported 95% CIs.

^b Unless otherwise specified, the results reflect the estimated percentage of all studies using the NIS data during the study period with the given act of nonadherence.

^c Value indicates the percentage of the estimated studies that included NIS data from 2012 or after.

^d Value indicates the percentage of the estimated studies whose analyses spanned through the sampling redesign.

Discussion

This cohort study suggests that issues of widespread nonadherence to NIS methodological standards persist in contemporary studies. In this follow-up, fewer than 1 in 10 studies followed all 7 required practices, without significant improvement over time. Notably, studies frequently failed to account for the complex multistage probability sampling design of the NIS.⁶ Improving the quality of research using national survey databases requires coordinated efforts from investigators, reviewers, and editors, which journals could further strengthen by requiring checklist-based methodological documentation during peer review. Limitations included that estimates for low-IF studies may not be fully representative, despite random sampling. Given that inaccurate assessments may produce misleading conclusions that shape policy, guidelines, and patient care, it remains essential to ensure appropriate use of national databases such as NIS in clinical research.

Supplement 1.

eMethods.

eReferences.

Supplement 2.

Data Sharing Statement