Stakeholder Perspectives on Minimum Clinically Important Difference and Noninferiority Margin for Hospital-Free Days to Assess Interventions

Abstract

This survey study examined perceptions of patients, caregivers and health care professionals on the number of hospital-free days required for detection of a minimum clinically important difference or noninferiority margin of new interventions.

Hospital-free days (HFDs), or days alive and outside the hospital, is increasingly reported as a research outcome.^1,2,3 This outcome is a patient-centered and pragmatic measure,⁴ but no consensus exists regarding how many HFDs represent a minimum clinically important difference (MCID), or the smallest change in an outcome that a patient considers meaningful.⁵ Similarly, there is uncertainty regarding the noninferiority margin (NIM), or maximal difference in HFDs obtained by 2 or more treatments that would lead stakeholders (patients, caregivers, and health care professionals) to view a treatment as no worse than another.⁶ We surveyed stakeholders on the MCID and NIM for HFDs to guide design of future superiority and noninferiority trials.

Methods

We administered a 2-part survey to individuals with lived or professional expertise in critical and serious illness (eMethods in Supplement 1). Lived experience experts included survivors of critical illness, people with chronic, life-limiting illness, and their family members. Professional experience experts included clinicians, researchers, and executives at large health systems and payers. This study was approved by the University of Pennsylvania Institutional Review Board. Written informed consent was waived for this minimal risk study. We followed the CROSS reporting guideline.

In part 1, participants watched a video describing the MCID and NIM using plain language and animated graphics (https://vimeo.com/798458323). Participants were next asked to indicate the MCID and NIM of a new treatment compared with a standard treatment that would achieve 120 HFDs during 6 months of follow-up and to justify their response. In part 2, a subset of participants attended a virtual group discussion followed by multiple-choice questions with options for MCID and NIM informed by results from part 1. We used RedCap (Vanderbilt University) and Zoom polling software (Zoom Video Communications).

Quantitative responses were summarized using descriptive statistics. Data were analyzed from December 15, 2022, to January 25, 2023, using Stata 17 (Stata Corp, Inc). Medians were compared using the Wilcoxon rank sum test. Qualitative explanations were summarized by major themes. A 2-sided P < .05 was considered statistically significant.

Results

Of 79 participants recruited, 65 (82%) completed part 1 (mean [SD] age, 47.3 [12.2] years; 40 women [61.5%], 20 men [30.8%], and 5 individuals [7.7%] with missing data or no response regarding gender) (Table 1). Median (IQR) MCID was 20 (13-30) days, and median (IQR) NIM was 5 (0-14) days. Lived and professional experience experts rated MCID (median [IQR], 23 [12-32] vs 20 [12-30]; P = .42) and NIM (median [IQR], 6 [0-30] vs 5 [0-10]; P = .35) similarly. Qualitative analyses revealed that participants’ responses were driven primarily by subjective gestalt assessment, although some clinicians and researchers referenced a percentage change from baseline (Table 2).

Table 1. Participant Characteristics and Survey Responses.

Characteristic	Participants, No. (%)a
	Lived experience	Professional experience
Participant demographic characteristics
No.	36	35
Age, median (IQR), y	51 (42-59)	41 (36-46)
Gender
Women	23 (66)	18 (54)
Men	10 (29)	12 (34)
Missing data or preferred not to answer	2 (6)	4 (11)
Raceb
American Indian or Alaska Native	1 (3)	0
Asian Chinese	0	3 (9)
Asian Indian	0	3 (9)
Black or African American	3 (9)	1 (3)
White	29 (83)	23 (74)
Missing data or preferred not to answer	2 (6)	5 (14)
Ethnicityb
Hispanic or Latinx	2 (6)	2 (6)
Not Hispanic or Latinx	31 (89)	28 (80)
Missing data or preferred not to answer	2 (6)	5 (14)
Region
North America	32 (91)	32 (91)
Europe	3 (9)	1 (3)
Australia	0	2 (6)
Lived experience
Former patient	25 (71)	3 (9)
Family member	16 (46)	4 (11)
Educational level
High school graduate or GED	3 (9)	NA
Some college, no degree	2 (6)	NA
Associate’s degree	2 (6)	NA
Bachelor’s degree	11 (31)	NA
Graduate or professional school	15 (43)	NA
Missing data or preferred not to answer	2 (6)	NA
Median household income
≤$34 999	5 (14)	NA
$35 000 to $99 999	6 (17)	NA
$100 000 to $149 999	6 (17)	NA
≥$150 000	8 (23)	NA
Missing data or preferred not to answer	10 (29)	NA
Employment status
Full time	15 (43)	NA
Part time	5 (14)	NA
Unemployed	9 (26)	NA
Missing data or preferred not to answer	7 (17)	NA
Professional experience
Clinician	NA	27 (77)
Physician	NA	21 (78)
NP or PA	NA	2 (7)
Social worker	NA	1 (4)
Respiratory therapist	NA	1 (4)
Physical therapist	NA	2 (8)
Researcher	NA	19 (54)
Health care system executive	NA	2 (6)
Insurer/payer	NA	1 (3)
Experience, median (IQR), y	NA	9 (7-12)
Survey responses
Part 1
No.	35	35
Minimum clinically important difference, median (IQR), d	23 (12-32)	20 (12-30)
Noninferiority margin, median (IQR), d	6 (0-30)	5 (0-10)
Part 2
No.	27	21
Minimum clinically important difference, d
1-7	4 (15)	4 (19)
8-14	12 (44)	9 (43)
15-21	7 (26)	6 (29)
22-28	1 (4)	1 (5)
≥29	2 (7)	1 (5)
Noninferiority margin, d
1-7	20 (74)	16 (76)
8-14	3 (11)	4 (19)
15-21	1 (4)	0
22-28	1 (4)	1 (5)
≥29	2 (7)	0

Abbreviations: GED, General Educational Development; NA, not assessed; NP, nurse practitioner; PA, physician assistant.

^a Sum may not equal 100% because participants could have both lived and professional experience.

^b Race and ethnicity were self-reported and collected to provide insight into the representativeness of this survey sample and were not analyzed.

Table 2. Participants’ Explanations for the Noninferiority Margin and Minimum Clinically Important Difference of Hospital-Free Days.

Decision factor	NIM	MCID	Explanation
Subjective gestalt or belief	5	5	“5 Days sounds like a long time when you're stuck in a hospital or rehab bed.” (researcher)
	0	10	“Even 10 more days at home is very meaningful. …Seven days is probably also meaningful, especially in serious illness near end of life.” (clinician)
	7	14	“Less than 1 week seems within margin of error, but 2 weeks is a meaningful difference.” (clinician)
	5	20	“For me to feel like the research treatment is about the same as the standard treatment, I would expect that the patient experience about the same number of hospital-free days. For me to feel like the research treatment is definitely better than the standard treatment, I would expect that patients are experiencing a considerably more amount of hospital-free days.” (critical illness survivor)
	0	20	“Gestalt.” (clinician)
	10	20	“My gut is telling me that a range of ±10 days would be reasonable to say that the new treatment was about the same as the standard treatment…. A difference of ±20 would feel more significant to me and allow me to say that the research treatment was either definitely worse or definitely better.” (clinician)
	0	24	“I think 2 weeks improved quality of life at home is significant for a patient and clinically meaningful.” (clinician)
	10	30	“I think give or take 10 days for a new research is reasonable. To say it's better than standard I think it needs to be another month out of the hospital.” (clinician)
% Change from baseline	6	12	“As a rule of thumb, I decided improvements ≤5% wouldn't count as real improvement, whereas at 10% or more there is definite improvement. I'm assuming that P values are robust enough that those seemingly small differences are statistically significant, not due to random variation. (I would like to see P < .001.)” (research and critical illness survivor)
	5	12	“Showing superiority would have to document an improvement in hospital-free days of greater than approximately 10%.” (clinician and researcher)
	12	18	“For the [NIM], I'm using about 10% relative difference. So if the treatment gives you the same number of HFDs [within] 10%, I think that's a pretty reasonable margin to say that they're similar enough, although you could be more conservative. For something that's definitely better, I would say the bar has to be a little higher, so a 15%-20% relative difference. But if the wording were changed to probably better, then I'd probably lower that percentage.” (clinician and researcher)
	0	37	“A 30% improvement vs standard treatment seems significant to justify change in process or additional investment.” (critical illness survivor)

Abbreviations: HFDs, hospital-free days; MCID, minimum clinically important difference; NIM, noninferiority margin.

Forty-four of 57 recruited participants (77%) completed part 2. The most common response for MCID was 8 to 14 days (41% of respondents). The most common response category for NIM was 1 to 7 days (73% of respondents). Response patterns were similar between lived and professional experience experts.

Discussion

As trials using HFDs proliferate, values selected to convey statistical significance have varied widely and without stakeholder input.^1,2,3 By eliciting stakeholder perspectives on the MCID and NIM for HFDs, we believe this study provides important guidance for future investigators.

Findings of this study suggest the feasibility of involving patients and families in formative research design. Generalizability of results was augmented by not referencing clinical details or specific treatments in the scenario. This study has limitations. Our convenience sample may not be representative of other stakeholders. Additionally, participants’ comprehension was not assessed. However, responses were internally consistent (MCID was greater in magnitude than NIM) and open-ended explanations were generally logical. Finally, because stakeholders were anchored to 120 HFDs in the survey, their responses may not be generalizable to populations with different average HFDs.

These findings provide benchmarks for the numbers of HFDs through 6 months to use in calculating power when designing clinical trials. Future work is needed to validate these findings and assess whether MCID and NIM differ based on population and follow-up duration, how differences in HFDs might alter treatment, and whether HFD quality changes over time.

Supplement 1.

eMethods.

Supplement 2.

Data Sharing Statement