Systematizing and Optimizing Collection of Patient-reported Outcomes after Community-based Care: A Quality Improvement Project

Abstract

Introduction:

Healthcare systems routinely assess patients’ perspectives on inpatient and outpatient care. Still, patients cared for outside of these clinical settings have limited opportunities to share experiences and provide feedback. Current data collection strategies may also under-capture feedback from marginalized or hard-to-reach populations, potentially reinforcing existing inequities in care. This project aimed to (1) systematize the collection of feedback from participants in a pediatric hospital-based violence intervention program (HVIP) providing community-based case management services following violence-related injury and (2) optimize collection methodology to achieve and maintain a quarterly response completion rate of 50%.

Methods:

Collection of the HVIP Client Satisfaction Questionnaire (HVIP-CSQ), a brief survey assessing satisfaction with and perceived short-term outcomes of HVIP participation, began in July 2021. Caregivers and patients self-completed surveys electronically in response to text message or email invitations. In March 2023, a rapid-cycle quality improvement initiative was initiated to optimize response rates. Using an expanded data collection methodology, an outreach coordinator invited nonrespondents to complete surveys via phone and text message.

Results:

During the initial HVIP-CSQ data collection, quarterly response rates were suboptimal. On average, 29.5% of caregivers and 23.5% of patients completed the HVIP-CSQ. Following improvements to the collection methodology, average quarterly response percentages exceeded the set targets, reaching 78.4% for caregivers and 51.8% for patients.

Conclusions:

Multimodal outreach strategies improved response rates for patients and caregivers. Although automated and self-directed survey methods offer advantages in reach and efficiency, they may not sufficiently engage respondents. Flexible, multimodal feedback systems can enhance data completeness and support more inclusive quality improvement efforts.

INTRODUCTION

Healthcare systems routinely solicit patient feedback on clinical experiences and outcomes to monitor the quality of service delivery, ensure patient-centered care, drive research and innovation, and meet regulatory requirements for service reimbursement.¹ Until recently, however, efforts to assess perceptions of care experiences outside inpatient or outpatient medical settings have been limited.² These exclusions risk omitting patient experiences with hospital-partnered programs, such as those that provide critical community-based services to address social determinants of health outside of discrete clinical settings.^1,3

Given the growing need for healthcare services to bridge the clinical-to-community divide, programs are rapidly expanding to support physical and psychosocial health and to deliver services outside the hospital setting.^4,5 One such care model, hospital-based violence intervention programs (HVIPs), has rapidly expanded to support violently injured patients with their holistic mental and physical recoveries following their release from hospital care.⁵ At present, 70 HVIPs operate in the United States and Canada, with program models that differ in population focus (eg, children/adolescents versus adults only).^6–8 Broadly, however, HVIPs are multidisciplinary programs embedded within trauma centers that serve as a bridge between acute postinjury medical care and community-based services. HVIP case managers connect with potential participants at the patient’s bedside or shortly after discharge and support connections to longer term services, including outpatient medical care, mental health treatment, education, employment, social connection, legal aid, and housing and food security. Through this long-term wraparound support, HVIPs ultimately seek to bolster protective factors—including self-efficacy in accessing community resources, psychosocial well-being, and socioeconomic stability—thereby potentially mitigating future violence and fostering positive longer term trajectories.^5,7,9

However, the systematic collection of patient feedback in community-based care settings, even in hospital-adjacent settings such as HVIPs, lags behind the more routine feedback systems for patients who receive traditional healthcare. For HVIPs specifically, assessment of program quality and efficacy relies heavily on unobtrusive administrative data sources (eg, medical records) and focuses on outcomes distal to care (eg, reinjury).^10–12 The resulting gap in documentation of patient-reported experiences and short-term outcomes hinders HVIPs’ use of such data to continually assess program quality and respond to patient needs and preferences. A recent development—a brief 12-item HVIP-specific patient questionnaire, the HVIP Client Satisfaction Questionnaire (HVIP-CSQ)—offers a novel tool to capture patient- and proxy-reported satisfaction with and more proximal outcomes of community-based care.¹³ Development of the HVIP-CSQ has been previously published,¹³ and both the patient and proxy caregiver instruments are publicly available. In brief, the HVIP-CSQ assesses satisfaction with service quality and volume, the extent to which staff were responsive and respectful, perceptions of services’ ability to meet patient needs, and program-derived knowledge of, and self-efficacy for, accessing community resources. As such, HVIP-CSQ responses may position HVIPs to modify program services or resources to better meet patient needs. Responses may also inform opportunities for staff training or education, and for assessing whether the HVIP is increasing patients’ and families’ ability to navigate community resources independently.

The development of the HVIP-CSQ and other patient-reported outcome measures¹⁴ is a promising advancement in understanding patient experiences with HVIP services. Yet the ability of such assessments to guide patient-centered care depends on the completion of measurements, which requires thoughtful consideration of data collection methods to balance response rates with resource constraints. Over the last decade, survey data collection has become increasingly automated and digitized, with respondents self-reporting their experiences via web- or mobile application–based tools (eg, REDCap, Twilio).^15–17 These automated and self-directed processes demonstrate adequate patient acceptability and may contribute to cost savings in data collection and to broadening the reach of survey sampling.¹⁸ However, response rates, completion time, and data quality may be jeopardized, especially among patients from marginalized or underrepresented populations, such as minorities, younger patients, and those with lower socioeconomic status or public insurance. Individuals within these populations may experience inequities in digital access, have lower health literacy levels, and hold underresearched preferences regarding automated, electronic data collection, which could contribute to decreased survey response rates.^17,19–21 Such factors may introduce biases into data collection and evidence-informed decisions that consider only the perspectives of a small number of patients, failing to represent the range of patient experiences and outcomes.^22,23

Guided by the ultimate objective of monitoring program quality and enabling future data-driven improvements, our single pediatric trauma center-affiliated HVIP launched a quality improvement (QI) initiative. The subsequent multiphase project sought to systematize and maximize the collection of patient- and caregiver-reported outcome metrics following HVIP participation through 2 key aims: (1) to integrate the HVIP-CSQ into routine program operations, and (2) to optimize collection methodology to achieve a targeted quarterly response completion of 50% of eligible participants.

METHODS

Setting

The initiative took place within a pediatric HVIP affiliated with the Children’s Hospital of Philadelphia (CHOP), a large, urban, level I pediatric trauma center. Established in 2012, the HVIP supports patients aged 8–18 years who receive medical care following injuries from nonfamilial, interpersonal violence. The HVIP provides voluntary trauma-informed, community-focused case management and mental health services to pediatric patients and their families to promote physical and psychosocial healing. Trained case managers work with patients and their caregivers to identify and address families’ support needs related to physical security, psychosocial well-being, and/or cross-system navigation.²⁴ Families typically remain in the HVIP for approximately 4–7 months. Between 2021 and 2024, the program served and discharged an average of 85 patients annually.

In addition to case managers providing direct services to program participants, the HVIP staff includes individuals with expertise in program evaluation, QI, and program management, who are responsible for quality assurance and oversight of direct services and who served as the “project team” for the QI initiative that follows. The CHOP Institutional Review Board deemed work to enhance HVIP-CSQ response rates as QI before initiating any process changes (Institutional Review Board no. 23-021252).

Phase 1: Launching HVIP-CSQ with Automation

The CHOP HVIP launched the HVIP-CSQ at the beginning of quarter 3 of 2021. Given the program’s pediatric setting, in which services are delivered through a family-centered care model, the project team invited both caregivers and patients to complete the HVIP-CSQ. Although the CHOP HVIP serves patients aged 8–18, only adolescents aged 12 years and older received invitations to provide feedback. This decision was made given the prerequisite developmental maturity to provide critical program feedback and the likelihood that adolescent patients engage more independently with their case manager, without their caregiver. Patients younger than 12 years of age typically participate in program services exclusively with their caregiver.

The project team used REDCap, an electronic data capture platform,^25,26 to collect and manage all HVIP-CSQ data. They additionally used Twilio, a third-party web service integrated into REDCap, to automate survey delivery (Twilio, Inc., San Francisco). Following discharge, an HVIP staff member input families’ information and preferred contact modality (text message or email) into the HVIP-CSQ REDCap database. Caregivers and eligible patients then received automated survey invitations via text message and/or email, along with up to 2 automated reminders during an approximately 10-day period. When unique contact information was available for adolescent patients and caregivers, each received invitations directed to their personal contact information; otherwise, invitations for both caregivers and eligible patients were sent to the single available contact.

Phase 2: Enhancing Automation with Phone Outreach

During quarter 2 of 2023, the project team critically reviewed HVIP-CSQ response patterns and identified suboptimal response percentages that routinely fell below the 50% target for both youth and caregivers. As shown in Figure 1, the team conducted a mapping exercise to align the key drivers of HVIP-CSQ completion with potential interventions to enhance response rates. They determined that a single process change—outreaching to nonrespondents via a personal phone call or text message—addressed key drivers and aligned interventions. Phone outreach provided a variety of HVIP-CSQ administration methods to accommodate variable completion preferences in the sample, an opportunity for the program to gather missing direct contact information for eligible youth participants, and an opening for an HVIP staff member to reiterate the value of feedback to program improvement.

Fig. 1.

Key drivers for HVIP-CSQ completion.

Figure 2 depicts the evolving survey administration and delivery processes for the HVIP-CSQ, with phase 1 showing the initial collection processes and phase 2 showing the modifications to integrate phone outreach, beginning on June 1, 2023. During phase 2, patients and caregivers continued to receive automated invitations using the procedures described in phase 1. However, if a survey remained incomplete after approximately 5 business days, an HVIP outreach coordinator (OC) without any case management responsibilities received an automated alert. During the following 2 weeks, the OC then made at least 3 personalized outreach attempts, via phone call or text message, to both caregivers and eligible patients with incomplete responses. Upon reaching respondents, the OC offered to administer the survey by telephone in real time or resend an electronic survey link for self-completion. The OC recorded all outreach attempts, including the outreach date, whether successful contact was made or a response was received from the respondent, and whether the HVIP-CSQ was administered by telephone or the survey link was resent for self-administration.

Fig. 2.

Data collection workflow for the HVIP-CSQ.

Data Management and Analysis

The team used a dedicated Qlik Sense dashboard (Qlik Technologies, Inc., King of Prussia, PA) to track HVIP-CSQ completion proportions and generate process control charts (p-charts) that monitored completion proportions by respondent group (eg, caregiver, patient) at quarterly intervals. Due to variable sample sizes across quarters, p-charts included exact upper and lower control limits for each quarter. Following the recommendations outlined by Wheeler et al,²⁷ the team applied the Aggregate Point Rule to identify a centerline shift before the accumulation of 8 consecutive data points and aligned with our process change on June 1, 2023. Additionally, the team descriptively summarized response patterns by respondent demographic characteristics within each phase. For phase 2 respondents, the team summarized responses by respondent group (caregiver, patient) and survey administration mode (self-initiated and administered, staff-administered during outreach, or self-administered after outreach). All descriptive summaries were prepared using R version 4.5.2.

RESULTS

In the 3.5-year HVIP-CSQ data collection period, the CHOP HVIP discharged 305 patients. A total of 294 caregivers and 254 patients received invitations to complete the HVIP-CSQ. Table 1 reflects the demographic characteristics by phase of data collection, respondent group (caregiver versus patient), and HVIP-CSQ completion (responded versus no response). Across phases, eligible caregivers were demographically homogeneous, with most being female-identifying (n = 281, 95.6%) and parents of the HVIP patient (n = 276, 93.9%). Eligible patients were roughly half female-identifying (n = 117, 46.1%), the majority Black or African American (n = 213, 83.9%), and had a mean age of 14.6 years (SD = 1.6 y).

Table 1.

Comparison of Phase 1 and 2 HVIP-CSQ Response Rates by Respondent Group and Demographic Characteristics

Eligible Caregiver Respondents (n = 294)	Phase 1 (n = 132)		Phase 2 (n = 162)
	Responded	No Response	Responded	No Response
Caregivers overall, n (%)	39 (29.5)	93 (70.5)	127 (78.4)	35 (21.6)
Caregiver role, n (%)
Parent	35 (28.2)	89 (71.2)	117 (77.0)	35 (23.0)
Relative (grandparent, aunt, uncle, sibling)	4 (50.0)	4 (50.0)	10 (100)	—
Caregiver Sex, n (%)
Female	35 (28.0)	90 (72.0)	121 (77.6)	35 (22.4)
Male	4 (57.1)	3 (42.9)	6 (100)	—
Child age, y (M, SD)	13.6, 2.76	13.8, 2.34	13.7, 2.50	13.7 (2.69)
Eligible Patient Respondents (n = 254)	Phase 1 (n = 115)		Phase 2 (n = 139)
	Responded	No Response	Responded	No Response
Patients overall, n (%)	27 (23.5)	88 (76.5)	72 (51.8)	67 (48.2)
Sex, n (%)
Female	12 (23.1)	40 (76.9)	34 (52.3)	31 (47.7)
Male	15 (23.8)	48 (76.2)	38 (51.4)	36 (48.6)
Race, n (%)
Black/African American	24 (24.7)	73 (75.3)	60 (51.7)	56 (48.3)
White	2 (33.3)	4 (66.7)	9 (69.2)	4 (30.8)
Other/multiracial	1 (8.3)	11 (91.7)	3 (30.0)	7 (70.0)
Ethnicity, n (%)
Hispanic	2 (20.0)	8 (80.0)	9 (52.9)	8 (47.1)
Age, y (M, SD)	14.8, 1.74	14.3, 1.57	14.9, 1.65	14.6, 1.71

The proportion of surveys completed by quarter and respondent group is presented as p-charts in Figure 3 (caregivers) and Figure 4 (patients). Although response percentages were initially robust during phase 1, when data collection was wholly automated and self-directed, percentages routinely fell below our 50% target, averaging 29.5% for caregivers and 23.5% for patients before the implementation of personalized outreach efforts, which began on June 1, 2023. Beginning during Q2 2023 (post-June 1, 2023), when phase 2 procedures began, response percentages improved dramatically. Our team documented a centerline shift in mean response percentages for both respondent groups, with percentages rising from 8.0% and 14.3% in Q1 2023 to 75.7% and 56.3% in Q3 2023 for caregivers and patients, respectively, indicating special-cause variation. Beginning in June 2023 and continuing into the 6 consecutive quarters that followed (Q3 2023–Q4 2024), the centerline shift was sustained, and response percentages consistently met or exceeded the 50% target, averaging 78.4% for caregivers and 51.8% for patients (Figs. 3, 4). Response percentages increased for all demographic cross-sections of caregivers, by caregiver role and gender, between phases 1 and 2 (Table 1). Similarly, all demographic cross-sections of patients—by sex, race, and ethnicity—showed increases in response percentages between phases. In phases 1 and 2, patient response percentages were consistent across sexes; however, patients identifying as Black or African American responded at lower percentages than their White peers (24.7% versus 33.3% in phase 1 and 51.7% versus 69.2% in phase 2, respectively). The mean age of patients was slightly younger among nonrespondents than among respondents.

Fig. 3.

Proportion of completed HVIP-CSQ by quarter (caregivers). CL, center line; LCL, lower control limit; UCL, upper control limit.

Fig. 4.

Proportion of completed HVIP-CSQ by quarter (patient). CL, center line; LCL, lower control limit; UCL, upper control limit.

Table 2 highlights variations in HVIP-CSQ response mode by respondent group and demographic characteristics during phase 2. Response percentages for self-initiated surveys in phase 1 were comparable to those in phase 2 (caregivers: 29.5% versus 36.2%; patients: 23.5% versus 22.2%). Most responses during phase 2—63.8% of caregivers and 77.8% of patients—were obtained through personalized outreach, though the preferred survey completion modality differed between caregivers and patients. Most caregivers (n = 73, 90.1%) completed the survey in real time during a phone call with the OC. In contrast, most adolescent patients (n = 34, 60.7%) opted to have the survey link resent for self-administration. In comparing response modes by patient gender during phase 2, male patients self-initiated surveys less often than their female counterparts (n = 4, 10.5% versus n = 12, 35.3%); however, after outreach, male and female patients had proportionally similar preferences for staff- versus self-administered completion with 38.2% of male patients (n = 13) and 40.9% of female patients (n = 9) opting for staff-administration. Observing differences in response mode by patients’ racial identity was limited by the degree of homogeneity in these characteristics within the sample. Notably, among Hispanic patients, nearly all (n = 8, 88.8%) completed the HVIP-CSQ following outreach. Real-time survey administration between the OC and the respondent lasted approximately 5 minutes; similarly, self-administration lasted an average of 5 minutes for caregivers and 3 minutes for youth.

Table 2.

Response Modes in Phase 2 by Respondent Group and Demographic Characteristics

Caregiver Respondents (n = 127)	Response Mode
	Self-initiated and Administered	Staff-administered during Outreach	Self-administered after Outreach
Caregivers overall, n (%)	46 (36.2)	73 (57.5)	8 (6.3)
Caregiver Sex, n (%)
Female	43 (35.5)	70 (57.9)	8 (6.6)
Male	3 (50.0)	3 (50.0)	—
Patient Respondents (n = 72)		Response Mode
	Self-initiated and Administered	Staff-administered during Outreach	Self-administered after Outreach
Patients overall, n (%)	16 (22.2)	22 (30.6)	34 (47.2)
Sex, n (%)
Female	12 (35.3)	9 (26.5)	13 (38.2)
Male	4 (10.5)	13 (34.2)	21 (55.2)
Race, n (%)
Black/African American	16 (26.7)	16 (26.7)	28 (46.7)
White	—	3 (33.3)	6 (66.7)
Other/multiracial	—	3 (100)	—
Ethnicity, n (%)
Hispanic	1 (11.1)	4 (44.4)	4 (44.4)
Age, y (M, SD)	15.19, 1.28	15.09, 1.60	14.65, 1.82

DISCUSSION

This QI project demonstrates how personalized telephone outreach can supplement automated collection of patient-reported quality-of-care and outcome surveys to enhance completion rates while moderating resource use (eg, staff effort). During the project, caregiver and adolescent patient responses to a postprogram satisfaction questionnaire increased dramatically—from fewer than one-third of caregivers and one-quarter of patients when survey delivery was entirely automated to more than three-quarters of caregivers and one-half of patients after integrating personalized outreach and assisted survey completion. The ability to sustain response rates over time is foundational to ongoing monitoring of caregiver- and patient-reported program satisfaction and short-term outcomes. By maintaining robust response rates, the HVIP is positioned to monitor whether services are responsive to families’ needs and routinely incorporate families’ perspectives into continuous program evaluation and improvement efforts. These data are essential to overcoming potential limitations or biases of data collected through self-directed processes alone.

Suboptimal response rates for patient satisfaction surveys, particularly those administered through mail or web alone, are well documented.²⁸ Results from this QI project highlight the benefit of offering patients multiple survey response modes to increase participation across demographic subgroups. These findings are consistent with prior research that identified assisted or sequential mixed-mode survey administration as key strategies for improving patient survey response rates across age groups, insurance types, and racial identities.^28–31 Such multimodal data collection strategies may be especially valuable when designing practices to address low response rates of potentially marginalized or underrepresented patient populations.^23,32,33 Without these voices, decision-makers risk perpetuating disparities and inequities in healthcare delivery. Previous work has identified other potential solutions that may be coupled with multimodal data collection to further enhance the demographic representativeness of patient surveys. Pairing surveys with hospital discharge,^31,33 translating surveys for non-English speakers,³⁴ reducing survey reading levels,³⁵ and offering nominal monetary incentives²⁸ have all demonstrated some success in improving response rates for hard-to-reach groups in other clinical settings and warrant further exploration.

Tailoring outreach and engagement strategies to respondents’ needs and preferences is critical to overcoming nonresponse. Prior work has documented variations in survey preferences among patient subpopulations.²¹ Given the limited demographic diversity in the CHOP HVIP sample, the project’s findings offer incomplete insight into preferences for administration modalities among specific subpopulations. However, the findings highlighted clear generational preferences in our sample: caregivers more frequently opted for assisted real-time survey completion, whereas adolescent patients more frequently elected to complete self-directed surveys. Due to the limited variability in racial and ethnic identity within our program (consistent with many HVIPs), we are unable to draw inferences regarding differences across racial and ethnic groups and response rates. We believe this is an important area of continued study, namely, ensuring evolving data collection practices contribute to the representativeness of responses and mitigate potential nonresponse bias.

Due to the high-contact nature of our pediatric HVIP case management services, in which managers maintain frequent telephonic and face-to-face contact with patients and caregivers, our program keeps contact information up to date for most participants. Coupled with high rates of program completion—locally, less than 15% of participants are lost to follow-up before program discharge—our robust participant contact facilitates successful postdischarge HVIP-CSQ data collection.³⁶ Although strong ongoing relationships between staff and participants beyond their hospital care are foundational to the design of the HVIP model and intended outcomes,^37–40 continued engagement in HVIP services is a documented challenge in HVIP settings, particularly among participants experiencing housing instability.^41,42 For HVIPs operating in different communities or contexts, we acknowledge that modifications to data collection processes alone are likely insufficient to overcome the complex structural and individual barriers that inhibit participants’ engagement with the program, let alone postprogram data collection. Such complexities have been explored^38,41,43,44 and may continue to be more fully understood using other methodologies beyond brief satisfaction questionnaires (eg, qualitatively).

Limitations

This QI initiative took place within a single, well-established HVIP in a single geographic location, with high levels of patient and caregiver engagement. This context may limit the feasibility of the initiative’s activities or the replicability of the findings in other HVIPs or different community settings. However, this initiative highlights the importance of allocating resources to support and sustain more equitable data collection and, ultimately, data-informed decision-making. Given the project’s narrow focus on improving the systematic and robust collection of the HVIP-CSQ, future work is needed to examine how data collection processes may underlie not only response rates but also the nature of responses themselves. Examining how survey modality (assisted versus self-directed) may be associated with data quality and/or reported satisfaction levels with program services may guide further refinement of data collection processes and data interpretation for ongoing program improvement activities.

CONCLUSIONS

Multimodal data collection processes that infuse higher touch, more personalized outreach into existing self-directed processes may enhance survey completion while balancing the personnel effort required to sustain robust data collection. Beyond the HVIP where these efforts were focused, this QI project provides a model for healthcare and community-based programs to optimize survey response rates. By adopting data collection processes that include both self-directed and assistive modalities, programs may collect data that reflect a fuller range of patient perspectives and be better positioned to evaluate the quality of care and make patient-centered, equitable programmatic decisions.

ACKNOWLEDGMENTS

The authors acknowledge the patients and caregivers who graciously shared their experience and feedback with us.