Engaging Public Health Alumni in the Tracking of Career Trends: Results From a Large-Scale Experiment on Survey Fielding Mode

Abstract

Objective:

We sought to understand the relative impact of fielding mode on response rate among public health alumni.

Methods:

As part of the 2021 Career Trends Survey of alumni from the University of Minnesota School of Public Health, we designed a fielding mode experiment to ascertain whether a paper survey, a postcard with a custom survey link (“postcard push-to-web”), a mobile telephone call or text (mobile), or an email invitation would garner the highest response rates. Invitations were randomly assigned from available contact information.

Results:

Of 8531 alumni invited, 1671 alumni (19.6%) completed the survey. Among the initial fielding modes, the paper survey had the highest response rate (28%), followed by mobile (19%), email (10%), and postcard push-to-web (10%). More robust recent engagement with alumni relations, paper survey invitation or mode switch, and recent graduation were all significantly associated with a higher likelihood of response.

Conclusions:

Paper and mobile invitations had the highest response rates to our survey among public health alumni. Findings from this fielding mode experiment are relevant to schools and programs of public health seeking to capture similar information among their alumni, especially given current trends in investment in the public health workforce.

Unlike almost every other field or discipline in the health sector, public health does not know where its students end up.^1-5 This was not always so; students graduating from the 1950s through the 1980s from schools and programs of public health were tracked systematically through alumni surveys throughout their career.^3,6 While early first-destination studies of public health graduates have been revitalized recently in the United States, ⁷ long-term studies on career trends have not been conducted nationwide since the late 1980s. These surveys were critical in understanding the trajectory of academic public health in the United States, especially in categorizing the shift from alumni being employed in government to outside it. ³ In particular, about 56% of graduates in the 1950s worked in governmental public health as their first job, compared with fewer than 20% in their first job as of 2021.^1,6 Information on where students go after their first job is lacking, and this lack of information may pose a particular challenge to a field that is now seeing substantial investment flowing to public health workforce development.^8-11 To understand what early-, mid-, and late-career public health graduates are doing, the field must ask them. ¹

Substantial changes have emerged in survey methods since alumni studies were routinely conducted in the past century; any national effort to engage public health alumni at a broad scale would necessarily require an updated approach to efforts used in the 1950s through 1980s. Data collection approaches have changed with the advent of electronic data collection, and the number of public health degrees awarded annually (38 000 in 2020 vs 770 in 1960) and the number of schools and programs of public health awarding these degrees (>500 in 2020 vs 14 in 1960) have increased substantially.^12,13

The question of fielding mode and response rate is common in survey research. Using multiple modes of contact can increase response rates and representativeness while decreasing costs in surveys of the general population and increasing response rates and representativeness of professional populations.^14-19 Most often, studies that use more than 1 mode show the impact of having a final form of outreach in a different mode; this mode switch is particularly important in a postcard with a custom survey link (“postcard push-to-web”), in which a final mailing of a paper-and-pencil survey is seen as a critical component. ²⁰

Because the last large-scale national studies of public health alumni were conducted almost 50 years ago, and because fielding mode may be tied to response rate and overall survey utility, we engaged in a single-site, large-scale fielding mode experiment as part of a broader alumni survey effort at the University of Minnesota School of Public Health. Our objective was to assess how best to engage alumni in career trends–type studies, with the hope that our pilot would have utility for other schools and programs of public health engaging in similar surveys or for national efforts.

Methods

We designed the University of Minnesota Career Trends Survey (CTS) to engage all alumni who had active contact information, resided in the United States, and were at least 12 months postgraduation. Of 9524 alumni who were initially included, 707 were later discovered during fielding to have bad contact information or had previously opted out of surveys, and 286 completed the survey outside the fielding time frame, resulting in an adjusted analytic sample of 8531 alumni. The CTS had 4 major domains: employment outcomes, education perceptions, debt, and demographic characteristics. We wrote the survey in English, designed to take 15-20 minutes to complete, and had form-equivalent versions administered through a web-based platform (Qualtrics LLC) and a machine-readable paper survey format (papersurvey.io). The University of Minnesota Institutional Review Board determined this project to be not human subjects research and waived institutional review board review (STUDY00012098).

To better understand how to engage alumni in the CTS, we designed a fielding mode experiment, contingent on availability of contact information. Across the 8531 living alumni in the United States with presumably usable contact information, available contact modes varied substantially: 3146 alumni provided a mobile telephone number, email address, and postal address; 2481 provided postal and email addresses; 884 provided a postal address and mobile telephone number; 91 provided an email address and mobile telephone number; 59 provided an email address only; 84 provided a mobile telephone number only; and 1786 provided a postal address only (eTable 1 in Supplemental Material). Nineteen percent of alumni who graduated in 1990 or earlier provided an email address, postal address, and mobile telephone number, compared with 33% of alumni who graduated during 1991-2010 and 65% of alumni who graduated during 2011-2019.

The experimental design used the same fielding mode (eg, email) for an initial invitation and 2 reminders, spaced approximately 2 weeks apart, with a switch to a different fielding mode for a third and final reminder (eFigure 1 in Supplemental Material). Alumni were randomized, based on contact mode availability, into groups for an invitation and mode switch (eFigure 2 in Supplemental Material). Modes included the following: (1) an email invitation to participate in a web-based survey, (2) a mobile telephone invitation to participate in a web-based survey (with shortened URL), (3) a postcard push-to-web (with shortened URL), and (4) a paper survey with included business reply envelope to cover postage. In partnership with the University of Minnesota Foundation and the school’s alumni relations group, the survey was conducted from January 25 through March 25, 2021.

Analysis

The intention of the fielding design was to compare various methods of survey administration, with a focus on evaluating response rates across the following core characteristics: year of graduation, degree type, and level of alumni engagement with the University of Minnesota alumni relations group. We directly measured all characteristics from survey response or administrative data, with 1 exception: alumni engagement during the previous year was operationalized by the University of Minnesota Foundation as interactions with various alumni activities and participation in University of Minnesota programs or events (eg, sports season ticket holder, museum membership participation) and was stratified into no engagement (32% of alumni), low engagement (37%), medium engagement (23%), and high engagement (7%). The Alumni Foundation determined engagement levels.

For bivariate inferential comparisons, we used the Pearson χ² test and Cochran–Armitage test for trend, and we used logistic regression fit with the dependent variable as survey completion within the study; P < .05 was considered significant. Independent variables included past calendar-year alumni engagement, year of graduation, degree type, the available number of contact modes, the mode associated with the initial survey invitation (and reminders 1 and 2), and the mode associated with reminder 3 (mode switch). The primary research questions studied were: (1) How did initial fielding mode affect survey response rates in a public health alumni career trends survey? and (2) How did a fielding mode switch affect survey response rates?

We were also interested in how respondent behavior related to survey completion on desktop versus mobile devices, because the impact of mobile outreach on survey response has not been well-characterized to date. ²¹ As such, we analyzed completion by mode, conducted partial and nonresponse analysis, and analyzed the operating system that was used. We managed and analyzed data in Stata version 16.1 (StataCorp LLC).

Results

A total of 1671 of 8531 alumni completed the CTS within the fielding experiment’s time frame (19.6% response rate overall; eFigure 2 in Supplemental Material). Response rates varied across alumni demographic characteristics, including year of graduation, degree type, and level of alumni engagement with the University of Minnesota in the calendar year prior to survey fielding (Table). Compared with alumni who graduated before 1980 (342 of 1561; 21.9%), response rates were significantly lower among respondents who graduated during 2001-2010 (292 of 1775; 16.5%; P < .001) and significantly higher among respondents who graduated during 2016-2019 (280 of 1212; 23.1%; P < .001). Alumni of master of public health (MPH) programs had the highest response rates (1056 of 4813; 21.9%), with other degree types having approximately a 16% response rate (P = .01). Previous engagement as an alumnus was highly correlated with survey completion. Compared with the no engagement group (382 of 2774; 13.8%), the high engagement group had a significantly higher response rate (251 of 601; 41.8%; P < .001). The high engagement group represented approximately 7.1% of alumni graduating during 2010-2019, 6.0% of alumni graduating during 1991-2010, and 8.7% of alumni graduating during or before 1990 (P < .001).

Table.

Demographic characteristics of alumni of the University of Minnesota School of Public Health who responded to the University of Minnesota Career Trends Survey, January–March 2021

Characteristic	Total invited, no.	Completed survey, no. (%)
Year of graduation
≤1980	1561	342 (21.9)
1981-1990	1423	286 (20.0)
1991-2000	1386	250 (18.0)
2001-2010	1775	292 (16.5)
2011-2015	1174	221 (18.8)
2016-2019	1212	280 (23.1)
Total	8531	1671 (19.6)
Degree type
Master of health administration	2103	352 (16.7)
Master of public health	4813	1056 (21.9)
Master of science	1303	213 (16.3)
Doctor of philosophy	312	50 (16.0)
Total	8531	1671 (19.6)
Alumni engagement level in previous year a
High	601	251 (41.8)
Medium	1975	528 (26.7)
Low	3182	510 (16.0)
Not engaged	2774	382 (13.8)
Total	8531	1671 (19.6)

^aAlumni engagement during the previous year was operationalized by the University of Minnesota Foundation as interactions with various alumni activities and participation in University of Minnesota programs or events (eg, sports season ticket holder, museum membership participation).

Prior to mode switch, response rates were highest among alumni who received a paper survey by postal mail (574 of 2063; 27.8%), followed by mobile telephone (263 of 1357; 19.4%), email (284 of 2896; 9.8%), and postcard push-to-web (211 of 2215; 9.5%) survey. A mode switch to paper survey garnered an increased response rate of 12%, compared with mobile telephone (2.7%), email (0.8%), and postcard push-to-web (0.4%; Figure 1) survey. Among respondents who initially received email invitations, 22.5% were associated with the initial invitation, 26.9% with reminder 1, 10.3% with reminder 2, and 40.3% with the mode switch. Among mobile invitations, 53.6% of responses were associated with the initial invitation, 23.4% with reminder 1, 12.2% with reminder 2, and 10.9% with the mode switch. Among paper surveys, 39.2% were associated with the initial invitation and reminder 1 (tracked together), 49.6% with reminder 2, and 11.1% with the mode switch. Among the postcard push-to-web invitations, 39.1% of responses were associated with the initial invitation, 22.5% with reminder 1, 21.7% with reminder 2, and 16.6% with the mode switch.

Figure 1.

Response rates of alumni of the University of Minnesota School of Public Health to the University of Minnesota Career Trends Survey, January–March 2021, by fielding mode.

In the logistic regression analysis, with the dependent variable being response to the survey within the study time frame, several independent variables were associated with increased adjusted odds ratios (aORs) of response: high engagement (vs no engagement) with University of Minnesota alumni relations in the previous calendar year (aOR = 6.3; 95% CI, 5.0-8.6; P < .001), paper survey (aOR = 3.4; 95% CI, 2.7-4.4; P < .001) or mode switch (aOR = 1.7; 95% CI, 1.3-2.1; P < .001) as initial invitation mode (vs email), and mobile telephone number (vs email) as initial invitation mode (aOR = 1.7; 95% CI, 1.4-2.2; P < .001) (Figure 2 and eTable 3 in Supplemental Material). Compared with graduates from before 1990, those graduating during 1990-2009 had lower adjusted odds of responding (aOR = 0.8; 95% CI, 0.7-0.9; P = .001). Similarly, compared with MPH graduates, master of health administration (aOR = 0.7; P < .001), master of science (aOR = 0.7; P < .001), and doctor of philosophy (aOR = 0.6; P = .005) graduates all had lower adjusted odds of responding. The number of available contact modes was not significantly associated with survey completion.

Figure 2.

Odds of completion of the University of Minnesota Career Trends Survey by alumni of the University of Minnesota School of Public Health, January–March 2021. The dependent variable was survey completion within 10 weeks for the web-based survey and completion of the paper survey (accounts for mail delays during the survey period). Odds ratios were adjusted for engagement with University of Minnesota alumni relations in the previous calendar year, fielding mode of invite, and fielding mode of switch, graduation year, degree type, and number of contact options (eg, email only, email + mobile telephone, email + mobile telephone + postal mail). Results from adjusted odds ratios from logistic regression were significant at P < .05. Error bars indicate 95% CIs. Constant not shown (adjusted odds ratio = 0.07; 95% CI, 0.04-0.11; P < .001). Abbreviations: MHA, master of health administration; MPH, master of public health; MS, master of science; PhD, doctor of philosophy.

In our analysis of operating system use and partial response by operating system, among partial respondents in the web-based survey platform (n = 150), the largest number of dropouts was associated with initial job information and competency-oriented questions (eFigure 3 and eFigure 4 in Supplemental Material). Multipart questions, matrices, or sections contingent on display logic (eg, contingent on year of graduation) also appeared to have substantial dropoff. Approximately 72% of email-based responses were associated with a desktop system versus a mobile one, as were 76% of postcard push-to-web–based responses, compared with 9% of mobile responses, indicating that 91% of responses associated with mobile invitations had been completed on a mobile device. However, while mobile invitations represented 32% of potential contact points with potential respondents, they represented 63% (94 of 150) of partial or incomplete responses (P < .001). Among participants who received an initial email invitation, 75% did not open the email, although 5% clicked through and 4% completed the survey. Among participants who received an email mode switch, 78% did not open the mode switch mail, 3% clicked through, and 1% ultimately completed the survey.

Discussion

Substantial investment flows into the public health workforce through academic public health workforce grants, scholarship expansion programs, and other opportunities.^4,22-26 With increased investments, graduate outcomes will undoubtedly be of renewed interest to students and funders alike. Especially now, as the number of awarded public health undergraduate degrees has surpassed the number of awarded public health master’s degrees for the first time in the United States, ²⁷ substantial work is needed to track long-term trends among alumni to understand the value of their public health education and the extent to which alumni are contributing to essential public health operations in the United States. However, given the decades-long break in systematic studies of career trends, much is unknown about how best to execute these data collection initiatives. ¹ A recent scoping review of career outcome studies in public health showed a deficiency in standards and comparability across institutions. ²⁸ It is our hope that the University of Minnesota CTS can yield insight into optimal ways to communicate with professional public health alumni with broad relevance to the public health community, provide career-long information on almost 2000 alumni to our school, and promote the need for standardized data collection.

The fielding experiment highlighted in this study shows that even with the advent of electronic survey technology, paper surveys still garner the strongest response rates across all groups of interest. This finding is consistent with previous research that found that highly educated populations, such as physicians, are receptive to or prefer a mailed, self-administered form of a survey over electronic forms.^12,17,18 Work on community-dwelling population studies routinely demonstrate that the mail mode continues to have a higher response rate than the web mode does. ²⁹ In elite populations, such as physicians, there is some indication that receptivity to nonmail forms of data collection is increasing. For example, studies by Beebe et al^17,18 found that the use of an initial mailing of a self-administered form followed by a web survey to nonrespondents provided slightly higher response rates and a more representative sample than one that started with a web survey and ended with a mailed survey. In these studies, the authors observed evidence suggesting that physicians retain mailed surveys longer than they do emailed or other electronic surveys. Specifically, they found that approximately one-quarter of the physicians assigned to the mail/web mode sent in a mailed survey after the web survey was sent. These professionals may delete their email messages right away but hold onto their postal mail a bit longer. At a more general level, other evidence suggests that while response rates have declined over time and across all populations, response rates to mailed surveys have remained relatively stable.^29,30

One expected contribution of this fielding experiment was to assess the utility of inviting alumni via mobile telephone messages to participate in a survey. The finding that mobile telephone invitations garner proportionately more respondents than email or postcard push-to-web invitations indicates a potentially important shift in our understanding of conducting survey research in our study population. Moreover, completion of the survey on mobile devices by 91% of mobile telephone respondents was unexpected, although it is difficult to know how much of this shift is attributable to the COVID-19 pandemic versus other secular trends. This substantial partial response among mobile telephone users does not necessarily indicate a preferred mode of communication but rather that the novelty of text-based outreach captured attention in a way that email or postcard invitations did not. Given the observed declines in survey participation over time, including some of the most expensive and robustly designed federal surveys, ³¹ it is important to continue the exploration of optimal approaches to maximize survey participation, because a low response rate can lead to the introduction of selection bias and reduce external validity. ³²

Limitations

This study had several limitations. First, while the fielding design was intended to randomize invitation and mode switch paths among alumni wherever possible, the design was fundamentally contingent on the availability of contact information from alumni. Because contact paths were consistently available across alumni by graduating year, potential differential nonresponse by mode was possible, and our analytic approaches assessed and attempted to address this bias where possible. Second, because the fielding experiment used alumni engagement pathways at the University of Minnesota, norms and conventions had to be followed that caused a necessary departure from the optimal design. For example, the project team learned after fielding that several hundred addresses were not contacted because they had previously indicated they had opted out of surveys; this, too, introduces the potential for differential nonresponse. A third set of limitations related to accessibility. This survey was available only in English, and little accommodation could be made for disability. While we could enable a manual switch (eg, from paper to email) for the handful of respondents requesting a change in mode, more robust disability accommodations were not available. The effect of such accommodations on survey administration could be studied as a cause of nonresponse in a future study.

Conclusion

Short-term investment from the federal government holds the potential to grow the workforce and the field’s understanding of the public health workforce. ³³ One major area of interest, as outlined by the Institute of Medicine, ⁴ is the role of academic public health in training the next generation of public health practitioners. Yet, research has shown that while public health graduates in the United States enter the workforce across a myriad of sectors including health care and nonprofit organizations, few work for the government. ²⁸

This experiment was conducted primarily so that the field of academic public health might better understand how to effectively reach alumni to capture data on career trends, which will be critical to workforce planning efforts in the United States. Paper surveys continue to garner the highest response rates, although cost and effort are substantial. Email and postcard push-to-web surveys yielded relatively low participation. Although mobile telephone push-to-web outreach had relatively high participation, questions remain as to whether mobile telephone users want to complete these types of surveys on mobile devices. As we consider implications for national efforts or future surveys at schools and programs of public health, tradeoffs between efficiency and response rate likely remain at the fore. Paper surveys garnered substantially higher response rates but also required modest financial outlays and staff time to process responses compared with electronic approaches, which leveraged institutional subscriptions to survey software and largely came in “clean” and required little data management. Given the challenging environment for conducting surveys and consequent low response rates, moving beyond paper surveys is not recommended; even younger alumni responded at proportionately higher rates to these surveys compared with email-based invitations (although mobile performed best in these groups). Increasing alumni engagement in advance of these surveys is also a clear means of improving overall response rates, relative to other strategies.