Using Longitudinal Data on Career Outcomes to Promote Improvements and Diversity in Graduate Education

The Ph.D. degree was once considered to be the gateway to academic employment, and degree holders are still considered “Stewards of the Disciplines.” Ph.D. graduates have reached the pinnacle of their field who will train the next generation of scholars and also set the trends within their discipline. However, the reality is that most Ph.D. graduates don’t follow academic career paths. They work in industry, they start companies, they take positions within the government and not-for-profit sector. Smart, critical thinkers with deep training follow many different trajectories after graduation.

Data from the National Institutes of Health (NIH) on biomedical Ph.D. recipients contradict this myth that the majority of our doctoral graduates pursue academic careers (Tilghman et al., 2012). These data show that only 23% biomedical Ph.D. recipients go on to tenure-track faculty positions at research universities and another 20% work at primarily undergraduate institutions with the remaining 57% of all doctoral degree holders pursuing non-academic careers including working in for-profit companies, entrepreneurship roles, government positions, and other pathways.

Even with these data, the debate about the career paths of our doctoral students is often more anecdotal than data-driven, and there is often resistance to accepting this reality from the broader doctoral training community. Most universities have relatively little information about their doctoral alumni and their career paths, especially those that do not involve more traditional activities of research and teaching at institutions of higher education (Allum, Kent, & McCarthy, 2014; Conti & Visentin, 2015; Silva, Des Jarlais, Lindstaedt, Rotman, & Watkins, 2016). Moreover, such data, even when available, are typically spotty, decentralized and short-term, focusing on initial job placements rather than longitudinal success of our alumni.

At Wayne State University, we have undertaken the Student Lifecycle Project to understand the careers of our Ph.D. graduates and their career trajectories over time. It is a story of data aggregation, as well as finding and reconnecting with our alumni. It is also a tale of how universities can tell the story of career success for doctoral degree holders and to inform current and former students on the myriad of careers our students pursue. It also describes the use of these data to inform strategic decision-making across the university.

These data have huge implications for how we design new programs and adapt existing ones to maximize the career success and satisfaction of our students. This requires setting up a unified mechanism to collect, aggregate, and disseminate those data to key stakeholders ranging from undergraduates who are considering graduate training, to faculty, and to funding agencies as well as employers.

Genesis of the Student Lifecycle Project

The Student Lifecycle Project at Wayne State University (WSU) starts with some questions: What would we like to know about our students’ careers and the impact that graduate education had upon their trajectory? Can we ultimately develop data about what students do in graduate school to get a sense of the key elements of graduate training that affects their careers? Do they start with the idea that they are pursuing careers at major research universities, or are they in graduate school to achieve other goals? Teaching at a small college? Working in the pharmaceutical industry?

We need to respect the motivation and intention of our students. Real success is not just placing students in one career track but rather having students feel that they accomplished (or are on track to accomplish) their own goals, whatever those may be. Too narrow a focus on academic reproduction overemphasizes a one-size-fits all career trajectory that might be imposed upon them by faculty mentors more comfortable with the traditional academic pathway than the true career goals of their mentees.

In 2014, the WSU Graduate School set out to understand the longitudinal career trajectories of our doctoral alumni. The project was designed to teach us about the career paths of our former students, but it served several additional goals. The project helped to inform us about the technical resources necessary in terms of both personnel and data structures to track alumni. The project also allowed us to use the data to initiate conversations with our current students about career paths of Ph.D. holders and to use alumni within specific workforce domains to mentor current students with specific career trajectories in mind.

Project Scope and Data Collection

Wayne State University (WSU) is a large urban research university of approximately 27,000 students, including about 7,000 master’s students and 2,000 doctoral students. We graduate approximately 200 Ph.D.s per year from 79 individual programs across 12 schools and colleges. Thus, with the goal of understanding the career outcomes of Ph.D.s awarded over the 15-year window of 1999 – 2014, we were committed to finding approximately 3,000 doctoral degree holders.

Why focus on a 15-year window? Windows that are less than 15 years are far too short as many Ph.D. holders remain in post-doctoral and other temporary training positions for 3–5 years; thus one gets a highly skewed perspective on career outcomes with a shorter window. Recognizing this fact, the National Institutes of Health has also proposed lengthening career tracking to at least 15 years.

Similarly, with the rise of adjunct faculty and short-term employment, it is not uncommon, even for top-tier scholars, to take several positions in the period immediately after graduate school. A 15-year window provides the opportunity for doctoral degree holders to complete their training and enter the non-temporary workforce and for those pursuing academic options to achieve tenure. For those entering non-academic positions, by 15 years post degree they have had the opportunity to firmly establish themselves in a career or even go through several cycles of personal reinvention.

WSU, like many universities, lacked central repositories for career outcomes data. Such knowledge often resides with individual faculty mentors and sometimes in departments. Rarely does the university aggregate such data. Our data collection project therefore entailed several discrete phases and was completed in 18 months from initial inception to public reporting of the findings through university-wide dashboards. Figure 1 illustrates a timeline for the project and the specific program milestones at each stage.

Figure 1.

Timeline for the launch of the alumni census project.

There were three phases to our data collection project:

Phase 1 of the project involved surveying faculty and departments on current information they had about their graduates. Some departments, especially those in the laboratory sciences, were able to effectively identify a large number of their graduates, since faculty develop close rapport with their trainees over years of close mentorship. Other departments with less formal structures often had little or no knowledge of how to find their former students. Over the entire university, about 20% of the students were identified through departmental contacts. The largest gaps occurred when faculty had passed away or otherwise left the university, and thus the informal contact network no longer maintained the information.

Phase 2 data collection, using five person-months of effort, enabled us to identify current employment of nearly 90% of 3,000 doctoral alumni from the 15-year window. However, while contact information was found, only a fraction of those alumni had electronically accessible email addresses in the public domain. One of the most significant hurdles was identifying individuals employed in large organizations and/or foreign companies who often did not post email addresses on the websites associated with their current employment. There are ongoing efforts to fill these gaps by asking colleagues from the same program to help us reconnect with these missing individuals.

Phase 3 involved a direct survey of a subset of our alumni (650/3,000). This survey asked our alumni to validate their current job placements and update them if necessary. They were also asked to answer a series of questions about their career trajectories including information on their first placement, their current placement, the length of time they have been with their current employer as well as their various job titles over time. The data provided a rich view of the career progression.

We also collected data on the perceived value of their Ph.D. and its value relative to Ph.D. graduates from other institutions. Did they work with colleagues who had trained at other universities and did they feel their training at WSU was better than, equal to, or lesser than the training that the other colleagues had received in various categories? Nearly 90% of those who were contacted responded to the survey.

Data Systems

A key element for both the success of a project like this as well as its long-term sustainability is the development and integration of suitable data systems. Most universities have invested heavily in powerful Student Information Systems (SIS). Such repositories house registration and transcript information, human resources files, financial aid data and much more.

These repositories are often less well-structured for the more granular data we wished to collect on current graduate students, such as publications, fellowships, professional presentations at conferences, participation in professional development activities, etc. Therefore, we opted to use a contact customer relationship management platform, Salesforce.com to address these needs. Adapted to accommodate data on all phases of the student lifecycle, the system allows aggregating data on K-12, undergraduate, graduate, post-doctoral, professional and career information in one place.

Integration via application program interfaces (APIs) connects these aggregated data to the SIS to allow mirroring of official university data. This provides opportunities to capture data from diverse external information sources, electronic forms associated with annual student progress review, and survey software. As an academic resource, these data allow us to see the entire pipeline of programming that affects workforce development.

This information is important as we create professional and academic programs to help students prepare for multiple career pathways. This data system has also become an important resource for tracking the long-term outcomes of university affiliated K-12 programs, undergraduate programs that provide a pipeline to graduate school, and ultimately the aggregation and return of de-identified data to graduate programs regarding the collective stories of career impact of their trainees.

Integration with software that seamlessly generates and permits the use of web-based forms reduces the paperwork associated with student management. These web forms can be easily accessed and maintained in the database. For example, web forms can be used if a student wants to change advisors or take his or her qualifying exam. Information from qualifying exams, dissertation committee meetings and the dissertation defense can be provided via the web forms instantly. This information can include any feedback to the student, sign-off by committee members, and automatic notification to the graduate program staff of outcomes before faculty members even leave the exam room. Similarly, a Committee member can join an exam by SKYPE and then simply sign the form electronically, a feature that is especially ideal for online and hybrid programs. Web forms are also used to log into the student record and automatically trigger any requests for follow-ups and reminders to students and advisors regarding required action items.

Closing the Loop – Making Actionable Data Available to Stakeholders

A common criticism of data collection efforts is that information is not shared back with the people who need it. Our goal as part of this process is to ensure that all members of the graduate community have immediate and direct access to the information they need to make data-informed decisions, rather than having to request data from the Graduate School.

This vision has two parts. First, by providing timely and actionable information to the programs, we improve the ground level buy-in of students, faculty, and staff for the data collection efforts. At the same time, we streamline some institutional practices by making them paperless so that students don’t need to worry about physical signatures from faculty across campus or carrying the paperwork to a specific office during working hours. For faculty interested in data for training grant applications or other programmatic needs, information access is dramatically improved. Second, for the Graduate School, granular data on the student experience, both while they are students at WSU and following graduation, can be easily mined such that we can better understand the career trajectories of our alumni.

In 2014, WSU made the preparation of an individual development plan (IDP) mandatory for all Ph.D. students. This allows the Graduate School and individual programs to get a sense of the career directions our students wish to pursue. Paired with the alumni outcomes information, this has allowed the Graduate School to create a Graduate Professional Development Workshop Series that helps our students build the transferrable skills that employers seeks across fields. It also helps students to recognize the transferability of skills they have already mastered as part of their doctoral training.

These data are become indispensible for graduate recruiting as well. Graduate programs have historically used standardized test data (for example the GRE) in a manner that specifically disadvantages some students. The outcomes data we have collected can be used to help illustrate for our faculty the disconnect that exists between such admissions criteria and eventual student success and career outcomes. These local data can thus help create institutional buy-in for policy reform related to efforts such as holistic review of admissions processes.

Finally, by analyzing the career trajectories of our alumni, we can project local trends related to current and future job markets for doctoral degree holders. We know that a large number of our STEM Ph.D. graduates transition to data science as they move into the private sector. This trend was apparent in job data from around 2011 and 2012, and programs that acted quickly to provide the relevant training did a great service for their students while students in programs that did not react to these needs were left largely to their own resources. Adapting training paradigms to better align with the shifting workforce needs is a crucial use of data collection and analyzing efforts.

Of perhaps greater interest is anticipating next trend in job markets. Since it typically takes 4–6 years to earn a Ph.D., the degree programs must project 7–10 years into the future to understand employment trends and position themselves for new opportunities. Graduate programs are traditionally not very nimble, and in most instances beleaguered graduate program directors do not have the time or resources to closely follow economic indicators and employment trends. By using these data and making rapid adjustments to programs, the graduate training community can better prepare highly adaptable graduates who will not only follow trends, but create them, inventing new career paths that are both engaging and exciting.

Telling Stories with the Data

What kinds of stories can we take away from our longitudinal data set that might resonate with various stakeholders? Several pieces surprised us, even if they might not surprise others.

For example, we often perceive doctoral students as being part of a national or international talent pool with great geographic mobility. However, our data showed that over the 15-year window we analyzed, more than 50% of our doctoral alumni remained in the state of Michigan. Recent data from a University of Michigan study substantiated that finding and showed that WSU was not unique (Buffington, Harris, Jones, & Weinberg, 2016; Weinberg et al., 2014). This is especially important for state public institutions to demonstrate to lawmakers the impact of graduates on the state’s economy and to garner state funding to support graduate education.

A second surprise was related to the job titles associated with our doctoral alumni. More than 75% of the doctoral alumni working at primarily undergraduate institutions were in tenured or tenure track positions, or had moved into an administrative role in their institution (Dean, Provost, President, etc.). The success of our alumni in these venues showed far greater career progression relative to those doctoral alumni who took positions at Ph.D. granting institutions. This may be because the timeline for advancement in careers of research university faculty may have more significant post-doctoral training and thus a delayed entry into tenure track positions. This is still a story we can share with our current students that helps shape their career goals.

Expressing the data visually was a goal from the start, and using resources already available we were able to tell stories through a series of dashboards. Specifically, SAS Visual Analytics (VA) is a powerful in-memory processing tool that allows visitors to visually access, explore and analyze data interactively. Designed to go beyond the traditional tabular presentation of the data, VA offers a series of visual techniques with filters and drill-through capabilities so that the stories we tell relate to specific demographic groups of students by college, department, and programs that could not have been told using simple crosstabs. In the future, as we build on to the data set, updates and changes can be made with minimal effort.

For example, Figure 2 depicts a screenshot of a Senke Diagram that shows, by each of our schools and colleges, career trajectories of our 15 year doctoral alumni in the broad employment sectors such as Business, Higher Education in PhD granting institutions, Higher Education in non-PhD granting institutions, Government, Not-for-profit organizations, Hospitals, K12, and Unknown (where data are not publicly available). The data are derived from the alumni census described above. The width of the “ribbon” or pathway relates to the numbers of individuals in that employment/job sector area and therefore changes over time as alumni transition between careers longitudinally. “Mousing over” each ribbon in the live dashboard reveals those numbers in each band. This relationship diagram is therefore a capture of the actual evolving career trajectory of our alumni.

Figure 2.

Senke diagram showing the flow of doctoral alumni from various schools and colleges at Wayne State University to specific employment sectors.

Similarly, Figure 3 shows a screenshot of the highest level of job attainment by our doctoral alumni in the Higher Education employment sector. This includes both PhD granting institutions and non-PhD granting institutions. The data are for all doctoral alumni ranging from zero-15 years past graduation and shows the longitudinal nature of the career progression.

Figure 3.

Data dashboard screenshot showing job titles for WSU doctoral alumni employed in Higher Education (both PhD granting and non-PhD granting institutions).

Data Privacy

Privacy and access control to personal information is obviously a major concern in today’s society. Educational data collected for internal program improvement is no exception. Data retained in these systems must follow the university data retention policy while recognizing that the whole point of this structure is the longitudinal analysis of career outcomes tied to student educational experiences. Therefore, the sunset date on retaining the information in active systems is longer than for many other university data stores. Specific fields within the data set can be tagged for privacy concerns, and tighter restrictions on data visibility can be set by the individual student. Thus, students and alumni can opt out of individual visibility by university personnel but not from aggregate reporting. We can then determine, for example, how many of our alumni have elected to not be contacted by the system for annual employment survey and thus statistically correct our reported data for those omissions.

Future Prospects

The highly granular data on what doctoral students do in graduate school is often ephemeral. The fact that a student attended six workshops on career trajectories or hosted a seminar series within their department is lost to history. Yet, the experiences that develop transferrable skills are exactly the stories that lead to future opportunities and career directions. In real time, we don’t necessarily know which of these experiences will be important and thus the data collection process should happen as seamlessly as possible, allowing analysis at a later time, aggregated and de-identified, to understand the broader trends.

The stories we tell from these data speak to larger issues in graduate education. Specifically, through these statistics we need to give students voice as they articulate their career goals, how they changed over time, and their overall satisfaction with both the training/mentoring they received and the way in which it shaped their career. Students pursue higher degrees for a wide variety of reasons. The real success in graduate education is watching the student set lofty personal, intellectual and professional goals, overcome setbacks and challenges along the way, and ultimately achieve their aims. Using data to improve the outcomes is an opportunity and an obligation that universities owe to themselves and their students.

In Short.

• Data from federal funding agencies demonstrate that the majority of doctoral alumni pursue careers outside academia.

• However, institutions have not tracked the career outcomes of their own doctoral alumni to get an accurate picture of placement of their graduates.

• Lack of data has resulted in continued use of the same training models that may not adequately support the needs of the majority of doctoral students who plan on pursuing careers outside academia.

Biographies

Andrew L. Feig is Professor of Chemistry and Associate Dean, Graduate School. He serves as PI of grants for both his research and educational activities. Together with colleagues from the Cottrell Scholars Collaborative, he co-founded the CSC New Faculty Workshop for Chemistry professors and was the President’s Council for the State Universities of Michigan Professor of the Year in 2015.

Leah Robinson is Director of Academic Support, School of Medicine. She has worked on startup initiatives including the McNair Scholars program, REBUILDetroit, and Wayne Med-Direct. She oversees the programming and curriculum for the premedical post baccalaureate program as well as STEM related pipeline programs

Song Yan is Director of Institutional Research and Analysis. She has worked at Wayne State University since 2010

Mark Byrd is Assistant Vice President for Institutional Research & Analysis. He is a member of the Michigan Association of State Universities Analytics Studies Committee and sits on the Executive Board for the national SAS Users Group for Institutional Researchers. He is a regular contributor to the field as a presenter and reviewer at state and national conferences on topics related to applied institutional research, predictive analytics, visual analytics, data management and assessment

Ambika Mathur is Associate Provost, Office of Scientific Training, Workforce Development and Diversity; and Dean, Graduate School; and Professor of Pediatrics. She served as Associate Dean, the first director of the WSU M.D./Ph.D. program, first director of the Office of Postdoctoral Affairs and is P.I. of both the NIH-funded WSU-BEST Program and the NIH-funded ReBUILDetroit Program that supports the development of underrepresented students who seek to pursue careers in biomedical research