The Annual BUILD Snapshot: Tracking Alumni Outcomes

Abstract

As the number of alumni of the CSULB BUILD Student Training Program continues to grow, it has become vital to develop a systematic way to track each trainee’s graduate school enrollment and persistence. Developing a system that tracks post-graduate outcomes is not only important for determining the success of the program, but it also creates opportunities for the program to continue supporting its former trainees. A major challenge to tracking is that alumni are not very engaged in the process. To address this challenge, we developed the Annual BUILD Snapshot, a personalized unique Excel file designed to collect information on student activities during their time in the BUILD Program and after graduation. In this paper, we describe the development and implementation of the Annual BUILD Snapshot. We also discuss the strategies we used to launch the Snapshot, the administration process, and the outcomes and lessons learned from the process. Our findings have implications for similar training programs that need to track the short-term and long-term outcomes of their students and aim to remain connected to their alumni in unique and creative ways.

1. The BUILD Program

1.1. Goal of the BUILD Program

The CSULB BUilding Infrastructure Leading to Diversity (BUILD) initiative, funded by the National Institutes of Health (NIH), provides an intensive research training program for undergraduate students pursuing a biomedical or behavioral graduate degree [1]. One of the primary goals of the BUILD Program is to introduce students to research and help them enter graduate school, ultimately increasing the diversity of the biomedical workforce. The CSULB BUILD Program is implemented in two distinct funding phases, with BUILD I (2014–2019) which focused on the ramp up and implementation of various institutional, faculty and student programs, and BUILD II (2019–2024) which aims to refine, institutionalize, and disseminate activities developed during BUILD I that are identified to be effective and sustainable beyond the NIH funding period. For student training, this means that we need to determine how successful the program is in providing enriched research training to undergraduate students, especially those from historically underrepresented minority groups, and in making them competitive for Ph.D. programs in biomedical and behavioral science disciplines. Thus, CSULB BUILD tracks its trainees’ undergraduate and post-undergraduate research activities and outcomes. This paper describes the creation and implementation of a personalized unique Excel file designed to collect data on student outcomes during and after BUILD, known as the Annual BUILD Snapshot.

1.2. Importance of Tracking BUILD Alumni

The NIH Diversity Program Consortium includes all ten BUILD Programs across the U.S., which are evaluated at the consortium level to unveil the interventions necessary to help increase the diversity of the biomedical workforce [2]. In addition, each BUILD institution conducts its own local evaluation. This program assessment is based on data from active and former student trainees. Various types of data are gathered from active trainees including indicators of personal background (e.g., gender, race/ethnicity, first-generation status), academic performance (e.g., GPA, GRE scores), research interests and discipline(s), research productivity (e.g., conference presentations and publications), off-campus summer research experience participation, and graduate school outcomes (e.g., application and acceptance).

Alumni are also an important resource as they can provide valuable feedback about how the BUILD Program impacted their postgraduate progress as well as other insights regarding the effectiveness of program components. In fact, undergraduate research training programs are known to miscalculate the time it takes for underrepresented students to be fully prepared for graduate programs; after all, graduation does not equate preparedness [3]. With this in mind, BUILD’s efforts began to focus on its alumni. Maintaining a relationship with alumni comes with major benefits for both parties. The program benefits by obtaining updates on alumni’s post-graduation experiences such as matriculation, research experiences, and career progress. Research experiences may include awards, recognitions, and publications. Career progress can include the completion of milestones in graduate school and beyond. These updates allow programs to update their data for program evaluation and for dissemination efforts. Other lesser known nuances critical to the program’s success may be obtained from alumni such as the time it takes to complete graduate degrees and graduate attrition [4]. Finally, obtaining continued feedback from alumni may help the program make changes to its curriculum [5] and may foster a community for alumni support [6]. Benefits to alumni may include the opportunity to mentor active trainees in the program [7] and participate in networking events such as conferences [6].

Efforts to strengthen a better relationship with alumni were developed (e.g., BUILD Fall Virtual Reunion, BUILD Summer Virtual Reunion). In terms of data collection, BUILD focused on data that could answer important questions related to the success of the program, such as: How many of its trainees entered a graduate program? How many graduated with their Master’s or Ph.D. degrees? What career paths did they pursue? Although data about graduate school admissions were obtained from students upon completion of the program, long-term tracking is key to determining graduate school matriculation and attrition rates as well as late graduate school entry and other matriculation updates (e.g., going from a Master’s to Ph.D.). Another marker of success for BUILD is trainees’ entrance into the biomedical research workforce. Finally, professional achievements such as research publications and awards (e.g., scholarships, fellowships, grants) are evidence that speaks to the program and trainee success.

1.3. Previous Approaches to Alumni Tracking Methods

Obtaining data from active trainees is straightforward because they are in regular communication and contact with the program faculty and staff while they are in the program. In contrast, it is challenging to obtain data from alumni who typically move away from their undergraduate institution and are busy with adjusting to graduate school or other post-baccalaureate work. By the beginning of Phase II, the BUILD trainees’ composition shifted to having more alumni than active trainees. In anticipation of the increasing complexity of data management, due to the growing number of BUILD alumni, a Program Data Management (PDM) team was created at the start of Phase II that would manage all aspects of data collection. As such, the BUILD leadership and the PDM team evaluated the strategies that were previously used to collect data from active trainees and alumni. It is important to note that the PDM team may be unique to the CSULB BUILD Program. Not all research training programs have the resources to form a PDM team. We hope that by describing our previous data collection and how it influenced the development of the Snapshot, other training programs can leverage the lessons learned from our efforts.

Prior to the Snapshot, BUILD staff relied on several informal methods to gather alumni data, such as obtaining updates from faculty mentors who maintained a relationship with their trainees or periodically checking social media (e.g., LinkedIn). However, as the number of alumni began to grow, these methods were too time-consuming to perform and not sustainable to capture alumni outcomes over the long run. By the end of BUILD I, an online survey was designed to have alumni directly fill out the information being tracked. This survey was used to streamline the alumni data collection during BUILD II. In May 2019, the Alumni Qualtrics Survey was administered to 176 BUILD I alumni (i.e., students who completed BUILD as of spring of 2019). Our initial plan was to administer the survey over a two-week window and provide one reminder e-mail a few days before the survey closed. However, we only received 12 responses by the survey closing date. Due to the low response rates, we followed up with each alumnus with the survey link using a personalized e-mail invitation. Although this additional step was time-consuming, 98 alumni (56%) eventually completed the online survey.

While we had a fair response rate, we discovered several limitations with this data collection method. First, we realized that collecting longitudinal data from our BUILD alumni in future years would be far more time-consuming if we continue to send individual e-mails to each alumnus. Second, we realized that our BUILD I alumni were not a homogeneous group, meaning they were completing our program at various points in their undergraduate schooling (see Table 1). Because the BUILD Program consists of a variety of training programs (lower-division vs. upper-division programs), we needed to devise a method that could capture our alumni’s unique post-undergraduate experiences without overwhelming them. For example, the 2019 Alumni Qualtrics Survey included all post-CSULB outcomes (e.g., graduate school items, industry items, career plans, etc.,) and the survey became lengthy because alumni were required to complete all sections of the survey. It was necessary to display all possible outcomes because not all alumni entered graduate school after completing BUILD (e.g., entered multiple graduate programs; took time off school and then entered a graduate program; went straight into industry). With a survey, we also could not communicate to the alumni which sections we needed updates for, so we requested for the alumni to complete all the sections which led to a great deal of redundancy. For that reason, many alumni who participated in this survey reported having had a very frustrating experience. Lastly, the items in this survey were too general (e.g., How many Masters programs did you apply to and get accepted into?) and we ended up with responses that lacked context such as “Applied to zero master programs,” and “Accepted into one master program.” In this example, the BUILD staff was left wondering whether the trainee erroneously entered zero master applications or was rejected from a Ph.D. program but offered admissions to a master’s program instead. These types of data discrepancies led the program staff needing to further communicate with the alumni and/or their mentors via e-mail to fill in the gaps.

Table 1.

Students enter the CSULB BUILD program at different class standings: scholars and fellows divisions are upper divisions. The associates program is a lower division program.

BUILD I & II components/Programs	Entry class standing	Length of division	May transition to an upper division
Associates	Sophomores & Juniors	1-year	Yes
Scholars I & II	Juniors	2-years	No
Fellows	Seniors	1-year	No

Given all the limitations mentioned above, the PDM team identified several factors that can aid the planning of future alumni data collection: (1) Establish or re-establish rapport and transparency with the alumni in regards to what data the program is required to collect for NIH and program evaluation purposes; (2) Reduce survey burden by decreasing the number of repeated inquiries with each data collection attempt and shortening the time that it would take to update trainee data or enter new data into the database; and (3) Introduce the new data collection method to the active trainees so they could become familiar with the process while they are in the program. With these factors in mind, the PDM team developed a unique Excel file for each BUILD trainee (active trainee or alumnus) that included their information (i.e., BUILD Program data and outcomes) pre-populated. The purpose of the individualized Excel file is to serve as a unique data collection tool for active trainees and alumni, allowing them to easily provide new updates, changes, and milestones every year.

The Annual BUILD Snapshot.

The PDM team titled this new data collection method “The Annual BUILD Snapshot.” The Snapshots included specific outcome sections that covered all possible unique paths. This would give our students their own ‘Snapshot’ of their undergraduate experiences and post-BUILD trajectories. With this method of collecting data, our trainees would simply fill in the sections specific to their career paths at the time of Snapshot administration. These personalized Excel files would be stored in a secure data encrypted CSULB Microsoft One-Drive folder. With the Snapshots, the PDM would simply e-mail the active trainees and alumni a link to their individualized Excel file that is pre-populated with their data. We hoped this new data collection method would also help create transparency since students would be able to see the data that is collected and maintained. Furthermore, the use of this shared file may reduce the response burden for alumni because they would be able to update pre-populated data, add new data, and add comments to further clarify any section.

2. Method

2.1. Developing the BUILD Snapshot

BUILD Snapshot Design.

The design of the Snapshot began in the summer of 2020 and lasted a total of two months. This process can be summed up in three steps: (1) identifying the data to be collected; (2) designing and formatting the Snapshot for readability and usability; and (3) piloting and incorporating alumni feedback into the Snapshot template and instructions (discussed in Sect. 2.2).

Identifying Data to be Collected.

The PDM team first identified the program and outcome data needed for the Snapshot. For both phases of the program, the type of data collected was largely guided by the NIH requirements which included two categories: active trainee data and alumni data. The following data elements and outcomes were collected for active trainees and alumni. Refer to Table 2 for an outline of the data collected.

Table 2.

Data elements of the snapshot.

Data elements	Data collection/Verification method
CSULB E-mail & Personal E-mail	E-mails are the main contact source
LinkedIn URL	Alumni are contacted via LinkedIn if their e-mails are no longer active
Standardized Test Scores & CV Link	GRE scores (or official reports) and CVs are obtained via a Qualtrics Survey link
Estimated CSULB Graduation	CSULB graduation estimation is obtained at the application phase
Major while in BUILD	Majors are verified with transcripts
Official CSULB Graduation Date	Official graduation dates are verified with transcripts
Degree-Major and Minor at CSULB Graduation	CSULB degree, major(s), minor(s) are verified with transcripts
List of Undergraduate Research Experiences (URE)	URE Program/location (e.g., OURS, HRPG, BUILD Fellow) and participation date range are collected
List of Post-Bacc/Graduate Programs & Admission Statuses	Information on the first round of post-undergraduate program applications and admission status is obtained. (Subsequent application rounds are not obtained) If no applications are provided, we ask: “Are you planning to apply?” and “If yes, what year?” If trainees are not applying, we ask: “What is the reason?”
Postbaccalaureate/Graduate Program Matriculation and Program Status	All programs students matriculated into are obtained. This includes the name of institution, the field of study, the type of degree/certificate, start date, current status, and graduation date
Academic and Non-Academic Employment	Title, employer, date range are obtained after graduating
Conference Presentations	Presentations (title, type, authors, conference name, and dates) are obtained during BUILD
Publications	Publications (title, published date, authors, publication type, publisher, status) are obtained during BUILD and post-BUILD
Honors/Awards	Awards are obtained during BUILD and post-BUILD
Milestones & Accomplishments	Students are asked to share any milestones and accomplishments that could not be captured by the previous outcomes

In addition to the data needed for the annual reports to the funding agency, the Snapshot gathered data needed for internal evaluation purposes. For example, capturing the reasons trainees did not matriculate to graduate school could help the program improve resources and support. Additionally, the BUILD Program was interested in learning which graduate schools trainees applied to and whether they were accepted during their first application cycle. To capture this, the PDM team created a section in the Snapshot that included the universities and programs the trainees applied to and their application/admission status (e.g., not accepted, accepted and declined, accepted and enrolled). With these data, we will be able to identify trends and examine the acceptance and enrollment rates within our partner institutions (i.e., University of California, Irvine, and University of Southern California).

Designing and Formatting the Snapshot for Readability and Usability.

Using an Excel file that has been designed as a data collection tool is different from using a general spreadsheet. Thus, we developed Snapshot instructions to help the alumni and active trainees understand the layout and functions of the Snapshot. Two challenges that surfaced during the development of the instructions had to do with (1) figuring out whether new or updated data was entered by a participant and (2) telling apart participants who viewed their Snapshots but had no updates from those who have not had a chance to update their Snapshots. To address these concerns, we instructed participants to highlight new data entries or updates. We also instructed them to provide their initials and date of completion at the bottom of the Professional Scholar Activities sheet.

The instructions also helped trainees navigate the Snapshot. An example was the drop-down menu options in some of the Snapshot sections that were denoted with an asterisk in the header cell. The drop-down menus prevented trainees from entering data in their own words and instead forced them to choose an option from the drop-down menu. This was helpful in sections that focused on data that would be updated repeatedly over the years (e.g., graduate school application or matriculation status). A second Excel feature we used was the embedded messages. These messages allowed the PDM team to add specific directions to a particular cell or section. For instance, we embedded a message that read “Please enable pop-ups so you can click on the survey hyperlink.” The survey link would then direct participants to a Qualtrics survey that collected GRE/MCAT scores and CVs. This survey was designed to avoid displaying tests scores in the Snapshot and thereby ensure privacy.

As mentioned above, GRE/MCAT scores were not displayed in the Snapshots because tests scores are protected under FERPA. Program assistants were part of the data entry process which is another reason why the GRE/MCAT scores were obtained via a Qualtrics survey. Lastly, the PDM team incorporated BUILD colors and the Annual BUILD Snapshot logo (see Fig. 2) to make it aesthetically appealing.

Fig. 2.

Annual BUILD snapshot logo

2.2. Snapshot Pilot Test

Once a template of the Snapshot was fully developed, the PDM team pilot tested the Snapshot in mid-September 2020. Four BUILD I alumni were invited to pilot the Snapshot because of their uniquely complex post-graduate experience:

• Person A: Scholar I, Scholar II, and Scholar III Participant

  • Career path: accepted into a Ph.D. program but declined, pursued industry
• Person B: Associate, Scholar I, and Scholar II Participant

  • Career path: entered 2 master programs but left both, pursued industry
• Person C: Scholar I, Scholar II, and Scholar III Participant

  • Career path: entered Ph.D. program and transferred to a different Ph.D. program to continue research with mentor
• Person D: Scholar I and Scholar II Participant

  • Career path: completed master’s program and entered Doctor of Medicine program

To help structure the participants feedback, the PDM team designed a short 9-item Qualtrics survey that inquired about their experiences with navigating the Snapshot sections, usefulness of the examples provided, effectiveness of the guidance provided by the instructions, and likelihood that the alumni would complete the Snapshot annually. Two open-ended items asked for feedback on the specific sheets, Graduate Employment and Professional Scholar Activities. Lastly, one open-ended item asked about the parts of the Snapshot the alumni found challenging to complete. In addition to the survey, the PDM team scheduled a 30-min Zoom feedback meeting with each of the participants. The alumni provided in-depth feedback about the instructions, the organization and formatting of the template, and the template data fields.

Populating the Snapshots.

By early October 2020, the Snapshot template (see Fig. 1) and the instructions were finalized. The next step was to populate the Snapshots for all active trainees and alumni, meaning enter each individual’s data into the Snapshots.

Fig. 1.

Snapshot template illustrating the second sheet “Graduate and Employment Experiences” of a pseudo trainee’s Snapshot.

During this time, the PDM team used an excel file called the “Snapshot Master File” as a tracking system to help organize the individual Snapshots. The first sheet in this Excel file listed the names of the 331 BUILD active and former trainees that were being tracked at the time of the launch and their start date in terms of the BUILD award year. Two columns were created to help organize the trainees in terms of BUILD Phase (1 or 2) and Batches (smaller subset of data for management purposes). The BUILD Phase column consisted of BUILD I Alumni (award years 1–5), BUILD II Alumni (award years 6–10), and whether the student is an active trainee. Each trainee was assigned to their appropriate category. The Batches column assigned a number to trainees from 1–6 to divide them into smaller groups. Note that Award year 6 was a unique grant year as some trainees overlapped between BUILD Phases I and II.

Once this tracking system was set up, the PDM team obtained the existing program data from a database system that the BUILD Program was using called FileMaker® Pro. This database includes BUILD active trainee and alumni outcome data. Please see Table 2 for all the outcome data that were exported to the Snapshots. Upkeeping the existing data in FileMaker® Pro is an important task for the PDM team in order to provide the most updated data files to BUILD leaders and partners who plan on presenting or publishing. With this in mind, it is important to note that all data stored in FileMaker® Pro came from various sources such as the CSULB BUILD application, 2019 Alumni Qualtrics Survey, social media updates, and updates from BUILD faculty members. The Snapshot would be an additional source of data collection that would further update BUILD’s database. The existing data from FileMaker® Pro was used to pre-populated the Snapshots, a process that took about three weeks.

2.3. Snapshot Launch

By early November 2020, all Snapshots were pre-populated and ready for launch. Next, an e-mail was designed to help plan the mass e-mails for all active trainees and alumni. We first e-mailed the Snapshot links to the active trainees because this would allow the PDM team to test and adjust the e-mails before sending the e-mails to the alumni. We used the LISTSERV® Maestro E-mail Marketing Software to send out our e-mails. With this software, the PDM team was able to design an e-mail template with a header, structure the content, attached several attachments, specify students by their names, and include unique Snapshot Excel links. Another important feature of this software was the ability to send out e-mails on behalf of a specific CSULB BUILD Program leader that the students know.

A schedule was set with dates of when the e-mails would be sent. The BUILD Training Directors who had the most contact with the student trainees during their time in the program served as the “influencers” for the alumni e-mail launch. For the active trainees, an announcement was made during their Learning Community class. The Snapshots were scheduled to launch in early November and close in early December. The active trainees would be the first to receive their Snapshots followed by the alumni three days later. The schedule also included three e-mail reminders.

Once the schedule was set, the PDM team worked with the faculty and staff in the BUILD Student Training Program to obtain their insight on the e-mail content. First, they advised the PDM team regarding how to word the introduction in the e-mail launch to make it more sensitive to the students during the period of the COVID-19 pandemic and the Black Lives Matter Movement. The leadership recognized this sensitivity was necessary since some of the alumni just completed the BUILD Program in May of 2019 and there had been minimal alumni contact before this Snapshot launch. The Student Training faculty and staff also suggested emphasizing the purpose of this new data collection method and why BUILD was no longer using its Qualtrics survey. In addition, they indicated that reinforcing the purpose of alumni data collection was essential to include in the e-mail launch. As a result, the following phrase was added: “Keeping track of your progress in graduate school, career, and/or future career plans allows BUILD to modify student training and share your success with BUILD’s current trainees, program faculty and staff, as well as BUILD’s funder the National Institutes of Health (NIH).” During the discussion, the faculty and staff of the Student Training Program also suggested the need for a marketing plan. They believed there needed to be more awareness about the Annual BUILD Snapshot among the trainees since the PDM team was relying on the students’ personal e-mails on file as the only source of communication.

Marketing Campaign.

The BUILD leadership and PDM team decided that the Snapshot would be part of the annual data collection that takes place for active trainees in spring. Introducing the active trainees to the Snapshot before they leave the BUILD Program will help build familiarity with this data collection process. After they complete the program, they can easily update their personalized Snapshot. Considering that we were implementing a new program requirement and that the Annual BUILD Snapshot would serve as the catalyst to engage and re-build the connection and rapport among the CSULB BUILD alumni, we developed a marketing campaign to raise trainees’ awareness of the Snapshot and to reconnect with our alumni.

As part of this marketing campaign, the alumni would be provided with incentives if they participated in the Snapshot. Two incentive plans were implemented to thank the alumni for taking the time to update their Snapshot and increase response rates. For the first incentive, the first 40 alumni who completed their Snapshot received a $5 Amazon e-card. The second incentive was a raffle to win one of four $25 Amazon e-cards for alumni who completed the Snapshot by the deadline of December 9.

The marketing of the Snapshot also involved the creation of an Annual BUILD Snapshot logo (see Fig. 2). We embedded this logo in the Snapshot and all advertisements andpromotional marketing tools, including flyers, infographics, a 6-min video, and social media posts.

Flyers and an infographic (see Fig. 3) were created using Canva (a free online infographic template). The flyers introduced the Annual BUILD Snapshot and its launch and deadline dates, and described the two types of incentives to encourage alumni participation. The infographic provided active trainees and alumni with 3 main steps to help them fill out the Snapshot. Within each step, important sub-instructions were included. The first step focused on the resources (e.g., PDF instructions) we included in the e-mails that would guide participants when updating their Snapshots. The second step highlighted specific Snapshot sections that the PDM team expected to be overlooked by the alumni such as their updated contact information, standardized test scores and CV survey link. One of the sub-instructions directed their attention to the second sheet where additional outcomes could be updated. The third step reminded participants to initial and date at the bottom of the second sheet for the PDM team to recognize that they completed the Snapshot.

Fig. 3.

Annual BUILD snapshot infographic

Social media announcements helped reach out to alumni who were active on such platforms. During late October to early December flyers were posted on the program’s social media platforms (e.g., Instagram, Facebook, and LinkedIn) along with video clips (refer to the Snapshot Video section). We posted twice a week for the first month and reduced the number of posts to once a week for the final two weeks. Lastly, we posted a description that covered the purpose of the Snapshot in our LinkedIn account. In total, we made 17 social media posts (eight on Facebook, eight on Instagram, and one in LinkedIn) and a total of 14 short video clips, seven for each of the program’s Facebook and Instagram accounts, providing an overview and instructions for filling out the Snapshot.

Snapshot Video.

To simplify the Snapshot completion process, a video was created that described the Snapshot with tips to show how to properly complete the main sections. Training Directors of the Student Training Program narrated the video. The content of the video included:

• Introductions of the Training Directors

• Acknowledgment of the current events at that time: COVID-19 Pandemic and outcome of the presidential election

• BUILD’s support of the Black Lives Matter Movement

• Purpose of tracking data during and after completion of the BUILD Program

• Acknowledgement of previous collection of alumni data

• Introduction of the Annual BUILD Snapshot

• Heads-up of the Snapshot e-mail invitation

• Tips for filling out the Snapshot sections

The video was uploaded to the BUILD YouTube channel in mid-October. A link of the video was included within the influencer e-mails to our alumni.

Additional Snapshot Advertisements.

The PDM team advertised the Snapshot in other BUILD’s platforms and events including the alumni webpage where alumni events were posted. A description of the Snapshot with the title “Keep Us Updated!” was also included along with the Snapshot video and a link to the instructions. Another place we advertised was during the BUILD alumni winter event title “BUILD Virtual Reunion and Trivia Night” in early December. The PDM team announced the Snapshot during this event and reminded all attendees to complete the Snapshot.

Post-Snapshot Launch.

After the Annual BUILD Snapshot was launched, the PDM team added to the “Snapshot Master File” two columns, one indicating whether the individual Snapshot was complete and the other if it was incomplete. The Snapshots that were not opened were flagged for the next e-mail reminder date. By early December, the majority of the active trainees had completed the Snapshot, but there were several alumni Snapshots that were not accessed. The PDM team sent out the early bird incentives and the raffle incentive reminder. Next, the PDM team highlighted the incomplete sections in green and then e-mail these alumni asking them to complete the green sections. Alumni who had active LinkedIn accounts were contacted through this platform with similar messages.

By early February, the data collection for the Annual BUILD Snapshot was closed. To thank the alumni for their participation, the PDM team sent all the alumni who had completed or partially completed their Snapshot a short survey asking for their home address in order to receive a reusable BUILD face mask. Finally, the PDM team and one student assistant entered the data obtained from the Snapshots into FileMaker® Pro. An expiration date was added to all the Snapshot links to prevent the Snapshots from being accessed. The Snapshots were cleaned and saved in a new folder for the next administration.

3. Results

3.1. Overall Snapshot Participation

A total of 319 Snapshots were e-mailed to the CSULB BUILD active trainees (N = 58) and alumni (N = 261) in November of 2021 (see Table 3 for a breakdown of participation).

Table 3.

Snapshot participation for CSULB BUILD trainees

BUILD trainees	Did not participated		Participated		Total
	n	%	n	%
BUILD alumni	126	48.28%	135	51.72%	261
Active trainees	6	10.34%	52	89.66%	58
Total	132	41.38%	187	58.62%	319*

Note:

^*Total includes the 4 alumni who participated in the pilot phase

Of the total of 135 alumni who participated in the Snapshot, 52 (38%) had minor incomplete Snapshot sections. Active trainees were required to participate in the Snapshot as part of their Learning Community activities.

3.2. Comparison of Alumni Qualtrics Survey and Snapshot Response Rates

The response rates of the Alumni Qualtrics Survey and the Annual BUILD Snapshot were compared by using the sample of alumni (n = 176) who were previously invited to participate in the Qualtrics survey. This sample only includes BUILD alumni who completed the BUILD Program as of spring of 2019.

Both data collection methods had unique labor-intensive processes. For the Alumni Qualtrics Survey, it was the process of emailing every alumnus a personalized e-mail asking them to complete the survey. As described earlier, the Annual BUILD Snapshot included a targeted campaign that included incentives, e-mail reminders, and social media posts. Both methods produced fairly similar results (see Table 3), and each method required about a 4-month data collection window (see Fig. 4).

Fig. 4.

Participation in the Alumni Qualtrics Survey and the Annual BUILD Snapshot within a four-month period (excluding alumni who participated in the Snapshot pilot phase)

One unique component of the Snapshot campaign is the use of incentives. As seen in Fig. 4, the Snapshot received more than triple the number of responses compared to the Qualtrics survey in the first month. It appears that the incentives may have helped increase the alumni participation in the early weeks, compared to the Qualtrics survey where we had very low participation during the first month.

In Table 4 we provide the response rates for each data collection method with the sample of alumni (n = 176). The Alumni Qualtrics Survey (56%, n = 98) had a higher response rate than the Snapshot (53%, n = 93), but it is important to note that both data collection methods are quite different. A Qualtrics survey may be an easier process compared to the Snapshot as it requires reading instructions and understanding what can be added or updated in each of the sections.

Table 4.

Response rates by participation in the Alumni Qualtrics Survey and/or Annual BUILD Snapshot

2019 Qualtrics vs. 2020 Snapshot Participation
	n	%
2019 Qualtrics only	46	26.14%
2020 Snapshot only	41	23.30%
Both	52	29.55%
Neither	37	21.02%
Total	176	100%

Observing the response rates for the Scholars and Fellows cohorts is important because these upper-division programs provide students with intensive research training. Since the Fellows program started in the fourth year of the BUILD I phase with a small number of students during its first two years, we are not able to compare participation rates by cohorts for this group. Participation rates for the three Scholars cohorts are outlined in Table 5.

Table 5.

Comparing participation in the Qualtrics survey and the Snapshot by the BUILD Scholars Cohort

Scholar cohorts	Participation		Response rate
	Qualtrics	Snapshot	Qualtrics	Snapshot
Cohort 1 (n = 40)	21	22	52.50%	55.00%
Cohort 2 (n = 38)	24	22	63.15%	57.89%
Cohort 3 (n = 35)	12	30	34.28%	85.71%
Total (n = 113)	57	74	50.44%	65.48%

Obtaining data from older alumni cohorts is relatively challenging, however the two data collection methods provided similar response rates for the first and second Scholars cohorts. For the last cohort, the Snapshot response rate of 85.71% is much higher than the 34.28% of the Qualtrics survey. This low response rate for the Qualtrics survey may be explained by timing. The Qualtrics survey was administered right after the third Scholars cohort had completed BUILD. The timing of the Qualtrics survey was probably too soon for this group of alumni since the BUILD Program collects active student data during the spring. Among the Scholars cohorts, the Snapshot has a higher response rate compared to the Qualtrics survey, 65.48% and 50.44% respectively.

3.3. Benefits of Snapshot

Unfortunately, a limitation in our process of comparing the Qualtrics survey to the Snapshot is not being able to quantify or provide the unique quality of data obtained by the Snapshot. Despite this limitation, the response rates suggest that the Snapshot is as successful as a Qualtrics survey when reaching out to alumni. Because of this promising finding, we believe it is important to highlight several potential benefits when using Snapshots as a data collection tool. First, using Snapshots to collect data from both alumni and active trainees does create transparency between the program and trainees. Participants of the Snapshots were able to review the data that was populated in their own Snapshot which included BUILD data and post-BUILD data. We did not receive e-mails from alumni asking us to clarify why we were requesting such data or ensuring we had previously obtained such data from them in the past. These types of inquiries were prominent during the Qualtrics survey administration. We believe this transparency led to better communication between the program and alumni, our second benefit. Communication between the PDM team and the alumni was smooth because the Snapshot allowed participants to include comments in several key outcome sections (e.g., matriculation, honor/awards, publications, etc.,). Lastly, the Snapshot allowed us to capture dynamic data, meaning the unique career paths of our alumni. Because alumni data is nonlinear, having a data collection tool that allows participants to enter new data and update old entries is essential as this may help programs obtain better quality data.

4. Lessons Learned & Recommendations

As other programs have discovered, collecting data from alumni is a challenging task and regardless of the data collection method, it requires a lot of work and planning. With the Qualtrics survey, it was apparent that e-mail reminders alone were not sufficient to connect with the alumni. The alumni were more responsive to personal requests from their faculty mentors and we had to contact the trainees’ mentor to solicit their help with data collection. Going forward with the Snapshot, we knew that the first administration of this new data collection method would require significant effort. We also knew that in order to jumpstart this process, we would have to incentivize completion of the Snapshot and bring awareness by launching a social media campaign. This also took a lot of work, but we were fortunate that the BUILD Program already had a social media presence and platform to post Snapshot information and details. Additionally, reminder e-mails were strategically scheduled.

One discovery that was made during the Snapshot administration was that the BUILD Program was contacting many of the alumni via an old e-mail or a mailbox that the alum did not check often. We suspect that we may not be connecting with many of the alumni because of this problem. Going forward for the next administration of the Snapshot, it would be worth the effort to inquire about any e-mail address changes at alumni events to update our contact list.

Additionally, to increase participation in the Snapshot from all programs in the upcoming years, we may have to be more strategic with our data collection procedures. It would be beneficial to directly e-mail the alumni who have not been in contact with BUILD to help increase participation, instead of including them in the mass e-mail list.

We believe that over time the Snapshot may reduce redundancy in reporting and increase transparency between the program and alumni. We will continue to use the Snapshots for the remaining BUILD II years to evaluate the effectiveness of our program. We believe that programs like CSULB BUILD can benefit from implementing the Snapshot if their trainees enter the program at various undergraduate stages (e.g., Juniors, Seniors, etc.,) because it will streamline the data collection process. An important consideration is the timing needed to develop a template of the Snapshot and an efficient Snapshot tracking system. Once this is set up, the maintenance of the Snapshot is fairly simple. Another time-consuming aspect of the Snapshot is entering the data from the Snapshot into a database system for data storage. Our PDM team entered the data manually from the Snapshots to the database, but this process may be simplified if an Excel code is applied to help generate reports. Lastly, to collect alumni data, program leadership should introduce the alumni data collection methods to active trainees and continue a relationship once trainees become alumni. These efforts will lessen the burden of creating an intensive marketing campaign. The next steps for our program are to continue our long-term alumni tracking efforts and establish engaging alumni events. Over time, we believe the BUILD Program will benefit from the quality of data obtained from the Snapshot. The accuracy and completeness of the Snapshot data will be examined in a future paper.