Synthesizing Research Narratives to Reveal the Big Picture: a CREATE(S) Intervention Modified for Journal Club Improves Undergraduate Science Literacy

ABSTRACT

Communicating science effectively is an essential part of the development of science literacy. Research has shown that introducing primary scientific literature through journal clubs can improve student learning outcomes, including increased scientific knowledge. However, without scaffolding, students can miss more complex aspects of science literacy, including how to analyze and present scientific data. In this study, we apply a modified CREATE(S) process (Concept map the introduction, Read methods and results, Elucidate hypotheses, Analyze data, Think of the next Experiment, and Synthesis map) to improve students’ science literacy skills, specifically their understanding of the process of science and their ability to use narrative synthesis to communicate science. We tested this hypothesis using a retrospective quasi-experimental study design in upper-division undergraduate courses. We compared learning outcomes for CREATES intervention students to those for students who took the same courses before CREATES was introduced. Rubric-guided, direct evidence assessments were used to measure student gains in learning outcomes. Analyses revealed that CREATES intervention students versus the comparison group demonstrated improved ability to interpret and communicate primary literature, especially in the methods, hypotheses, and narrative synthesis learning outcome categories. Through a mixed-methods analysis of a reflection assignment completed by the CREATES intervention group, students reported the synthesis map as the most frequently used step in the process and highly valuable to their learning. Taken together, the study demonstrates how this modified CREATES process can foster scientific literacy development and how it could be applied in science, technology, engineering, and math journal clubs.

INTRODUCTION

Science literacy skills have been receiving increased attention recently, particularly in the context of improving undergraduate science education. Vision and Change underscored the need to focus on competencies such as the ability to apply the process of science, scientific communication, and collaboration (1). Teaching science undergraduates these skills is necessary not only for their success in a variety of careers, but also to produce informed citizens capable of engaging in discourse about science in society (2). Howell and Brossard have put forth that “science literacy is best seen as skill-building that can help people navigate diverse science-related issues across the science information life cycle to avoid misinformation and make informed decisions at the individual and collective level (3).” Additionally, science literacy as a term has been defined throughout the literature in a variety of other ways, including the abilities to interpret and evaluate the credibility of scientific information, engage with scientific literature, “think like a scientist,” understand how scientists do science, and communicate about science, particularly with the public (4–7). Given these broad definitions, for the purpose of this study we chose to use the Program for International Student Assessment (PISA) definition of science literacy, which includes three competencies: to explain phenomena scientifically; to interpret data and evidence scientifically; and to evaluate and design scientific inquiry (2).

One strategy that instructors have used to increase students’ science literacy skills is incorporating primary literature into their classes (8–15). Journal club presentations, in which students read primary scientific literature and present a summary of an article to their peers, add further opportunities for gaining science literacy and science communication skills beyond simply reading and discussing primary literature (16). When students are asked to present a paper, they demonstrate additional skills, such as expressing complex ideas and identifying the essential aspects of the scientific process and findings that need to be conveyed (17–21). Successful communication of a paper through a journal club allows instructors to assess many of the skills associated with scientific literacy; in order to present a scientific article, students need to interpret, contextualize, and evaluate the scientific literature and understand how the knowledge was acquired through the process of science. We previously found that requiring students to give multiple scientific presentations, including a journal club presentation in the context of a course-based undergraduate research experience (CURE), was critical to student learning and development of science process skills (22).

While the potential for supporting students’ science literacy is significant, a journal club, and interpreting primary literature in general, also comes with challenges for both students and instructors (12, 13, 23–26). Instructors have indicated that the time-consuming nature of teaching science process skills and the need for more content coverage can be barriers to implementation of primary literature approaches (26). Students report feeling intimidated, confused by jargon or technical language, and finding it difficult to identify the big picture of primary articles (12, 27). Understanding experimental data, scientific techniques, and drawing or evaluating conclusions have also been identified as challenges (12, 13, 24). Journal clubs have, however, been shown to lessen students' self-reported levels of stress and frustration while reading scientific literature (28).

Journal clubs vary significantly in their implementation, in both the format or level of structure and the population involved, but they tend to be less commonly used in the undergraduate classroom setting, given the difficulties faced by novices discussed above (17, 18, 25, 27–32). Consistent with other reports, we found that our students could demonstrate basic content knowledge gained from primary literature, but many students struggled with higher-order cognitive skills, including analysis, interpretation, and evaluation of the scientific methods and data (33, 34). Additionally, most students needed extensive individual coaching sessions to feel comfortable enough with interpreting a scientific article to present it in a journal club, resulting in a significant time burden on the instructor. Even with coaching from the instructor, students’ presentations tended to lack a clear narrative flow and often demonstrated a limited understanding of the scientific process (e.g., how methods and data connect back to the question or hypothesis), both of which are key components of scientific literacy. Given these challenges, we aimed to implement and assess a structured learning activity that could support undergraduate students’ ability to read, analyze, and explain the narrative of scientific papers in a journal club format.

The CREATES intervention to support development of science literacy skills

With the aim to enhance students' development of their scientific literacy and communication skills in a journal club, a modified version of the CREATE process (Consider, Read, Elucidate the hypotheses, Analyze and interpret the data, and Think of the next Experiment) was implemented as a learning activity for students’ presentation assessments (14, 15). The original journal club assignment was a low-structured process, where students read scientific articles and prepared their presentations on their own, then met with the instructor to discuss the slides before their formal presentation to the class. We reshaped this into a highly structured process that incorporated the CREATE approach for analyzing scientific literature, an activity which has been shown to enhance student skills associated with scientific literacy (15, 35). The CREATE approach to analyzing primary scientific literature improves students' ability to interpret and evaluate data and increases their understanding of the process of research (14, 15, 36–38). By completing CREATE, students actively apply the process of science: they understand questions being asked and can develop and test hypotheses, analyze figures, and draw conclusions (14, 15, 36, 37, 39, 40). These science process skills are key facets of scientific literacy (2, 13). The popularity of CREATE has led to a variety of modifications of the approach, many of which have been shown to maintain the positive student outcomes of the original method (36–38, 41, 42).

Although the traditional CREATE approach does include opportunities for discussion of the primary literature, presentation of the entire paper by individual students through a journal club is not part of the established method. Therefore, we modified the process to meet our needs, reordering or separating several of the steps (Fig. 1). We refer to our modification as CREATE(S), or simply CREATES, to emphasize to students the significance of the detailed concept map they create at the end of the process (the Synthesis map step), which they use to develop a narrative flow for their journal club presentation. Other scholars have implemented versions of synthesis maps as tools in other contexts to help students develop a narrative organizational framework of complex topics (43–46). CREATES was implemented to guide students’ conceptualization of the overarching narrative of an entire project, supporting the scientific literacy concept of understanding and contextualizing how the process of science can lead to new scientific knowledge.

FIG 1.

The CREATES process.

Our synthesis map step occurs after the TE step (Think of the next Experiment) rather than as part of the Analyze step in the original implementation (14). This order change was intentional, as it allowed students to use the work done on prior CREATE steps to synthesize the entire paper in a detailed visual concept map format, with a focus on making connections to understand the flow of hypotheses, experiments, results, and next steps (Fig. 1). The students then use their synthesis map as an outline to create their journal club presentation, which replaces the final steps in the original CREATE process. Our intention is that having students create this map by hand will support their synthesis of a research narrative as they process content, while also drawing out and creating diagrammatic linkages between concepts (47). This final step in the CREATES process aligns with our definition of scientific literacy, where students are asked to interpret, evaluate, and communicate scientific inquiry.

For our intervention, we first introduced CREATES in a scaffolded manner to ensure that students were familiar with the process and could effectively apply it to the primary literature. We started by modeling each step of the CREATES process in the first week of class, and the students worked through a practice article together in groups of 3 or 4. The students received extensive formative feedback on each component of the CREATES process from the instructor, with a focus on helping students develop a narrative synthesis map for the practice article. Journal club presentations took place over roughly weeks 3 to 7 of the 10-week quarter and began with individual students choosing a primary article with guidance from the instructor. They then independently completed the CREATES process.

Students met with the instructor 1 week before their journal club presentation to get feedback on their CREATES materials and discuss questions about the article. The format of these 1-h sessions was for students to walk the instructor through their synthesis map, referring to the other steps’ materials as needed, in order to explain the scientific narrative of the article (example synthesis maps are provided in Text S1). The students then used their synthesis map as an outline to create a journal club presentation using PowerPoint, with an additional optional 30-min meeting with the instructor to get feedback on their presentation slides. The students then delivered their 20-min journal club presentation to the class, followed by 5 min of questions and answers with their peers.

We hypothesized that applying this modified CREATES process as an activity to support students' journal club presentation assessments would improve science literacy skills as demonstrated by their ability to interpret and communicate a scientific paper. Our research aims for this project were to (i) assess how the CREATES process impacted student learning outcomes (SLOs) in journal club presentations, and (ii) determine which steps of the CREATES intervention the students perceived to be the most valuable to their learning. We tested our hypothesis by mapping each of the CREATES steps to lower- and higher-order cognitive skill (LOCS and HOCS)–categorized SLOs in the journal club assignment and then used rubric-guided assessment of presentation slides to gather direct evidence of learning gains. We also used student reflections to examine which individual steps of the CREATES process students found to be the most useful for interpreting primary literature. To our knowledge, this is the first study to apply a modified CREATE process to journal club presentations, as well as the first study to assess students’ perceptions of how individual steps in the CREATE process support their learning.

METHODS

To test our hypotheses, data were collected in the context of the second term of two different upper-division two-quarter CUREs in the Microbiology, Immunology, and Molecular Genetics Department at UCLA (previously described in detail by Shapiro et al. [22]). Data for the study were collected from 2013 to 2015, and the instructor of record for the courses in this study (J.M.P.) remained consistent over the entirety of the study.

We used a retrospective quasi-experimental study design to explore the impact of CREATES on students' science literacy skills. The retroactive comparison group in this study included three cohorts of students from the terms just prior to the introduction of the CREATES in Fall 2014, and the intervention group consisted of two cohorts of students from Fall 2014 and Spring 2015. Both groups received virtually the same guidelines for the journal club assignments and had mandatory one-on-one preparatory sessions with the instructor. For the comparison group, the one-on-one sessions to review the article were very instructor-centered, with the instructor either answering questions or, more frequently, walking the student through the article and explaining the experiments and results. The CREATES intervention preparatory sessions were more student-centered, with the student using their synthesis map to walk the instructor through the article and the instructor asking the student questions.

Table 1 provides a summary of demographic information for the students included in this study, with 196 study participants between the comparison (N = 87) and intervention groups (N = 109).

TABLE 1.

Student demographics

Variable	% in comparison group (N = 87)	% in intervention group (N = 109)
Year in school
2nd Year	0.0	0.9
3rd Year	37.9	71.6
4th Year	58.6	24.8
5+ Years	3.4	2.7
Pell grant recipient	40.2	39.4
URGa	10.3	21.2*
Female	52.9	54.1
Transfer	29.9*	17.4
Cumulative college GPA	3.27	3.28

^aURG, underrepresented racial or ethnic group (American Indian or Alaskan Native, Black or African American, and Latina/o/x). *, P < 0.05.

Ethics statement

All elements of this research were approved by the Institutional Review Board (protocol 10-000904).

Assessing Research Aim 1: How does the CREATES process impact student learning outcomes in journal club presentations?

To directly measure the impact of the CREATES process on SLOs, we collected and analyzed data from students’ individual journal club presentation slides. This embedded course assessment was selected as a data source because it allowed the research team to assess science literacy learning outcomes related to interpreting and communicating science literature and was a consistent assignment completed by students both before and after CREATES was implemented.

Direct evidence assessment rubric development

A five-point rubric was developed to assess the presentation slides, with the SLOs grouped by six components roughly corresponding to the steps of the CREATES process (Table 2). The rubric was originally developed as a tool to both provide guidelines for students and facilitate grading of the individual journal club presentations The full rubric is available as Text S2. Prior to assessment of the presentations, two researchers (J.M.P. and E.C.G.) independently assigned a level of Bloom’s taxonomy to each of the rubric items using a framework based on the Blooming Biology Tool (33, 34). Researchers compared their independent Bloom’s assignments and discussed any discrepancies to reach consensus on the final Bloom’s level assignment. Rubric items were additionally assigned lower-order or higher-order cognitive skill labels, with “remember” and “understand” automatically designated LOCS, “analyze,” “evaluate,” and “create” designated HOCS, and “apply” items assigned to either LOCS or HOCS, depending on the perceived level of complexity for each item (Table 2). The rubric was designed to evaluate only the presentation slides; therefore, no audio or video presentation recordings were used in this analysis.

TABLE 2.

Student learning outcomes and Bloom's levels

Category	SLO	Bloom category		Example indicator
		Level	Activity
C, Concept map the introduction	Describe background information that is relevant to the project	LOCS	Understand	Concisely summarize information about the organism(s) or experimental system
		HOCS	Analyze	Make an obvious connection between the research question, hypothesis and background information
		HOCS	Analyze	Extensively incorporate background information describing the history of discovery and the establishment of techniques or research strategies used in the project (i.e., what has been done in the field? How did the discipline get here?)
		LOCS	Apply	References are cited for all background information
	Discuss big picture research questions being addressed by the project	LOCS	Understand	Clearly state research questions or problems
		HOCS	Analyze	Establish significance of the research question or problem (i.e., why is it important? Why are scientists motivated to study the research question?)
		HOCS	Analyze	Address broader impacts of the project by showing relevance of the research question or problem (i.e., how would answering this question or addressing this problem potentially impact modern science? Society? Our daily lives? Who benefits?)
R, Read and diagram methods	Summarize specific aims (experimental approaches) of the article	LOCS	Understand	Show clear list of aims or experimental approaches designed to test hypothesis or address research question
		HOCS	Analyze	Describe experimental approaches designed to address specific aims or hypotheses, including diagrams, flowcharts, or illustrations (break down steps in project plan, incorporating exptl detail for key steps)
	Explain why an experimental approach is appropriate for addressing the specific aims or hypotheses of project (rationale for methods)	HOCS	Evaluate	Cite examples from the literature as evidence
		HOCS	Analyze	State expected outcomes (results)
		HOCS	Analyze	Relate how an outcome (result) could address a hypothesis or model
		LOCS	Understand	Explain the theoretical basis of techniques or technology used in the article (i.e., appropriate background for potentially unfamiliar techniques)
	Critically think through experimental approaches	HOCS	Evaluate	Discuss procedural sources of bias or methodological limitations in detail
		HOCS	Evaluate	Acknowledge the challenges associated with all experimental protocols
E, Elucidate hypotheses	State project goal(s)	LOCS	Understand	Clearly state the overall goal of the project as it relates to the research question or problem
		LOCS	Understand	Give quantitative indication of project scope (i.e., the no. of genes, organisms, or assays under study are indicated)
		LOCS	Understand	Give qualitative indication of project scope (i.e., overall experimental approach; experimental flowchart)
	Rephrase the article's specific hypotheses	HOCS	Analyze	Hypothesis explicitly stated (well thought out, highly developed, engaging and interesting; not confused with specific aims)
		LOCS	Understand	Predict outcomes if results support the hypothesis. (For “If… Then…” statement, “then” is identified correctly as the expected outcome)
		LOCS	Apply	Use literature as rationale
A, Analysis	Analyze data	HOCS	Analyze	Show relevant data that addresses the hypothesis(es)
		LOCS	Apply	Figures and graphs are titled and axes labeled clearly
		HOCS	Analyze, evaluate	Results are presented with sufficient detail (i.e., controls, experimental conditions, key comparisons, etc., are indicated)
	Interpret and discuss results	LOCS	Understand	Concisely and insightfully summarize trends or patterns from graph or table
		HOCS	Evaluate	Present key results; do not focus on extraneous data
		HOCS	Analyze	State relationship between data and controls, standard thresholds, and/or statistical significance
TE, Think of the next Experiment and significance	Understand significance of results	LOCS	Understand	Summarize key results. Is there a “take-home message”?
		HOCS	Evaluate	Relate the results to the original hypothesis (support? refute?) or research questions
		HOCS	Evaluate	Recognize discoveries as novel or results as interesting or point out results support previous findings
		HOCS	Create	Generate future directions for research; future directions are reasonable and sufficiently detailed, using creative license
S, Synthesize a narrative	Create a cohesive presentation	LOCS	Understand	Entire presentation has the same style (writing and graphics), format, tone, and organization (consistent look or flow)
	Develop a narrative to connect scientific ideas into a cohesive story	HOCS	Analyze	Background has a narrative flow connecting the big picture to specific aims or hypotheses
		HOCS	Analyze	Content well-organized, concise, and presented in logical progression
		HOCS	Analyze	Transitions are used between key ideas or results. What do we know? What is the next question?

Journal club presentation data and analysis

We randomly selected 48 journal club presentations from students in the BL courses, including 23 presentations from students who had received the CREATES intervention and 25 presentations from the comparison group. To analyze whether these selected journal club presentations were representative of the larger courses, we compared the grades assigned to the slides from our randomly selected presentation to the grades assigned to slides not selected for assessment in this study, and we found no significant differences (t = 0.71, P > 0.05). Presentations were randomized and stripped of all identifying information. For consistency, all the presentations were assessed by a single researcher (E.C.G.), who was trained to use the rubric through a round of practice scoring and discussion of two presentations with another member of the research team (J.M.P.). The researcher routinely checked for internal consistency by rescoring presentations; in total, 10% of the presentations were scored twice, allowing the researcher to confirm that their interpretation and application of the scoring rubric were consistent throughout the process.

Welch’s two-sample t tests were used to compare scores between students in the intervention and comparison groups on journal club presentations for each of the six SLOs, as well as to compare scores on LOCS items, HOCS items, and all items combined. We also calculated effect size using Cohen’s d to evaluate the scale of the impact of the intervention, by considering the standardized mean difference in scores between the intervention and comparison groups. While t tests reveal statistical significance, understanding the scale of the difference an intervention makes is equally, if not more, useful in determining the quality of an intervention (48, 49).

Assessing Research Aim 2: Which aspects of CREATES do students use the most and perceive to be most valuable?

Research Aim 2 assessed which steps in the CREATES process students were most likely to use extensively and which steps they perceived to be the most valuable. The embedded course assignment included both qualitative and quantitative data gathered via students’ written reflections after they completed their journal club presentations. Students were asked to reflect on the extent of their completion of each component of the process on a Likert-like scale of 1 to 5 (with 1 = didn’t use, 5 = used extensively). Then, follow-up open-ended questions asked students to explain how or why they selected their use scores and what aspects of the process were the most valuable. This mixed-methods approach was useful to both the researchers and the students. It allowed the researchers to quantify the extent the students used the steps, while the open-ended reflections provided the research team with details on what specific aspects of the CREATES process students found most useful. This assignment also provided students the opportunity to develop their metacognition by reflecting on what they have learned through the CREATES process, how they applied it, and what use it may be to their work in the future (50–52).

Two methods were used to analyze these data. First, summary statistics were calculated for Likert-like data to provide a quantitative assessment of which aspects of the CREATES process students reported using the most. Student’s open-ended responses were analyzed using Dedoose qualitative coding software (https://dedoose.com/). Coding was guided by existing literature on the CREATES framework (14, 15) and included both a priori and emergent codes. Two researchers (C.S. and B.T.) independently coded and discussed the same set of responses to establish an initial codebook for categories within the CREATES framework (53, 54). As an example, for the “Reading the Methods and Results” component of the framework, researchers used a set of a priori codes from the literature for strategies within this category, including (i) drawing or cartoon method, (ii) annotating figures, and (iii) rewriting or paraphrasing titles. A set of subcodes was then established based on student comments about the usefulness of this component of framework, including (i) helpful for meta-analysis, developing overall narrative, (ii) helpful in creating their presentation, or (iii) not useful. Codes were continually added, combined, and refined using a constant comparative method to establish a final set of codes and subcodes for categories in the framework that most accurately captured students’ comments (53). Strong interrater reliability was established by testing the codebook in Dedoose (Cohen’s kappa = 0.80) (55). The final codebook is included as Text S3.

RESULTS

The CREATES journal club intervention improves science literacy learning outcomes

We conducted a series of t tests to understand in what ways the CREATES intervention resulted in a statistically significant difference on students’ journal club assessment scores. We first conducted three tests to understand if individual scores varied for students on all items scored for the assessment, on items specifically designed to measure LOCS and on items specifically designed to measure HOCS. Students who received the CREATES intervention had significantly higher average mean scores on all items combined than students in the comparison group (t = −2.19, df = 41.18, P = 0.034) (Fig. 2A, Table 3). When items were disaggregated by LOCS and HOCS, we found that only student scores for HOCS items were statistically significant (t = 2.25, df = 41.02, P = 0.03) (Fig. 2C, Table 3). In addition to establishing statistical significance, we used effect sizes to evaluate whether these differences were practically meaningful. There was a medium effect size for all three comparisons (all items together, only LOCS items, or only HOCS items; Cohen’s d = 0.64, 0.52, and 0.65, respectively). The larger effect size seen for HOCS items compared to LOCS items suggested that the CREATES intervention may have a stronger impact on students’ higher-order cognitive skills.

FIG 2.

The CREATES intervention significantly increased students’ scores across all items and for items measuring higher-order cognitive skills (HOCS), with a medium effect size. Points on these violin plots show individual student scores on all items (A) and disaggregated by LOCS (B) and HOCS (C) items for students in the comparison and intervention group, while the shaded area depicts the density distribution of the student scores. The horizontal line on each plot represents the mean student score for students in the comparison and intervention groups.

TABLE 3.

Journal club presentation scores for the comparison and intervention groups^a

Test	Comparison group mean (SD)	Intervention group mean (SD)	Welch's t test	Cohen's d (effect size)
All items	3.63 (0.46)	3.97 (0.60)	t = −2.19, df = 41.18, P = 0.034	0.64 (medium)
LOCs	4.06 (0.44)	4.29 (0.47)	t = −1.80, df = 44.78, P = 0.078	0.52 (medium)
HOCs	3.27 (0.56)	3.69 (0.73)	t = −2.25, df = 41.02, P = 0.030	0.65 (medium)
Introduction	3.99 (0.66)	3.94 (0.87)	t = 0.22, df = 40.95, P = 0.827	0.06 (minimal)
Hypotheses	3.48 (0.77)	3.95 (0.81)	t = −2.02, df = 45.11, P = 0.050	0.58 (medium)
Methods	2.03 (0.58)	2.75 (0.88)	t = −3031, df = 37.74, P = 0.002	0.96 (large)
Analyses	3.89 (0.70)	4.22 (0.80)	t = −1.52, df = 44.05, P = 0.135	0.44 (small)
Significance	3.85 (0.64)	4.21 (0.77)	t = −1.73, df = 42.74, P = 0.090	0.50 (medium)
Narrative synthesis	3.53 (0.80)	4.11 (0.93)	t = −2.30, df = 43.81, P = 0.026	0.67 (medium)

^aWelch's t test and Cohen's d both assumed unequal variance in the samples. Cohen (48) recommends effect size values of 0.2, 0.5, and 0.8 be considered “small,” “medium,” and “large,” respectively, for Cohen's d.

We additionally conducted t tests on items disaggregated by SLO, to understand which aspects of the journal club presentation resulted in statistically significant increases with the CREATES intervention (Fig. 3, Table 3). We found that students who received the CREATES intervention scored significantly higher on their ability to explain the methods used in the study (t = 3.31, df = 37.74, P = 0.002), with a large effect size (d = 0.96) (Fig. 3C; Table 3). Students receiving the CREATES intervention also scored significantly higher in their ability to clearly identify the study hypothesis (t = 2.02, df = 45.11, P = 0.049) (Fig. 3B; Table 3) and to convey the overall narrative synthesis of the research study (t = 2.30, df = 43.81, P = 0.026) (Fig. 3F; Table 3). These are both considered to be medium effect sizes, with Cohen’s d values of 0.58 and 0.67 for SLOS related to study hypotheses and narrative synthesis, respectively (Table 3). Though not statistically significant, the intervention group also scored higher in their interpretation of the significance of the research study that they described in their journal club presentations, and this difference represented a medium effect size (d = 0.50) (Table 3).

FIG 3.

The CREATES intervention significantly increased students’ ability to convey hypotheses, methods, and narrative, with differences that correspond to medium to large effect sizes. Points on these violin plots show individual student scores for each learning outcome (A-F) for students in the comparison and intervention group, while the shaded area depicts the density distribution of the student scores. The horizontal line on each plot represents the mean student score for students in the comparison and intervention groups.

The CREATES process helps students interpret and present primary literature

We used a post-journal club reflection assessment to identify which steps of the CREATES process the students used the most and found to be the most valuable. Students indicated their level of completion of each step on a Likert-like scale of 1 to 5 (1 = didn’t use, 5 = used extensively) and answered open-ended questions prompting them to explain their responses. Table 4 highlights the relatively high levels of completion for nearly all the CREATES steps in preparation for their journal club presentations. Based on this scale, those steps with the highest completion rates (M of >3.50) were included for further analysis of students’ open-ended responses (N = 59): Read the methods and results, Elucidate hypotheses, Think of the next Experiments, and the Synthesis map (Table 4, bolded steps). Text S3 provides an overview of the final codebook and student responses.

TABLE 4.

Level of completion of CREATES process steps

Stepa	No. of respondents	Meanb	SD
C, Concept map the introduction	74	3.49	1.46
R, Read methods and results	74	3.65	1.10
E, Elucidate hypotheses	74	3.96	1.10
A, Analysis templates	73	2.79	1.49
TE, Think of the next experiments	74	3.63	1.18
S, Synthesis map	74	4.09	1.26

^aSteps shown in boldface indicate CREATES steps that a majority of students found to be useful.

^bMean score on a Likert-like scale of 1 to 5 for the level of completion of each step (1 = didn't use, 5 = used extensively).

Reading methods and results helped students better understand scientific methods

Nearly 90% of respondents (N = 53) shared insights about how they engaged with the reading methods and results step of the CREATES process. Coded responses indicated that students used various strategies outlined in this step, including annotating figures, drawing or cartooning the methods, and writing their own titles. More than 60% of respondents indicated that at least one of the strategies in the read step helped them break down the experiments in their scientific paper and better understand the methods and results for each. As one student commented, “I found this to be very helpful in breaking down what was happening in each figure. Oftentimes when I read scientific papers I get overwhelmed and intimidated by the figures, but this helped.” Another said, “I was able to better understand the purpose and conclusions of the figures by translating captions/legends into my own words and annotating the most important parts of the graphs.” These activities aligned with key components of scientific literacy that give students the opportunity to engage in primary scientific literature by examining scientific methods and better understanding the process of scientific inquiry (2).

Elucidating hypotheses encourages examination of methods and meta-analysis

Among students who commented on their use of the elucidate the hypotheses step (N = 54), more than 81% (N = 44) indicated that this step in the CREATES process helped them to either (i) better understand the methods of each experiment in the paper or (ii) identify and articulate the overarching narrative of the study, and nearly half of these students reported that it supported both (N = 21). As previously stated, the elucidate the hypotheses step encourages students to consider why the researcher conducted their experiments, what questions they were trying to answer, and what were the anticipated outcomes. One student stated, “This portion of the process was hugely successful for me as it helped me frame the importance of each experiment done in the paper. Each individual experiment had a particular focus and reason behind it and writing a hypothesis for each really helped map out the narrative of what the researchers were trying to explore.” Another student commented, “It was very helpful to elucidate the hypothesis for each experiment because it helped me understand why the researchers were testing or comparing certain things and helped me tie it back to the main hypotheses for the paper.” By completing this step, students had the opportunity to explore the mindset of a scientist to understand the process and goals of their work.

Thinking of the next Experiment led to deeper examination of research

In the fifth step of the CREATES process, students are asked to identify any questions and critiques they have about the experiments in the study and to consider how researchers might go about addressing them. A majority of responses (N = 36) included general comments about the ways that the Think of the next Experiment (TE) step helped students identify future directions for the research, while more than a quarter talked specifically about how they designed new experiments. Forty percent of respondents (N = 20) indicated that this step helped them to make connections between experiments and to consider the broader implications of the research, while more than a quarter (N = 13) talked about how this step helped them develop their final presentation. The following student comment exemplifies a number of these themes:

“Formation of new experiment ideas is quite important for understanding the significance and shortcomings of each research article. By trying to come up with our own future experiments, we were put in a position where we were forced to understand the big picture, what needed to be done to fully understand the topic, understand what the paper had not gone over, and what is still left uncovered. I thought this portion of the CREATES process made sure everything came together in the end.”

This step in the CREATES process pushed students to use their own skills and knowledge as scientists to critique the study and consider next steps to further the research.

The final Synthesis map clarified the big picture and presentation

The two major themes that emerged from student comments about the synthesis map step (N = 55) focused on the ways that this component of the process helped them to (i) look at the big picture of their research article (N = 30) and (ii) prepare for their journal club presentation (N = 31). As previously stated, the goal of this new and final step in the CREATES process is to weave together the various components of their research article and map out a clear scientific narrative. One-third of student respondents who commented on the synthesis map step (N = 18) indicated that working through this process helped them with both developing a strong understanding of the big picture of the study as well as preparing for their final presentation. One student commented, “This [step] prompted my ‘ah ha!’ moment in which the whole paper came together for me and I recognized the narrative I would follow in my presentation. This was worth all of the rest of the CREATE process and I cannot stress enough how helpful this was!” Similarly, another student said, “This was by far the most helpful part of the CREATE process. This helped me connect different parts of the paper and aided me in preparing my presentation slides.” Completing this step was an important learning opportunity for students to deepen their understanding of the main points of the research study and effectively communicate scientific information to their peers through the final presentation.

DISCUSSION

In this study, we hypothesized that our modified CREATES process would improve students’ scientific literacy skills, which we assessed through student journal club presentation assignments. We also used student reflections to assess which steps of the CREATES process they used the most extensively in preparation for their presentations and what aspects of the process they found to be the most valuable. We used rubric-guided quantitative assessments to measure student gains in SLOs categorized by six components of scientific presentations and cognitive skills designated lower-order or high-order skills according to Bloom’s taxonomy (33, 34). Our SLOs are representative of common scientific literacy goals (2, 4, 5, 13, 16), including those set forth in our operating definition of science literacy (2). Our SLOs also aligned closely with the science literacy learning objectives identified by Krontiris-Litowitz, who developed a set of homework assignments to teach science literacy skills and a Bloom’s-aligned rubric to assess those skills (13).

A unique aspect of our research is that we individually assessed SLOs mapped to each step of the CREATES process. Students showed significant gains with medium to large effect sizes in the methods, hypotheses, and narrative synthesis SLO categories (Fig. 3; Table 3), which align to key competencies in the PISA definition of scientific literacy (explain phenomena, interpret data, and evaluate and design inquiry) (2). The methods and hypotheses categories assessed students’ abilities to summarize the experimental approaches in an article and evaluate whether they were appropriate to address the research questions of the article, as well as identify and clearly state the goals and hypotheses of the article. The narrative synthesis category assessed students’ abilities to develop a narrative to connect scientific ideas into a cohesive story. These results were expected because they correlated directly with steps in the CREATES process in which intervention students received explicit training but were not emphasized in the comparison group cohorts. Indeed, we found that students reported using the “Concept Map the Introduction” and “Analyze the Data” steps the least (Table 4), and consequently SLOs related to the introduction and analysis were the two areas where we saw the least differences between the comparison and intervention groups (Fig. 3; Table 3). This supported the conclusion that the degree to which students engaged in each of the CREATES steps directly impacted their ability to interpret primary literature. The particularly low use of the analysis step was possibly because students found the analysis templates confusing (56), or that the analysis templates were most useful for novice readers of primary scientific literature (14) while students in the intervention group were able to generate their own analyses after exposure to data analysis throughout the CURE curriculum (57).

We found that students who received the CREATES intervention demonstrated an overall improved ability to interpret and explain primary literature, as measured through their higher mean scores across all SLOs. Additionally, we found that when the individual learning outcomes were disaggregated from their presentation category and analyzed by cognitive skill level, the CREATES intervention had a significant positive impact on students' overall HOCS scores (Fig. 2C). This finding aligns with several previous studies that found the original CREATE method and its adaptations increased students’ critical thinking skills (14, 15, 38, 39).

Student’s written reflections about specific steps in the CREATES process helped illuminate the ways that a more structured and scaffolded approach to reading, analyzing, and presenting primary literature in a journal club can enhance learning and engagement. In alignment with the quantitative analysis, student comments indicated that the CREATES process helped them to break down and interpret the methods and findings presented in the primary literature (R), examine the researchers’ motivations and anticipated outcomes (E), and critically evaluate the study with an eye toward future research (TE). These findings collectively also aligned with students’ open-ended responses from previous studies on CREATE, and specifically the methods “cartooning” step I, as an aspect of the CREATE process students liked best (14, 56, 58). Interestingly, for many students the final step of creating a synthesis map (S) was instrumental in both creating linkages between concepts to better understand the big picture and articulating the overarching narrative of the study in preparation for their journal club presentation. These findings indicate that the CREATES process, and the final synthesis map step in particular, help students to more thoughtfully interpret and critique research methods and procedures, make connections to broader implications of scientific inquiry, and develop a cohesive narrative to share with their peers, all in alignment with the goals of scientific literacy.

Limitations

Given the nature of our retrospective quasi-experimental study, it is important to highlight some limitations. The participants in our comparison and intervention groups were determined by which year students enrolled in the class, and while all students were taught by the same instructor, there could have been differences in the experiences of the students or instructor that we are unable to control for in this work. We found little difference in students’ background and educational experiences between the comparison and intervention groups, with the exceptions that the intervention population included a higher proportion of students from underrepresented racial or ethnic groups and the comparison group had a higher proportion of transfer students (Table 1).

Finally, we need to consider the possibility for both type I (false-positive results) and type II (false-negative results) errors in our findings. By conducting multiple t tests on the same set of data, we increase our likelihood of generating type I error. Common type I error corrections are known to impose a severe penalty on smaller-sample studies, such as our own (49, 59, 60). Using a correction for type I error in studies with limited sample sizes (common among behavioral studies and education research studies) significantly decreases the power of our analyses, which increases the likelihood of type II error, that we would dismiss potentially meaningfully findings (49, 61). Statisticians have proposed that psychological and behavioral studies have previously overemphasized the importance of P values and argued that in some circumstances overcorrecting for type I error can be worse than allowing the possibility of type II error, because it leads us to dismiss potentially meaningful findings in smaller, exploratory studies (48, 49). Given these considerations, and the medium-to-large effect sizes which indicate that the CREATE intervention likely has a meaningful impact on students’ scientific literacy, we chose in this work to emphasize effect sizes while highlighting the potential in our study for both type I and type II error (49, 59).

Considerations for CREATES implementation

One limitation of our intervention is the amount of time that it takes to teach students about the CREATES process. To facilitate this introduction, we (J.M.P.) developed a website (https://uclalibrary.github.io/creates/) in collaboration with the UCLA Library which guides users through CREATES, with detailed instructions and samples of student work. Anecdotally, we find that this website is an excellent resource for both students and faculty. The user-friendly website walks students through CREATES with easy-to-follow steps, and the examples provided reduce student confusion and questions about what is expected from each step.

In our study, students reported that the R, E, TE, and S steps were the most helpful components in preparing for journal club (Table 4). In future iterations of CREATE or CREATES, faculty with limited time may therefore choose to implement some or all of these particular steps and skip the concept map of the introduction and/or the analysis templates. This decision should consider not only class timing but also student familiarity with research literature; first-year students with limited exposure may find the analysis templates to be very helpful.

Conclusions

This study was the first to apply a CREATE(S) modification to scaffold the process of training students to prepare and deliver a journal club presentation and, to our knowledge, the first to quantitatively and qualitatively assess each step of the CREATE(S) process. Students reported that the final synthesis map in particular clarified the big picture and helped them prepare for the presentation. Journal clubs provide an excellent opportunity for students to demonstrate their scientific literacy and scientific communication skills (17, 19–21, 27). An additional necessary skill for students participating in CUREs is to not only understand and interpret the research process behind primary scientific literature, but to also understand and narrate how their own research projects in the CURE follow similar authentic scientific research processes. In the future, we plan to assess whether our CREATES process results in transferable skills when applied to students’ own CURE research projects.