Interventions for Supporting and Assessing Science Writing Communication: Cases of Asian English Language Learners

Abstract

In seeking to support diversity, one challenge lies in adequately supporting and assessing science cognitions in a writing-intensive Biochemistry laboratory course when highly engaged Asian English language learners (Asian ELLs) struggle to communicate and make novice errors in English. Because they may understand advanced science concepts, but are not being adequately assessed for their deeper scientific understanding, we sought and examined interventions. We hypothesized that inquiry strategies, scaffolded learning through peer evaluation, and individualized tools that build writing communication skills would increase confidence. To assess scientific thinking, Linguistic Inquiry Word Count (LIWC) software measured underlying analytic and cognitive features of writing despite grammatical errors. To determine whether interventions improved student experience or learning outcomes, we investigated a cross-sectional sample of cases within experimental groups (n = 19) using a mixed-methods approach. Overall trends of paired t-tests from Asian ELLs’ pre/post surveys showed gains in six measures of writing confidence, with some statistically significant gains in confidence in writing skill (p=0.025) and in theory (p≤0.05). LIWC scores for Asian ELL and native-English-speaking students were comparable except for increased cognitive scores for Asian ELLs and detectable individual differences. An increase in Asian ELLs’ cognitive scores in spring/summer over fall was observed (p = 0.04), likely as a result of greater cognitive processes with language use, inquiry-related interventions, and peer evaluation. Individual cases further elucidated challenges faced by Asian ELL students. LIWC scores of student writing may be useful in determining underlying understanding. Interventions designed to provide support and strengthen the writing of Asian ELL students may also improve their confidence in writing, even if improvement is gradual.

INTRODUCTION

Diversity originates from the richness of culture, language, and insights from different perspectives. Yet, when not communicated effectively through scientific writing, even novel discoveries can be lost. To solve global challenges, there are calls to action for broadening participation and supporting diversity in science, technology, engineering, and mathematics (STEM) graduates (1, 2). Nationally, the percentage of STEM bachelor’s degrees awarded to Asians is increasing (3). For this reason, the National Science Foundation’s classification of underrepresented groups (4, 5) does not include students from 48 different ethnicities who identify as Asian-American, Pacific Islander, Southeast Asian (Hmong, Cambodian, Laotian, and Vietnamese), some of whom may nonetheless be underserved. Even some who identify as East Asian (Chinese, Japanese, Korean) or South Asian (Asian Indian and Pakistani)—whether native, foreign-born, or visa-holders—may be among the growing under-supported and overlooked populations of English language learners (ELL), or L2 learners, for whom English is a second language (6). Their increased participation in higher education does not accurately represent challenges many face as Asian-Americans working in STEM fields in the US; in academia, as graduate teaching assistants, postdoctoral fellows, or faculty; or as global partners after graduation (7).

Regardless of their origin, we must support our next generation of scientists. We have the opportunity in a writing-enriched laboratory course to use inquiry and various tools to assess writing, both formatively and summatively. As part of a University of Minnesota campus-wide Writing Enriched Curriculum, the College of Biological Sciences (https://wec.umn.edu/college-biological-sciences) builds upon introductory first-year and lower-level Writing-Intensive (WI) courses to culminate with upper-division WI credits in the major. The students in our Biochemistry laboratory course for WI and laboratory credit have varied backgrounds: continuing education, transfer students having taken prerequisite biochemistry courses elsewhere, and UMN students with varying numbers of WI credits. It is expected that students have moved beyond being novice writers—and most have. However, just-in-time interventions are needed for students who still struggle with science communication in English, the de facto language of global science. In addition, educators need methods to assess deeper knowledge and understanding of scientific conceptions, cognitive reasoning, and analytic skills, and to measure student ability to synthesize key concepts and draw supported conclusions.

The crux lies in assessing highly engaged learners who are technically well versed and may have mastered advanced science concepts, both cognitively and analytically, but struggle in their pursuit of science because of enduring language deficits in English. The transition from thinking in L1 (their native language) to making novice errors when communicating in L2 (English) can result in lower grades, lower confidence, and higher stress. Educators need to know which interventions to recommend for increasing student confidence and ability, and how to grade objectively—a challenge for educators with which computer-assisted tools can help.

Different approaches for student-centered learning support the rationale that ELL students become key developing practitioners of learning rather than targets of teaching (8). We adopted this exploratory practice framework: learners are unique; those who take learning seriously learn effectively with peers; and those able to make independent decisions as practitioners of learning are key in their own development within educator-developed interventions. This guided our study to look beyond group trends. We focused on individual barriers and assessed different interventions for effectiveness in building communication. Traditionally, the linguistic characteristics of L2 learners have been studied based on individual differences, treating each learner as distinct using computer-assisted technologies for assessment (9). Educators can use resources about common errors in syntax specific to L1 languages to understand the patterns of language use to help students in L2—Chinese (10), Vietnamese, Korean (11)—or common writing errors in the adopted language while switching between L1 and L2 (12). We used such resources in the individualized Asian ELL interventions which launched our studies to examine hidden themes within poorly written text. Each language has its own style, and learners their own persona. More densely packed informational writing, such as research reports, may present a greater difficulty for L2 learners, but latent understanding can be measured objectively (13).

Early studies of L2 learners proposed that latent comprehension begins with vocabulary and prior knowledge: understanding a concept, knowing the theory, and learning within a context. Within this knowledge hypothesis (14), simple assessment methods used word analysis to understand ELL behavior during reading and writing in L2 (15). Schema theory, which states that knowledge is structured as lexical units, puts language development and comprehension at the psycholinguistic level in memory. People use prior knowledge to understand the gist of meaning. It is easier for native speakers with more lexical units to fill in the gaps than it is non-native speakers. Hence, even when L2 writers use proper terminology, the semantics may change. Access to language affects the quality of the comprehension and the cognitive strategies for analysis, including cognitive roles of selecting, grouping, predicting, interpreting, and justifying terms found on a grading rubric (16). When students have greater prior knowledge and context, they may be able to compensate for lack of vocabulary and skills to aid in comprehension.

Likewise, students’ writing, despite grammatical errors, can still reveal deeper cognitions through word usage. This underlying capacity can be measured by discourse study, such as Latent Semantic Analysis, which calculates similarity between words and texts (17), or Linguistic Inquiry and Word Count (LIWC), which categorizes word use into psychological categories based on coded dictionaries useful for studying emotional, cognitive, and structural components (18). These analyses use a body of text, such as student reflective writings or other writing assignments. Just as poems can be studied to identify patterns of their writers, corpus linguistics (from the Latin corpus, meaning body) is a methodological approach to the study of language which involves transcribing and coding a body of text or oral language. Semantic content analysis uses automated text analysis of L2 learners’ work to understand deeper linguistic and cognitive relationships; it is not used to study language construction, but the learners themselves by their use of words. By using dictionaries and themes to detect word use, computer-assisted technologies allow objective standardized study of cognitive processes (19). Useful information could be extracted from large corpora of text by combining automated scoring with profiles of writing behavior, patterns, or tendencies representative of groups (20, 21)

In the Handbook of Discourse Analysis (22), several authors reveal how communication “beyond the sentence” involves not only the language, but also the nonlinguistic cultural and social practices surrounding language. When taken together in communication, it is natural, then, that different language learners would write with differences in tone, semantics, including concepts, feeling, culture and personality, and even shift from their native writing cultural persona to write in L2 (23).

Predictably, not all ELL students exhibit the same traits. By examining their writing samples, first without looking at student names or demographics and observing patterns, then more deeply through case-study analysis (compared with native-speaking lab partners), we chose a corpus linguistics method to reveal deeper cognitions. Based on previous analysis of a variety of writing samples (B.L. Smith-Keiling and H.F. Hyun, submitted for publication), we posited that we could objectively measure the comprehension of lab exercises through analytical skills and cognitions needed in writing a manuscript-style discussion section, even with grammar and scientific writing difficulties.

Developed by Pennebaker and extensively validated, Linguistic Inquiry and Word Count (LIWC) has been used to detect psychological processes underlying language learning, both emotional affect and cognition (18, 19, 24–26). Writing in L2 can induce stress and anxiety if mastery is not achieved. This is measurable using word counts. The LIWC2015 software (27, http://liwc.wpengine.com/) has four summary language variables: Analytical Thinking, Clout, Authentic, and Emotional Tone. Within these variables are domains with groups of dictionary words in sub-dictionaries creating word categories such as Cognitive Processes, Comparisons, and Quantifiers—all dictionaries that we predicted might capture the skills used in scientific writing per our grading rubric (Table 1A, B) based on previous examination of raw scores, algorithmic scores for variables, and percentage of word use for categories (B.L. Smith-Keiling and H.F. Hyun, submitted for publication). Recent use of the LIWC shows the possibility of using analytic programs to seek different cognitions of students in personal writing samples despite ELL barriers (28, 29).

TABLE 1.

Sample LIWC2015 code dictionaries (A) and matching categories in our rubric (B).

A)
Category	Abbrev	Examples	Words in category
Word count	WC
Analytical thinking	Analytic
Linguistic Dimensions
Total function words	funct	it, to, no, very	491
Total pronouns	pronoun	I, them, itself	153
Articles	article	a, an, the	3
Other Grammar
Common verbs	verb	eat, come, carry	1,000
Common adjectives	adj	free, happy, long	764
Comparisons	compare	greater, best, after	317
Interrogatives	interrog	how, when, what	48
Numbers	number	second, thousand	36
Quantifiers	quant	few, many, much	77
Psychological Processes
Affective processes	affect	happy, cried	1,393
Anxiety	anx	worried, fearful	116
Cognitive processes	cogproc	cause, know, ought	797
Insight	insight	think, know	259
Causation	cause	because, effect	135
Discrepancy	discrep	should, would	83
Tentative	tentat	maybe, perhaps	178
Certainty	certain	always, never	113
Differentiation	differ	hasn’t, but, else	81

B)

DISCUSSION Purification conclusion in 1st sentence Data of purification Purification evaluation Limitations of techniques Protein isolation BH comparison Protein isolation expressed comparison Comparison of manual vs automated purification Data support to back up comparison Comparison: kinetics results (Vmax, Kcat, Km) published values Comments (interpretation): discrepancies between results and literature What could improve accuracy of your measurements Comment on linear and non-linear regression models and which is more accurate and why Data and discussion of electrophoresis Data and discussion of immunoblot What does each piece of data tell you (interpretation) about purity of sample What purity questions remain unanswered (conclusion) Style, clarity, details discussed, past tense and switch to present or future studies

A) The LIWC2015 table adapted with permission from the LIWC Language Manual (27) has summary language variables Analytical Thinking and categories Cognitive Processes, Comparisons, and Quantifiers—all dictionaries that we predicted might capture the skills used in scientific writing in addition to Word Count per B) our grading rubric, which matched to categories in the formative and summative grading rubric used in peer-evaluation interventions. LLR discussion rubrics required conclusion, supporting data, evaluation, limitations, comparison, interpretation of discrepancies, improvements of accuracy, “results tell you” interpretation, “unanswered” along with measures of style, clarity, and grammar. The total percentage score of the discussion sections was 27% of the total LLR score. The LLR was 15% of the grade, peer-evaluation participation 3%, other writing 12%, so writing totaled 30% of the overall course grade. LIWC = linguistic inquiry word count; LLR = large lab report.

The focus of this cross-sectional study was to examine a sample of cases from existing experimental groups of our total student population under investigation. We measured student confidence coming into the course and after interventions for those who faced additional challenges and demonstrated the greatest need for support with English L2. Our goals were to determine types of interventions on an individual level for Asian ELL students and all students (Table 2) using strategies for 1) building inquiry, 2) developing scaffolding for learning analysis, and 3) building writing ability to communicate science. We employed a LIWC computer-assisted assessment strategy to measure cognitive, analytic, and other features despite L2 challenges in scientific writing, and to measure anxiety in a post-study free response survey.

TABLE 2.

Metacognitive intervention process included plan, monitor, and assess steps.

Plan	Monitor	Assess
Students began with self-aware process thinking about their skills to plan: pre-self-ranking, pre-skills tests, assessing draft and rewrite of BQA, and planning improvement.	Interventions to scaffold learning allowed students to self-monitor: inquiry next-step discussions, formative rubric use in self-evaluation, office hours, etc.	Through repetitive peer-evaluation, students reflected and assessed: post-self-ranking, postskills tests, and final assessment with LLR rubric.
Pre/post survey for self-ranking student confidence (Likert 1=low, 5=highly competent): “Indicate for each of the following skills how competent you feel knowing the theory” “Indicate for each of the following skills how competent you feel actually doing the skill” Three scientific writing skills (this study) seeking primary literature reading primary literature scientific writing and citation	Interventions Inquiry – questions at the end of each lab to promote student development of ideas for their next step of the lab protocol discussed, checked-out with TAs and spring/summer students wrote ideas in weekly lab notebooks open ended and not assessed once students were provided with the next lab protocol, they could determine how close their ideas were to the next lab exercise	Same post-survey for self-ranking Inquiry – enhanced in the spring/summer intervention by building on weekly discussions of inquiry-based lab assignments
Other confidence and pre-test skills measured (not this study) mathematical calculations molarity dilutions polymerase chain reaction (PCR) primer design solving a case study Writing skills assessed Biochemical Question & Answer (BQA) stimulated reflective curiosity as students posed a biochemical question and scientifically supported answer sought primary literature, read, and wrote scientifically written and cited answers popular/news, layperson and scientific writing, properly citing sources, and nuances of writing grammar, mechanics, and style Writing errors were used in selection of cases Students were invited to discuss plan	Peer evaluation – beyond simply tutoring, this provided feedback for the student to do self-checks draft and rewrite of multiple writing samples smaller scaffolding assignments prior to a larger (LLR) synthesis project, weekly peer-evaluation in spring formative use of rubrics	Peer evaluation – used a formative guideline rubric/checklist to inform students for assessing their own writing instructor and peer evaluation feedback for monitoring progress the same rubrics used formatively in monitor stage were used summatively for final LLR grades
	Individual interventions office hours with writing exercises recommended writing center services individual online tools (turnitin.com similarity score plagiarism checker, Grammarly.com grammar checker) individual reflections help from friends	Interventions Peer-evaluation in spring was assessed by TAs for a participation grade

BQA = biochemical question and answer; LLR = large lab report.

We hypothesized that these practices would not only increase confidence in ability (self-efficacy), but also that we could use the analytic LIWC2015 software to read beyond the sentence, based on a coded dictionary to measure analytical thinking, cognitive, comparison, and quantifier words in scientific writing (27) (http://liwc.wpengine.com/), separated from reflective styles measured more for affect. This mixed-methods approach addressed our primary outcome of providing strategies to assess and support ELL students in science.

METHODS

Student demographics and sampling case selection

As part of a larger study, all enrolled students were invited on the first day to participate in course-specific educational research and gave informed consent per our IRB protocol. Those who did not consent participated in all class activities and assessments, but they were not enrolled in the study, and their data were censored from analysis. Participants were de-identified and given ID numbers to keep the instructor blinded in the larger study, but as students were identified with writing difficulties, they were selected for individual scrutiny for this study. Students entering and exiting Biochemistry lab courses in fall, spring, and summer of 2016–2017 (N = 186) provided demographic and other data through paper-based pre/post surveys (Fig. 1). Of these, 22 students, or 12%, identified as both Asian and ELL, our target population. This number narrowed to 19 potential cases who completed all assessment data in digital format for analysis (fall, n = 11; spring/summer, n = 8). Those who met selection criteria with respect to need became sample cases (n = 6) for deeper cross-sectional analysis through discussions with the instructor and additional individual writing interventions. Fall semester students had no writing interventions and served as a baseline control group. Interventions for all spring/summer students included inquiry discussions, formative rubrics, and peer-evaluation discussions for a final written report, which was submitted in digital format and analyzed by LIWC for comparison of Asian ELL and their native-English-speaking lab partners. Cases 1 to 6 received individualized interventions (Fig. 1).

FIGURE 1.

Demographics and sampling process from fall control and spring/summer interventions for representative cases. Biochemistry lab courses fall, spring, and summer, 2016–2017 (N = 186), pre-course confidence and demographic surveys measured all students’ confidence in theory and skill, matched to skill test measures. Demographics identified 12% who identified as both Asian and ELL. Writing sample BQA1 and 2 drafts identified those with greatest need in scientific writing in English. From potential cases, six cases were selected from those who came to office hours or discussed writing. LIWC analysis of LLR ranges: word count 566 (low) – 2,297 (high); analytic formal scientific 83.8–98 (reflective 18–50 not shown); cognitive processes 8.5–18.2%; comparison 3–7%; quantifier 2–5%; “I” words 0 to pronouns <1% (not shown). Free-response range: anxiety words 0–6.25. Representative quotes from 3/22 Asian-ELL various communications including a post-course written free response prompt: “Consider a time in lab that you felt stress, or a challenge. Write about it, how you felt and what you wish you had heard/experienced/done instead to reduce your stress or to overcome your challenge.” ELL = English language learner; LIWC = linguistic inquiry word count; LLR = large lab report; BQA = biochemical question and answer.

Quantitative pre/post confidence in theory (knowledge) and skills (self-efficacy)

Pre/post surveys of confidence were matched with skills assessments for several mathematical, lab, and writing tasks (Table 2). Based on Bandura’s Social Cognitive Theory and outcome expectations that confidence in writing impacts actual writing skill (30, 31), we examined constructs of confidence in theory (knowledge) and confidence in being able to do the skill (self-efficacy) using a series of “How confident are you…” questions modeled after a previously validated survey (32). This followed the metacognitive process of students thinking about their skills, working on skills, revising, and reflecting again at the end (33). From the Asian ELL students (n = 19) in fall (baseline) and spring/summer (with interventions), arithmetic mean scores for the 15 students who completed both pre- and post- surveys and answered all six survey questions were compared to demonstrate trends (Fig. 2A, 3A). Values were mean 5-point Likert-scale responses. Due to the small sample size, chi-squared analysis was not feasible, so trendlines of the six measures and frequency of responses illustrated gains for each category of three theory questions and three skill questions using SAS statistical software version 9.4 (Fig. 2B, 3B). Paired two-way t-tests were performed to compare the means between pre- and post- survey answers; for fall and spring/summer semesters, unpaired two-way t-tests were performed.

FIGURE 2A and 3A.

Pre/post confidence trends measured confidence in knowing theory and confidence in doing skills in three writing skills. From (n = 19) pre/post-confidence measured, those who completed both pre- and post-student competency responses were used to compare the fall control group with the spring/summer with interventions. Student self-reported responses (Likert Scale 1 to 5) for confidence in theory and confidence in skills for writing showed positive trends in both semesters; fall showed higher overall incoming confidence than spring/summer (not significant); reading and searching primary literature higher than writing; greater increases measured in fall for theory (p ≤ 0.05); and greater gains seen in scientific writing skills for spring/summer students (p = 0.025).

FIGURE 2B and 3B.

Frequency graphs of student responses to confidence in theory and confidence in skill in writing. The upward trends represented with stacked graphs show frequency patterns of responses for nine fall students and six spring/summer students who answered all survey questions. Higher frequency resulted after the course.

Writing-skill assessment for case selection

Several writing forms were used as part of the formative learning process to promote practice and the metacognitive approach of evaluation and revising. To qualify as WI, the course met several university criteria including rubrics and rewritten work, with enhanced spring interventions including formative use of rubrics in self and peer evaluation (Table 2) (34–38).

The LIWC2015 was vetted for use with these writing samples (B.L. Smith-Keiling and H.F. Hyun, submitted for publication). Short, one-page samples collected in the first four weeks of the course were used to assess writing, citing, and drawing conclusions concisely (Table 2). Biochemical Question and Answer (BQA) papers began with students posing a biochemical question (Q) followed by a scientifically supported answer (A). These provided our first glimpse at writing challenges students faced. Although samples were not masked before scoring to avoid bias, upon reading the first BQA samples, the instructor identified students who exhibited writing errors—all of whom were Asian students in need of ELL support. Other ELL students and native speakers did not show grammar and writing deficiencies.

To confirm no bias, after the end of both semesters, the instructor and assistant re-examined electronically stored BQA draft assignments and qualitatively coded errors in the Q sections, with > 95% reliability between the two coders. Common ELL errors included grammar mechanics: fragments, run-on sentences, misuse of tenses, prepositions, and subject-verb agreement of singular and plural (10, 12). If sequentially reviewed BQA drafts showed grammatical errors, we coded yes. If we coded no errors, the student was not selected. From the number of possible student cases with detected errors per selection criteria, these were further narrowed to six students who had discussed their writing with the instructor (Fig. 1).

All students were invited to discuss writing during office hours. A simple random-sampling study design method allowed for the probability of any student coming to open office hours, making an appointment, or discussing writing briefly with the instructor when writing assignment drafts and finals were returned in lab. Student cases were chosen by convenience sampling. To assure sampling was representative of the true population for inferential statistics validity, the instructor surmised (based on discussion) and the assistant later identified those in both Asian and ELL classifications. Frequency of those with errors and response to an invitation to office hours determined selection of cases. Only those who submitted final digital work were selected. Because Asian names can identify ethnicity, we assigned non-Asian pseudonyms for Asian ELL cases fitting these criteria and selected for further analysis: fall (Dayton), spring/summer combined to maintain more anonymity for the smaller summer population (Bob, Mike, Henrietta, Hope, and Zach) (Fig. 1). These students’ native-English-speaker lab partners were included for comparison in later analysis.

Analysis of LLR discussion using LIWC (analytical thinking, cognitive processes, comparisons, and quantifiers)

In addition to BQA papers, weekly lab reports (data and written components) culminated in a comprehensive manuscript-style large lab report (LLR) which integrated deeply with the laboratory exercises for a semester-long protein purification and gene expression project. Fall labs followed traditional cookbook-style, rote protocols. Interventions for spring/summer students included discussion of next-step inquiry questions and evaluation with peers (Table 2); in addition, the LLR sections were peer-evaluated with rubrics including cognitive and analytic terms in comparison with LIWC terms (Table 1). Students brought material for weekly peer review. Graduate teaching assistants (TAs) and the instructor commented. Students were asked to bring a final copy for proofreading the week before the due date.

Students uploaded final LLR submissions electronically to be checked for plagiarism (using Turnitin, https://turnitin.com) and brought a printed copy for grading. For consistency, TAs each graded one section using the rubric. After the courses, for this study, the assistant downloaded the discussion sections from the LLRs into separate Word documents, with student ID numbers only, and placed them into separate folders: Asian ELLs, native-English-speaker lab partners, all native English speakers, and other ELLs. When opened in the LIWC analytic program, Excel (2016) files were generated as output and imported into SAS for statistical comparison. Fall and spring/summer Asian ELLs and their native-speaker lab partners were compared and analyzed using SAS Software version 9.4, and frequency graphs were created in GraphPad Prism version 5.

Two-sided t-tests were used to compare fall and spring/summer semester Asian ELL students with their native-speaker lab partners, with significance alpha set at 0.05. Regression analysis performed in SAS explored the relationship between the five LIWC categories we studied (Figs. 4, 5, 6).

FIGURE 4.

LIWC analysis of LLR discussion comparing fall Asian-ELL and native-speaking lab partners. Box and whiskers plots show mean and median values. Fall Asian ELL (blue) compared with fall native-English-speaking lab partners (yellow) showed comparable variation of variables A) Word Count, B) Cognitive Processing, C) Analytical Thinking, D) Comparisons, and E) Quantifiers. LIWC = linguistic inquiry word count; LLR = large lab report; ELL = English language learner.

FIGURE 5.

LIWC analysis of LLR discussion comparing spring/summer Asian ELL and native-English-speaking lab partners. Box and whisker plots show mean and median values. Spring/summer Asian ELL (green) compared with spring/summer native-speakers (yellow) showed comparable variation of variables A) Word Count, C) Analytical Thinking, D) Comparisons, and E) Quantifiers. A significant difference (p = 0.008) in B) Cognitive Processing was detected between Asian ELL and native-speakers. When comparing Fig 4. fall Asian ELL and Fig 5. spring/summer Asian ELL, there was a significant difference (p = 0.04) with effect size (1.54) in Cognitive Processing. LIWC = linguistic inquiry word count; LLR = large lab report; ELL = English language learner.

FIGURE 6.

LIWC analysis of LLR discussion of individual Asian ELL and their native-English-speaker lab partners (NLP). Variation in student LLR discussion section was measured in LIWC for A) total Word Count, B) percent Cognitive Processes, C) Analytical Thinking score, D) percent Comparisons, and E) percent Quantifiers. Fall Asian ELL (blue) and spring/summer Asian ELL (green) were compared with NLP (yellow). Arithmetic means (indicated with a black bar) showed individual patterns could be detected across the five variables e.g., individual Bob (with lower grade performance) showed low word count, low analytical, and high cognitive. This differed from another, Mike (with hand-graded, deeper meaning detected), with average word count, average analytical, and average cognitive. Others with high word count, average analytical, average cognitive, but also high comparison and quantitative scores matched high-quality writing and scores (e.g., Zach). Dayton’s, Henrietta’s, and Hope’s scores fell within predictable ranges. LIWC = linguistic inquiry word count; LLR = large lab report; ELL = English language learner.

The LLR discussion sections were analyzed using LIWC for four variables most applicable to scientific writing: summary variable Analytical Thinking; three word category variables, Cognitive Processes, Comparisons, and Quantifiers; and other indicators such as pronouns determined of interest for scientific writing (B.L. Smith-Keiling and H.F. Hyun, submitted for publication). A free version of LIWC software on Pennebaker’s site (https://liwc.wpengine.com/) was initially trialed in the analysis of a separate study; the free version is designed to become cumbersome with higher data input to limit misuse. The most current LIWC2015 is easily purchased at low cost to educators.

Individual case analysis and qualitative, open-coding word analysis

The six individual cases were considered representative of issues common to Asian ELLs. The instructor, no longer blinded to the Asian ELL sample, examined the specific intervention examples based on what the case students had chosen in order to draw generalizable conclusions: office-hour editing discussions, practice sentences, writing center help, online tutors, translators and grammar checkers, along with inquiry and peer-evaluation interventions introduced in spring/summer. The instructor re-read each case student’s LLR discussion for reliability of TA scoring and to seek discernable patterns compared with interventions and LIWC scores.

Stress/anxiety in Asian ELLs was measured in two ways: 1) qualitative coding from various forms of written communication (reflective writings, e-mail communication, and a post-survey free response), and 2) quantitative analysis of the same free-response answer using the LIWC categorical variable anxiety (Table 1). The latter yielded a percentage score for each case (Fig. 1). Free responses were transcribed with spelling and punctuation fidelity and coded from a post-course prompt about overcoming challenges. Qualitatively, as part of a larger study (unpublished data), two independent coders simultaneously used quasi-deductive coding to develop a codebook searching for deductive codes as inductive codes were revealed, agreed upon codes with >90% reliability, and organized axial coding with subcategories as themes. Here, the instructor and assistant re-coded by looking deductively for three codes (writing, editing, communication) associated with stress or challenge and noting representative quotes from 3 of the total 22 Asian ELLs specifically referring to writing and communication stress.

RESULTS

Quantitative analysis of pre/post self-reports of confidence in theory and skills

In a series of pre-competency skills tests coupled with measures of student self-report, positive trends of overall confidence for six measures of theory and skills matched our writing skill findings. Pre-competence questions for indicators of writing confidence about theory and skills showed fall Asian ELL were generally more confident coming into the course than spring students, were generally more confident seeking and reading literature than writing, and more confident with theory than skill. Fall (Fig. 2A) had higher pre-confidence than spring/summer Asian ELL (Fig. 3A).

To determine whether interventions to promote inquiry and peer-evaluation discussions and the formative use of rubrics impacted confidence and skills, fall was compared with spring. Whereas rubrics were provided to fall students during the writing phase as a checklist, spring students used rubrics to peer-review each other’s lab reports in addition to engaging in inquiry and peer-evaluation discussions.

Confidence in writing skills increased after the course, according to follow-up post-competency skills tests and student self-report. Paired t-tests showed increases in means from the fall semester pre-to-post surveys for confidence in theory of searching primary literature, reading primary literature, and scientific writing and citation (p ≤ 0.05). Spring/summer semester increases were significant only for confidence in skill of scientific writing and citation (p = 0.025). Unpaired t-tests comparing fall and spring semesters were not statistically significant. Frequency graphs demonstrated counts of each response (Figs. 2B, 3B).

Analysis of LLR discussion using LIWC (analytical thinking, cognitive processes, comparisons, and quantifiers)

To determine whether there was an association between LLR scores and interventions, LIWC analysis of LLR discussion sections yielded values expressed as total word counts, an Analytical Thinking algorithmic score, or percentages of Cognitive, Quantifier, and Comparison words (according to the LIWC dictionary) out of the total number of words analyzed.

Asian ELL students and their native-English-speaking lab partners were compared and analyzed in five LIWC categories for fall (Fig. 4A–E.) and spring/summer (Fig. 5A–E.). Although variation existed, there were no statistically significant differences between group means except for cognitive (p = 0.008 for spring/summer Asian ELLs), suggesting outlier scores could be used to detect individual differences on a case-based level rather than group trends. Values for cognitive processing were statistically different (p = 0.04) between fall and spring/summer semesters for Asian ELL students (Fig. 5B), suggesting impact from interventions of inquiry and peer evaluation.

It was important to look at individual differences with deeper qualitative grading by the instructor to discern these patterns. LIWC quantitative scores validated that students wrote scientifically: Analytic score approaching 100 with more formal writing; low use of pronoun “I” (close to zero) (data not shown); Comparison and Quantifier words at 5%–6% of total word count; and Cognitive Processes ranging from 8.5%–18%, indicating higher cognitive load of Asian ELLs compared with their native-English-speaking lab partners showed discernable patterns (Fig. 6). Regression analysis revealed a statistically significant relationship between the Analytical Thinking score and the Cognitive Processes score (p = 0.0012) and Comparison score (p = 0.0007), which mathematically supported patterns observed and suggested that, with further analysis, this method could be further developed as a potential heuristic predictor.

Detection of individual errors in Q and A for ELL sampling and individual case interventions

From a total of 186 students’ writing samples, all students with errors detected in the Q of BQAs were selected. These were all Asian ELLs with writing challenges consistent with thinking in native L1 while writing in L2: 9/13 = 69% in fall, 9/9 = 100% in spring/summer (Fig. 1) (12, 39, 40). Two fall students responded, only one of whom turned in the final assessment digitally for analysis. Spring/summer had greater response, with five Asian ELL students meeting selection criteria for our case analysis. Anecdotes ranged from greater improvements in cases Dayton, Hope, and Zach to persistent problems in cases Bob, Henrietta, and Mike. Case Dayton showed early errors and revealed that not using translator programs and instead relying on native-English-speaking friends corrected his errors. Bob, who did not seek further help, did not improve. Samples of Mike’s intervention writing exercises (Fig. 7), Henrietta’s intervention writing center help (Fig. 8), and Mike’s final LLR sample (Fig. 9) were summarized (Table 3).

FIGURE 7.

Example of case Mike’s BQA Q Section and A section practice exercise intervention. A) Common errors were left as empty circles by the instructor; B) writing practice exercises emphasizing subject-verb agreement tasks were tried by the student. Despite numerous attempts, the issue persisted throughout the semester. BQA = biochemical question and answer.

FIGURE 8.

Example of case Henrietta’s BQA A section writing center intervention. Student had previously tried office hours for the draft, had used a free online app (grammarly.com), and had tried to get help from the on-campus Writing Center. These results were from an edited paid service (unknown) for the final product. None of the services could adequately get at the true meaning of some of the scientific content. To do so, the instructor verbally discussed, and assisted with the rewrite of the final product, but a reduced score and issues persisted throughout the semester. BQA = biochemical question and answer.

FIGURE 9.

Sample introductory paragraph of case Mike’s LLR discussion instructor assessment. Originally this discussion section was marked as being “incomprehensible” for grading by an assistant lab instructor and scored very poorly. This showed that not all graders may have the same viewpoint or tolerance for grammar errors. Because the LIWC scores detected a pattern comparable to high-scoring native English speakers with comparable values around the mean for word count, comparison, and quantifiers, it was re-graded by the instructor. Scoring 96% analytical and 13% cognitive (around the mean), even without proper grammar and writing, the instructor saw evidence fitting the rubric for understanding, cognitive reasoning, analytical thinking, quantifiers, comparisons of lab techniques, and other key components throughout the section. Some specific understanding was still lacking, but in the later portion the author proposed a new idea—one that would have been missed if scientists fail to read between the grammar errors, or scientific communication is not improved. The score was raised with re-grading based on the LIWC scores. When the instructor re-graded a different LLR discussion (Bob’s), also with a poor score, his LIWC scores were not near the mean compared with native English speakers. In fact, some scores were outliers with low word count, low analytical 84%, and high cognitive 18%, indicating greater ELL challenges and possibly lab comprehension challenges since the depth of content was not matching requirements in the rubric. The low score remained because the LIWC scores did not indicate otherwise and were consistent with the instructor’s judgement. LLR = large lab report; LIWC = linguistic inquiry word count; ELL = English language learner.

TABLE 3.

Case histories and intervention recommendations.

Cases	Case History	Helpful Recommendations
Dayton	Case Dayton’s BQA2 draft first showed L1 to L2 switch issues. It was observed during grading; grammar was less noticeable in the scientifically written A section than the more reflective personal curiosity Q section. He shared that to express himself in personal writing he relied on computer translator programs like Google translator, and these errors were more pronounced. Errors crept into scientific writing less frequently because more word patterns were recognized i.e., lexical units. Improvement occurred; further errors were not detected when Dayton stayed in L2 for reflecting and writing and used other grammar checks instead of the translator program.	recommend not using translator programs stay in L2 language for thinking and writing in L2 English-speaking friends helped proofread
Bob	Case Bob’s BQA drafts had several markups for correction, yet he did not respond after one office hour. Rewritten BQAs also had errors even with instructor feedback. In-lab performance relied heavily on his lab partner; Quantifiers and Comparison terms were comparable in LIWC scores with probable comparable data. LIWC writing scores showed low Word Count, low Analytical and high Cognitive in comparison with mean. Score matched final LLR score by instructor analysis.	increase invitations to office hour help peer-evaluation, scaffolded assignments, draft and rewrite, and instructor feedback
Henrietta (Fig. 8)	Case Henrietta’s BQA drafts provided intervention examples of seeking Campus Writing Center and online writing assistance with grammarly. com. In both situations the scientific meaning was misunderstood. She reported frustration with not being able to make a timely appointment, or find a scientifically-trained tutor. While it is possible to request science-specific help, timing of assignments and availability were deterrents. Only when the instructor verbally discussed the science content and what the student intended to say could the corrections be made; however, errors persisted in new writing.	Time is a necessary factor for proofreading and editing drafts and final products for quality work increase time between assignments for rewriting Seeking help at Campus Writing Centers and online required an appointment, and work completed in advance. increase time to make appointments Writing tutors could not always get at the scientific meaning embedded in the grammar corrections. more scientifically trained writing center support
Mike (Fig. 7, 9)	Case Mike’s BQA1 draft showed evidence of significant grammar errors. In weekly office hours, the instructor provided intervention markups for practice exercises; errors were marked as empty circles. Exercises emphasized subject-verb agreement tasks. He was assigned the task of going to the Campus Writing Center, but follow through was lacking. While anecdotally he reported the metacognitive process increased his awareness, despite numerous attempts, the issue persisted throughout the semester.	Practice writing exercises helped teach, but, in Mike’s case, were not able to overcome the four most common errors (mechanics, tenses, prepositions, and subject-verb agreement) during a semester course. longer time than a semester is needed; develop a series of science-related practice writing exercises
Zach and Hope	Both showed high motivation for English Early challenges in first year courses and strong motivation with tutors helped prepare. All case anxiety scores in post-free response varied.	Earlier LLR writing start times, more peer-evaluation, fewer BQAs could help.

BQA = biochemical question and answer; LIWC = linguistic inquiry word count; LLR = large lab report.

Qualitative search for deeper understanding despite writing style and grade outcomes

TAs used the same rubric/checklist given to the students to grade the different LLR sections (introduction, experimental protocol, results, and discussion) and to record scores. In the case of Mike’s LLR, despite numerous ELL difficulties, the writing in sections graded by other TAs was adequate; however, the TA grading the LLR discussion sections marked Mike’s as incomprehensible for having too many grammatical errors. A deeper look by the instructor yielded examples where biochemical knowledge, understanding, and critical thinking were supported. Despite ELL writing challenges, deeper meaning and new ideas were evident (Fig. 9). This raised the question of subjective scoring, and whether analyzing the discussion by LIWC could yield objective data indicative of Mike’s analytical thinking and cognitive skills (described above).

Analysis of a post-course free response survey using LIWC (anxiety)

Anecdotally from the first-day ice-breaker, students were asked to “share an example of a challenge they had faced and how they overcame it,” students identified English barriers and the culture shock of moving to the United States. A post-course, free-response survey question measured confidence and prompted the first-day question of stress, challenges, and overcoming them. Quasi-deductive qualitative coding of BQA Q, reflective, personal statements, and emails revealed additional stressors and challenges in 3 of the 22 Asian ELLs due to writing and communication.

Compared with the students’ scientific writing, LIWC analysis of their free responses showed higher use of “I,” and yielded affect and anxiety scores under summary variable Emotional Tone. Quotes and anxiety scores from Asian ELL students showed that some issues highlighted as writing and communication stressors had not yet been resolved (Fig. 1).

DISCUSSION

Motivated to retain more diverse students, we used a mixed-methods approach to examine an experimental, cross-sectional, case-control sample to find ways to support and assess Asian ELL students. Pre/post confidence trends showed gains for six related measures of scientific writing. Gains in the spring/summer group could be due to several interventions. Enhanced inquiry within the lab discussions may have strengthened student confidence and skills as students talked with peers and TAs, thus practicing their ability (self-efficacy) to generate new ideas and communicate them. Peer-evaluation and discussions of how to analyze results and write scientifically provided further practice while also scaffolding analytical interpretation and writing skills. Use of formative rubrics building on lab analysis in lab notebooks and weekly lab reports benefited Asian ELL students with the greatest need. From these trends, it appeared that students expected that they had gained knowledge and skills in writing even if their grammar in the final LLR showed deficiencies. This supported Bandura’s outcome expectancy beliefs that expecting the outcome leads to self-efficacy (30, 31, 41). Higher gains supported our hypothesis that these practices increased confidence and skills. Even if Asian ELLs’ writing was not perfect, the scientific writing ability improved. Although fall students showed higher writing ability than spring students, and fewer sought assistance, there were Asian ELL students who demonstrated need but did not come to office hours. We consider it possible that the use of more inquiry and discussion in the spring could have prompted more response.

Grouped comparisons of the LIWC analysis of LLR discussions showed no noticeable differences between means except for the Cognitive Processes. LIWC did, however, detect differing patterns between individual students. Overall variability existed in both Asian ELL and native-English-speaking lab partner populations. Even between two lab partners with similar lab results, we observed similar patterns and extreme outliers. This validated for us that the LIWC scores could be used on an individual basis as indicators for grading. LIWC scores measuring underlying cognition and analytical skills, and correlating to hand-grading and assigned grades per the rubric by TAs and the instructor supported our hypothesis that this tool could be useful in objective assessment of L2 learners. We expect this would be generalizable to other ELL and students overall, which we hope to study further with larger populations of underrepresented groups.

LIWC Quantifiers and Comparison scores for the LLR discussions were less varied, perhaps due to the similarity in weekly lab-exercise data between lab partners. It is too early to propose a heuristic model. If student scores fell lower than the mean, it could act as an indicator. More meaningful patterns or tendencies of low word count, low analytical scores, and high cognitive scores acted as potential L2 markers. Higher-than-average cognitive scores signify greater challenges resulting from L1-to-L2 switching (8, 20–22). Despite a small sample size, the effect size for this increase perhaps indicates they struggled more and their gains could be more significant as a result of the interventions received. This would explain the significantly higher cognitive scores in spring/summer students who had greater ELL challenges and underlying psychological affect (23), represented by Bob’s case. This suggests that even lower writing skills in an ELL population, which may be challenging for a TA or instructor to grade objectively, can be detected.

Conversely, if the patterns of word count and other scores are closer to the mean, or if the analytical scores are even higher, these could signify that more formal, higher-ordered thinking occurred and that the writing is at least worthy of a second look (by the instructor) for deeper meaning beyond the grammar difficulties. This was particularly true in Mike’s case. Since these are early heuristic models still under development, we caution using them for a graded judgment. The patterns we observed showed potential in assisting qualitative grading and suggest that further studies in this area are worth pursuing.

Within the exploratory practice framework (8), looking at individuals and a metacognitive process, we gained information about intervention usefulness. Based on case discussions with Dayton, Hope, and Zach, we recommend against using translator programs. As the brain switches between L1 and L2, cognitive load increases, which can result in more errors (40); it is better to stay in L2 (English) when writing. Based on case discussions with Henrietta and Mike, practice exercises and writing sessions may help in the long run, but the time frame of one semester was too short to see grammar improvements. Writing centers can assist with proofreading and some grammar mechanics, but we would recommend trained science tutors also skilled in the mechanics of writing.

Perhaps, too, peer evaluation contributed to growth not only by providing social support, but by providing opportunities to try out new lexical units of communication and by affirming the value in students’ attempts at analysis and writing. Practice and feedback are key in learning. As this took place while lab experiments were running, all students had access. Spring/summer interventions using a guided rubric and peer-evaluation impacted Asian ELL students positively.

One of our limitations was in the ELL dilemma of not fully understanding directions: some students did not complete the survey, some did not have a completed LLR ready for proofreading one week before the due date, and some did not turn in the final report electronically for LIWC analysis. A limitation of our study was small sample size, prompting our unconventional use of arithmetic mean to describe non-continuous Likert values. Even with possible bias in only investigating those few who came to office hours or talked to the instructor about scientific writing, a strength was the random sampling selection method as representative of the true population of all Asian ELLs in class who might struggle. This might also help other ELLs.

When scientific writing is suboptimal, redesigning rubrics and using LIWC could perhaps help assess cognitive and analytical skills more accurately. Before failing students on a writing task, or judging them to be poor scientists, perhaps if we take the time to use an analytic program to complement our grading rubric, we could assess them higher and more on par with native-English-speaking peers in some areas, even if their work doesn’t read well.

Finally, while this study stemmed from the observation of writing challenges in one specific population in our classroom, we would like to understand how both the teaching practices and the LIWC assessment practices impact other underrepresented groups. Initial studies show promise with our small population size, and further studies are ongoing. Together these findings suggest methods useful in supporting and assessing scientific writing communication for design of future instruction and other support mechanisms to benefit Asian ELL students—and perhaps all students.