Evaluating the Use of Automated Feedback on Student Motivation

Abstract

Objective

Autonomy-support and quality feedback have been shown to significantly impact student learning. Unfortunately, quality feedback is often very time consuming. The goal of this study was to examine the autonomy-supportive potential from feedback provided via technology, specifically the ExamSoft Strength and Opportunities report.

Methods

Students were divided into two groups: one received enhanced performance feedback via the ExamSoft Strength and Opportunities report, the other received traditional grade-only feedback in the form of grades and course averages, via ExamSoft. The Learning Self-Regulation Survey (SRQ-L) was administered to 139 third-year doctor of pharmacy students. The survey contained 14 statements asking students to rate their reasons (relative autonomy) for reviewing ExamSoft feedback, three items related to use of the ExamSoft technology, and five demographic items.

Results

A statistical difference, t(52) = -2.07, p =0.043; d = .577, was reported between the two groups indicating enhanced feedback via ExamSoft had a moderate impact on students’ autonomy.

Conclusions

Students who received enhanced ExamSoft feedback reported significantly higher relative autonomy index (RAI) for feedback review than students who did not receive the feedback. This suggests that the enhanced ExamSoft feedback was autonomy-supportive.

INTRODUCTION

Ask any faculty what motivates his/her students and you may get a myriad of answers. Students can participate in class activities for any number of reasons including good grades, recognition, a deep desire to learn, or even parental approval. Self-Determination Theory (SDT)¹ offers a framework to explore the extent to which individuals willfully engage in activities and actions.² SDT states all individuals have three basic psychological needs: need for competence, need for relatedness, and need for autonomy. Experiences that support these three needs are believed to be the most motivating, increasing the likelihood that the individual will return to the activity.

Though it began as a study in motivational psychology, SDT³ quickly became a prominent theoretical framework for research in education, healthcare, and athletics. Action is the result of choice, which can be either internally or externally motivated. Choices, the product of motivation, can be internal or external. The extent to which motivation is internal or external defines the self-determination of the choice. Self-determined choices are made when an individual willfully engages in an activity because it meets all or some of the three psychological human needs. Choices that are made due to external pressures (e.g. parental approval, external recognition, etc.) are less self-determined.² In other words, when external motivation is the primary cause for activity, individuals are less likely to be invested in their choice because little, if any, of their psychological needs have been met.

Although each psychological need is valuable to the self-determination of choice, autonomy, specifically, has been a focus of educational research because it is believed to be the most influential in increasing motivation and deep learning.² Most faculty would attest that motivated students of all types are easier to teach; however, research indicates that students with high levels of autonomy are higher achievers.³

Students with high levels of autonomy come to the classroom with a keen interest in the subject and pursue it because of an intrinsic desire to learn rather than a desire to receive reward (e.g., grades) or subvert punishment. These students either believe the subject to be naturally interesting or see it as a natural progression to their ultimate goal. Unfortunately, few, if any students, find every activity or topic to be equally motivating or interesting. Luckily, when autonomy is not innate, it can be influenced through autonomy support, resulting in the same positive educational outcomes that are often associated with the most intrinsically motivated students. Autonomy support has been shown to increase: student satisfaction;^4–5 student persistence;⁶ student creativity;⁷ the students’ engagement and enjoyment in the course and program;⁸ and learning and achievement.^9–11 The positive consequences associated with autonomy support should encourage faculty to seek ways to increase this construct for all students.

There are a variety of ways to support student autonomy. The primary way in which faculty support student autonomy is through provision of choice, rationale, and empathy.¹² When students are able to act on the choices they make,¹³ they exercise their autonomy, but in those cases where choice is unavailable, faculty should provide rationale to the students.¹⁴ Other ways to offer autonomy support to students include connecting future learning and professional experiences to the activities of the class,¹⁵ and providing autonomy-supportive feedback.¹⁶

Like autonomy support, feedback has also been shown to positively affect student success. Among the many factors that influence learning (student study behavior,¹⁷ instruction quality,^18–19 and innate student characteristics such as personality and personal interests²⁰) feedback is recognized as one of the most impactful. A synthesis of over 500 meta-analysis studies on the factors that influence achievement²¹ concluded that feedback is one of the best ways to positively impact student learning, with an average effect size of 0.79; however, through further investigation the study concluded that not all feedback is created equally. Feedback that offered punishment or reward had little positive impact on learning, whereas feedback that provided students with an “informational component” and was related to learning outcomes was the most effective.

It has been said that feedback should offer students insight on how their current work relates to the goals and standards of learning.²² However, Hattie and Timperley²³ asserted that the best feedback has a third component; in addition to communicating the goals of the course or assignment (feed up) and providing students with information on their current achievements (feedback), good feedback should also provide students with information on how to improve (feed forward). When feedback follows this format it provides students with enough information to make decisions on how to increase achievement. This provision of choice encourages student participation²⁴ in the process of remediation and improvement. Thus, feedback that follows the feedback, feed up, and feed forward model could be said to be autonomy-supportive.

Research indicates that there are many benefits to providing students with autonomy-supportive feedback. Unfortunately, providing autonomy-supportive feedback takes time – more time than pharmacy faculty might be prepared to give. Large class sizes and additional responsibilities make it difficult for tutors and faculty to provide the detailed feedback students need.²⁵ Technology may offer a solution to this problem. When technology is incorporated into the assessment process, it can allow faculty to spend more time on teaching, learning, and consulting with students, and less time on the administrative tasks required of the job.²⁶ The purpose of this study was to examine if feedback provided by a computer-based assessment and analytics platform, specifically ExamSoft (Dallas, TX), can be autonomy-supportive. The study was granted an exempt status by the University of Kentucky Institutional Review Board.

METHODS

This study was incorporated into the 2015 fall semester of the third professional year Patient Care Laboratory (PCL) course. The PCL course sequence spans the first six semesters in the doctor of pharmacy curriculum. Classes meet twice per week, once for seminar and another for practice. The practice sessions are divided into four sections conducted on the same day, two in the morning and two in the afternoon. Students in the third year self-select their preferred lab section time before the semester begins. Lab sections are designed to be equal with an enrollment of approximately 35 each. Students in all sections received the same instruction, topic sequence, and assessments.

Students (N=139) were divided into morning and afternoon sections to create two groups for the study. Students in both groups were provided feedback via ExamSoft, an online assessment tool. Students in the morning sections received two ExamSoft reports which consisted of traditional feedback which included only grades and class averages on course assignments and activities. Students in the afternoon sections received two enhanced feedback via the ExamSoft Strengths and Opportunities report. In addition to the student’s grade and the class average, the ExamSoft Strength and Opportunities report provided three additional pieces of information that the morning students did not receive: 1) a list of outcomes associated with the course and activities; 2) information on students’ numeric performance in each outcome; and 3) a color-coded icon (red, yellow, or green) to indicate which content areas were in need of remediation.

To help understand the effect of ExamSoft feedback on motivation, students in all four sections were asked to complete a survey asking them to indicate why they review ExamSoft feedback in the PCL course. The Learning Self-Regulation Survey (SRQ-L) served to make up the majority of the survey tool. The questionnaire can be adapted to reflect the learning-related activity of interest. In the case of this study, students were asked the SRQ-L items to capture student motivation for reviewing ExamSoft feedback.

The SRQ-L is a validated tool^27–29 consisting of 14 items, which ask students to indicate the various reasons they engage in a targeted learning-related activity. The SRQ-L requires respondents to score statements on a seven-point scale, indicating how like or unlike them the statement was. Per the scale instructions, a score of one on a statement was “Not at all true” for the student, a score of four was only “somewhat true” for the student, and a score of seven was “Very true” for the student. The survey’s questions are equally divided into two sub-scales: autonomy regulation (seven items) and controlled regulation (seven items). A high autonomy regulation score indicates that the student is more motivated by intrinsic factors (e.g., personal interest); a high controlled regulation score indicates that the student is more motivated by extrinsic factors (e.g., grades, status).

Using Qualtrics (Provo, UT), the survey was sent via email to all 139 students enrolled in PCL near the end of the semester. In keeping with the research goal of supporting autonomy, survey completion was not mandatory. Students were reminded via email and in class to take the survey, but no efforts were made to pressure students to complete and no time was given during class to submit the survey. Eighty-one students submitted the survey electronically for a response rate of 58%. Because of the scale used to conduct analysis, only complete surveys with no omitted items from the SRQ-L were included in analysis. For seven cases, a single item from the SRQ-L was left unmarked which left a total of 74 usable responses, 44 from the AM (traditional ExamSoft feedback) group and 30 from the PM (enhanced ExamSoft feedback) group.

Though the survey response rate was lower than generally desired, the numbers were higher than expected given the timing of the survey (near finals) and the fact that completion was neither mandatory nor rewarded. In an effort to validate the results of the survey, statistical analysis was conducted to determine if the respondents were representative of the pharmacy year three population. A significance of a < .05 was set a priori. First, using SPSS an independent samples t-test was conducted to compare both the cumulative and semester GPA of the respondents (n=81) to the non-respondents (n=58). No statistical difference was found in semester GPA, t = 1.345, p = 0.181, or cumulative GPA, t = 1.137, p = 0.258, which indicated that the respondents were academically equivalent to the non-respondents. Second, a Chi Square Goodness-of-Fit test was conducted to compare the demographic characteristics of the respondents (n=81) and the pharmacy year three population (N=139). Neither gender (Male/Female), X2 (1, N=81) <0.001, p =0.996, nor ethnicity (White/Non-white), X2 (1, N=79) = .540, p > 0.462, were shown to be significantly different between the observed and expected populations. It was concluded that the survey’s respondents were generally as expected in regard to demographic characteristics when compared to the population of the pharmacy year three. Though the response rate is lower than desired, analysis indicated the respondent population was representative of the course’s enrolled population.

Survey responses were calculated for the SRQ-L’s two sub-categories: autonomy regulation and controlled regulation. Item scores for each sub-category were summed to create two scores: Autonomy Regulation and Controlled Regulation. For each student the Relative Autonomy Index (RAI) was also determined by subtracting the student’s controlled regulation score from his/her autonomy regulation score for each feedback type. RAI indicates the relative strength of an individual’s autonomy regulation for a specific activity. Positive RAI scores signify the student has greater autonomous regulation (intrinsic motivation) for the activity,²⁷ whereas, negative RAI scores indicate a student has greater controlled regulation (extrinsic motivation) for the activity.²⁸ A difference in RAI between the two groups would indicate that one activity was more autonomy-supportive than the other. To determine if RAI scores were different between the two groups, an independent samples t-test was conducted using all RAI scores in the sample.

To measure internal consistency of the scale, Cronbach’s alpha was calculated. The alpha coefficient for all ExamSoft items was 0.868, suggesting that items had a high internal consistency. Data analysis was conducted using independent samples t tests. Statistical analysis of data was conducted using SPSS, v22 (IBM, Armonk, NY) and an online calculator (University of Colorado, Colorado Spring, CO). The level of significance (two-tailed) was set at a < .05.

RESULTS

Table 1 provides a summary of student (n=74) responses to the SRQ-L surveys addressing student motivation for review of the two feedback types. Generally, students in the PM group rated the autonomous-regulated items higher than their peers in the AM group. By subtracting the sub-score for controlled regulation from the sub-score for autonomous regulation, RAI was calculated for each student and ranged from -14 to 28 for the students in the AM group and -8 to 28 for students in the PM group. A significant difference was found in the RAI for ExamSoft feedback between the AM group (M = 7.48, SD = 8.02) and the PM group (M = 12.13, SD = 10.36), t = -2.075, p = 0.043, indicating that students in the PM (enhanced feedback) group were more autonomously regulated to review feedback given in ExamSoft than the AM (traditional feedback) group. Table 2 provides a summary of RAI scores for each feedback group.

Table 1. Results of SRQ-L ExamSoft Survey Items for Patient Care Laboratory Students by Group.

	AM group			PM Group
Item Response Scores	N	Mean	SD	N	Mean	SD
I will participate actively in reviewing my ExamSoft feedback:
*Because I feel like it’s a good way to improve my understanding of the course’s and program’s content.	44	4.16	1.829	30	4.70	1.985
Because others would think badly of me if I didn’t.	44	2.43	1.576	30	1.93	1.701
*Because knowing about my achievement in learning outcomes is an important part of becoming a pharmacist.	44	4.77	1.641	30	5.37	1.866
Because I would feel bad about myself if I didn’t look at these reports.	44	2.55	1.635	30	2.23	1.654
I am likely to follow my instructor’s suggestions for reviewing ExamSoft feedback: Because I would get a good grade if I do what he/she suggests.	44	3.64	1.989	30	3.63	2.008
*Because I believe my instructor’s suggestions will help me learn effectively.	44	4.41	1.633	30	4.77	1.431
Because I want others to think that I am a good student.	44	2.86	1.622	30	2.43	1.832
Because it’s easier to do what I’m told than to think about it.	44	3.52	1.923	30	3.07	1.999
*Because it’s important to me to do well at this.	44	5.25	1.644	30	5.77	1.478
Because I would probably feel guilty if I didn’t comply with my instructor’s suggestions. The reason that I will continue to review	44	3.32	1.865	30	2.67	1.605
ExamSoft feedback in the future is: *Because it’s exciting to learn about my strengths and weaknesses on OSCEs.	44	3.82	1.808	30	4.20	2.074
Because I would feel proud if I did continue to improve in the course.	44	4.84	1.584	30	5.20	1.919
*Because it’s a challenge to really understand what I am learning.	44	3.93	1.717	30	3.73	2.033
*Because it’s interesting to use the results to try to identify my strengths and weaknesses in the curriculum.	44	4.30	1.651	30	4.77	2.079

^aScale: 1(Not at all true) – 7 (Very true)

^*Autonomous-regulation items

Table 2. Relative Autonomy Index (RAI) for ExamSoft Feedback by Type.

	N	Min	Max	Mean	SD
RAI – Traditional	44	-14	28	7.48	8.016
RAI – Enhanced	30	-8	28	12.13	10.355

In order to determine the effect that the enhanced ExamSoft feedback had on student motivation, Cohen’s d was calculated. Using Cohen’s guidelines for interpreting effect size³⁰, a result of 0.577 indicates that provision of enhanced feedback via ExamSoft had a moderate impact on students’ autonomy.

To better understand the influence of the autonomy-supportive feedback on student performance, independent samples t-tests were conducted to analyze the grades for each group on the PCL objective structured clinical exam (OSCE) and the overall course grade. No statistical difference was found for any of the t-tests, which indicates that students in both the traditional feedback sections and the enhanced feedback sections performed similarly. Table 3 provides a summary of the t-test results for student grades (N=139).

Table 3. Results of Independent Samples t-Test for PCL Grades by Type.

Traditional Feedback				Enhanced Feedback
	M	SD	n	M	SD	n	t	df
OSCE Exam	73.09	10.78	70	73.04	14.25	69	.979*	137
Final Score	91.22	2.45	70	91.01	3.28	69	.683*	137

DISCUSSION

Although not required for use, ExamSoft allows for the alignment of assessment items (i.e., test questions and rubric criteria) to course, program, and/or institutional outcomes. Programs have chosen a variety of ways to implement the alignment feature, ranging from no alignment (i.e. grade only), to simple alignment (i.e. aligned to course-related outcomes/topics only), and for some, complex alignment (i.e. aligned to a range of course, program, and institutional outcomes). This study attempted to compare two of the ways in which faculty might choose to implement ExamSoft: no alignment/grades only, and simple, course-related alignment.

The simple alignment process required faculty in the PCL course to create categories for alignment in the ExamSoft system and to align individual rubric criteria and questions to the appropriate categories. The entire simple alignment process took approximately an additional five minutes per assessment. Once aligned, the provision of system-generated feedback could be accomplished in a few clicks of the mouse. There was no additional time associated with providing reports that included enhanced feedback compared to grade-only feedback. For a small investment of faculty time, students reported higher levels of autonomy when reviewing the enhanced ExamSoft feedback.

The enhanced ExamSoft feedback most likely increased student perception of autonomy-support due to the additional information it provided. ExamSoft’s enhanced feedback contains three key components necessary for autonomy-supportive feedback: a list of relevant outcomes and topics aligned to the assessment; grade and class average information; and color-coded icons that indicate if a student performed well (green), moderately (yellow), or poorly (red) at achieving the goals of the assessment. These three aspects of the feedback fit the autonomy-supportive model. First, by providing students with a list of assessment-specific outcomes, the report feeds up - students know what it is they are trying to accomplish. Second, by providing grade and class average, the report feeds back – students know how they have done toward the goals of the course and/or assessment. Finally, by providing students with color-coded icons the report feeds forward – students know what to work on in order to improve. Traditional reports containing grade and class averages only, give little information the students can use to improve or interpret their performance.

Though previous research indicates that autonomy support improves student learning, this study found no improvement in grades between the two groups. The feedback likely had no impact on grades because the instances of autonomy-supportive feedback were too few to impact overall course performance and because grades in the course are generally quite high (average final course grades ranging from 92% to 94% over the past five years) even without autonomy-supportive feedback. Although we would not expect grades to increase significantly in any circumstance for this course, the impact of more frequent implementation of autonomy-supportive feedback on overall learning and retention is an area that warrants exploration in future research.

As educators with increasingly limited access to resources, we should be more cognizant of how we use our time, money, and supplies to benefit student achievement. Although grades are the traditional measure of student learning, they are not the only metric we wish to impact through autonomy-supportive feedback. In fact, in a program in which students achieve high marks consistently, we may never witness an increase in grades due to the implementation of these interventions. Possibly even more important for a professional education program, we also want to implement autonomy-supportive feedback because it has been shown to promote student satisfaction; ^31–31 student persistence;³³ student creativity;³⁴ the students’ engagement and enjoyment in the course and program;⁸ and learning and achievement.^{10, 35}

The findings of this study are meaningful to pharmacy faculty who are seeking time-efficient methods of delivering, substantive feedback to simultaneously meet student requests and accreditation standards. Not only do students desire more frequent feedback, the Accreditation Council for Pharmacy Education (ACPE) Standards 2016³⁶ articulate expectations that programs provide students with formative performance feedback in didactic and experiential courses alike. Due to the time-consuming nature, robust student-specific feedback is often difficult to provide consistently in large clinical patient care simulation courses. Autonomy-supportive feedback via ExamSoft is one method of providing formative performance feedback that is both timely for the learners and efficient for faculty. Faculty should feel empowered that a relatively small intervention can provide an efficient means of delivering the increased feedback students desire.

Standards 2016 also expect programs to instill in students the attitudes and behaviors compatible with a commitment to lifelong professional development.³⁶ A primary concern of the program is to produce life-long learners with a deep love and dedication to the profession. Efforts to support student autonomy stem more from a desire to increase student performance over the course of their career rather than through increased grade averages in a single course. The authors believe frequent use of autonomy-supportive feedback can foster the development and improved self-awareness of personal knowledge, skills, and abilities by allowing students to choose areas for personal development or remediation. Although additional elements are required for the full maturation of professional attitudes and behaviors, enhanced self-awareness is essential and requisite to the development of a commitment to lifelong learning.

Although the research has reached its aims, there were some unavoidable limitations. First, the sample size was small. Only one cohort from one school was included in this research. In addition, the response rate was lower than desired. Future research should seek to include students from a variety of cohorts and institutions to verify the results of this study. Another limitation is the lack of prior research in pharmacy related to the topic of autonomy support. This topic has been thoroughly researched in other fields, including medicine, but there was little research found related specifically to pharmacy education. We believe that this research serves to help fill a gap in the literature, but we also note that it provides little ability to compare results in the profession. We suggest that additional research on autonomy support be conducted in pharmacy education to help educators better understand how their education practices affect student motivation.

CONCLUSIONS

Students who received enhanced ExamSoft feedback reported significantly higher relative autonomy (RAI) for feedback review than students who did not receive the feedback. This suggests that the enhanced ExamSoft feedback was autonomy-supportive. These results indicate enhanced ExamSoft feedback can be autonomy-supportive and should be used when faculty have access to the tool.

Funding/Support

None

Other Disclosures

None