Feedback Support for Training: Accounting for Learner and Task

Abstract

Years of research on feedback has not produced universal prescriptions for feedback during training. Results are split in two directions; those recommending more feedback during training and those recommending less. This has resulted in no unified theory and little understanding of other factors that might affect feedback mechanisms. The current experiment manipulated the cognitive load of the task and measured the working memory capacities of the learners to see whether forcing the learner to self-evaluate affects learning of a cognitive task and how this might be moderated by learner ability and task load. Results generally showed that reducing task load through supportive feedback resulted in more learning.

INTRODUCTION

Many feedback parameters have been studied, including frequency, immediacy, and content. Results are mixed: at times more frequent, immediate, and informative feedback improved learning (e.g., Tuovinen & Sweller, 1999; Wishart, Lee, Cunningham, & Murdoch, 2002), while in other studies less frequent, less immediate, and less informative feedback improved learning (e.g., Schmidt & Wulf, 1997; Schroth, 1997; Wishart, Lee, Cunningham, & Murdoch, 2002). Competing theories each explain some results, but not all.

Proponents of more feedback explained their results via cognitive load theory (Sweller, 1988; Sweller & Chandler, 1991, 1994). Cognitive load theory suggests that learning improves when the need to integrate elements of the task during training is reduced. Feedback that reduces the most extraneous cognitive load should result in increased performance during acquisition as well as higher performance on retention and transfer tests, compared to feedback that does not reduce (or increases) cognitive load.

Studies that found more learning through less feedback explained their results via two possible mechanisms. The first was transfer-appropriate processing, where training that most closely approximated what was be expected of the learner in retention or transfer should result in the most learning (Bransford & Schwartz, 1999; Schmidt & Bjork, 1992). Another possible explanation was that a learner must be challenged appropriately; tasks below a certain threshold do not elicit the attention required for learning (Guadagnoli & Lee, 2004; Snow, 1989).

A recently proposed model may explain these conflicting results (McLaughlin, Rogers, & Fisk, 2006). In a review of the literature, it was found that more feedback was beneficial to learning for cognitively demanding tasks or resource- limited learners (e.g., remedial learners, older adults, or novices). Thus, the benefit of more or less supportive feedback appeared resource-based: learners with low cognitive resources benefited from increased feedback support as did those confronted with a taxing task. On the other hand, most individuals (cognitively limited or not) needed more feedback for cognitively demanding tasks. These patterns appeared across feedback parameters.

OVERVIEW OF THE STUDY

The model proposed by McLaughlin et al. (2006) was tested in the current study by manipulating feedback support and task complexity for learners with either high or low working memory capacity. Thus, high working memory participants learned either a simple or complex task and received either highly supportive feedback during acquisition or minimally supportive feedback. Highly supportive feedback provided both knowledge of correct response (KCR) and information on every possible case for that trial. Less supportive feedback provided only KCR. Participants returned after one week to perform retention and transfer tests with no feedback present.

A limitation of previous feedback studies was that task load and learner capability were not manipulated or measured. The current study controlled these variables to understand the link between feedback and available cognitive resources. It was expected that learning would vary according to the support of the feedback provided, and this effect would differ by WMC groups depending on task load. According to discovery learning theory and “desirable difficulties” (Bjork, 1994), in a simple task, high working memory capacity (WMC) individuals should learn more with less supportive feedback whereas the opposite should be true for low-WMC individuals. However, in a complex task, high-WMC individuals should also benefit from increased feedback support. Cognitive load theory predicts all groups would benefit from highly supportive feedback (Sweller, 1988).

METHOD

Participants

Twenty-eight high-WMC and twenty-eight low-WMC young adults participated (18-35 years of age) and were paid $50 for their time. WMC was assessed using the automated- O-span test (Unsworth, Heitz, Schrock, & Engle, 2005) with high- and low-WMC defined as the first and fourth quartiles of performance compared to a community sample (Turner & Engle, 1989; Unsworth et al.). The O-span requires the participant to hold letters in working memory while solving math problems.

Task

Working memory capacity is a reliable, stable indicator of how quickly an individual may learn a task as well as being highly correlated with numerous abilities (Engle & Kane, 2004). An experimental task was chosen that loaded this ability: the logic-gate task (Carlson & Yaure, 1990; Schneider, 1985). The logic gates task can impose low or high working memory load depending on the number of interacting elements in a trial (Figure 1). Each gate (e.g., “NAND”) has a rule (e.g., “if no inputs are 1 then the output is a 1”) that is used to solve the output to the gate based on the inputs that enter the left side of the gate (Figure 1). Participants learned these rules from feedback provided in the task.

Figure 1.

Example feedback presentation for complex logic gate task. Shaded boxes not present in actual task.

Materials

Participants provided demographic information, had their vision tested, and completed several abilities tests (Table 1).

Participant Characteristics

	Low WMC n = 28				High WMC n = 28
	Simple Task		Complex Task		Simple Task		Complex Task
	M	SD	M	SD	M	SD	M	SD
Gender	5M 9F		5M 9F		7M 7F		7M 7F
Age	26.21	4.89	23.71	4.70	24.93	3.56	22.64	4.81
Near Vision1	22.31	5.99	21.54	3.76	23.57	4.97	20.71	2.67
Far Vision1 Shipley	19.54	7.47	19.93	7.32	23.07	7.43	16.93	3.81
Vocabulary2 Digit Symbol	27.43	3.39	26.86	3.23	29.50	4.00	28.71	8.43
Substitution3 Reverse Digit	65.93	16.89	66.00	12.31	71.00	8.12	71.93	9.08
Span3* O-span	6.50	1.99	7.21	2.29	9.29	2.20	9.71	2.61
absolute4* Trials to	12.86	8.38	14.43	8.60	59.57	6.38	59.21	4.76
criterion5*	2.43	1.16	2.14	1.03	1.57	0.65	1.36	0.63

Note.

^*indicates significant differences between WMC groups (p < .05).

¹Vision scored as Snellen acuity.

²Shipley, 1986.

³Wechsler, 1997.

⁴Unsworth et al., 2005.

⁵Number of attempts to complete logic-gate knowledge test prior to experimental task. There were no differences between simple and complex task groups (all p's < .05).

Tests

The first retention test occurred after a 10-minute filled interval following acquisition trials. The second retention test occurred seven days after the acquisition session. The transfer test required solving logic gates using “L” (for low) and “H” for high in place of the 1’s and 0’s learned previously. Logic gate inputs were also converted.

Equipment

The experimental task and tests were performed on IBM- compatible computers (3.2 GHz Pentium 4, 1 GB RAM). Screen size was 19” with a resolution of 1024 × 768 pixels and a refresh rate of 85 Hz.

Design and Procedure

The study was a 2 (Task load: simple, complex) × 2 (WMC: low, high) × 2 (Feedback support: KCR, truth table) factorial with feedback support as a within-participant factor, task load a between-participant factor, and WMC a quasi-independent grouping variable. The dependent variable was accuracy on the logic gate retention and transfer tasks (though acquisition effects are also reported).

Task Load

Task load was manipulated as the number of mental computations required to solve a single trial of the logic gate task. The low-load condition (simple) required solving one logic-gate whereas the high-load (complex) condition required solving three logic-gates per trial wherein the answer to the third gate depended on the output of the first two gates.

Feedback support

“Low support” was KCR and “High support” was truth-table information added to the KCR that provided all possible inputs and outputs for a gate (Figure 1). Participants received low feedback support for half of the ten gates and high support for the other half.

Session one

Participants gave informed consent and completed abilities tests. They were allowed five minutes to study the logic gate rule-sheet before taking a matching test of gate and definition. Once criterion was reached (100% correct) the experimenter demonstrated solving logic gates. As indicated in Table 1, low-WMC participants required more trials to reach the criterion. Practice consisted of participants solving ten trials in their assigned load condition using the rule-sheet with definitions.

In the acquisition phase, participants performed 10 blocks of 20 trials each followed by an 11^th block with no feedback present. They then completed the Shipley vocabulary test and then the 10-minute retention test block.

Session two

Participants returned after one week to perform 80 trials of the logic gates task with no feedback followed by the transfer test. Last, participants filled out the exit interview, received debriefing and payment for their time.

RESULTS

Accuracy data were divided into acquisition and retention/transfer sessions for analysis. Simple task trial. Complex task accuracies were computed as correct if the last gate in the problem were answered correctly.

A three-way analysis of variance (ANOVA) was conducted to evaluate the relationship between WMC, task load, and feedback condition for each phase of learning and on transfer tests. First and last block of acquisition were included in the acquisition analysis as a repeated measure.

Acquisition

Averaged over acquisition there was a marginal interaction of feedback × WMC where low-WMC participants benefited more from the truth table feedback than the KCR and high-WMC participants performed similarly with both, F(1,52) = 3.80, p = .057, ηp² = .07, (Figure 2).

Figure 2.

Performance and learning data from start of acquisition through 1-week retention interval. Bars represent standard error.

Retention and Transfer

10-minute retention

Feedback interacted with WMC, F(1,51) = 8.85, p= .004, ηp² = .15. Low-WMC participants’ performance on truth table gates was lower than their performance with KCR, t(26) =3.36, p = .002; There was no difference in high-WMC participant performance between feedback conditions (p > .05). There were only marginal differences between WMC groups for KCR feedback (p = .078). However high-WMC participants were significantly more accurate than low-WMC participants when provided with truth table feedback, t(53) =−4.53, p < .001).

For this initial retention measure, high-WMC participants appeared able to use the additional information contained in the truth table feedback to increase their learning. Low-WMC participants did not show this same benefit and were more accurate with the less supportive feedback condition. It is possible the low-WMC participants were overwhelmed with the information available in the truth table feedback.

1-week retention post score

After one week, a feedback main effect was found in favor of the truth table condition, F(1,52) = 4.90, p = .031, ηp² = .09. The low-WMC participant benefit for KCR at the 10-minute retention interval was no longer evident.

Retention change score

A change score was calculated by subtracting accuracy on the 10-minute retention test from accuracy on the 1-week retention test. A significant interaction between feedback, WMC, and task load indicated differences in retention change scores, F(1,51)= 5.27, p =.026, ηp² =.09. In the simple task, low-WMC participant performance did not change across the retention interval for truth table gates (p=.41). However, there was a significant loss for KCR performance (t =−2.42, p = .031). Accuracy in both feedback conditions declined for the low-WMC participants in a complex task (p’s >.05). High-WMC participants showed no change with either task load or feedback type (p’s >.05). There was a main effect of feedback where performance declined more for the KCR gates, F(1,51) = 5.70, p= .021, ηp² =.10.

In summary, participants performed more accurately in the truth table condition. High-WMC participants retained the task well over the week interval, demonstrating no decline in performance. Low-WMC participants benefited most from KCR at the 10-minute retention interval, however, in one week gate accuracy in the KCR condition declined while performance on truth table gates did not change.

Transfer

Truth table gates were solved more accurately than the KCR gates on the transfer test, F(1,51) = 7.84, p =.007, ηp² =.13. Contrasts revealed low-WMC participants were more accurate on truth table gates on the complex task than for KCR gates, t(13) = −3.78, p =.002.

The increased information present in the truth tables resulted in more accurate performance on the content transfer test. In summary, when there were effects of feedback (for retention and transfer), there was a benefit for the more supportive truth tables over KCR.

Subjective Data

Participants were asked which feedback level helped them to learn the gates most thoroughly and why. Participants preferred the truth table feedback, but mentioned different reasons. Low-WMC participants frequently mentioned trying to memorize outcomes from the tables (e.g., “table” “Helped me memorize them.”) while high-WMC participants mentioned using the tables to discover patterns, re-discover a forgotten rule, or develop a personal rule from the tables (e.g., “table” “I had information in front of me to put a theory together.”) However, many participants answered that both types of feedback helped them or provided no useful information (e.g., “Just the kind of person who likes doing well.”) which made firm conclusions from the subjective data problematic.

DISCUSSION

The results of this study support the idea that reducing task load via increased feedback support results in more learning. When there were differences between WMC groups, high-WMC participants were better able than low-WMC participants to use the increased feedback to improve their performance in acquisition, retention, and transfer. However, low-WMC participants also benefited from the increased support of the truth tables, if not as much as high-WMC participants. As stated previously, feedback was not only support for performance; it may be a tool to be used by those able to take advantage of it.

That the additional support resulted in learning for both high- and low-WMC groups may be explained through their different use of the same feedback. There is no one-size-fits-all design for training or even for specific populations. It is most useful to examine what feedback is doing or allowing under high and low support. The following is an analysis of how different learners could have utilized feedback in the current experiment.

This experiment included highly capable learners (i.e., the high-WMC group). Though these participants learned the logic gate task under all feedback support levels, they learned better via more supportive feedback (i.e., the truth table). There were sometimes no differences between feedback conditions, but there was no case in which they learned more from the less supportive feedback. These results may be explained in their ability to utilize all information in the feedback. Subjective measures indicated that these learners looked for patterns or developed concepts as to how the logic gates worked. They were then able to test and re-evaluate upon further feedback receipt. Participants did not report using the additional information as much for memorization of answers as they did for hypothesis testing and model development.

Pattern-detection, hypothesis testing and model development all require meta-cognition, self-analysis, and problem solving. Thus, few of these benefits would necessarily be available to learners with low working memory capacity. Low-WMC participants in the current studies also showed more learning under conditions of increased feedback, so the question remained that if they were not also developing and testing hypotheses, what were they doing?

Feedback provided guidance through the task as well as a clear standard of performance. It may also simplify the task by removing the component of self-assessment. There was no time limit on looking at the feedback, so the additional processing load may have been minimal and the benefit of not having to generate their own feedback large. Thus, even without the meta-cognitive analysis or complex problem solving strategies available to high ability learners, the low ability learners could still benefit from increased feedback support for two reasons: (1) when more feedback support was available, the reduced load allowed low ability learners to maintain some performance while learning the task and (2) it allowed the learner to experience what a correct response looked like. Because of the limited subjective data available, these explanations need to be investigated in further feedback research designed to reveal the strategic methods of low and high ability learners

Cognitive load theory supports the first assumption that a reduction in task load should result in more learning (Sweller, 1988), true for the low-WMC participants. Of course, this may also explain the better performance of the high-WMC learners under additional feedback support, but it seems likely their use of strategies upon evaluating their own performance played a role as well.

Perhaps the most interesting finding is the lack of superiority for the relatively less feedback condition. Many articles from the last two decades are adamant about the superiority of “less” feedback compared to more. Once explained, it appears to make sense that those with the resources to do so might learn best with little external aid. The current studies controlled for cognitive resources, and yet there was no case in which less feedback outperformed more, despite numerous predictions to the contrary.

If too much feedback becomes a crutch, or inextricably part of the task, then KCR should have resulted in better retention than truth tables, however this was not the case for any condition or group. If feedback kept learners from internalizing their own feedback, one would expect superior learning from KCR compared to truth tables. Again, this was not the case. Next, if feedback may block other information- processing activities that could result in the capability to perform when feedback is withdrawn, less information should result in better retention. However, the information in the feedback only increased retention performance. Discovery learning predicts KCR should have outperformed the truth tables because KCR provides needed information but requires the learner to “discover” the other cases for a gate. However, providing those cases explicitly through a truth table resulted in better retention. Last, there are many proponents of desirable difficulties in instruction because they supposedly lead to transfer appropriate processing through retrieval practice (Bjork, 1988). However, in all groups and conditions, performance increased as difficulties were removed. Even high-WMC participants learning a simple task benefited from a decrease in difficulty during acquisition.

In conclusion, learners benefit from increased feedback support in multiple scenarios. First, users who are very capable may think about their own performance, the feedback, and how the two match or do not match and why. Because they can see patterns in their performance they may benefit from additional information provided. This is most likely to happen when the task is not overwhelming, so there is time and attention remaining for considering their performance. This scenario fits with the objective and subjective data obtained on learners with high-WMC in the current experiment. Second, learners can benefit from increased feedback support because the task load is reduced by the feedback. This scenario fits with both the WMC groups included in the current study, but especially for the low-WMC group.