Understanding how to enhance efficacy and effectiveness of feedback via e-portfolio: a realist synthesis protocol

Abstract

Introduction

The validity of feedback as one of the defining components for electronic portfolios (e-portfolios) to be effective and efficacious has yet to be demonstrated. While the literature has shown individual beneficial features of e-portfolios and feedback per se, evidence of feedback as mediated through technology directly resulting in improved educational practice is scarce. The explanation of how feedback via e-portfolio improves educational practice is particularly vague.

Methods and analysis

The aim of this research is to unpack how and why feedback via e-portfolio is likely to flourish or wither in its path. Given the complexity of intervention, we will apply a theory-driven approach for evidence synthesis called realist synthesis. Informed by realist philosophy of science, it seems the most appropriate method because it explores observed outcomes (O) in terms of causal relationship between relevant contexts (C) and generating mechanisms (M). Initial programme theory will be developed through literature scoping. Later on it will be tested against purposively gathered evidence (through database and journal search), which simultaneously will be evaluated for rigour and relevance (whether method used are trustworthy and whether data contributes to theory building). We strive to (1) uncover ‘context sensitive’ mechanisms that generate feedback via e–portfolio to be (in) effective and (2) define in what circumstances is this mostly likely to occur.

Ethics and dissemination

The synthesis report will be written according to the RAMESES guidelines and its findings will be published in peer reviewed articles and presented at relevant conferences. The aim is to inform: (1) policy and decision makers for future-course design; (2) medical educators/clinical supervisors and learners for improved educational use. No formal ethical approval is required.

PROSPERO registration number

120863.

Strengths and limitations of this study.

• With realist synthesis we account for the breadth and depth of analyses appropriate for complex educational interventions.

• No prior realist synthesis has been undertaken on the topic of how feedback via electronic portfolios works effectively.

• In developing our initial programme theory we include stakeholder group inputs.

• Content experts are not included in programme development.

• Only studies published in English language will be searched.

Background

Introduction

Despite variations in content and format, portfolios are essentially a means through which healthcare learners can report on work done, feedback received, progress made and their plans for improving competence.¹ Portfolios in postgraduate healthcare education can be employed for a range of end-purposes including reflective practice and assessment (summative and formative), and act as an essential connection between workplace learning organisationally and individually.² As such, the content of a portfolio may vary according to the requirement of an organisation and the design of the training programme. For example, the content of medical trainees’ electronic portfolios (e-portfolios) may include quantitative assessments (eg, the Mini-Clinical Evaluation Exercise, Direct Observation of Procedural Skills, Case-based Discussion and 360-degree Evaluation), reflective writing (eg, a medical ethics and legislation report, healthcare quality report and personal development report) and an evidence-based medicine report. In the context of such a portfolio, clinical teachers are required to provide appropriate feedback for trainees on their assessment and reports contained within.³ Finally, portfolios can be either physical documents, or can be managed online (known as an e-portfolio).

The interest in e-portfolio use in healthcare education has been on the rise. This is probably because both, portfolios in general and electronic versions in particular, have shown to be beneficial to the user. In all their complexity of design, content and interface,^{1 4} what makes them stand out from other educational tools is their ability to encourage reflective practice and self-directed learning,^{5 6} which caters perfectly to the educational discourse that emphasises competence-oriented, individualised learning styles. By emphasising feelings of ownership and personal development,⁷ they encourage learners to become more self-aware of their learning process and more responsible for their own creation, maintenance and presentation.⁸

Contextual use of electronic portfolios in healthcare education

E-portfolio in healthcare education is foregrounded in its flexibility of access, repository, and content.1 2 9 10 When explaining its usage, scholars tend to emphasise its contextualisation. For instance, the nature of implementation, design and content^11–14 and the individual perceptions of ease of use and usefulness¹⁵ are all important facets affecting the e-portfolio use and its potential to fundamentally transform the learning process.

Rather than dwelling on the notion of e-portfolio being merely a combination of portfolio and technology,¹⁶ in this paper, we try to argue how organisational, cultural and individual factors present a significant entry point for theorising the e-portfolio use. More importantly, we do so by focusing specifically on feedback portrayed via e-portfolio. We aim to understand (1) in what circumstances feedback via e-portfolio works most effectively and (2) whether this relates to fortunes and mishaps of e-portfolio use?

Effectiveness of feedback via e-portfolio

Feedback plays an influential role in educational achievements¹⁷ and when employed in healthcare settings, it is indispensable for successful learning, clinical teaching and improved clinical performance.^{18 19} Surprisingly, in healthcare education, little is known about how feedback can be used to maximise its impact on learning, behaviour and improved practice, and much less so when talking about technology-enhanced feedback.

One reason for this might be that the majority of research papers on feedback published between 1980 and 2015 used the lowest of Kirkpatrick’s levels of evaluation—assessing reactions to feedback—and amongst all the studies, only 7% out of 650 included articles were about computer-based feedback.²⁰ Literature interpreting feedback as a one-way, educator-driven processes, with a focus on best delivery practices only, might be another reason. Indeed, educational studies have shown time and again that the high variability of effective feedback is too complex for it only to be explained with the notion of delivery processes.^{17 21} The many facets of learners’ feedback seeking behaviours3 20 22 23 as well as the gaps occurring between mentor’s and learner’s perceptions of the quantity, quality and efficacy of feedback have to be reconsidered if we are to completely understand feedback practice. Indeed, feedback via e-portfolios can occur variously, including: as asynchronous written feedback in which the educator leaves their comments for the learner to find and read, as synchronous technology-enhanced feedback, as synchronous face-to-face feedback, as mandatory or voluntary and as open access or not.

The aim of this research is to develop a model to facilitate feedback via e-portfolio, and thus enhance/improve the responsiveness and use of feedback; meaning, we need to understand the contextual workings for giving and receiving feedback in a technology enhanced environment. In addition, we have to consider not only the provision of information, but also the influence of the manner in which feedback is provided, the recipient’s decision to receive feedback and all the contended responses which might subsequently arise.

Methods

Aim

Focusing on higher educational settings internationally, we aim to understand why and how feedback via e-portfolio might produce different outcomes. For this purpose, we plan to use the Kirkpatrick hierarchy model modified by Tochel et al ² and distinguish outcomes that describe the impact of intervention in terms of:

1. Participants’ reactions (eg, their views on learning experiences, attitudes towards e-portfolio use and usefulness, aspects on the nature and efficiency of feedback).

2. Changes in participants’ attitudes and learning (eg, changes in perceiving e-portfolio or feedback as useful, acquisition of new concepts, improvement of skills).

3. Changes in participants’ behaviours (motivational changes for further learning, active engagement with agency, e-portfolio content, application of new knowledge).

4. Changes in organisational practices and any improvements in the health and well-being of patients occurring because of the intervention.

Research questions (RQ)

(RQ1) What outcomes are identified resulting from feedback via e-portfolio, and at what level do they occur?

(RQ2) What mechanisms are identified that relate to: (1) positive outcomes of feedback via e-portfolio, (2) negative outcomes of feedback via e-portfolio?

(RQ3) What are the contexts within which the mechanisms trigger these outcomes, and for whom?

Realist synthesis

To address our RQs within a rapidly developing methodological field of data synthesis,²⁴ we choose a theory driven approach called realist synthesis. Underlined by realist philosophy of science, the method’s hallmark is in its generative understanding of causality. It holds that outcomes (O) of events are generated by/through underlined mechanisms (M), which may or may not occur in certain context (C).²⁵ Mechanisms are not ‘visible’—having their rooting in individual tendencies—and are ‘context specific’—and changeable according to the opportunities provided by specific context(s). Realist synthesis thus looks for interactions among the resources provided by the intervention and the reasoning and/or responses of the participants.²⁶ Rather than assessing variables associated with a particular outcome, the method’s strength is in its ability to (1) explore generative mechanisms that underline main causes of (un)intended outcomes and (2) highlight the circumstances in which these mechanisms are triggered.

Realist synthesis starts with a programme theory and ends, if it has been successful, with a ‘revised, more nuanced and more powerful programme theory’.²⁷ (Re)building programme theory means to draw from theoretical descriptions of C-M-O relationships (middle-range theories) that are close enough to the data that allow empirical/hypothetical testing. In our case, by synthesising the data, we will compare how feedback via e-portfolio was intended to work to the empirical data on the actuality in different situations—all with C-M-O relationships. In this manner we might explain some contingencies that influence the prospect of feedback via e-portfolio generating its intended outcomes.

Study design

The study follows the iterative steps suggested by Pawson et al,²⁵ as well as two realist synthesis protocols: one by Wong et al ²⁸ and the other by Pearson et al ²⁹ (see online supplementary appendix 1 for Diagram of the project). We plan to report the actual realist synthesis according to RAMESES publication standards³⁰ and use a modified flow diagram.^{28 31}

Step 1: clarify the scope, locate existing theories and develop programme theory

The objective in the first step will be to conduct an exploratory (informal search) for various ‘working theories’,²⁵ helping us tobuild an initial programme theory. In realist terms—underlining the relationships between the context, mechanisms and outcomes27 30 32—we are to explore ideas around how feedback via e-portfolio is intended to work and why sometimes things go astray. When getting a feel for the literature (its quality, quantity, as well as its boundary scope),²⁵ we will be mindful not to foreclose potentially important perspectives.²⁹ Therefore, we will conduct a broad electronic database scan for evidence with no quality assessment in mind.³³ While the body of references will be narrowed down in Step 2, the documents in this stage will only need to contain information on e-portfolio-related instruments (ie, e-logbook, personal digital assistants, personal development plans) and feedback/assessment/ evaluation. To further test the developing theory, we will also conduct face-to-face interviews with e-portfolio users (clinical teachers and postgraduate trainees) as well as engage in discussion with the research team, who are familiar with the e-portfolio and feedback literature.

Initial programme theory

We have started work on this stage and have a number of potential theories that might help explain the mechanisms underlying the effectiveness of feedback via e-portfolio (see online supplementary appendix 2 Initial programme theory).

Theories of technology adaptation explain how perceptions of e-portfolio correlate to behavioural changes of e-portfolio usage,^34–36 for example, the possibility of motivational mechanisms (such as self-efficacy, subjective norms, level of e-learning enjoyment, experiences and computer anxiety) and their impact on perception of (O1, O2) and intention to use (O3). These theories can shed light on whether the specific technology adopted might in any way affect the effectiveness and efficacy of feedback portrayed.

Other potentially valuable sources for our programme theory development are theories on feedback responsiveness and seeking behaviours.^37–39 Assuming that responses to feedback arise solely from one’s sense of self-worth (mediated as mechanisms of fear from criticism, longing for appraisal, expectation of recognition), individuals are more likely to effortfully engage with technology/agency (O3) when they perceive feedback as being congruent with their selfhood (regardless of the intervention context). On the other hand, individuals might be able to self-regulate their motivation in relation to a specific context. As regulatory focus theory explains,^{39 40} it is the ‘promotion’ or ‘prevention foci’ of the context that will dictate the nature of engagement with technology/agency. In realist terms, high engagement and behavioural changes (+/−O3) might occur only when positive aspects of the intervention are conducted in promotion-aroused conditions (C), those regulated by wishes and desires, or when negative aspects of the intervention are given in prevention-aroused conditions (C), those regulated by obligation and necessity. For example, in a ‘promotion foci’ implementation context—such as where e-portfolio is voluntary, a part of formative assessment, the mentor’s comments on the learner’s tasks are positive—the learner will likely want to engage (M) with the mentor in an effortful manner (O3), or perhaps vigorously seek (M) new creative ways to continue the work (O3). By contrast, in a ‘prevention foci’ implementation context—such as where e-portfolio is mandatory, part of summative evaluation, mentor gives negative comments—the learner will perhaps become extra hard-working (M)/hypervigilant just to avoid (M) punishment and rectify (M) the situation. In this situation, a negative aspect of the intervention (C) might lead to positive learning, behavioural changes (O3). On the other hand, if the mentor praises learner’s assignments/performance (C), it is more likely that the feeling will be that no additional effort is needed (M, relaxation, indifference, disengagement), leading to no behavioural changes and low engagement with self, the mentor or e-portfolio (O)

Finally, the educational alliance theory states that behavioural changes to feedback happen according to learner’s evaluation of mentor’s credibility in a supervisor–trainee relationship.^{41 42} This might be another source for potential theory development. For example, learners trusting in the credibility of the mentor (clinical competency, content credibility, personal characteristics), and the relationship (meaningfulness and authenticity), will more likely contemplate feedback in an effortful manner, which will also probably lead to behavioural changes (O3).

The initial theories uncovered during our searches will be reconsidered against the empirical data. As such, it is possible that only a small number will be prioritised for synthesis, based on their greater resonance with that data.

Step 2: search for evidence

Using a more formal search for published literature in four bibliographic databases (Web of Science, Scopus, Medline+Journal@Ovid, Wiley Online Library), we will look for sufficient evidence to refine, confirm or refute our initial programme theory (see online supplementary appendix 3 Example search strategy for Medline+Journals@Ovid). Specifically, we will look for: (1) empirical (peer reviewed full articles) and non-empirical literature (eg, review, opinion pieces, editorials, commentaries, abstracts from conferences, process evaluations, programme manuals) as long as they comply with our rigour and relevance criteria;^{30 32} (2) studies of all types of research design will be included; (3) articles published in English; (4) articles published between 2008 and 2017; (5) with participants (learner and educator roles) in healthcare and higher educational settings in Taiwan and abroad (see online supplementary appendix 4 Definitions of concepts and supplementary appendix 5 Inclusion/exclusion criteria for formal search).

Because there is no finite set of relevant papers that can be strategically defined and found, compared with a more traditional systematic review, realist synthesis adopts an iterative approach to searching for multiple types of evidence.²⁵ In order to explore the literature deeper for theoretical elements which might help to explain new findings, or re-examine certain aspects of the developing theory,³² we expect to undertake additional inquires such as: (1) hand searching relevant journals (related to e-learning, e-portfolio or feedback in an educational setting, such as British Journal of Educational Technology, Australian Journal of Educational Technology, Electronic Journal of e-learning, International Journal of ePortfolios); (2) using citation tracking (pearling); (3) skimming through various grey literature platforms (https://www.jisc.ac.uk/) and (4) coming across evidence by chance. Additional searches will be purposeful, focusing on relevant sources for developing programme theory. For all searches, we will make augments in our preliminary criteria (eg, include papers that are missing sufficient data, or not in the timeframe).

Step 3: study selection procedure and appraisal

After importing references into Endnote 9, we will undertake the study selection in two phases. First, we will screen based on title and abstract, excluding all references not specifically mentioning web/online portfolios and the feedback, assessment, evaluation portrayed in it. Second, we will look at the full text documents to further exclude based on the following questions: Does this paper (or a section of it) involve feedback via e-portfolio, that (1) is described as an ongoing (direct or indirect) interaction between receiver and giver using e-portfolio as an educational tool: (2) takes place in higher (healthcare) educational setting? Using the preliminary set of inclusion/exclusion rationales, the lead researcher (LVM) will check a randomly selected sample of 20% of the identified documents. The remaining will be screened by two reviewers. Any discrepancies will be discussed until an agreement is reached.

Aligned with the RAMESSES standards and proposed quality judgements,^{30 32} we will appraise the quality of included content of a section of a text as: (1) relevant, if they address or contribute to theories we are exploring, and (2) rigour, if the methods used to generate that particular data are credible and trustworthy. Quality judgements will be made on ‘the level of arguments and theory’ rather than merely on ‘the level of data’, allowing us to consider evidence seemingly of lesser quality yet potentially relevant to programme theory development.³³ However, to give an indication of the ‘coherence, plausibility and appropriateness”³⁰ of our selection, we will (1) apply elemental methodological questions⁴³ for rigour and (2) use a hybrid tool^{29 44 45} to distinguish conceptually thick (rich) material from conceptually thin (weaker) according to its ability to provide explanations to developing programme theory. This tool has been shown to be useful in theory-driven synthesis just because it gives the option to focus on richer sources of programme theory without denying the weaker ones as well⁴⁶ (see online supplementary appendix 6 Test for assessing relevance and rigour).

Step 4: data extraction and organisation

For the included full text papers, we will develop a data extraction sheet to provide an accessible overview of our findings (see online supplementary appendix 7 Data extraction table) as well as importing them into Atlas.ti V.8 for further coding of the themes. While coding, we will consider the raw data, textual descriptive findings as well as authors’ interpretations written in the results or discussion section. For non-research papers, we will consider various forms of textual descriptions. All relevant sections —relating to context, mechanisms and their relationships to outcomes—will be coded deductively (conceptual themes/codes created from initial programme theory developed prior to data extraction) and inductively (conceptual themes/codes recognised during the process). Should the paper contribute to only one specific element of the C-M-O, we will not discard it, as we will be able to make inferences from other sources.

Step 5: data synthesis

To refine and further explain the developing programme theory through the data synthesis process, we will simultaneously analyse evidence for potential C-M-Os and organise them in themes and semi predictable patterns.

• To identify potential C-M-Os, we will think ‘‘backwards’ from the outcome’⁴⁷ and will try to identify the causal mechanisms alongside the contexts within they are associated. We will be careful not to presume there is only one outcome within the chain of events.

• When thematically organising the data, we will take a similar approach to that described by many other researchers:27 29 44 45

Juxtapose sources of evidence, for instance, when data about the effects of feedback via e-portfolio in one paper will allow an insight on its effective patterns in another paper; Reconcile sources and identify differences, such as, understanding why different results might occur in apparently similar situations; Adjudicate sources of evidence and make judgements between studies based on their methodological strengths and weaknesses; Consolidate sources of evidence, by creating a multifaceted explanation of the intervention. That is, whenever we have different outcomes in particular contexts, we will try to explain how and why this might occur. Situate sources of evidence, for example, when a particular mechanism is triggered in context A, while another mechanism might only occur in context B.

During this stage, the programme theory will be redeveloping and in its refinement. As we delve into our included studies and beyond, we will be mindful of unexpected patterns, which might inform us of new middle range theories, thereby further explaining dynamics around e-portfolio being an effective means for the feedback process. Considering we expect to find limited data specific to our enquiry, we recognise that some of the theoretical assumptions we will make might be weakly supported. Nevertheless, throughout our work we will be fully transparent about the levels of evidence available to support/refute our hypotheses, giving the reader the space to decide exactly how much of it is relevant.

Patient and public involvement statement

This realist synthesis around feedback via e-portfolio will be done without patient and public involvement. Our rationale for this is that, to the best of our knowledge, patients are not typically involved in this aspect of clinical education. As such, patients will not be invited to contribute to study design, interpretation of the results, or help with writing or editing of the document. Also, we will not include them when developing dissemination strategy.

Ethics and dissemination

No formal ethical approval is required for this synthesis. We aim to publish our findings in at least one peer reviewed journal as well as present them to relevant bodies including broader educational institutions. At present, we have a fairly vague understanding of the complex dynamics between e-portfolio and feedback; even more unclear are all contingencies closely linked to it. By applying a method that has the analytical strength to provide insight into the complexity,²⁷ we hope to pinpoint the most valued educational features of effective feedback via e-portfolio in a contextual manner. With a forward-looking perspective, we aim not only to inform the educational community, but also to give practical guidance and recommendations to policymakers on how to re-enact the context, or even provide enhanced resources in the future.