The use of ‘Think-Out-Loud’ methodology in the development of teaching materials for abbreviated breast Magnetic Resonance Imaging scan (FAST MRI) interpretation, and a comparison of the learning experience of two reader cohorts

Abstract

Aim

FAST MRI is an abbreviated breast MRI technique, proposed as a screening tool for breast cancer. This study aims to explore how ‘Think-Out-Loud’ (TOL) methodology can effectively develop and shape training to interpret standard breast MRI on readers’ learning experience.

Materials and methods

TOL methodology asks people to speak their thoughts while performing a task; to say whatever they are looking at, thinking, doing, and feeling at each moment. It helps determine expectations and identify aspects of confusion. This methodology was adopted with two groups of image readers to develop training materials for FAST MRI interpretation.

Eight image readers (readers); 4 NHS Breast Screening Programme (NHSBSP) breast MRI and mammogram readers (Group 1) and 4 NHSBSP mammogram readers who do not read breast MRI (Group 2) were audio and video recorded during training. Content analysis was undertaken.

Results

TOL was well received by the readers and allowed iterative development and refinement of a training package. Challenges relating to introducing new concepts and managing uncertainty were identified through the analysis of the TOL audio data. Additionally, Group 2 asked for more clarification than Group 1, of both the teaching materials and how to use the image-manipulation software.

Conclusion

TOL methodology allowed effective training of both groups of readers. The iterative nature of individual training ensured production of user-friendly materials including frequently-asked-questions sheets. It allowed for each person to feel that their views were fully listened to and incorporated into the training package, allowing for training materials with high face validity.

1. Introduction

Finding breast cancers early saves lives [1] and is the purpose of the NHS Breast Screening Programme (NHSBSP). Screening with mammograms is cost effective but imperfect, as it results in both over-diagnosis (finding biologically-irrelevant cancers that if left alone without treatment would not have caused harm during the woman’s lifetime) and under-diagnosis (failing to find aggressive, biologically-relevant cancers that if left alone without treatment will cause harm to the woman) [2,3]. It also uses ionising radiation and carries a very small risk of radiation-induced breast cancer [4]. An ideal screening test for breast cancer would be able to selectively detect aggressive, biologically-relevant cancers without detecting biologically-irrelevant ones. It would also not use ionising radiation, be acceptable to patients, and no more burdensome than mammography to acquire or to report.

FAST MRI is the core component of abbreviated breast MRI techniques and has been proposed as a screening tool for breast cancer that might potentially prove more cost effective than mammograms for some women [5,6]. The NHSBSP relies on a multi-professional workforce of image readers, some who interpret both mammograms and breast MRI, and others who interpret mammograms but have minimal or no experience of breast MRI interpretation. Integration of any new imaging test for breast cancer into the existing NHSBSP framework will require the workforce of readers of all levels of experience with breast MRI to undergo additional training [7]. It is likely that readers with different levels of prior knowledge will require different amounts of training to achieve a similar level of performance at FAST MRI interpretation.

The authors wished to discover whether brief training could enable NHSBSP readers of differing levels of prior knowledge of breast MRI to effectively interpret FAST MRI [8]. This research required the development of a standardised training package. During the design and development of the teaching tool it was important to ensure that the produced training materials were accurate and fit for purpose; in that the end-users could directly relate the new learning to clinical tasks.

‘Think Out Loud’ (TOL) methodology has traditionally been used as a psychological research method [9], but it has been embraced for the practical evaluation of human-computer interfaces [10]. The main advantage of the TOL methodology is that it pinpoints user misconceptions reliably, quickly and cheaply [11]. This qualitative methodology allows understanding of participants’ thought processes as they conduct a particular task without the researchers disturbing ongoing processing [12]. The TOL method captures the problem-solving process contemporaneously by asking participants to verbalise their thoughts as they occur [13]. We demonstrate how TOL methodology can effectively shape training and enable exploration of the influence of pre-existing ability to interpret breast MRI on readers’ learning experience.

2. Materials and methods

Health Research Authority (HRA) and ethical approval was obtained (REC reference: 17/SW/0142) and all participants gave written informed consent.

2.1. Participants

Eight NHSBSP mammogram readers from a single centre were recruited as readers. These eight practitioners comprised four readers who interpreted diagnostic breast MRI as part of their clinical practice (Group 1; range of years of mammogram-interpretation experience 5-11; range of numbers of MRIs read per year 100–225, and mammograms read per year 5000–6500) and four readers who did not interpret breast MRI in their normal clinical practice (Group 2; range of years of mammogram-interpretation experience 2–28 and numbers of mammograms read per year 5000–18000).

2.2. Training procedure and training materials design

A training package, comprising guided use of the software and image interpretation, supporting documentation, and (following a request for clarification of some factors) a PowerPoint presentation, was constructed.

A guided element of the training package, comprising a set of 14 FAST MRI teaching images with a range of normal and abnormal appearances and a “script” of exact words the trainer should use during the teaching session, was created with additional instructions to the trainer (for example about when to manipulate the images on the workstation themselves and when to encourage the trainee reader to do this for herself). This was delivered to each of the readers during the one-to-one teaching session as the FAST MRI teaching images were displayed. The content of the training package was constructed collaboratively from the existing professional knowledge of two researchers who were consultant radiologists, experienced in breast MRI and mammogram reporting. Teaching practical skills required for image manipulation was achieved as follows: during the training session, readers were shown how to optimise the image, enabling windowing, centring and magnification of the image. They were instructed in how to scroll through a stack of image slices and to change the orientation of each image from axial to sagittal and back. They were encouraged to demonstrate to the trainer how they would achieve image optimisation for subsequent teaching cases. Lastly their ability to manipulate the image on the workstation to achieve specified goals, such as identifying the nipples, was formatively tested on a third training image. For each aspect of the teaching the trainer built upon the trainee’s existing knowledge of mammogram interpretation, comparing the appearances of any given pathology on the two modalities and drawing parallels between, for example, how to manipulate a mammogram or digital breast tomosynthesis image and how to manipulate a FAST MRI image.

The training package was delivered, iteratively one at a time to four of the eight readers. The previous image reading experience of the first four trainees was; MRI reader, non-MRI reader, non-MRI reader, MRI reader respectively. This allowed for variation in their prior exposure to similar software. The feedback from their individual training sessions informed the content and development of the training package. The remaining four readers were presented with the finalised training package, while TOL continued to be encouraged.

During all eight the training sessions, readers were shown the series of 14 FAST MRI images and given interactive instruction and formative assessment. This enabled hands-on experience of image manipulation on the workstation that was to be used for FAST MRI interpretation, and allowed them to learn interpretation skills needed when working with the FAST MRI images. The first trained reader suggested the development of supporting documentation that could be referred to whilst learning to use the software and interpret the images. The first four readers iteratively offered refinements to both the documents and the need for additional information to be included in the training package, which were integrated into the scripted training and developed the documents prior to the next person being trained. The supporting documents were introduced to the readers during the taught element of the training session and were freely available to them during their formative assessment, as they would be when undertaking image interpretation during normal practice.

After eight weeks of experience interpreting FAST MRI an additional PowerPoint presentation was offered to all the readers as a group. The content of this presentation included aspects of FAST MRI interpretation that had been informally identified by the readers as being difficult for them in practice, namely, assessment of movement artefact and assessment of the degree of background glandular enhancement. Seven of the readers attended this session, with one reader from group two unavailable, however the presentation and the script was given to the reader to review in her own time. This presentation session including the questions and answers that followed was recorded. At the time of the group presentation, two readers from group 2 informally expressed their continued lack of confidence in their ability to interpret FAST MRI, and a second one-to-one session was offered to all readers. However, only one reader (from Group 2) took up the offer and had a second training session that comprised formative assessment of the reader’s interpretation of a further 13 FAST MRI scans, presented in batches of 4, 5 and 4 scans respectively. Fig. 1, shows a flow chart of the developed training package, and Fig. 2 is an example of one of the developed resources.

Fig. 1.

Flowchart of the development of the training package.

Fig. 2.

shows an example of one of the developed resources (“Resource name - BIRADS categories of background parenchymal enhancement").

2.3. Data collection and analysis

The training sessions were video and audio recorded, with the author (SH) present to prompt trainees in the TOL method and to take field notes.

The formative assessment section of training was transcribed verbatim. The full audio files were listened to and sections identifying difficulties, of FAST MRI image manipulation, interpretation or of missing or unclear information on the supporting documentation were transcribed, as was positive feedback. Audio files were interrogated for information provided by the participants, immediately following completion of the training, once offered the opportunity to reflect on what they had experienced. Video data was used to add descriptive detail when readers referred to areas of the computer screen, the developed materials and aspects of the environment. The video data offered additional information to allow for triangulation of utterances with gestures and the developed learning materials.

Content analysis was used to categorise the data from the audio recordings and the supplementary information provided by the video and field notes. The following steps were taken [14]:

• •The interview transcripts were read several times
• •The text was separated into categories with similar meaning
• •The condensed meanings were abstracted with codes
• •The codes were arranged into subcategories and categories, based on assessments of their similarities and differences

The first author (SH) analysed the results independently to recognise initial codes and then consensus and refinement was reached through discussion with the co-authors. This was then fed back into the training materials and supporting resources. The credibility of the data was established through member and peer checking. Informal commentary from participants verified that their perceptions were exactly represented.

3. Results

A difference in the duration of the training between Group 1 and 2 was found. Training Group 1, those with previous MRI reading experience, took a mean average of 38 min less than Group 2 (122 and 160 min respectively). An additional 50 min of training was provided to seven out of the eight readers, and one reader Group 2 had a further 90 min individual session, having expressed a lack of confidence in her skill to read the new image format. However, independent samples median test showed no significant difference (p = 0.486).

Qualitative analysis of the data collected during training found that, members of both groups, sought clarification of both information within the teaching materials and how to interface with the image-manipulation software. The two categories that were represented in the qualitative data were introducing new concepts and managing uncertainty.

3.1. Introducing new concepts

Comments from participants highlighted aspects of the training, supporting materials and image interpretation that they found interesting and engaging as well as occasional misunderstandings, and confusions. Elements of content and instructions on how to navigate the image display software were therefore modified to improve participants’ experiences of using the software and to improve their ability to interpret the displayed images.

Table 1 presents selected quotes from participants, illustrating how their perspectives were used to suggest changes to the teaching materials.

Table 1.

Quotes from participants.

Topic	Quote	Modification
TM – Cysts	I’m not sure what I am an seeing on this image, it might be a cyst, I’m not sure if you would be able to see a cyst on this type of imaging	Clarification added
TM – Nodes	I think that might be a lymph node, but I am not sure, I am wondering how I can check if this is a lymph node and if it is how I should be reporting it	Clarification added
R – grading MRI	I now need to assign a grade to the FAST MRI. I am wondering if it is the same as when I do it with a normal MRI. I just tend to go with my gut, I’m not sure what percentage ranges go with which MRI grade.	Add guidance on what percentage range match MRI categories
R & ST – Crib sheet	Having real case images on the crib sheet would be handy. It would be like a refresher and clarifying stuff like movement and calcification	Add images to Crib sheet
N - Image manipulation	Why do the breasts look like that and why can’t I see the nipples on this image?	Notes added to TM to include this during training
N - Image manipulation	I need to turn the image round so I can look at it from a different angle, how do I do that?	Additions to the TM

Key: TM = Teaching Material, R = Supporting resources, ST = Stylistic changes, N = Navigation of image software.

Content changes to the teaching materials and resources included: adding a PowerPoint presentation to the teaching materials to explain identified challenges of image interpretation. These challenges included the presentation of abnormalities such as calcification. Visual examples of such abnormalities were added to resource sheets to allow readers to compare what they are looking at on screen with known abnormalities and difficulties of interpretation.

Stylistic Changes included: ensuring consistent alignment and fonts throughout as participants noticed inconsistencies. Participants also said that images included in the printed resources need to be high resolution, otherwise the images will not be helpful.

Navigational changes included: describing how FAST MRI images are captured and how to navigate through the software to change image views. Instructions on image manipulation were specific to the hardware and software used in this study, descriptions were added to the training materials and developed resources to explain the practicalities of software manipulation.

3.2. Managing uncertainty

Readers in Group 2, those without previous MRI reading experience, frequently verbalised how they were applying their experience of reading mammograms to their interpretation of FAST MRI. Participants indicated that in some circumstances they would attempt to seek further information either by applying their existing knowledge about imaging or considering the training and referencing the supporting materials. Members of both groups also asked for additional information about the demographics of the women whose images they were interpreting, a common question being how old is the woman?

Not all participants sought further information to inform their decision making. Factors influencing this appeared to relate to, previous image reading experience, and how much time they had to undertake the training. Including the TOL methodology increased the time it took to undertake the training of the readers. This meant that some participants expressed that they were rushing the last parts of the training due to other commitments but wanted to complete training rather than scheduling a further session.

Members of both groups of reader asked the trainer about how to assign a grade to FAST MRI and the process of second reading. They expressed concern about ‘what happens if I get it wrong?’ and required reassurance that if FAST MRI became standard within breast screening practice then double reading would be used as it is with mammograms currently.

4. Discussion

We have developed a new training package which conveys how to read FAST MRI in a way that engages readers and enables them to make quick and accurate judgements when interpreting this new image modality FAST MRI [7]. The development process drew on the evidence-base for FAST MRI [5], professional knowledge of full protocol breast MRI reporting, pragmatics of training an NHSBSP mammogram reader workforce within the current NHS context, and qualitative research to maximise the effectiveness of training and supporting resources. Strengths of this project include utilising the whole multi-professional team and the combination of evidence theory to draft the initial training scripts and resources, and person-based (TOL) approaches. Team members, including both facilitators (LJ & RG) and IRs contributed expertise in image manipulation and interpretation, anatomy, physiology and pathology of the breast, communication, and teaching, and represented diverse professions including psychology, radiology, radiography and medical education. All participants helped ensure the final teaching package was accessible but not patronising. By drawing flexibly on a combination of approaches we were able to incorporate evidence, theory, and users’ perspectives throughout the training development process.

Within the current research, training was scheduled around each individual’s clinical commitments. This led to some sessions having the formative assessment feedback element rushed. There was no evidence of poor performance in those with an abbreviated feedback, but best practice dictates that future training should be undertaken in a ring-fenced period of time, where the importance of the training is emphasised and recognition of the training is provided to the attendees. Recognition in the format of continuing professional development points (CPD) may be effective.

The TOL methodology used during the development of the specific training presentations and supporting materials provided reach data that ensured an effective, efficient and accessible programme of education was produced. Further use of the training and resources is required to gain accurate timing of the teaching process without the TOL methodology and also to evaluate the effectiveness of group training in comparison to the individualised sessions. The TOL methodology would not be practicable within a group setting and therefore the corrective/explanatory statements made possible during the present study would be very limited when the training package is delivered to a group.

Think Out Loud generally has some limitations as a methodology. Asking people to sit and talk to themselves is an unnatural task. This makes it hard for the participants to keep up the required monologue. An author was present to prompt and encourage reflection and participants were typically willing to try their best.

A further challenge in undertaking this methodology is in interpreting the data collected. Participants are instructed to say things as soon as they come to mind rather than to reflect on their experience and provide an edited commentary after the fact. However, unconsciously people want to appear knowledgeable and astute, and thus there's a risk that they won't speak until they've thought through the situation in detail. During the present project, prompts were used to keep participants talking, but care was taken to avoid interruptions that could have changed participants’ behaviours.

5. Conclusions

Think Out Loud methodology proved to be a powerful tool in understanding the strengths and weaknesses of the developed training package and ensuring the final package was comprehensive and also positively received by the readers. Following training all 8 readers completed an interpretation task of an enhanced dataset of 125 FAST MRI examinations which demonstrated that the training of both groups had been effective [8]. Think Out Loud methodology allowed effective and efficient training across groups of readers with different professional backgrounds and different prior knowledge and skills. The iterative nature of individual training ensured production of user-friendly support materials including frequently asked questions sheets. Differences in time taken to train achieved between the two groups have implications for the design, provision and resources needed for the group teaching of FAST MRI interpretation.

Funding

This research has been carried out with the support of North Bristol NHS Trust Research Capability Funding.