A review of the PERFORMS scheme in breast screening

Abstract

This review details the aetiology of the PERFORMS self-assessment scheme in breast screening, together with its subsequent development, current implementation and future function. The purpose of the scheme is examined and the importance of its continuing role in a changing screening service described, together with current evolution.

Introduction

Every year more than 2 million females have breast cancer screening in the UK,¹ while approximately one in eight are diagnosed with breast cancer during their lifetime.² Breast screening was introduced in the UK in 1988 following the Forrest report³ which recommended screening every 3 years for all females aged 50 to 64. Screening is crucial to early breast cancer detection; however, the low incidence rate during screening compounds any difficulty readers have in identifying early signs of abnormality readily and also for them to be aware of details of their own performance. Females aged 50 to 70 are currently invited for screening^4,5 with further age extensions of 47 to 73 years under investigation via the AgeX trial.⁶

Screening is a two-stage process. A female is first imaged using two view full-field digital mammography (FFDM), and her case reported as either “return to screen” or if the mammographic appearance is suspicious then she is “recommended for assessment”. Subsequent further imaging and investigations at assessment determine whether her mammographic appearance is abnormal and what action to take.

Since its inception, the UK screening programme has always aimed for a low recall for assessment rate which has been found to be circa 4.56%⁷ and to maintain this level necessitates that high-quality abnormality detection decisions are made concerning which cases merit recall. In 2016 in England, some 7.8% of prevalent round females were recalled for assessment with 3% of incident screened females recalled.⁵ A breast screening reader, typically either a radiologist or specially trained advanced practitioner radiographer, gains rapid feedback on whether their decisions to recommend a case for assessment were appropriate. However, with a 3-year screening interval, there is no swift feedback for a case considered normal or definitely benign and consequently not recalled for assessment. In such instances, appropriate feedback to the reader cannot occur until the female presents for the subsequent screening round 3 years later or if she presents with an interval cancer in the intervening period.

To help maintain and improve radiological skills in determining which cases truly require recalling the PERFORMS scheme^8,9 was developed. In the UK, each breast screening reader has been recommended to report 5000 cases per year,¹⁰ thereby on average is only presented with some 35 cancer cases annually. By also examining the PERsonal perFORmance in Mammographic Screening (PERFORMS) test sets, they could additionally read 80–90 challenging cancer cases per annum.

The PERFORMS scheme

The PERFORMS scheme was invented and developed by A.G. Gale, working with E.J. Roebuck¹¹ one of the pioneers of screening in the UK. It began as a small-scale unfunded research project in the East Midlands region of England with the motivation of modelling variations in radiologists’ skills in identifying key mammographic appearances of early breast cancer. This led to its development as a self-assessment scheme which was then gradually established nationally in parallel with the introduction of UK nationwide breast cancer screening. This development was in conjunction with the Royal College of Radiologists (RCR) and the UK NHS Breast Screening Programme which originally funded the programme nationally. The scheme has been implemented across the UK continuously for over 30 years. It is the leading and, first, national self-assessment scheme in radiology.^12–14 It provides an individual with self-assessment on imaging decisions as judged against: expert radiological opinion, known case pathology, peer opinions, and also provides personalised training and additional screening programme quality assurance functions.

Participation in the scheme has always been highly recommended by the RCR¹⁵ and more recently has become mandated by Public Health England (PHE)¹⁶ with readers receiving appropriate professional CME (Continuous Medical Education) points for taking part. It has also been independently recommended to improve reader skills following an external review of a screening incident which had resulted in some 61 cancers being missed.¹⁷

Screening in the UK originally used a single Medio-Lateral Oblique (MLO) mammographic film of each breast and so the PERFORMS scheme was first designed and deployed using radiographical copies of single MLO images, together with a paper-based response system where readers recorded their decisions on the cases.¹⁴ This information was mailed back for analyses, with subsequent detailed feedback again being returned to the reader. Inevitably this offered readers delayed and simplistic performance analyses. Sets of screening cases were couriered around the UK, coupled with reporting books which readers completed. Two view screening (MLO and CC; Cranio-Caudal) was introduced in 1995 and the scheme developed in parallel. It subsequently evolved using various personal computer systems to provide rapid feedback to readers immediately after they had read a set of test mammographic cases. As all screening centres progressed after 2010 in using two-view FFDM, then a cloud-based approach for the scheme became feasible.

Current implementation

De-identified recent potential test FFDM cases are sourced widely from UK screening centres, complete with the original radiological opinion and any pathological report. A panel of over 10 expert breast radiologists then individually identify key mammographic features in each case and rate case density, case difficulty and other factors, determining whether each case is challenging enough (i.e., approximately 75% of participants could possibly report it incorrectly) to be included in the scheme. In collaboration with further experts, we then derive a consensus opinion per case. Finally, set of cases are constructed which include a mixture of normal, benign and malignant cases with various features including subtle or interval cancers, bilateral appearances, multifocal tumours, etc. Case sets are updated annually.

PERFORMS is now¹⁸ a complex online training scheme with extensive educational feedback and is undertaken annually by all (almost 1000) screening readers nationally as well as symptomatic radiologists (these are radiologists who do not participate in the national screening programme but do read symptomatic breast cases). The scheme has been developed to be as simple and transparent in use to readers as possible. With a range of different vendor workstations in use across the UK, it is imperative that readers employ their own workstations and associated familiar viewing software to examine the FFDM test cases, so that the recorded readers’ performances reflect their everyday screening behaviour. Readers simply use whatever workstation software they usually employ to view and examine the test cases and then use the PERFORMS App, typically running on a laptop or other computing device beside the workstation (although it can also run on the workstation itself if this is internet enabled), to report the cases and receive detailed feedback. The scheme then easily accommodates any vendor software updates or change in vendor supplier by a breast screening centre. It has purposefully not sought to be a standalone DICOM (Digital Imaging and COmunications in Medicine) viewer and reporting application running on the reader’s workstation.

Every reader has their own PERFORMS confidential web portal which provides detailed ongoing performance data and from where they can log in to the dedicated web-based reporting App; download numerous sets of test cases; report these cases; and receive feedback. Using the App, readers locate and identify several key mammographic appearances, rating each feature in terms of level of suspicion. They also classify whether to recall or not recall each breast image using the UK RCR screening classification rating scale¹⁹ or when the scheme is being used outside the UK, they use the appropriate Breast Imaging Reporting and Data System (BIRADS) rating scale.²⁰ The two scales having been shown to be well related.²¹

The App provides immediate feedback and readers can revisit any, or all, cases and examine annotated expert radiological pictorial and textual feedback as well as associated relevant pathology information. After taking part, they can, via their web portal, view personalised performance reports; enter into online peer and expert discussions on cases, and other associated activities. Once all readers have completed a scheme then all data are statistically analysed, and each reader receives detailed and anonymised web-based reports regarding their performance in comparison with their colleagues nationally. Management reports for the screening programmes are also regularly produced.

Of necessity, the large number of readers need to take part over several months as they schedule participation around their normal screening workload. Consequently, daily updated comparative and anonymised peer review is available where a user can keep revisiting the website to compare their own case decisions to an ever-growing number of decisions of their peers. Various interactive performance tables are available which highlight cases which were most misreported (e.g., false negatives, false positives) as well as other factors such as most missed abnormalities. Ongoing current work is examining the possible use of augmented reality as part of enhanced educational feedback to readers.²²

A key aspect of the PERFORMS scheme is that it can statistically identify mild and severe underperforming outliers and then follow-up these individuals with suggestions of how to improve their performances (which can entail detailed visual search recording in our laboratory) as well as monitor them on future test sets. Typically, a few outliers are found annually out of the hundreds of readers taking part. In general, substandard performance is found to relate to an individual’s clinical workload at the time of reading the case set. As all reader responses are timed then any outlying performance can be investigated to firstly determine how long the user spent reading whole case sets, reading specific cases and the time of day when reporting. Any outlier typically is found to improve their performance on subsequent iterations of the scheme. Previously, outliers have only been able to be identified when all readers have taken part in a particular PERFORMS case set. However, we have successfully demonstrated that we can now statistically predict potential outliers only after a few readers have taken part which means that potential outliers can be identified earlier whilst a scheme is being implemented and so given help much sooner.²³

The PERFORMS scheme is deliberately designed to allow readers both to participate when they like and also to start, stop and restart at will. This allows readers to fit taking part into their clinical work and at a time that suits them. However, extending the test set reading over time may also introduce other factors into the performance measurements. In practice, the majority of UK participants read the 60 test cases in one continuous reporting session. We have examined data on readers taking breaks while taking part and found little or no effect whether they took mini-breaks when reading a case set or whether they spread reading a case set over clinical reporting sessions. When readers read the case set in one complete session, without mini-breaks, more errors occurred as reading time increased.²⁴ This indicates possible effects of fatigue.

PERFORMS and real-life screening data

An important issue is how an individual’s performance on the PERFORMS scheme relates to their performance in screening practice. Inherently, reporting selected test cases is essentially the same task as routine everyday reporting and so similar performance values between the two situations might be expected. However, differences between the two situations exist that must be acknowledged when interpreting information such as the reader’s knowledge before reporting that these are carefully selected test cases. While the actual variables recorded in PERFORMS are based upon the routine measures reported in everyday UK breast screening, the particular exemplar cases used in the scheme are carefully selected to be challenging enough to be able to tease out potential performance differences between participants. Each set of difficult test cases is carefully derived by a panel of experts as case difficulty can be very subjective.²⁵ Apart from the test cases being more difficult than found in routine screening, the case sets are necessarily weighted with more benign and malignant cases so as to elicit variance in performance measures. In the UK, with an individual reading 5000 cases a year, we estimate that in one year our scheme would give them the equivalent experience of challenging abnormal cases as they would experience when reading several years’ worth of routine screening cases.

In the UK, readers’ usual screening performance is monitored and audited annually, with data concerning every screening centre published. This provides important information for individuals concerning their performance. However, using the PERFORMS test sets additionally to assess aspects of their performance objectively is important, not only because these data can give rapid feedback on an individual’s skills but also because it is virtually impossible to reliably assess an individual’s performance based solely on annual real-life data. Instead, their real-life screening performance must be based on several years of screening data in order to encompass them reporting a representative number of cancer cases. This is because the number of abnormal cases which a reader will see per annum is approximately 35 per 5000 cases screened (based on an approximate cancer detection rate of 7 per 1000 screened cases). This means that an individual’s annual sensitivity measure would be based solely on a very small number of abnormal cases out of the 5000 cases examined. Therefore, picking up, or missing, a very small number of these abnormal cases would lead to a relatively large difference in the sensitivity measure in their annual performance assessment.

Over the years of implementation various comparisons have been undertaken between the data from the PERFORMS scheme and real-life screening. This has included comparisons between the mammographic features missed in PERFORMS and those that were recorded in an interval cancer database. The finding that similar proportions of features were missed in both situations demonstrated that readers had comparable difficulty with these features, both in real life and in the scheme. Additionally, readers’ real-life screening recall rates correlated with PERFORMS measures of correct recall and correct return to screen.²⁶ Subsequently the relationship between different types of readers in one UK health region was studied using data from the national NHS National Breast Screening System (NBSS) database and comparable PERFORMS data for the similar time frame. Significant positive correlations were found between the PERFORMS data and most of the comparable real-life measures.²⁷ More recently, the Breast Screening Information System (BSIS) has been introduced across England, which provides screening programme information as well as individuals’ real-life performance data over 3-year periods. Consequently, real-life screening performance data from over 450 screeners in England have been examined²⁸ over such a 3-year period and compared to their PERFORMS data over a similar time frame. Significant positive correlations were found between real-life cancer detection rates, recall rates and PPVs (Positive Predictive Value) and the equivalent PERFORMS measures.

Taken together, all these data demonstrate that an individual’s performance on the PERFORMS scheme can be seen as a useful surrogate indicator of their real-life screening performance. Importantly, this measure can be achieved quickly, without having to wait years for real-life data to be amassed, and so if an underperforming individual is found on the PERFORMS scheme then additional training can readily be offered to improve their real-life screening performance.

Factors which underlie screening performance

The theoretical underpinning of the performance measures used in the PERFORMS scheme is based upon an active vision approach to perception.²⁹ Although mistakes occur in any task for a variety of reasons,³⁰ the key to understand radiological performance is basically how individuals detect and identify known features indicative of normal, benign or abnormal appearances.^31,32 Visual search of the medical image is a key factor in this visuo-cognitive process and is complex,³³ primarily being composed of a series of saccadic eye movements which serve to shift the fovea to fixate upon specific areas in the image. As saccades are ballistic then these movements are pre-planned. The schematic map’s³⁴ theoretical approach well describes this process where the first fixation on the image establishes an initial image gist³⁴ which is followed by a series of eye fixations driven by the developing cognitive schema of the image at each subsequent eye fixation which determines the next fixation location and so on.^35,36 Researchers^37–39 have argued that medical image inspection involves both global and focal processes underlying such medical image search behaviour. Many studies investigating visual search and expertise in different scenarios⁴⁰ have shown that experienced individuals fixate important image areas faster and for longer periods than naïve observers, who make more and shorter disorganised fixations per unit time.⁴¹ Such an approach describes both the acquisition of radiological knowledge as well as how experienced radiologists’ image examination is subsequently almost automatically and quickly performed.

In breast screening research, most interest is in false-negative decisions where an abnormality is missed. Such errors can be classed as due to visual search, detection or classification.³⁷ Various research studies have demonstrated that, depending on the study, approximately 24–30% of false-negative errors are due to errors of visual search, some 25% perceptual and 45–52% cognitive.^37,3838 The PERFORMS scheme enables clarification of any errors as due to detection or classification. By inspecting a reader’s feature identifications against the expert panel decisions, it is possible to assess whether they have identified appropriate features or not (detection error) and if appropriately identified have they then reported them suitably (interpretation error). In order to determine whether any errors are due to visual search per se, then the eye movements of the reader need to be recorded and this is something we have done with outliers to give them better insight into this aspect of their performance.

The type of reporting error can be directly related to the type of mammographic features^42–45 identified,with some features being more likely to be undetected and others misinterpreted. In addition to the importance of key mammographic features, breast density⁴⁶ is an important underlying factor which increases the risk of breast cancer and can also mask its appearance. High breast density is associated with a greater chance of developing breast cancer, but unfortunately dense images are much harder to interpret, therefore making it more difficult to identify breast cancer accurately.⁴⁷ We have examined how density was reliably reported across individuals and across different participant groups⁴⁸ with better agreement being found amongst radiologists on case density ratings as compared to advanced practitioners.

The usual approach to determining correct feature identification is to assume a circular area of interest (AOI) centred around the abnormality, even when the abnormality has been clearly demarcated by experts as irregularly shaped. This is a reasonable working approach and a response within the AOI is then taken as a correct abnormality identification and detections outside the AOI marked as false positives. However, an improved technique has been developed²³ which more closely follows the contours of an abnormality and therefore can more accurately determine whether a reader’s response should truly be considered as a correct detection or not.

Expertise and reading case volume

The development of expertise in breast screening is a complex process, comprising both reading a large number of cases per annum and also years of reporting cases. Several studies^48–51 have tried to tease apart whether it is years of experience or volume of cases which is the more important by examining data from the scheme. In the UK, all readers are recommended to read at least 5000 cases per annum as a means of developing and maintaining expertise in the domain.¹⁰ By doing so they then inspect many normal cases and develop an appreciation of the wide range of normal appearances. Increasing expertise serves to develop appropriate cognitive schema for normal and abnormal appearances which then reduces the time required to examine a case as experienced readers search and concentrate upon high probability areas of an image where an abnormality may reside, as well as develop experience in identifying early features of abnormality.

A very different approach is in the USA where the continuing experience requirement is to read 960 cases over 2 years.⁵² The role of case volume has been examined by studying UK and American radiologists reading PERFORMS cases, with the American group split into high-, medium- and low-volume readers. Data demonstrated that the lower volume readers had significantly lower sensitivity than high-volume readers or the UK radiologists supporting the need for high volume.⁵³ Further comparative studies have shown largely similar performances between radiologists from the two countries, as well as from Europe, but with differences in recall decisions interpreted as reflecting different health management approaches.^54–56

Emergent future roles

PERFORMS was developed with the aim of helping readers identify early signs of breast cancer. As the UK screening programme has matured such a need has been maintained as more new readers, particularly advanced practitioners, join the screening programme and PERFORMS data have shown how well such practitioners can perform. Not surprisingly, mean values for cancer detection rate and correct recall rate improve as individuals’ experience in screening grows. Readers learn many abnormality identification skills fairly quickly in their first year of screening.⁵⁷

High cancer detection performance should be equivalent to good correct recall performance; however, this is not always the case as good abnormality detection can be achieved by simply over reading cases which simply increases assessment numbers. While PERFORMS has successfully aided readers to identify early difficult abnormality appearances, it is important in actual screening that the number of cases recommended for assessment are both minimised and fully appropriate. Consequently, PERFORMS is developing targeted test sets of cases which emphasise key benign appearances with the aim of increasing knowledge of the range of benign presentations which do not actually merit recall.

The PERFORMS scheme initially grew out of earlier work⁵⁸ which had developed a computerised decision aid based upon accurate radiological identification of a small number of key mammographic features. However, such accurate feature identification across readers is not feasible, due to intra- and inter-individual variability in reporting, which limited the generalisability of such an approach. More robust computer-aided detection and decision systems have been developed with variable success, tending to target-specific mammographic features.⁵⁹ A UK trial (CADET II) demonstrated that CAD systems could potentially replace a reader in a double reading scenario.⁶⁰ However, CAD approaches have never had any great impact or widespread support within the UK screening community.

In marked contrast, the subsequent growth of artificial intelligence (AI) and machine learning approaches has come at a time when the UK is struggling to recruit radiologists, and these may therefore potentially offer considerable help to the screening programme.⁶¹ There are several ways in which an appropriate AI approach could be implemented in UK breast screening, the most promising may be to filter out a large number of normal cases so reducing the readers’workload.⁶²

AI approaches are developed using a large database of training cases and then the developed algorithms are tested using a separate large set of test cases with the algorithm’s performance generally being taken as its success on these test cases. However, there is a growing realisation of the need for a third stage of AI algorithm validation which is where the PERFORMS scheme is valuable. By applying AI algorithms to sets of cases where the clinical outcome is both known and which large numbers of radiologists have read, then it becomes possible to rate algorithms’ performance against health professionals’ skills.

Consequently, the PERFORMS AI Grand Challenge⁶³ has been established to allow developers to assess their software on the same challenging cases, both against other AI systems, and also against a large cohort of experienced breast screeners. AI developers can therefore directly see how their system would potentially perform in the real screening environment and possible purchasers can know whether systems are truly ready for market and fit for purpose.

The PERFORMS scheme was established specifically for mammography in breast screening. However, the approach has since been extended to encompass digital breast tomography (DBT). A current major UK trial (PROSPECTS) of DBT in screening is underway in which the scheme is involved. It is also embedded as part of a major EU H2020 international trial (MyPEBS) of personalized breast screening across Europe where the role of the scheme is to monitor performance across hundreds of readers from different countries over the course of the trial. Based on some aspects of the PERFORMS model, some other countries (e.g., Australia, Netherlands) have subsequently developed their own breast screening monitoring approaches. The PERFORMS scheme is available internationally and has also been developed for other radiological domains (www.iperforms.com).

Conclusions

PERFORMS is a UK national self-assessment and educational scheme which has been implemented alongside the breast screening programme, providing peer review and quality assurance functions. The scheme quickly identifies individuals who are underperforming and facilitates their improvement. Through a process of continuous development, the scheme maintains advantage to readers and the screening programme. The scheme is implemented internationally as well as being developed across other radiological domains.