ROCView: prototype software for data collection in jackknife alternative free-response receiver operating characteristic analysis

Abstract

ROCView has been developed as an image display and response capture (IDRC) solution to image display and consistent recording of reader responses in relation to the free-response receiver operating characteristic paradigm. A web-based solution to IDRC for observer response studies allows observations to be completed from any location, assuming that display performance and viewing conditions are consistent with the study being completed. The simplistic functionality of the software allows observations to be completed without supervision. ROCView can display images from multiple modalities, in a randomised order if required. Following registration, observers are prompted to begin their image evaluation. All data are recorded via mouse clicks, one to localise (mark) and one to score confidence (rate) using either an ordinal or continuous rating scale. Up to nine “mark-rating” pairs can be made per image. Unmarked images are given a default score of zero. Upon completion of the study, both true-positive and false-positive reports can be downloaded and adapted for analysis. ROCView has the potential to be a useful tool in the assessment of modality performance difference for a range of imaging methods.

The success of an imaging technique should not be judged by physical measures, such as signal-to-noise ratio or contrast resolution, alone. Human observations must be considered because the reader is an integral part of the diagnostic process. For the latter, receiver operating characteristic (ROC) methods successfully quantify the combined performance of imaging techniques and readers [1].

Traditional ROC analysis simply required readers to state whether they thought a case was normal or abnormal, using a rating scale to indicate decision confidence [2]. A notable development in ROC methodology has been the free-response ROC (FROC) method. Unlike ROC methods, FROC uses location information to resolve ambiguities in detection (lesion or lesion mimic) to prevent false identification of pathology, resulting in a true-positive result [3].

Jackknife alternative free-response ROC (JAFROC) methods [4] are the latest evolution of FROC methods in which multiple readers and modalities are compared with greater sensitivity to differences in performance from other ROC methods [5]. Table 1 lists the terms associated with JAFROC analysis. JAFROC methods demand a precise response from the reader, in which location and confidence information must be supplied for each lesion within a case [6]. This identification and scoring is typically referred to as a “mark-rating” pair [7].

Table 1. The terminology associated with jackknife alternative free-response receiver operating characteristic analysis.

Term	Alternative meaning/comment
Reader	Observer, participant
Case	Image
Lesion	Pathology
Modality	Variable
Case matched	Same image displayed
Lesion localisation (LL)	Successful identification of pathology (TP)
Non-lesion localisation (NL)	Unsuccessful identification of pathology (FP)
Truth	The true site of pathology in a case
Signal	A stimulus within background noise stimulating a localisation
Case memory	The likelihood of a reader remembering the appearance of cases shown in sequential order

Those setting up JAFROC studies are encouraged to allow mouse clicks to record reader responses and an acceptance radius around each area of pathology to classify each response as either a lesion localisation/true positive (LL/TP) or a non-lesion localisation/false positive (NL/FP) [8].

JAFROC methods can be valuable in the evaluation of different techniques in radiology. Like all observer performance studies, achieving optimal statistical power can require a large number of readers and cases [9,10], and a 50:50 ratio of normal and abnormal cases [11]. However, JAFROC methods achieve greater statistical power than ROC for the same combination of readers and cases [7]. This in turn requires a reliable method of image display and response capture (IDRC). All ROC methods require accurate data recording; ROCView is proposed as a computer-based solution to accurate IDRC in JAFROC analysis—the current end point of the free-response paradigm.

Why develop ROCView?

A recent study of dose optimisation in CT [12] prompted investigation into a reliable solution for IDRC suited to JAFROC methodology. Although other software exists [13,14], the authors took the opportunity to develop ROCView to suit the needs of their current research. As a web-based service ROCView would have excellent availability with no software download or installation required. A requirement of ROCView would be to produce data suitable for analysis via JAFROC [15] and Dorfman–Berbaum–Metz Multi-reader Multi-case (DBM-MRMC) software [16] for direct comparison of the performance of multiple readers and modalities.

Gathering suitable readers in a single centre for an observer study also presented itself as a boundary to conducting JAFROC research. Consequently, a geographically independent method of IDRC was required to allow readers to complete the observation—this prompted the development of a web-based solution. ROCView meets the demands of geographical independence and IDRC with the option to store resultant data in a form suitable for DBM-MRMC analysis. The nature of the software solution required a simplistic method to maintain the integrity of the data, as there would be no on-site support. Figure 1 describes the process of perception that readers will use when assessing studies on ROCView.

Figure 1.

The process used by readers attempting a study on ROCView.

Design and development of ROCView

ROCView is implemented as a web application, with reader evaluations performed in a front-end display system (browser/client), with a web server supplying image data and recording reader responses into a database. A second web interface provides administration facilities: user management, modality and case management, creation of the “truth” and report generation. Guidance provided by Chakraborty [17] in relation to conducting a JAFROC study aided the design of ROCView. The development of a successful IDRC system required an understanding of the following key principles:

• image display

• comparing modalities

• setting the “truth”

• acceptance radius criterion

• scoring confidence

• data recording.

The following description and the flow diagram shown in Figure 2 explain the process for setting up and running a JAFROC study on ROCView.

Figure 2.

The administration process used to create a study on ROCView.

Image display

Image display is fairly straightforward for a JAFROC study. Cases from multiple modalities are displayed, with quick progression between them. ROCView automatically randomises the images from all modalities in order to reduce case memory. As a web-based application it is important to ensure that the viewing conditions are consistent and to a certain standard. A variety of monitor test patterns can be used to ensure monitor performance [18], in which it is also important to ensure a consistent display response to allow a consistent response by the observer. Methods to assess the adequacy of display performance in accordance with the digital imaging and communication in medicine (DICOM) greyscale standard display function (GSDF) are available [19].

Comparing modalities

To allow direct comparison of two modalities, such as a variation in image acquisition parameters, ROCView allows the upload of a large number of image data sets via file transfer protocol (FTP) [20]. All uploaded modalities should be named or numbered appropriately and all cases should be numbered and case-matched across all modalities. When assigned to the appropriate study, cases are automatically randomised. From this point the study is ready for the “truth” to be set.

Setting the “truth”

A suitably trained person should access the study as an administrator and create the “truth”, which is stored in a database on ROCView to act as a standard with which all further reader responses are compared, via the implementation of a pre-defined acceptance radius arising from the pixel that is marked on the image. When performing this task the administrator is able to view the modality and case identifier alongside the image, together with the localisation marker showing the acceptance radius, in order to minimise error in lesion localisation. This is particularly useful for phantom studies in which the lesion sites are known. All readers are blinded to modality and case identifiers. The administrator can make up to nine lesion localisations per case.

Acceptance radius criterion

An acceptance radius is a region surrounding a lesion that allows slight error in localisation. ROCView allows adjustment of the acceptance radius in the source code. The acceptance radius for current studies, determined by lesion size, is nine pixels’ radial distance from the centre of the localisation made by the administrator creating the “truth”. All marks within this radius are classified as LLs. All other marks are NLs, penalising the reader for inaccuracies and false localisations. This is, however, dependent on the resolution of the monitor and the image size. Currently, the largest size of image used has been 512×512 pixels, which allows for a 1.8% error with an acceptance of 9 pixels in all directions from the point of localisation. If monitor resolution does not allow the maximum image size then it will be scaled, with a 100% viewing window, and centred over the cursor, available in the top right of the screen. Figure 3 explains the implication of acceptance radius size; a clinical example would equate to a vessel (lesion mimic) next to a lesion in the thorax being incorrectly localised but being classified as LL owing to it falling within the acceptance radius of the lesion. Different values of radii clearly affect reader performance [21], where a relaxed criterion (20–40 pixels) improved apparent reader performance. Localisation accuracy is an important distinction between conventional ROC and free-response methods; consequently, this criterion, at the discretion of the researcher, must be controlled to ensure that JAFROC maintains precision. When a reader clicks on the image in ROCView a 19-pixel-diameter marker appears to aid the reader's judgement of accuracy (Figure 4). Existing phantom-based ROCView studies have not used simulated lesions that are >19 pixels in diameter. Therefore, localisation of the centre of the lesions would always result in a TP/LL result.

Figure 3.

The implication of acceptance radius. A larger acceptance radius could result in a greater number of incorrect true-positive results owing to the marking of a lesion mimic that falls within the acceptance radius. A small acceptance radius can reduce this risk and would increase the number of false-positive results owing to lesion mimics and reader inaccuracies.

Figure 4.

Screenshot of ROCView. The main image (scaled) shows a reader mark that prompted the confidence scale to appear. Note the option to remove unwanted clicks. The image in the top right of the screen shows the main image at 100% size, without reader marks for re-evaluation of uncertain areas. Progress through the observation challenge is also displayed.

Scoring confidence

Many variations of confidence scales are described [22-24] and one must be mindful that one scale may not be suited to all investigations. To address this, ROCView offers two types of scale: continuous and discrete (Figure 5). When creating a study the administrator can choose which type of scale will be presented to the reader following a mark made on an image. The discrete scale appears as a pop-up box showing a series of ranked radio buttons (1–5, low to high confidence) from which readers can make their selection. The continuous scale presents as a slider bar that can function as a 101-point scale similar to those in use [24]. The data can be resampled to an 11-point discrete rating scale if there is not a good distribution of responses—a problem consistent with this type of scale arising from the inability to estimate within a few percentage points [23].

Figure 5.

Screenshot of ROCView showing the continuous-style confidence scale variant. A mouse click prompts the slider bar to appear. The slider starts at the left side of the bar (score 1.0/low confidence) and can be moved to any point along it to a maximum of 10.0 (high confidence).

Data recording

Once confidence has been scored, mark-rating pairs are stored in a database (TP or FP) in the format shown in Tables 2 and 3, which can then be downloaded and analysed via JAFROC methods.

Table 2. True-positive database format as recorded by ROCView.

Reader ID	Modality ID	Case ID	Lesion ID	Reader mark (x, y)	Truth (x, y)	Proximity (pixels)	TP rating
Name	Variable	Value	Value	Value	Value	Value	Confidence
Example
a@b.com	High resolution	4	1	121 293	120 294	1.4	4

ID, identity; TP, true positive.

HTML format, all fields; CSV format, exclude italic fields.

Table 3. False-positive database format as recorded by ROCView.

Reader ID	Modality ID	Case ID	Reader mark (x, y)	FP rating
Name	Variable	Value	Value	Confidence
Example
a@b.com	Low resolution	39	236 129	3

FP, false positive; ID, identity.

HTML format, all fields; CSV format, exclude italic fields.

For each successful localisation five integer values are recorded in the TP sheet. Four integers are recorded for each incorrect localisation in the FP sheet. The following data need to be recorded as identifiable integers:

• reader ID

• case ID

• modality ID

• lesion ID (1, 2, 3 etc.)—not used in the FP worksheet

• confidence scale rating (TP or FP).

When no mark-rating pairs are made, a default score of zero is stored for each case and lesion. These results are not required for JAFROC analysis but have been included in the reports to validate the results recorded by ROCView.

Currently the reader ID is recorded as the login details (email address) of the reader. Once the data have been downloaded, this can easily be converted to an integer in Microsoft Excel® (Microsoft Excel, Redmond, WA). Once downloaded, TP and FP results need correlating with a “truth” sheet in a single Excel file as described by Chakraborty and Yoon [25]. This data file can then be analysed using JAFROC v. 4.0 [15] to transform the FROC data to inferred ROC data (*.lrc file), which can be analysed using DBM-MRMC software [16].

Software development

When the application starts, it makes an asynchronous request to the web server to retrieve the modalities to be evaluated by the user. This is dictated by the activation key used when registering as a user, since ROCView runs multiple studies concurrently. Cases are displayed in turn, and, as the user creates mark–rating pairs, each response is sent asynchronously to the server and is stored immediately in the appropriate database (TP or FP). The software has been made functional for general purpose personal computers, in terms of the processing power and memory required, as a result of the tools used to develop the software (Table 4). As with all FROC studies, monitor performance and viewing conditions remain a concern and these should meet the required standards consistent with the study objectives.

Table 4. Programming tools and applications used in the development of ROCView (open source components).

Tool	Function
Apache HTTP Server [26]+PHP module [27]	Web server
MySQL [28]	Server database
Flex 4 SDK [29]+jQuery [30]	Client applications, JavaScript library
Apache Ant [31]	Source file deployment
JSLint [32]	Code quality checking of JavaScript
JSMin [33]	Compression of client-side JavaScript code

The client has been designed to use the native HTML and JavaScript capabilities of browsers. Consequently, browser plugins are not necessary and the hardware requirements of the client remain low. The native solution is at risk of manipulation by the user; for example, by manipulation of the page uniform resource locators or by reloading the evaluation page in unsupervised studies. Therefore, a function is applied to the evaluation data to maintain integrity between server requests. This prevents the user from reloading the evaluation page and returning to evaluate previous images while ensuring that evaluations resume at the last stored point if they are not completed in one attempt (due to loss of internet connection or computer hardware failure).

Future application, flexibility and development

ROCView has the potential to deal with any images acquired in a manner suitable for FROC analysis. With the capability to display images from a wide range of acquisition methods (CT, ultrasound, MRI, nuclear medicine, mammography and computed/digital radiography) there is a relatively unlimited application for lesion detection-based studies. A phantom-based lesion detection study of CT acquisition parameters highlighted the type of research study for which this software is suitable [12]. Producing case-matched images is a significant challenge for MRMC studies, but if this can be achieved ROCView can be a powerful tool for the display of images and the accurate recording of reader responses.

A significant development to the software would see the creation of an acceptance volume, to allow classification of LLs and NLs, for lesions that cross multiple cross-sectional images. Software to allow this is available [14] and has been used in a FROC study of mammography techniques [34]. A further development would allow hybrid images to be displayed in their component parts (emission, transmission and fused) to allow novel FROC studies of diagnostic performance in single photon emission CT and positron emission tomography/CT.

There is also scope for the development of a region-of-interest style acceptance criterion to enable a form of JAFROC analysis suited to fracture detection in which the over-riding outcome would be a study in a similar vein to mammography PERFORMS [35] to quantify trainee performance in fracture detection. ROCView, as a web-based service, requires no download of software. ROCView is currently being used in a research programme but may become available commercially to other researchers in the future.