Skip to main content
PMC home page
JMIR Cancer logoLink to JMIR Cancer
. 2017 Jan 6;3(1):e1. doi: 10.2196/cancer.6750

Rotterdam Prostate Cancer Risk Calculator: Development and Usability Testing of the Mobile Phone App

Nuno Pereira-Azevedo 1,2,✉, Luís Osório 2, Avelino Fraga 2, Monique J Roobol 1
Editor: Han Wu
Reviewed by: Stacy Loeb, Wei Xie, Bing He
PMCID: PMC5367845  PMID: 28410180

Abstract

Background

The use of prostate cancer screening tools that take into account relevant prebiopsy information (ie, risk calculators) is recommended as a way of determining the risk of cancer and the subsequent need for a prostate biopsy. This has the potential to limit prostate cancer overdiagnosis and subsequent overtreatment. mHealth apps are gaining traction in urological practice and are used by both practitioners and patients for a variety of purposes.

Objective

The impetus of the study was to design, develop, and assess a smartphone app for prostate cancer screening, based on the Rotterdam Prostate Cancer Risk Calculator (RPCRC).

Methods

The results of the Rotterdam arm of the European Randomized Study of Screening for Prostate Cancer (ERSPC) study were used to elaborate several algorithms that allowed the risk of prostate cancer to be estimated. A step-by-step workflow was established to ensure that depending on the available clinical information the most complete risk model of the RPCRC was used. The user interface was designed and then the app was developed as a native app for iOS. The usability of the app was assessed using the Post-Study System Usability Questionnaire (PSSUQ) developed by IBM, in a group of 92 participants comprising urologists, general practitioners, and medical students.

Results

A total of 11 questions were built into the app, and, depending on the answers, one of the different algorithms of the RPCRC could be used to predict the risk of prostate cancer and of clinically significant prostate cancer (Gleason score ≥7 and clinical stage >T2b). The system usefulness, information quality, and interface quality scores were high—92% (27.7/30), 87% (26.2/30), and 89% (13.4/15), respectively. No usability problems were identified.

Conclusions

The RPCRC app is helpful in predicting the risk of prostate cancer and, even more importantly, clinically significant prostate cancer. Its algorithms have been externally validated before and the usability score shows the app’s interface is well designed. Further usability testing is required in different populations to verify these results and ensure that it is easy to use, to warrant a broad appeal, and to provide better patient care.

Keywords: mHealth, prostate cancer, nomogram

Introduction

Prostate cancer is a serious health issue, accounting for 14% of all new cancers and 6% of total cancer deaths in men worldwide [1]. With the expected increase in life expectancy, the disease’s burden is projected to increase substantially [2]. However, neither the optimal balance between screening intensity and the risk of overdiagnosis (ie, detecting indolent disease) nor the ideal prostate cancer screening test or combination of tests have been determined [3].

To address these issues, screening trials were initiated. Recently, the third analysis of the European Randomized Study of Screening for Prostate Cancer (ERSPC), the world’s largest prostate cancer screening study, has been published. Currently, with more than 13 years of follow-up, the updated results show a stable relative benefit of screening (relative risk=0.79, ie, a 21% prostate cancer mortality reduction in favor of screening) but a still increasing absolute benefit [3]. The recently published findings show that to avoid one prostate cancer death, 781 men would need to be invited to screening and 27 additional prostate cancer cases will be diagnosed compared with no screening, both decreasing as compared with previous reports with shorter follow-up [3]. In summary, the number needed to screen and to treat to avoid one death from prostate cancer is decreasing and is now lower than the reported number needed to screen in trials for breast cancer [4].

Currently, the decision to perform a prostate biopsy is mostly based on the outcome of the serum prostate-specific antigen (PSA) test. However, the serum PSA level can increase in many situations, including benign (eg, benign prostatic hyperplasia) and inflammatory conditions (eg, acute prostatitis). Moreover, the optimal cutoff value has not yet been established [5].

Leveraging the decision of performing prostate biopsy solely on the PSA value, using a PSA value greater than 3.0 ng/mL as indication for Bx, resulted in 76% negative biopsy results [6]. Conversely, using a higher PSA threshold can neglect prostate cancer cases [7]. To address this lack of specificity, it is recommended that the PSA value should be combined with other relevant patient characteristics, using so-called risk calculators [2]. Even though many are available, currently it is not possible to provide a clear recommendation about which one to use in which situation (eg, first prostate biopsy, repeated prostate biopsy, patient with small prostate) because there are no direct head-to-head comparisons [8]. One scientifically sound and extensively validated risk calculator is the Rotterdam Prostate Cancer Risk Calculator (RPCRC), based on the ERSPC Rotterdam data [9].

The RPCRC predicts the risk of a biopsy-detectable prostate cancer and also of potentially high-risk prostate cancer, defined as Gleason score ≥7 and clinical stage >T2b. This has important clinical implications as a way of decreasing overdiagnosis and overtreatment [3]. The different RPCRC algorithms provide an increasingly accurate risk estimation (ie, adding variables to the model increases its area under the curve, AUC). The algorithm uses information on PSA level, previous negative prostate biopsy, digital rectal examination (DRE) findings, prostate volume measurement, and transrectal ultrasonography (TRUS) findings. Additionally, the Prostate Health Index (phi), which aggregates the results from the Hybritech PSA, free PSA, and p2PSA (the [-2] form of proPSA), can also be used to further stratify prostate cancer risk [10]. All these different prediction models are available on the website of the Prostate Cancer Research Foundation (Figure 1) [11].

Figure 1.

Figure 1

Open in a new tab

Screenshot of the Prostate Cancer Research Foundation website showing the prostate cancer risk calculators.

At present, mobile health (mHealth), the delivery of health care services via mobile communication devices, is a growing trend with more than 160,000 medical apps available, and the number is expected to grow even further, expedited by the ubiquitous presence of mobile phones and the continuous improvements in hardware and software [12,13]. To increase its usability and accessibility, the originally Web-based RPCRC [11] has been redesigned as an app, which has several benefits for the user.

Even though the app uses the same algorithms as the available Web-based risk calculators [11], the app’s proprietary step-by-step workflow ensures that, depending on the available information, the most complete algorithm is always used. In contrast, the website user has to initially choose a specific RPCRC, which may not be the most comprehensive available and inadvertently dismiss known clinical data.

Another strength of the app is that the calculations are performed in the user’s mobile phone (ie, it works offline), which ensures a safe user experience, bypassing issues with website blocking (eg, some facilities constrain Internet access) and with infrastructure and Internet service providers (eg, slow intranet or low-speed Internet access).

Several studies have shown that mHealth was well received by users, including health care professionals and patients, in both urban and rural settings. Some examples include the use of mobile phone–based guidance for rural health providers in Tamil Nadu, India [14], and the use of a gestational diabetes app by pregnant women in Oxford, United Kingdom [15]. Moreover, it has been documented not only in young adults [16], but also in older adults—both had a high degree of acceptance of apps that promoted physical activity [17].

The aim of this study was to design and develop a mobile phone app for prostate cancer screening, based on the RPCRC algorithms. Moreover, we sought to evaluate the usability of the developed app using IBM’s Post-Study System Usability Questionnaire (PSSUQ) [18].

Methods

This study was structured according to the standard life cycle of system development: analysis, design, implementation, and evaluation, as shown in Figure 2.

Figure 2.

Figure 2

Open in a new tab

Study outline and research procedure. PSSUQ: Post-Study System Usability Questionnaire.

System Analysis

Knowledge and functional requirements for system implementation were assessed.

Knowledge Requirements

All risk calculator algorithms used in the app were developed based on the Rotterdam arm of the ERSPC, using the clinical data and prostate biopsy outcome from 3624 previously unscreened men and 2896 men with previous negative prostate biopsy. The following 4 models were built, with cumulative clinical information:

  • Model 1—PSA alone;

  • Model 2—PSA and DRE (normal/abnormal);

  • Model 3—PSA, DRE (normal/abnormal) and DRE-assessed volume;

  • Model 4—PSA, DRE (normal/abnormal), TRUS (normal/abnormal), and TRUS-assessed volume.

The predictive capability of the models within the RPCRC app were assessed in terms of discrimination (C statistic) for predicting the probability of both prostate cancer on biopsy and serious prostate cancer (defined as >T2b and Gleason score ≥7) [19]. Further details about the construction and the validation of the RPCRC algorithms have been previously published [19].

Functional Requirements

The system’s functional requirements were based on the available risk calculator algorithms that were developed by the Rotterdam ERSPC. To improve the RPCRC app usability, a unique decision tree was devised, with a multistep approach, to gather available clinical information: previous negative prostate biopsy, PSA value, DRE evaluation, TRUS evaluation, and phi value.

System Design

The app’s user interface was designed to ensure the best possible experience, according to Apple’s design guidelines. The interface was based on the RPCRC decision tree, taking into account the clarity and ease of use, and was designed using the GNU Image Manipulation Program (GIMP).

System Implementation

To ensure the best performance, a native iOS version was developed using Apple’s Xcode (Apple Inc), an integrated development environment that comprises a suite of software development tools, including debugging functions.

System Usability Evaluation

Usability is defined as the measure of the ease with which a system can be learned and used, including its safety, effectiveness, and efficiency [20]. Usability is also a measure of the effectiveness of the interaction between humans and computer systems (ie, how do users perform tasks in the system) [21]. The usability of the RPCRC app was evaluated using IBM’s PSSUQ, which is currently in its third revision and consists of 3 domains: system usefulness, information quality, and interface quality [18]. These 3 domains cover 16 questions, rated on a Likert scale from 1 (I strongly disagree) to 5 (I strongly agree; Table 1). In addition, users also had the option to write their own comments. The PSSUQ was chosen because it is a popular usability testing instrument that was validated and showed discriminative validity, discerning applications with recognizably different quality [22]. Moreover, it has been used in several other mHealth studies [16,23-25].

Table 1.

Means and standard deviations of the Post-Study System Usability Questionnaire result.

Category No. Item Mean SD
System usefulness 1 Overall, I am satisfied with how easy it is to use this application 4.67 0.557

2 It was simple to use this application 4.80 0.399

3 I was able to complete the tasks and scenarios quickly using this application 4.53 0.601

4 I felt comfortable using this application 4.55 0.747

5 It was easy to learn to use this application 4.80 0.426

6 I believe I could become productive quickly using this application 4.34 0.905
Information quality 7 The application gave error messages that clearly told me how to fix problems 3.85 1.398

8 Whenever I made a mistake using the application, I could recover easily and quickly 4.16 1.067

9 The information (such as on-line help, on-screen messages and other documentation) provided with this application was clear 4.43 0.701

10 It was easy to find the information I needed 4.47 0.654

11 The information was effective in helping me complete the tasks and scenarios 4.52 0.673

12 The organization of information on the application screens was clear 4.76 0.477
Interface quality 13 The interface of this application was pleasant 4.57 0.789

14 I liked using the interface of this application 4.51 0.819

15 This application has all the functions and capabilities I expected it to have 4.29 1.064
Overall 16 Overall, I am satisfied with this application 4.42 0.880
Total

4.48 0.832
Open in a new tab

Urologists, medical students, and general practitioners (GPs) were selected as end users; GPs were included because they are the first gatekeepers for prostate cancer screening, making the decision of whether or not to refer the patient to a urologist. Medical students’ evaluation is pertinent because they will be the urologists and GPs of tomorrow. An invitation to participate in the study was sent via email.

For the quantitative measurements (baseline characteristics, PSSUQ), means and standard deviations were calculated using software package IBM SPSS v20 (IBM Corporation).

Results

System Analysis

Knowledge Requirements

All risk calculator algorithms used in the app were developed based on the Rotterdam arm of the ERSPC, using the clinical data and prostate biopsy outcome from 3624 previously unscreened men and 2896 men with previous negative prostate biopsy [19].

In the original previously unscreened men, applying model 1 to model 4 resulted in AUCs from 0.69 to 0.79, respectively, for predicting prostate cancer and from 0.74 to 0.86, respectively, for predicting serious prostate cancer. In the previously screened group (men with at least one previous negative prostate biopsy), applying the same models, AUCs ranged from 0.62 to 0.69 for predicting prostate cancer and from 0.72 to 0.81 for predicting serious prostate cancer [19].

Several related papers that validate the algorithm of the RPCRC in different cohorts and compare the RPCRC with other calculators have been previously published, with good performance in the various settings [26-33].

Functional Requirements

A unique decision tree was designed to ensure the app would always use the most powerful risk calculator model, depending on the available information (Figure 3). This ensures that the most significant available data is used in the most complete algorithm to compute with greater reliability the probability of a positive prostate biopsy and the risk of aggressive prostate cancer.

Figure 3.

Figure 3

Open in a new tab

The Rotterdam Prostate Cancer Risk Calculator decision tree. PSA: prostate-specific antigen; DRE: digital rectal examination; phi: Prostate Health Index.

System Design

The app design can be divided into 6 interface categories: disclaimer, question, explanation, language, results, and about (Figure 4). The disclaimer must be accepted by the user before using the app. A total of 11 questions were built into the app, and, depending on the answers, one of the different algorithms could be used to predict the risk of prostate cancer and of significant prostate cancer. All question interfaces are designed in a similar way. For every question, there is an interface with an explanation of the question. The results (ie, risk of prostate cancer and risk of aggressive prostate cancer) are shown in numerical (percentage) and graphic forms. The “about” screen details the scientific background of the risk calculators and lists all contributions. The user also has the option to choose the default language: Chinese, Dutch, English, German, Portuguese, and Spanish.

Figure 4.

Figure 4

Open in a new tab

Screenshots of the Rotterdam Prostate Cancer Risk Calculator app, showing “About,” “Disclaimer,” “Explanation,” “Question,” “Results,” and “Language” screens.

System Implementation

The debugging of the app was performed within the Apple Xcode environment. All code errors were identified in a step-by-step approach, through the use of the intrinsic debugging tools, and were corrected according to Apple’s guidelines.

The functionalities of the app were assessed in various devices, namely, mobile phones and tablets, in the usability evaluation stage. Care was taken to ensure a consistent user experience across all devices.

System Usability Evaluation

A total of 92 participants evaluated the usability of the app (response rate = 11%), among whom 28 (30%) were urologists, 29 (32%) were medical students, and 35 (38%) were GPs. The mean age of participants was 31 years and 62% were female. The calculated mean and standard deviation of the PSSUQ 16 questions are presented in Table 1. “It was simple to use this application” and “It was easy to learn to use this application” had the highest rating among the 16 items, with 4.80 out of 5 possible points.

The final scores of the 3 domains evaluated (ie, system usefulness, information quality, and interface quality) are presented in Table 2. The highest score (92%) was reported for system usefulness, and information quality got the lowest score (87%). These results show that the participants were, overall, satisfied with the usability of the app.

Table 2.

Scores per evaluation category of the Post-Study System Usability Questionnaire.

Item category Actual score Possible score % Actual score
System usefulness 27.7 30 92
Information quality 26.2 30 87
Interface quality 13.4 15 89
Open in a new tab

Figure 5 shows the percentage of actual scores given by urologists, GPs, and medical students for system usefulness, information quality, and interface quality. The highest score was given for the system usefulness category by urologists.

Figure 5.

Figure 5

Open in a new tab

Percentage of actual score per item category and occupation of participants. GP: general practitioner.

Discussion

Principal Findings

Risk calculators are increasingly being used to stratify men at risk of prostate cancer. The RPCRC, previously only available digitally on the website [11], was based on the Rotterdam arm of the ERSPC, which started in 1993 in Europe to study the feasibility of population-based screening for prostate cancer and its effect on mortality [34]. This new app is publicly available on the Apple App Store [35].

To facilitate its use in clinical practice, we decided to create an mHealth version using the RPCRC algorithms. However, to simplify its use, a unique decision tree was created that offers a streamlined user experience, while incorporating additional information at every step. The app was well received by urologists and won the BJUI award for Best Urology App in 2015, presented at the American Urological Association Annual Meeting.

Starting with the total PSA value, a more complete assessment is built based on supplementary information regarding a previous negative prostate biopsy, DRE and TRUS findings, as well as phi value. Multiple external validations and comparisons of the RPCRC have shown that including more relevant information increases predictive capability [9].

This app builds on the ubiquitous presence of mobile phones to provide doctors and patients with a new way of using the RPCRC. Moreover, it maintains the ERSPC’s original goal to optimize prostate cancer screening, reducing unnecessary prostate biopsies and preventing the overtreatment of indolent prostate cancer while avoiding underdiagnosis. mHealth offers the opportunity to change the paradigm of health services, and prostate cancer, the second most common cancer worldwide, must be included in that effort [1].

In addition, it was designed and developed from day 1 by a multidisciplinary team, which included not only urologists but also other health care professionals, which has been shown to influence significantly the number of app downloads [36].

The strength of the RPCRC app is its development based on high-quality health information extracted from various published studies that validate the outcome of ERSPC risk calculator in multiple cohorts.

The IBM Computer Usability Satisfaction Questionnaire allowed the authors to obtain quantitative information regarding the app usability, which offered strong measures of usability. Moreover, taking into consideration that tests with only 5 participants are able to uncover 85% of usability issues, we believe most usability issues would be identified in this study, which included 92 users [37].

Limitations

In this study, we only discuss the development of the iOS app, but further studies are under way to replicate this for other mobile platforms. Only medical students and health care professionals took part in the usability testing, which may represent a selection bias. In the near future, a similar evaluation will be done for patients.

Conclusions

We created a scientifically valid and convenient mobile app for the RPCRC. The RPCRC has been designed to help patients and to assist health care professionals in the decision-making process. The app was found to be easy to use and, therefore, can be useful in the daily management of patients. The RPCRC app can be used in a clinical setting to better stratify the risk of prostate cancer, avoiding unnecessary biopsies and, consequently, reducing overdiagnosis and overtreatment.

Abbreviations

AUC

area under the curve

DRE

digital rectal examination

ERSPC

European Randomized Study of Screening for Prostate Cancer

GP

general practitioner

p2PSA

(-2) form of proPSA

phi

Prostate Health Index

PSA

prostate-specific antigen

PSSUQ

Post-Study System Usability Questionnaire

RPCRC

Rotterdam Prostate Cancer Risk Calculator

TRUS

transrectal ultrasonography

Footnotes

Conflicts of Interest: None declared.

References

  • 1.Jemal A, Bray F, Center MM, Ferlay J, Ward E, Forman D. Global cancer statistics. CA Cancer J Clin. 2011;61(2):69–90. doi: 10.3322/caac.20107. doi: 10.3322/caac.20107.caac.20107 [DOI] [PubMed] [Google Scholar]
  • 2.Mottet N, Bellmunt J, Briers E, van den Bergh RCN, Bolla M, van Casteren N. [2016-07-26]. Guidelines on prostate cancer. European Association of Urology. 2015. https://uroweb.org/wp-content/uploads/EAU-Guidelines-Prostate-Cancer-2015-v2.pdf .
  • 3.Schröder FH, Hugosson J, Roobol MJ, Tammela TL, Zappa M, Nelen V, Kwiatkowski M, Lujan M, Määttänen L, Lilja H, Denis LJ, Recker F, Paez A, Bangma CH, Carlsson S, Puliti D, Villers A, Rebillard X, Hakama M, Stenman U, Kujala P, Taari K, Aus G, Huber A, van der Kwast TH, van Schaik RH, de Koning HJ, Moss SM, Auvinen A. Screening and prostate cancer mortality: results of the European Randomised Study of Screening for Prostate Cancer (ERSPC) at 13 years of follow-up. Lancet. 2014 Dec 6;384(9959):2027–35. doi: 10.1016/S0140-6736(14)60525-0. http://europepmc.org/abstract/MED/25108889 .S0140-6736(14)60525-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Independent UK Panel on Breast Cancer Screening The benefits and harms of breast cancer screening: an independent review. Lancet. 2012 Nov;380(9855):1778–1786. doi: 10.1016/S0140-6736(12)61611-0. [DOI] [PubMed] [Google Scholar]
  • 5.Roobol MJ, Carlsson SV. Risk stratification in prostate cancer screening. Nat Rev Urol. 2013 Jan;10(1):38–48. doi: 10.1038/nrurol.2012.225.nrurol.2012.225 [DOI] [PubMed] [Google Scholar]
  • 6.Schröder FH, Hugosson J, Roobol MJ, Tammela TL, Ciatto S, Nelen V, Kwiatkowski M, Lujan M, Lilja H, Zappa M, Denis LJ, Recker F, Berenguer A, Määttänen L, Bangma CH, Aus G, Villers A, Rebillard X, van der Kwast T, Blijenberg BG, Moss SM, de Koning HJ, Auvinen A. Screening and prostate-cancer mortality in a randomized European study. N Engl J Med. 2009 Mar 26;360(13):1320–8. doi: 10.1056/NEJMoa0810084.NEJMoa0810084 [DOI] [PubMed] [Google Scholar]
  • 7.Thompson IM, Ankerst DP, Chi C, Lucia MS, Goodman PJ, Crowley JJ, Parnes HL, Coltman CA. Operating characteristics of prostate-specific antigen in men with an initial PSA level of 3.0 ng/ml or lower. JAMA. 2005 Jul 6;294(1):66–70. doi: 10.1001/jama.294.1.66.294/1/66 [DOI] [PubMed] [Google Scholar]
  • 8.Louie K, Seigneurin A, Cathcart P. Do prostate cancer risk models improve the predictive accuracy of PSA screening? A meta-analysis. Ann Oncol. 2014;17 doi: 10.1093/annonc/mdu525. [DOI] [PubMed] [Google Scholar]
  • 9.Roobol MJ, Schröder FH, Hugosson J, Jones JS, Kattan MW, Klein EA, Hamdy F, Neal D, Donovan J, Parekh DJ, Ankerst D, Bartsch G, Klocker H, Horninger W, Benchikh A, Salama G, Villers A, Freedland SJ, Moreira DM, Vickers AJ, Lilja H, Steyerberg EW. Importance of prostate volume in the European Randomised Study of Screening for Prostate Cancer (ERSPC) risk calculators: results from the prostate biopsy collaborative group. World J Urol. 2012 Apr;30(2):149–55. doi: 10.1007/s00345-011-0804-y. http://europepmc.org/abstract/MED/22203238 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Roobol M, Nieboer D, Houlgatte A, Vincendeau S, Lazzeri M, Guazzoni G. Reducing unnecessary biopsies for suspicion of prostate cancerxtension and validation of an ERSPC based risk calculator with phi and comparison with the PCPT Risk Calculator including %free and -2propsa. The J Urol Suppl abstract. 2013;189(4):2054. [Google Scholar]
  • 11.SWOP - The Prostate Cancer Research Foundation. [2016-10-27]. Your Prostate Cancer Risk Calculator. http://www.prostatecancer-riskcalculator.com/assess-your-risk-of-prostate-cancer .
  • 12.Torgan CE. [2016-03-08]. The mHealth Summit: Local and Global Converge. Kinetics. http://caroltorgan.com/mhealth-summit/
  • 13.Reseach2Guidance. [2016-12-23]. mHealth App Developer Economics 2015: The Current Status and Trends of the mHealth App Market. 2015 Nov. http://research2guidance.com/r2g/r2g-mHealth-App-Developer-Economics-2015.pdf .
  • 14.Gautham M, Iyengar MS, Johnson CW. Mobile phone-based clinical guidance for rural health providers in India. Health Informatics J. 2015 Dec;21(4):253–66. doi: 10.1177/1460458214523153.1460458214523153 [DOI] [PubMed] [Google Scholar]
  • 15.Hirst JE, Mackillop L, Loerup L, Kevat DA, Bartlett K, Gibson O, Kenworthy Y, Levy JC, Tarassenko L, Farmer A. Acceptability and user satisfaction of a smartphone-based, interactive blood glucose management system in women with gestational diabetes mellitus. J Diabetes Sci Technol. 2015 Jan;9(1):111–5. doi: 10.1177/1932296814556506. http://europepmc.org/abstract/MED/25361643 .1932296814556506 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Al Ayubi SU, Parmanto B, Branch R, Ding D. A Persuasive and Social mHealth Application for Physical Activity: A Usability and Feasibility Study. JMIR Mhealth Uhealth. 2014;2(2):e25. doi: 10.2196/mhealth.2902. http://mhealth.jmir.org/2014/2/e25/ v2i2e25 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Hong Y, Goldberg D, Dahlke DV, Ory MG, Cargill JS, Coughlin R, Hernandez E, Kellstedt DK, Peres SC. Testing Usability and Acceptability of a Web Application to Promote Physical Activity (iCanFit) Among Older Adults. JMIR Hum Factors. 2014 Oct 13;1(1):e2. doi: 10.2196/humanfactors.3787. http://humanfactors.jmir.org/2014/1/e2/ v1i1e2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Lewis JR. Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies. Int J Hum-Comput Int. 2002 Sep;14(3-4):463–488. doi: 10.1080/10447318.2002.9669130. [DOI] [Google Scholar]
  • 19.Roobol MJ, van Vugt HA, Loeb S, Zhu X, Bul M, Bangma CH, van Leenders AG, Steyerberg EW, Schröder FH. Prediction of prostate cancer risk: the role of prostate volume and digital rectal examination in the ERSPC risk calculators. Eur Urol. 2012 Mar;61(3):577–83. doi: 10.1016/j.eururo.2011.11.012.S0302-2838(11)01251-6 [DOI] [PubMed] [Google Scholar]
  • 20.Preece J, Rogers Y, Sharp H, Benyon D, Holland S, Carey T. Human-computer interaction. Workingham, England: Addison-Wesley Publishing Co; 1994. [Google Scholar]
  • 21.Butler KA. Usability engineering turns 10. Interactions. 1996;3(1):58–75. doi: 10.1145/223500.223513. [DOI] [Google Scholar]
  • 22.Rosa A, Martins A, Costa V, Queirós A, Silva A, Pacheco RN. European Portuguese validation of the Post-Study System Usability Questionnaire (PSSUQ). Proceedings of the 10th Iberian Conference on Information Systems and Technologies (CISTI); 2015; Aveiro. 2015. pp. 17–20. [DOI] [Google Scholar]
  • 23.Landman A, Neri PM, Robertson A, McEvoy D, Dinsmore M, Sweet M, Bane A, Takhar SS, Miles S. Efficiency and usability of a near field communication-enabled tablet for medication administration. JMIR Mhealth Uhealth. 2014 Jun 02;2(2):e26. doi: 10.2196/mhealth.3215.v2i2e26 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Steele GC, Wodchis WP, Upshur R, Cott C, McKinstry B, Mercer S, Palen TE, Ramsay T, Thavorn K, Project Collaborators And Technology Partner‚ QoC Health Inc Supporting Goal-Oriented Primary Health Care for Seniors with Complex Care Needs Using Mobile Technology: Evaluation and Implementation of the Health System Performance Research Network, Bridgepoint Electronic Patient Reported Outcome Tool. JMIR Res Protoc. 2016 Jun 24;5(2):e126. doi: 10.2196/resprot.5756. http://www.researchprotocols.org/2016/2/e126/ v5i2e126 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Stellefson M, Chaney B, Chaney D, Paige S, Payne-Purvis C, Tennant B, Walsh-Childers K, Sriram P, Alber J. Engaging community stakeholders to evaluate the design, usability, and acceptability of a chronic obstructive pulmonary disease social media resource center. JMIR Res Protoc. 2015;4(1):e17. doi: 10.2196/resprot.3959. http://www.researchprotocols.org/2015/1/e17/ v4i1e17 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Cavadas V, Osório L, Sabell F, Teves F, Branco F, Silva-Ramos M. Prostate cancer prevention trial and European randomized study of screening for prostate cancer risk calculators: a performance comparison in a contemporary screened cohort. Eur Urol. 2010 Oct;58(4):551–8. doi: 10.1016/j.eururo.2010.06.023.S0302-2838(10)00556-7 [DOI] [PubMed] [Google Scholar]
  • 27.Jeong CW, Lee S, Jung J, Lee BK, Jeong SJ, Hong SK, Byun S, Lee SE. Mobile application-based Seoul National University Prostate Cancer Risk Calculator: development, validation, and comparative analysis with two Western risk calculators in Korean men. PLoS One. 2014;9(4):e94441. doi: 10.1371/journal.pone.0094441. http://dx.plos.org/10.1371/journal.pone.0094441 .PONE-D-13-40592 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.van den Bergh RC, Roobol MJ, Wolters T, van Leeuwen PJ, Schröder FH. The Prostate Cancer Prevention Trial and European Randomized Study of Screening for Prostate Cancer risk calculators indicating a positive prostate biopsy: a comparison. BJU Int. 2008 Nov;102(9):1068–73. doi: 10.1111/j.1464-410X.2008.07940.x. doi: 10.1111/j.1464-410X.2008.07940.x.BJU7940 [DOI] [PubMed] [Google Scholar]
  • 29.Trottier G, Roobol MJ, Lawrentschuk N, Boström PJ, Fernandes KA, Finelli A, Chadwick K, Evans A, van der Kwast TH, Toi A, Zlotta AR, Fleshner NE. Comparison of risk calculators from the Prostate Cancer Prevention Trial and the European Randomized Study of Screening for Prostate Cancer in a contemporary Canadian cohort. BJU Int. 2011 Oct;108(8 Pt 2):E237–44. doi: 10.1111/j.1464-410X.2011.10207.x. doi: 10.1111/j.1464-410X.2011.10207.x. [DOI] [PubMed] [Google Scholar]
  • 30.Oliveira M, Marques V, Carvalho AP, Santos A. Head-to-head comparison of two online nomograms for prostate biopsy outcome prediction. BJU Int. 2011 Jun;107(11):1780–3. doi: 10.1111/j.1464-410X.2010.09727.x. doi: 10.1111/j.1464-410X.2010.09727.x. [DOI] [PubMed] [Google Scholar]
  • 31.Lee DH, Jung HB, Park JW, Kim KH, Kim J, Lee SH, Chung BH. Can Western based online prostate cancer risk calculators be used to predict prostate cancer after prostate biopsy for the Korean population? Yonsei Med J. 2013 May 1;54(3):665–71. doi: 10.3349/ymj.2013.54.3.665. http://www.eymj.org/DOIx.php?id=10.3349/ymj.2013.54.3.665 .201305665 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Yoon DK, Park JY, Yoon S, Park MS, Moon DG, Lee JG, Schröder FH. Can the prostate risk calculator based on Western population be applied to Asian population? Prostate. 2012 May 15;72(7):721–9. doi: 10.1002/pros.21475. [DOI] [PubMed] [Google Scholar]
  • 33.Poyet C, Nieboer D, Bhindi B, Kulkarni GS, Wiederkehr C, Wettstein MS, Largo R, Wild P, Sulser T, Hermanns T. Prostate cancer risk prediction using the novel versions of the European Randomised Study for Screening of Prostate Cancer (ERSPC) and Prostate Cancer Prevention Trial (PCPT) risk calculators: independent validation and comparison in a contemporary European cohort. BJU Int. 2016 Mar;117(3):401–8. doi: 10.1111/bju.13314. [DOI] [PubMed] [Google Scholar]
  • 34.Schröder FH, Denis LJ, Roobol M, Nelen V, Auvinen A, Tammela T, Villers A, Rebillard X, Ciatto S, Zappa M, Berenguer A, Paez A, Hugosson J, Lodding P, Recker F, Kwiatkowski M, Kirkels WJ, ERSPC The story of the European Randomized Study of Screening for Prostate Cancer. BJU Int. 2003 Dec;92 Suppl 2:1–13. doi: 10.1111/j.1464-410x.2003.04389.x. http://onlinelibrary.wiley.com/resolve/openurl?genre=article&sid=nlm:pubmed&issn=1464-4096&date=2003&volume=92&issue=&spage=1 . [DOI] [PubMed] [Google Scholar]
  • 35.iTunes. [2016-10-31]. Rotterdam Prostate Cancer Risk Calculator. https://itunes.apple.com/us/app/rotterdam-prostate-cancer/id729313737?mt=8 .
  • 36.Pereira-Azevedo N, Osório L, Cavadas V, Fraga A, Carrasquinho E, Cardoso de Oliveira E, Castelo-Branco M, Roobol MJ. Expert Involvement Predicts mHealth App Downloads: Multivariate Regression Analysis of Urology Apps. JMIR Mhealth Uhealth. 2016;4(3):e86. doi: 10.2196/mhealth.5738. http://mhealth.jmir.org/2016/3/e86/ v4i3e86 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Nielsen J. [2016-07-26]. Why you only need to test with 5 users. Nielsen Norman Group. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

Articles from JMIR Cancer are provided here courtesy of JMIR Publications Inc.

RESOURCES