Abstract
Aims
To evaluate 3 Bayesian forecasting (BF) programs—TDMx, InsightRx and DoseMe—on their user‐friendliness and common liked and disliked features through a survey of hospital pharmacists.
Methods
Clinical pharmacists across 3 Australian hospitals that did not use a BF program were invited to a BF workshop and complete a survey on programs they trialled. Participants were given 4 case scenarios to work through and asked to complete a 5‐point Likert scale survey evaluating the program's user‐friendliness. Liked and disliked features of each program were ascertained through written responses to open‐ended questions. Survey results were compared using a χ2 test of equal or given proportions to identify significant differences in response.
Results
Twenty‐seven pharmacists, from hospitals, participated. BF programs were rated overall as user‐friendly with 70%, 41% and 37% (P = .02) of participants recording a Likert score of 4 or 5 for DoseMe, TDMx and InsightRx, respectively. Participants found it easy to access all required information to use the programs, understood dosing recommendations and visualisations given by each program, and thought programs supported decision‐making with >50% of participants scoring a 4 or 5 across the programs in these categories. Common liked features across all programs were the graphical displays and ease of data entry, while common disliked features were related to the units, layout and information display.
Conclusion
Although differences exist between programs, all 3 programs were most commonly rated as user‐friendly across all themes evaluated, which provides useful information for healthcare facilities wanting to implement a BF program.
Keywords: Bayesian forecasting, Likert scale, survey, therapeutic drug monitoring, user‐friendliness
What is already known about the subject
A potential barrier to the uptake of the Bayesian forecasting method of therapeutic drug monitoring is the availability of an easy‐to‐use program
A predictive performance evaluation of TDMx, InsightRx and DoseMe indicated that these programs are suitable for clinical implementation
A user‐friendly evaluation of these programs with end‐users has never been performed before
What this study adds
All 3 programs performed well in terms of their user‐friendliness
The study provides information that should be considered when planning to implement a Bayesian forecasting program in clinical settings
Increases awareness of the applicability of a Bayesian method in healthcare settings
1. INTRODUCTION
Bayesian forecasting (BF) is a mathematical approach that can be utilised to individualise drug dosing regimens when performing therapeutic drug monitoring (TDM).1, 2 This method has been implemented in several software programs available for use in clinical practice.3, 4 Initial dosage predictions, based on a population model5 can be made using this approach. As concentration measurements become available, pharmacokinetic parameter and exposure predictions can be updated to more individualised values.
This approach can be applied under complicated dosing regimens and non–steady‐state conditions and using single concentration measurements and samples taken at flexible times.3, 6 Despite these advantages, the uptake of BF methods into healthcare settings has been limited,7 even though some national guidelines recommended this approach.8 Potential barriers to uptake include the absence of high‐level pharmacokinetic and technical expertise at practice sites, lack of easy‐to‐use software programs and difficulty with validating licensed programs in the clinical setting.3, 9, 10
As with any decision support system, the user's experience needs to be considered from a design, implementation and operational perspective.11, 12 System usability is affected by software design, graphical user interface, setup assistance and system feedback.13, 14 Initial and on‐going costs, technical support and user training also affect the user's perception of a system's user‐friendliness.15, 16, 17 System usability and user‐friendliness may also be influenced by user demographics (such as age, qualifications and experience) and an individual's ability to adapt to new systems.16, 18 Thus evaluating user‐friendliness involves considering multiple factors in context of the user, their clinical setting and the system.
TDMx (http://www.tdmx.eu), InsightRx (https://insight‐rx.com/) and DoseMe (https://doseme‐rx.com/) are 3 internationally recognised BF programs currently available for use in the clinical setting. These programs utilise the same population pharmacokinetic model for tobramycin and were shown to have high accuracy and low imprecision in their predictions suggesting suitability for clinical implementation.19 Currently, no study has assessed the user‐friendliness of these programs during their application. Therefore, the aims of this study were to: (i) evaluate the user‐friendliness of TDMx, InsightRx and DoseMe; and (ii) identify common liked and disliked features of each program.
2. METHODS
2.1. Study design and survey administration
A cross‐sectional survey‐based study involving hospital pharmacists with TDM experience was conducted at the Queensland Children's Hospital (Brisbane), the Children's Hospital at Westmead (Sydney) and the Westmead Hospital (Sydney). Hospitals selected for this study were not using a BF program at the time. The study was approved by hospital and university Ethics Committees. As part of participant recruitment, the study was promoted at all 3 sites with detailed information circulated through staff emails and at staff meetings for 2 months. Four study investigators from the corresponding study sites were site‐specific investigators responsible for recruiting participants. Participation was voluntary and open to all pharmacists with TDM experience.
Four cystic fibrosis‐focused tobramycin‐based case scenarios were developed by study investigators for this study (see Supporting information Appendix S1), based on previously observed patient examples and their expertise.19 One of the cases was used as an exemplar to illustrate the use of each program. A survey was developed based on previously published system usability and clinician satisfaction surveys18, 20, 21, 22, 23, 24 (see Supporting information Appendix S2). Questions were scored on a 5‐point Likert scale, where 5 indicated the most positive response. The first part of the survey obtained participant demographic details. The second part consisted of 11 questions designed to obtain participants' perception on: (i) the user‐friendliness of each program and (ii) the liked and disliked features of each program. These questions were grouped into 7 categories; A: overall user‐friendliness (Q13); B: ease of data entry (Q14); C: availability of clinical information required by the program (Q15 and Q22); D: understanding dosing recommendations (Q16 and Q17); E: obtaining and interpreting visualisations (Q18 and Q19); F: adequacy of decision support (Q20 and Q21); and G: likelihood of recommending the tool (Q27). Questions 24 and 25 were open‐ended questions allowing participants to provide a written response around what they liked most and least about each program. The survey was piloted twice with experienced pharmacists and pharmacy students. Two questions were reviewed and reworded to increase clarity after the first pilot, no further refinements were required after a second pilot.
Study sessions were conducted in 3 phases: (i) an introductory presentation to potential participants at all 3 hospitals on tobramycin TDM, dosage adjustment tools and the BF methods; (ii) a 2‐hour workshop for survey participants (maximum of 4 participants per workshop); and (iii) completion of a paper‐based survey. The workshop started with a detailed introduction to each program using the same exemplar case scenario (case 1, Supporting information Appendix S1). This was followed by the participants using the programs for a further 3 case scenarios by themselves. The order in which each participant used each program was randomised, to help minimise recall bias. Participants were informed that certain program settings could be adjusted to suit their hospital requirements; however, for the purpose of the workshop and survey the standard form of each program was used without customised settings. TDMx (v0.95.5), InsightRx (v1.5.9) and DoseMe (v2.8.36) were used in the study.
2.2. Data analysis
Descriptive statistics were used to summarise participant demographic information. Each theme was analysed per BF program. Questions around user‐friendliness were evaluated by comparing frequency of response across the programs using a χ2 test of equal or given proportions to see if there were differences in results. Statistical differences in the proportion of participants who chose a Likert score 4 or 5 across the programs per theme was tested using RStudio (v0.99.491). The significance level was set to P < .05. Answers to the open‐ended questions were analysed by 2 investigators.25, 26 Questions around liked and disliked features of each program were evaluated by reading each written comment and grouping these into broader themes.27, 28 The percentage of overall written responses that were grouped into broader themes as a liked or disliked feature were then reported. This process was then reviewed by another investigator to confirm the written responses fitted the identified themes and responses had been grouped appropriately.
3. RESULTS
3.1. Participant demographic
Across the 3 hospitals 120 registered clinical pharmacist positions were invited to participate; however, of these, maximal 30% were available for research activities on any given day and as study participant. Forty pharmacists attended the introductory presentation and 27 of these completed the survey, with 13 not able to participate due to work‐related constraints. Characteristics of study participants are summarised in Table 1. Half of all participants were age >25 years with at least 5 years of registered pharmacy experience. Median self‐assessed TDM experience was 5 years and 89% of participants had a high to very high level of self‐assessed computer literacy according to their Likert response.
Table 1.
Participant demographic data and self‐reported experience, knowledge and computer literacy in the field
| Demographics | ||
|---|---|---|
| Sex: | Number, percentage | |
| Male | 8, 30% | |
| Female | 19, 70% | |
| Median | IQR | |
| Age (y) | 30 | 27–37 |
| Work experience (y) | 7 | 5–13 |
| TDM experience (y) | 5 | 3–10 |
| Number of weekly TDM episodes conducted | 5 | 3–10 |
| Computer work per working week (h) | 30 | 25–38 |
| Likert responsea | ||
|---|---|---|
| Median | IQR | |
| Computer literacy | 4 | 4–5 |
| Tobramycin pharmacokinetics knowledge | 3 | 3–4 |
| Knowledge on tobramycin dose adjustment tools | 3 | 2–4 |
| Knowledge on dose adjustment tools after the workshop | 3 | 3–4 |
Response recorded on a 5‐point Likert scale ranging from no knowledge at 1 to very high level of knowledge at 5.
IQR = interquartile range; TDM = therapeutic drug monitoring.
3.2. User‐friendliness of each BF program
Results on the user‐friendliness of the programs are summarised in Figure 1. The higher the Likert‐scale response, the more user‐friendly the program was perceived by participants. For categories A, B, D and G, a significant higher proportion of participants scored a 4 and 5 for DoseMe compared to InsightRx and TDMx: A: 70 vs 37 and 41%, respectively (P = .02); B: 78 vs 52 and 41%, respectively (P = .01); D: 75 vs 50 and 68%, respectively (P = .02); G: 70 vs 59 and 30%, respectively (P < .01). For categories C, E and F, no significant difference was seen between the programs. Participants found it easy to access all required information needed to use the programs, understood dosing recommendations and visualisations given by each program, and thought they supported decision‐making based on >50% of participants scoring a 4 or 5 across the programs in these categories.
Figure 1.
Bar plots displaying the survey response as per Likert score for each category and BF program together with the percent of participants scoring 4 and 5 labelled as liked, 3 labelled as neutral, and 1 and 2 labelled as disliked, displaying their frequencies for each program. Results are shown for the 7 themes for the 5‐point Likert scale
3.3. Liked and disliked features
Results from written responses to open ended questions on liked and disliked themes of each program are displayed in Figure 2. Data entry and graphical display of dose recommendations were liked feature across all 3 programs. The top 3 liked features of TDMx were: easy to use and navigate, freeware and graphical displays. Data entry was noted to be the most disliked feature of TDMx (32% of 28 comments) followed by units and layout and no file storage availability. Participants liked the graphical display of dose recommendations and data entry process in InsightRx and reported that it was an intuitive program to use. In contrast, the American date format (i.e. mm‐dd‐yyyy) used in InsightRx received unfavourable comments (44% of all dislike comments). A few participants (7%) specifically reported that InsightRx was visually appealing and appeared user‐friendly. A commonly liked feature of DoseMe was that it was easy to use and navigate (28% of all comments). Participants also liked its European date format (i.e. dd‐mm‐yyyy), its easy to understand summary page and availability of file storage. Participants had fewer comments about features they disliked in DoseMe with 50% of the 14 comments made, related to data entry and the number of computer‐clicks required to complete the tasks. Although, data entry processes also appeared as a liked feature of DoseMe with 4 positive comments. DoseMe was considered the preferred BF program by 67% of participants, InsightRx by 15% and TDMx by 15%, with 1 participant undecided. Few participants changed their mind when they were told which program was freeware (TDMx) and which were not (DoseMe, InsightRx).
Figure 2.
Summary of written comments on the liked (A) and disliked (B) features of TDMx, InsightRx and DoseMe divided into themes. The percentage value displayed is the proportion of times a particular feature was noted by participants as a liked or disliked feature. Not everyone commented and some participants made multiple comments. Total number of comments for liked features were 32, 27 and 33 for TDMx, InsightRx and DoseMe, respectively. Total number of comments for disliked features were 28, 34 and 14 for TDMx, InsightRx and DoseMe, respectively
4. DISCUSSION
This study is the first to evaluate the user‐friendliness of BF programs based on the opinions of potential future users and found that although differences exist between programs, all 3 programs were most commonly rated as user‐friendly across all categories evaluated. While 1 previous study has compared 5 different BF programs (Adult and Paediatric Kinetics, Best Dose, DoseMe, InsightRx, and PrecisePK) in terms of their adaptability, visual appeal, ease of use and company support, findings were based on study investigator rather than end‐user opinion.29
Date format and program layout were common factors influencing the user's experience. These are important design‐related features which help determine the intuitiveness of a program.4 Notably, after the completion of this study, InsightRx has been updated to include both the European and American date styles. DoseMe and InsightRx received a few unfavourable comments around system feedback and information display with a few users finding it difficult to identify the most appropriate dose from those suggested; this might be because some of these programs' features are for more advanced users with DoseMe providing 2 options and InsightRx providing 3 options for potential doses along with corresponding pharmacokinetic parameters to aid the user's clinical decision making. Participants may have found the data entry process somewhat more difficult in TDMx because they had to alternate between several tabs to obtain a potential dose.
The response rate in this study was considered positive given work demands on pharmacists and the total time required to complete the training workshop.
This study has some limitations. Participants who were more comfortable with pharmacokinetic calculations or with higher computer literacy may have been more likely to volunteer to participate in the study. Given the hospitals selected for this study did not use a BF program, most participants were likely to be first time users; however, this was not specifically ascertained. Given the likely BF naivety of participants, there may have been a preference towards straight‐forward processes with fewer options to choose from. However, with increased use, a program with expanded capacities and greater applicability across different clinical settings may become more favoured. A not applicable option was not provided in the survey and that is why not all items were answered by all participants (such as Q15 and Q22). DoseMe was originally developed in Australia and as such some participants may have had exposure to it prior to the study. Additionally, our workshop presentations did not highlight all features of each program, but focused on the basic functionalities and information needed for dosage adjustment.
This study provides valuable insight into use of BF program in the clinical setting and healthcare provider preferences. With overall positive results on the user‐friendliness of all 3 programs, it is anticipated that clinical uptake of BF programs should increase in the future.
COMPETING INTERESTS
There are no competing interests to declare.
CONTRIBUTORS
A.K., M.B., S.H., S.S., T.L., C.C., M.C., and C.S. are responsible for ethics application. T.L., I.S., S.S., C.C., and M.C. are responsible for recruitment. A.K. and M.B. collected the data. M.B., C.S., and S.H. designed the study. M.B., C.S., and S.H. contributed to the survey and case scenario design. A.K., C.S., and S.H. analysed the data. A.K., C.S., M.B., and S.H. reviewed the results. A.K., C.S., and S.H. wrote the manuscript. M.B., S.S., I.S., T.L., C.C., M.C., C.S., and S.H. reviewed the manuscript.
Supporting information
Data S1.
Supporting information
ACKNOWLEDGEMENTS
The study team would like to thank all participants at the 3 participating hospitals for their time and efforts as well as the pharmacists and students who volunteered during the pilot phase of the study.
We would like to acknowledge the Bayesian software tool distributors for providing their software for research purposes. None of them had commercial or scientific input on data entry, analysis nor interpretation of the results.
Furthermore, we acknowledge the contributions of the reviewers to improve the manuscript.
The data that support the findings of this study are available from the corresponding author upon reasonable request. S.H. was supported partly through a fellowship by the A.v. Humboldt Foundation, Germany.
Kumar AA, Burgard M, Stacey S, et al. An evaluation of the user‐friendliness of Bayesian forecasting programs in a clinical setting. Br J Clin Pharmacol. 2019;85:2436–2441. 10.1111/bcp.14066
Principle Investigator: The authors confirm that there was no PI for this paper, and no patients were involved.
REFERENCES
- 1. Sheiner LB, Beal S, Rosenberg B, Marathe VV. Forecasting individual pharmacokinetics. Clin Pharmacol Ther. 1979;26(3):294‐305. [DOI] [PubMed] [Google Scholar]
- 2. Thomson AH, Whiting B. Bayesian parameter estimation and population pharmacokinetics. Clin Pharmacokinet. 1992;22(6):447‐467. [DOI] [PubMed] [Google Scholar]
- 3. Barras MA, Serisier D, Hennig S, Jess K, Norris RLG. Bayesian estimation of tobramycin exposure in patients with cystic fibrosis. Antimicrob Agents Chemother. 2016;60(11):6698‐6702. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Fuchs A, Csajka C, Thoma Y, Buclin T, Widmer N. Benchmarking therapeutic drug monitoring software: a review of available computer tools. Clin Pharmacokinet. 2013;52(1):9‐22. [DOI] [PubMed] [Google Scholar]
- 5. Al‐Metwali B, Mulla H. Personalised dosing of medicines for children. J Pharm Pharmacol. 2017;69(5):514‐524. [DOI] [PubMed] [Google Scholar]
- 6. Hennig S, Holthouse F, Staatz CE. Comparing dosage adjustment methods for once‐daily tobramycin in paediatric and adolescent patients with cystic fibrosis. Clin Pharmacokinet. 2015;54(4):409‐421. [DOI] [PubMed] [Google Scholar]
- 7. Paviour S, Hennig S, Staatz CE. Usage and monitoring of intravenous tobramycin in Cystic Fibrosis in Australia and the UK. J Pharm Pract Res. 2016;46(1):15‐21. [Google Scholar]
- 8. Principles of aminoglycoside use Melbourne, Australia: therapeutic guidelines ltd 2018. Available from: https://tgldcdp‐tg‐org‐au.ezproxy.library.uq.edu.au/viewTopic?topicfile=aminoglycoside‐use‐principles#toc_d1e1496. Accessed December 12, 2019.
- 9. Donagher J, Martin JH, Barras MA. Individualised medicine: why we need Bayesian dosing. Intern Med J. 2017;47(5):593‐600. [DOI] [PubMed] [Google Scholar]
- 10. Darwich AS, Ogungbenro K, Vinks AA, et al. Why has model‐informed precision dosing not yet become common clinical reality? Lessons from the past and a roadmap for the future. Clin Pharmacol Ther. 2017;101(5):646‐656. [DOI] [PubMed] [Google Scholar]
- 11. Viitanen J, Hyppönen H, Lääveri T, Vänskä J, Reponen J, Winblad I. National questionnaire study on clinical ICT systems proofs: physicians suffer from poor usability. Int J Med Inform. 2011;80(10):708‐725. [DOI] [PubMed] [Google Scholar]
- 12. Petter S, Delone W, McLean E. The past, present, and future of "IS success". J Assoc Inf Syst. 2012;13(5):341‐362. [Google Scholar]
- 13. Sheikh ACK, Wright A, Bates DW. Key advances in clinical informatics transforming health care through health information technology. London, United Kingdom: Academic Press; 2017. [Google Scholar]
- 14. Saxena S, Dubey SK. Impact of software design aspects on usability. Int J Comput Appl. 2013;62(22):48‐53. [Google Scholar]
- 15. Kaipio J, Laaveri T, Hypponen H, et al. Usability problems do not heal by themselves: national survey on physicians' experiences with EHRs in Finland. Int J Med Inform. 2017;97:266‐281. [DOI] [PubMed] [Google Scholar]
- 16. Raglan GB, Margolis B, Paulus RA, Schulkin J. Electronic health record adoption among obstetrician/gynecologists in the United States: physician practices and satisfaction. Int J Health Care Qual Assur. 2017;39(3):144‐152. [DOI] [PubMed] [Google Scholar]
- 17. DesRoches CM, Campbell EG, Rao SR, et al. Electronic health records in ambulatory care‐‐a national survey of physicians. New Engl J Med. 2008;359(1):50‐60. [DOI] [PubMed] [Google Scholar]
- 18. Bani‐Issa W, Al Yateem N, Al Makhzoomy IK, Ibrahim A. Satisfaction of health‐care providers with electronic health records and perceived barriers to its implementation in the United Arab Emirates. Int J Nurs Pract. 2016;22(4):408‐416. [DOI] [PubMed] [Google Scholar]
- 19. Burgard M, Sandaradura I, van Hal SJ, Stacey S, Hennig S. Evaluation of tobramycin exposure predictions in three Bayesian forecasting programmes compared with current clinical cractice in children and adults with cystic fibrosis. Clin Pharmacokinet. 2017;57(8):1017‐1027. [DOI] [PubMed] [Google Scholar]
- 20. Carayon P, Cartmill R, Blosky MA, et al. ICU nurses' acceptance of electronic health records. J am Med Inform Assoc. 2011;18(6):812‐819. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Wakefield DS, Halbesleben JR, Ward MM, Qiu Q, Brokel J, Crandall D. Development of a measure of clinical information systems expectations and experiences. Med Care. 2007;45(9):884‐890. [DOI] [PubMed] [Google Scholar]
- 22. Sockolow PS, Weiner JP, Bowles KH, Lehmann HP. A new instrument for measuring clinician satisfaction with electronic health records. Comput Inform Nurs. 2011;29(10):574‐585. [DOI] [PubMed] [Google Scholar]
- 23. Top M, Gider O. Nurses' views on electronic medical records (EMR) in Turkey: an analysis according to use, quality and user satisfaction. J Med Syst. 2012;36(3):1979‐1988. [DOI] [PubMed] [Google Scholar]
- 24. Brooke J. SUS: A "quick and dirty" usability scale In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL, eds. Usability Evaluation in Industry. London: Taylor & Francis; 1996:189‐194. [Google Scholar]
- 25. The SAGE encyclopedia of qualitative research methods. Thousand Oaks, California: SAGE publications, Inc; 2008. [cited Open Coding pages 582‐582]. Available from: http://sk.sagepub.com/reference/research. Accessed December 12, 2019. [Google Scholar]
- 26. Sutton J, Austin Z. Qualitative research: data collection, analysis, and management. Can J Hosp Pharm. 2015;68(3):226‐231. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Dey I. Qualitative Data Analysis: A User Friendly Guide for Social Scientists. London, UNITED KINGDOM: Routledge; 1993. [Google Scholar]
- 28. J SCC‐B . Keyword analysis: a new tool for qualitative research In: The SAGE handbook of qualitative methods in Health Research [internet]. London: SAGE Publications Ltd; 2010. 2019/05/24 Available from: http://sk.sagepub.com/reference/sage‐hdbk‐qualitative‐methods‐in‐health‐research. Accessed December 12, 2019. [Google Scholar]
- 29. Turner RB, Kojiro K, Shephard EA, et al. Review and validation of Bayesian dose‐optimizing software and equations for calculation of the vancomycin area under the curve in critically ill patients. Pharmacother: J Hum Pharmacol Drug Ther. 2018;38(12):1174‐1183. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data S1.
Supporting information