Abstract
Objective
To determine the perceived barriers and enablers to efficient completion of the College of Intensive Care Medicine (CICM) of Australia and New Zealand Formal Project – a trainee research project mandated for award of CICM Fellowship – and to develop consensus-based recommendations to support Intensive Care trainees and supervisors.
Design
A two-stage modified Delphi study was conducted. In stage one, an anonymous electronic survey was distributed with three targeted open-ended questions relating to perceived key steps, barriers to, and improvements for efficient completion of the Formal Project. A thematic analysis used the survey results to generate a list of close-ended questions.
In stage two, a consensus panel comprising of 30 panellists including CICM trainees, Formal Project supervisors and assessors, and critical care researchers, underwent a Delphi process with two rounds of voting and discussion to generate consensus-based recommendations.
Setting
Surveys were distributed to Intensive Care Units across Australia and New Zealand. The consensus panel convened at the Queensland Critical Care Research Network Annual Scientific Meeting in Redcliffe, Queensland, Australia, on 9 June 2023.
Participants
CICM trainees, Formal Project supervisors and assessors, and critical care researchers in Australia and New Zealand.
Main outcome measures
Consensus-based recommendations for the CICM Formal Project.
Results
We received 88 responses from the stage one survey. Stage two finalised 22 consensus-based recommendations, centring on key steps of the research process, resources for trainees, and support and training for supervisors.
Conclusions
Twenty-two recommendations were developed aiming to make the process of completing the mandatory CICM research project more efficient, and to improve the quality of research produced from these projects.
Keywords: 4 Anaesthesia and intensive care, 4.24 Intensive care, 12 Education, 62 Miscellaneous
1. Introduction
The College of Intensive Care Medicine (CICM) of Australia and New Zealand requires trainees to complete a research project, known as the Formal Project. Successful completion of the Formal Project is mandated for award of CICM fellowship. Different types of research projects meet the requirements of the Formal Project, including clinical audits, systematic reviews, epidemiologic studies and clinical trials.1
The CICM's requirement to complete a research project during training is in line with most other Australian and New Zealand medical colleges.2,3 While previous reports have demonstrated trainees' desire for involvement in research,4,5 some have highlighted the barriers to conducting research during specialty training.[5], [6], [7], [8], [9] The most common impediments include competing clinical commitments,7,8 lack of time,6,8,9 and lack of availability of, or interest from, supervisors.5,8 In addition, there is a wide variability in publication productivity associated with award of CICM fellowship, with only a minority of CICM fellows having publications related to their Formal Projects.10
Similar barriers have been identified by trainees of other medical specialties in Australia and New Zealand3,[7], [8], [9] and Intensive Care trainees in the United Kingdom.5 There are geographic and framework differences in the UK, where Trainee-led Research Networks are available to provide trainees with opportunities to be involved in high-quality, multi-centre projects. Some other Australian and New Zealand colleges offer different research training requirements to CICM, such as an alternative coursework pathway.9 There has not yet been a study exploring the perspectives of CICM trainees and/or Formal Project supervisors and assessors, nor specific recommendations for improving the successful completion of the CICM Formal Project through consideration of trainee, supervisor, and college factors.
We performed this modified Delphi study to: (i) identify the barriers and enablers to undertaking the CICM Formal Project; (ii) identify and prioritise strategies that will increase the capability, capacity and opportunities for trainees to complete the CICM Formal Project; and (iii) develop consensus-based recommendations to support trainees and supervisors to facilitate a more efficient undertaking of the CICM Formal Project.
2. Methods
2.1. Study design
A two-stage modified Delphi process11,12 utilising both an electronic survey and consensus panel was employed. The Delphi methodology is a multi-stage process of generating consensus amongst experts and stakeholders.[13], [14], [15], [16], [17] After each round, obtained data were analysed, and then presented to participants in a structured manner, ultimately designed to combine opinions into group consensus. In a planned modification to the standard Delphi method, the list of questions presented to the consensus panel was pre-selected by the investigators based on a thematic analysis from initial survey results (instead of stemming from ideas or statements from the consensus panel). The discussion by the consensus panel was held in a hybrid face-to-face and virtual meeting (rather than panellists giving anonymous feedback). Consideration of an alternative pathway or alterations to the current guidelines were not within the scope of this study.
2.2. Study procedures
2.2.1. Stage one – survey
An anonymous electronic survey was administered online via Google Forms™. Participant demographic data including the respondent's role and prior research experience were collected. The survey included three targeted, open-ended questions:
-
1.
What do you think are the key steps that CICM trainees should follow to successfully complete the CICM Formal Project?
-
2.
What do you perceive to be the barriers to the successful completion of the CICM Formal Project?
-
3.
What do you think will be helpful to improve the successful completion of the CICM Formal Project?
2.2.1.1. Participants
We used convenience and snowball sampling techniques to distribute the survey. The denominator was difficult to ascertain due to the use of snowball sampling, but we estimate approximately 200 CICM trainees and 130 Formal Project supervisors and assessors received the survey. Professional and investigator networks (i.e. Intensive Care Unit Directors and Supervisors of Training across Australia and New Zealand, CICM Formal Project assessors, the Queensland Critical Care Research Network, and the Queensland Intensive Care Training Pathway) were used to engage with key stakeholders in the CICM Formal Project (i.e. CICM trainees, Formal Project supervisors and Formal Project assessors). The survey link was active for ten days from 14 to 23 April 2023. One reminder email following the initial email was sent during this period.
2.2.1.2. Thematic analysis
Survey responses were reviewed to remove answers irrelevant to the remit of this study, such as those proposing alternatives to, or removal of, the Formal Project. A thematic analysis18,19 was performed by AH. Data familiarisation through repeated reading of the survey responses was followed by the generation of initial codes. Codes that were derivative of each other were combined (e.g. “plan early” and “early planning” were combined). Codes were collated into themes, which were then reviewed and refined. The entire dataset was reviewed again to confirm thematic validity. The codes and themes were used to generate statements in relation to the question they were answering (i.e. key steps, barriers, improvement).
The investigator group refined the list of statements using an electronic survey administered via Google Forms™. The thematic statements were listed in random order and each was scored on a seven-point Likert scale (where 1 = strongly disagree and 7 = strongly agree). Statements with an average score of 4 or less were discarded. To ensure that the statements reviewed by the consensus panel were actionable, thematic statements which covered processes that were already mandatory, not measurable, too individualised, or not actionable for either the trainee, supervisor or college, were removed. The remaining items were then translated into close-ended questions.
The Delphi process and questions were piloted at two Queensland Critical Care Research Network (QCCRN) online forums on 25 May and 1 June 2023. Ten participants were involved in the meeting, including Intensive Care specialists and research coordinators from across Queensland. All questions were presented at the pilot sessions. One of the questions, “Choosing a supervisor or supervisory group is a key step to successful completion of the CICM Formal Project”, elicited discussion, specifically the term used to describe Formal Project supervisors. Subsequently, we performed a Principal Component Analysis (PCA)20 and retained three components, resulting in the terms “helpful”, “capacity”, and “experienced” being selected and incorporated into the question (see Supplementary Methods, Fig. S1 and Table S1).
2.2.2. Stage two – consensus panel
2.2.2.1. Participants
The consensus panel was convened at the QCCRN Annual Scientific Meeting in Redcliffe, Queensland, Australia on 9 June 2023. Members of the panel were automatically included based on their attendance at the QCCRN Annual Scientific Meeting, with the option to opt out. The panel included CICM trainees, Formal Project supervisors and assessors, and other critical care researchers. The panel discussion was conducted using a hybrid in-person and virtual meeting format, with online participants joining in via videoconferencing using Zoom®. The panellists were given the list of close-ended questions generated from stage one, one day before the conference (see Supplementary Table S2). Two Delphi rounds took place on the same day. In the first round, the list of close-ended questions was presented one at a time to the panellists. The panellists responded to the questions using a 6-point Likert survey where 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree, and abstain from voting). The participants had 1 minute per question to record an anonymous response via Mentimeter®, an online audience engagement platform. Due to the anonymous nature of the voting process, if a panellist had not voted by the end of 1 minute, their response was not recorded.
The results from the first Delphi round (i.e. the percentage distribution of votes for each Likert response) were presented to the participants prior to round two. Consensus was defined a priori:13,14,16,17 recommendation required 70% or more scoring 4 or 5, not for recommendation required 70% or more scoring 1 or 2, and any other result was recorded as did not reach consensus. Questions that did not reach consensus were discussed among the panellists, moderated by AH, a CICM trainee, and JAS, a research nurse.
To facilitate the effective participation of online attendees, one of the QCCRN members (FE) monitored the Zoom® chat platform. FE could signal the moderators (AH and JAS) to acknowledge and address comments from online participants. Additionally, Mentimeter® allowed participants to submit comments anonymously, which were monitored throughout the discussion (FE) to allow all comments to reach the moderators. After discussion, the questions that did not reach consensus were amended as per panel discussion and presented in the second Delphi round for re-vote. After presentation of round two results, questions that did not reach consensus were discussed by the panel.
3. Results
3.1. Stage one – electronic survey
Eighty-eight responses were received from 38 CICM trainees, 33 Formal Project supervisors, six Formal Project assessors, and 11 who were both supervisors and assessors. Respondent characteristics are detailed in Supplementary Table S3. Forty-three per cent of the respondents were CICM trainees. The majority (92%) of respondents had prior research experience. Of the 20 CICM trainees who had not yet completed the CICM Formal Project, six (30%) had at least one first-author publication in a peer-reviewed journal, three (15%) had a peer-reviewed publication but not as first author, six (30%) had performed unpublished research, and five (25%) did not have prior research experience.
The thematic analysis of the survey responses extracted 151 codes and 20 themes. Using the themes and the three survey questions, the codes were translated into closed questions. There were 43 questions relating to key steps, 50 questions relating to barriers, and 58 questions relating to improvements. These were refined to 22 questions by the investigator group for presentation to the consensus panel.
3.2. Stage two – consensus panel
There were 30 participants in the consensus panel, with 24 in-person and six online participants. The demographic profile of the Delphi panellists was similar to those who responded to the survey (Supplementary Table S4). There was a higher proportion of Formal Project supervisors (50%) present for the consensus panel compared to the electronic survey response. Four panellists did not fill in the demographic survey.
In round one, 19 of 22 (86%) questions achieved the criteria for recommendation; three questions (Questions 4, 5 and 20) did not reach consensus (Fig. 1 and Supplementary Table S2). These were modified through mediated discussion amongst the panellists. An additional question (5b) was proposed based on Question 5.
Fig. 1.
Delphi round one results. Twenty-two questions (Q) were presented to the consensus panel during round one of the Delphi process. Each stacked bar represents the proportion of votes for each answer. The red line represents the consensus threshold of 70%, defined a priori.
In round two, these four questions were presented for voting. Three questions (Questions 4, 5b and 20) achieved recommendation. Question 5a, “allocation of non-clinical time dedicated for project supervision for supervisors is likely to improve successful completion of the CICM Formal Project”, did not reach consensus (Supplementary Table S5).
3.3. Recommendations
The Delphi process resulted in 22 recommendations (Table 1). Key steps for trainees to follow are summarised in Fig. 2.
Table 1.
Final recommendations for the College of Intensive Care Medicine (CICM) Formal Project at the conclusion of the Delphi process.
| Number | Recommendation |
|---|---|
| R1 | Developing a mentorship program for trainees is likely to improve successful completion of the CICM Formal Project.
|
| R2 | Providing opportunities for resources aimed at improving research literacy amongst trainees is likely to improve successful completion of the CICM Formal Project.
|
| R3 | Providing opportunities for training to supervisors is likely to improve successful completion of the CICM Formal Project.
|
| R4 | Establishing a college framework and appropriate support for Formal Project supervisors. |
| R5b | Allocation of non-clinical time dedicated to the Formal Project for trainees is likely to improve successful completion of the CICM Formal Project. |
| R6 | Having a registry of potential supervisors is likely to improve successful completion of the CICM Formal Project |
| R7 | Choosing a supervisor or supervisory group with capacity is a key step to successful completion of the CICM Formal Project
|
| R8 | Choosing a helpful supervisor or supervisory group is a key step to successful completion of the CICM Formal Project.
|
| R9 | Choosing an experienced supervisor or supervisory group is a key step to successful completion of the CICM Formal Project.
|
| R10 | Planning to do the project early in Intensive Care training is a key step to successful completion of the CICM Formal Project. |
| R11 | Having a feasible project timeline is a key step to successfully completing the CICM Formal Project. |
| R12 | Having regular meetings between the trainee and the project supervisor is a key step to successful completion of the CICM Formal Project. |
| R13 | Planning staged feedback throughout the research project is likely to improve successful completion of the CICM Formal Project.
|
| R14 | Developing a research question in keeping with S.M.A.R.T. (specific, measurable, achievable, realistic, and timely) principles is a key step to successful completion of the CICM Formal Project. |
| R15 | Performing a literature review to inform the research question and study design is a key step to successful completion of the CICM Formal Project. |
| R16 | Devising a clear research plan in the form of a written protocol is a key step to successful completion of the CICM Formal Project. |
| R17 | Having the research protocol reviewed by experienced clinician-researchers is a key step to successful completion of the CICM Formal Project. |
| R18 | Seeking statistical support at the stage of protocol design, which may include development of a statistical analysis plan, is a key step to successful completion of the CICM Formal Project. |
| R19 | Achieving timely ethics and governance approvals where relevant is a key step to successfully complete the CICM Formal Project. |
| R20 | Developing the skill of scientific writing is a key step to successful completion of the CICM Formal Project. |
| R21 | Critical appraisal of the project, including methodology and results, by the trainee is a key step to successful completion of the CICM Formal Project.
|
| R22 | Ensuring that the chosen project adheres to the requirements as outlined by CICM is a key step to successful completion of the CICM Formal Project.
|
R: Recommendation.
Fig. 2.
Key Steps for the Formal Project for CICM Trainees based on recommendations from the Delphi process. The numbers within the square brackets correspond to the recommendation item in Table 1. S.M.A.R.T. refers to specific, measurable, achievable, realistic and timely.
Trainees should plan to do the Formal Project early in training and consider the timing of competing priorities. Two of the most described barriers in our survey were the college examinations and rotation across different clinical sites and terms. The project timeline should be feasible. In choosing a supervisory group, trainees should consider their capacity (i.e. time and resources to adequately supervise the trainee), helpfulness and experience. Trainees should meet regularly with their supervisors, and plan staged feedback during their projects.
A registry of potential supervisors published by CICM would be helpful to assist trainees in searching for a supervisor. To ensure that supervisors can support trainees adequately, we recommend that supervisors have access to training to further their research and supervision skills. The college should consider establishing a training framework for supervisors to ensure they receive support and are, in turn, empowered to provide appropriate support to trainees.
As the trainee commences the project, they should perform a literature review, develop a research question in keeping with S.M.A.R.T. (specific, measurable, achievable, realistic, and timely) principles,21 establish a clear research plan, and seek early statistical support. The research protocol must be critically reviewed by experienced researchers. The trainee and supervisory group should ensure that the project adheres to the CICM requirements. Ethics and governance reviews were frequently cited in the survey as a barrier to the timely completion of the project; the time to process and revise these applications should be considered during initial planning. During the project, trainees should critically appraise their project and develop the skills of scientific writing.
We recommend that the CICM support a mentorship program where trainees can benefit from the knowledge of more experienced trainees and other clinician-researchers. Trainees should access resources aimed at improving research literacy such as research methodology courses. We recommend trainees be allocated some non-clinical time dedicated to their Formal Projects.
4. Discussion
This modified Delphi study resulted in 22 recommendations to support trainees and supervisors in the efficient completion of CICM Formal Projects. In the initial survey, over half of the CICM trainee respondents had never published in a peer-reviewed journal, and only 30% had been the first author in a peer-reviewed publication. The barriers in performing research during training identified in previous studies[5], [6], [7], [8], [9] were echoed in the survey responses. Choosing the “right” supervisor can be challenging but critical in ensuring the feasibility of the research project.22 The supervisor's availability of time and resources should be investigated and taken into consideration. Existing literature highlighted that supervisors may have a high supervision load or varying expertise, and as such, trainees should consider one or more co-supervisors.22,23 Helpful supervisors and supervisors who have experience in research and publications is important for the success of the research project.22
A key recommendation of our study was to establish a feasible project timeline. Previous papers have described how to approach a research project and structure a scientific paper.24,25 Similar to our recommendations, they highlighted feasibility. The supervisor is crucial in ensuring the type and timeline of the project are appropriate.25 The research question should be guided by a literature review, which can be assisted by a research librarian.26 A clear and concise research protocol should be established25 and supported by qualified biostatistical support to prevent poor or invalid results, or the need for additional data collection after project completion.27,28 Ethics and governance reviews can be complicated and lengthy processes.29,30 These were frequently cited in the initial survey as a delay in the completion of the Formal Project. The time to process these applications should be considered during planning of the project, and local research coordinators may be engaged to assist in this process.
Our study also recommended steps outside of the research process that will support a trainee's Formal Project, such as the provision of resources to improve research literacy, and training and support for supervisors. Some studies have found that protected time for research is related to increased research productivity in trainees.5,31 While we acknowledge that this may be difficult to institute, we recommended increased allocation of non-clinical time dedicated to the Formal Project for trainees. Shaw and colleagues5 noted the challenge of rotational terms, and recommended sufficient advance notice of rotations and developing a means to look up research activities and contacts in each unit prior to placement. While we did not address advance notice of placements as rotations are determined by individual units, a formal vocational training pathway such as the Queensland ICU Pathway32 can assist in more advanced planning. A national or binational coordination of research projects may also assist trainees in planning; trainees can collaborate on a project, provided that each trainee submits a separate report for the CICM. Coordinated research across multiple sites may produce more meaningful research and reduce research waste.
We noted a few limitations of our study. Several biases are known to affect survey-based studies, such as respondent selection and survey fatigue. We used snowball sampling to distribute an anonymous survey and could not gauge exact response rates. During the consensus panel, there were fewer trainees than supervisors. Despite the hierarchical imbalance, we utilised anonymous voting during the Delphi process, and the discussions were mediated by a CICM trainee and a research nurse rather than an ICU specialist to minimise authority bias. The consensus panel may not be representative of the entire CICM trainee, supervisor and assessor population given self-selection bias. While the consensus panel included representatives from all three stakeholder groups – CICM trainees, Formal Project supervisors and Formal Project assessors – it was imbalanced with only three Formal Project assessors, and with 35% of panellists not affiliated with any of the stakeholder groups. All panellists practised primarily in Queensland, potentially limiting the generalisability of their perspectives to other regions of Australia or New Zealand. We did not address specific challenges that may arise for specific Intensive Care Units (e.g. rural sites) or individuals (e.g. gender or parental status). We did not explore alternative options to the Formal Project, but we noted that other colleges such as the Australasian College of Emergency Medicine provided an alternative coursework-based pathway, which was favoured by 79% of surveyed trainees.9 Despite these limitations, we engaged all stakeholders including trainees, supervisors and assessors in all stages of this Delphi study. We produced pragmatic, actionable recommendations.
In conclusion, this study produced consensus-based recommendations to support trainees and supervisors in efficient completion of quality CICM Formal Projects. A formal implementation and evaluation study may ascertain the validity of the recommendations.
Ethical approval
This project (2023/HE000748) has been reviewed by the Research Ethics and Integrity committee and is deemed to be exempt from ethics review under the National Statement on Ethical Conduct in Human Research and relevant University of Queensland policy (PPL 4.20.07).
CRediT authorship contribution statement
Ariel Ho – Conceptualisation, Methodology, Formal analysis, Investigation, Writing – Original Draft, Review & Editing, Visualization.
Kerina Denny – Conceptualisation, Methodology, Investigation, Writing – Review & Editing.
Kevin Laupland – Conceptualisation, Methodology, Investigation, Writing – Review & Editing.
Mahesh Ramanan – Conceptualisation, Methodology, Investigation, Writing – Review & Editing.
Alexis Tabah – Conceptualisation, Methodology, Investigation, Writing – Review & Editing.
James McCullough – Methodology, Investigation, Writing – Review & Editing.
Jessica Schults – Methodology, Investigation, Writing – Review & Editing.
Sainath Raman – Conceptualisation, Methodology, Formal analysis, Investigation, Writing – Review & Editing, Supervision.
Conflict of interest
None.
Footnotes
Supplementary data to this article can be found online at https://doi.org/10.1016/j.ccrj.2024.05.002.
Contributor Information
Ariel Ho, Email: ariel.ty.ho@gmail.com.
members of the Queensland Critical Care Research Network:
Yogesh Apte, Antony Attokaran, Stuart Baker, Roland Bartholdy, Neeraj Bhadange, Jane Brailsford, Katrina Cook, Alexandre David, Jayesh Dhanani, Felicity Edwards, Hatem Elkady, Tess Evans, Jane Hutchinson, Sean Lannon, Andrea Marshall, Philippa McIlroy, Elissa Milford, Lynette Morrison, Lauren Murray, Alyssa Serratore, Vikram Shah, Kiran Shekar, Stacey Watts, and Kyle White
Appendix A. Supplementary data
The following is the Supplementary data to this article:
References
- 1.College of intensive care medicine of Australia and New Zealand. Formal Project Requirements; 2018. https://www.cicm.org.au/CICM_Media/CICMSite/Files/Training/T-9-Formal-Project-Requirements.pdf Available from: [Google Scholar]
- 2.Roos D.E., Foley E.F., Wonders T.J. Trainee research requirements of Australasian medical colleges: how does the Faculty of Radiation Oncology compare? J Med Imaging Radiat Oncol. 2019;63(6):862–868. doi: 10.1111/1754-9485.12930. [DOI] [PubMed] [Google Scholar]
- 3.Stehlik P., Noble C., Brandenburg C., Fawzy P., Narouz I., Henry D., et al. How do trainee doctors learn about research? Content analysis of Australian specialist colleges' intended research curricula. BMJ Open. 2020;10(3) doi: 10.1136/bmjopen-2019-034962. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Jayasundera T., Fisk M., McGhee C.N. Attitudes to research and research training among ophthalmologists and ophthalmology trainees in New Zealand. Clin Exp Ophthalmol. 2003;31(4):294–299. doi: 10.1046/j.1442-9071.2003.00678.x. [DOI] [PubMed] [Google Scholar]
- 5.Shaw M., Harris B., Bonner S. The research needs of an ICM trainee: the RAFT national survey results and initiatives to improve trainee research opportunities. J Intensive Care Soc. 2017;18(2):98–105. doi: 10.1177/1751143716689276. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Gill S., Levin A., Djurdjev O., Yoshida E.M. Obstacles to residents' conducting research and predictors of publication. Acad Med. 2001;76(5):477. doi: 10.1097/00001888-200105000-00021. [DOI] [PubMed] [Google Scholar]
- 7.Thiruthaneeswaran N., Turner S., Milross C., Gogna K. Promoting a research culture among junior radiation oncologists: outcomes from the introduction of the Australian and New Zealand research requirement in training. Clin Oncol (R Coll Radiol) 2014;26(3):162–173. doi: 10.1016/j.clon.2013.12.007. [DOI] [PubMed] [Google Scholar]
- 8.Foley E.F., Roos D.E. Feedback survey on the royal Australian and New Zealand college of radiologists faculty of radiation oncology trainee research requirement. J Med Imaging Radiat Oncol. 2020;64(2):279–286. doi: 10.1111/1754-9485.12994. [DOI] [PubMed] [Google Scholar]
- 9.Mitra B., Jones P., Fatovich D., Thom O., Trainee Research Committee ACfEM Trainee perspectives on usefulness of the trainee research requirement. Emergency Medicine Australasia. 2014;26(4):392–397. doi: 10.1111/1742-6723.12251. [DOI] [PubMed] [Google Scholar]
- 10.Shen E., Dhanani J., Milford E.M., Raileanu V., Laupland K.B. Publication outcomes among intensive care trainees. Anaesth Intensive Care. 2024;52(1):45–51. doi: 10.1177/0310057X231194079. [DOI] [PubMed] [Google Scholar]
- 11.Gattrell W.T., Logullo P., van Zuuren E.J., Price A., Hughes E.L., Blazey P., et al. ACCORD (ACcurate COnsensus Reporting Document): a reporting guideline for consensus methods in biomedicine developed via a modified Delphi. PLOS Medicine. 2024;21(1) doi: 10.1371/journal.pmed.1004326. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Fletcher A.J., Marchildon G.P. Using the delphi method for qualitative, participatory action research in health leadership. International Journal of Qualitative Methods. 2014;13(1):1–18. [Google Scholar]
- 13.Nasa P., Jain R., Juneja D. Delphi methodology in healthcare research: how to decide its appropriateness. World J Methodol. 2021;11(4):116–129. doi: 10.5662/wjm.v11.i4.116. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Hasson F., Keeney S., McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–1015. [PubMed] [Google Scholar]
- 15.Jünger S., Payne S., Brearley S., Ploenes V., Radbruch L. Consensus building in palliative care: a Europe-wide delphi study on common understandings and conceptual differences. J Pain Symptom Manage. 2012;44(2):192–205. doi: 10.1016/j.jpainsymman.2011.09.009. [DOI] [PubMed] [Google Scholar]
- 16.Jünger S., Payne S.A., Brine J., Radbruch L., Brearley S.G. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706. doi: 10.1177/0269216317690685. [DOI] [PubMed] [Google Scholar]
- 17.Diamond I.R., Grant R.C., Feldman B.M., Pencharz P.B., Ling S.C., Moore A.M., et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401–409. doi: 10.1016/j.jclinepi.2013.12.002. [DOI] [PubMed] [Google Scholar]
- 18.Braun V., Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. 2006;3(2):77–101. [Google Scholar]
- 19.Beng T.S., Xin C.A., Ying Y.K., Khuen L.P., Yee A., Zainuddin S.I., et al. Hope in palliative care: a thematic analysis. J Palliat Care. 2022;37(2):177–182. doi: 10.1177/0825859720948976. [DOI] [PubMed] [Google Scholar]
- 20.Ringnér M. What is principal component analysis? Nature Biotechnology. 2008;26(3):303–304. doi: 10.1038/nbt0308-303. [DOI] [PubMed] [Google Scholar]
- 21.Doran G.T. There's a S.M.A.R.T. Way to write management's goals and objectives. Management Review. 1981;70(11):35–36. [Google Scholar]
- 22.Alak A., Jerzak K.J., Quirt J.A., Lane S.J., Miller P.A., Haider S., et al. How to succeed in research during medical training: a qualitative study. Clin Invest Med. 2014;37(3) doi: 10.25011/cim.v37i3.21378. [DOI] [PubMed] [Google Scholar]
- 23.Siskind D., Parker S., Loi S., Looi J.C., Macfarlane M.D., Merry S., et al. How to survive in research: advice for the novice investigator. Australas Psychiatry. 2015;23(1):22–24. doi: 10.1177/1039856214562078. [DOI] [PubMed] [Google Scholar]
- 24.Weinstein R. How to write a manuscript for peer review. J Clin Apher. 2020;35(4):358–366. doi: 10.1002/jca.21797. [DOI] [PubMed] [Google Scholar]
- 25.Barletta J.F. Conducting a successful residency research project. Am J Pharm Educ. 2008;72(4):92. doi: 10.5688/aj720492. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Pati D., Lorusso L.N. How to write a systematic review of the literature. Herd. 2018;11(1):15–30. doi: 10.1177/1937586717747384. [DOI] [PubMed] [Google Scholar]
- 27.Altman D.G. The scandal of poor medical research. Bmj. 1994;308(6924):283–284. doi: 10.1136/bmj.308.6924.283. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Lee K.J., Moreno-Betancur M., Kasza J., Marschner I.C., Barnett A.G., Carlin J.B. Biostatistics: a fundamental discipline at the core of modern health data science. Med J Aust. 2019;211(10):444–446.e1. doi: 10.5694/mja2.50372. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Duplancic C., Crough T., Bell S.C. Multi-centre ethics and research governance review can impede non-interventional clinical research. Intern Med J. 2019;49(6):722–728. doi: 10.1111/imj.14158. [DOI] [PubMed] [Google Scholar]
- 30.Fistein E., Quilligan S. In the lion's den? Experiences of interaction with research ethics committees. J Med Ethics. 2012;38(4):224–227. doi: 10.1136/medethics-2011-100124. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Laupland K.B., Edwards F., Dhanani J. Determinants of research productivity during postgraduate medical education: a structured review. BMC Med Educ. 2021;21(1):567. doi: 10.1186/s12909-021-03010-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.The State of Queensland (Queensland Health). Intensive care [Internet] c2023 [cited 2024 Jan 31] [Available from: https://www.careers.health.qld.gov.au/medical-careers/resident-medical-officer-rmo-and-registrar-campaign/medical-specialty-training-programs/intensive-care.
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.