Abstract
Introduction:
There is currently no agreed definition of social prescribing. This is problematic for research, policy, and practice, as the use of common language is the crux of establishing a common understanding. Both conceptual and operational definitions of social prescribing are needed to address this gap. Therefore, the aim of the study that is outlined in this protocol is to establish internationally accepted conceptual and operational definitions of social prescribing.
Methodology:
A Delphi study will be conducted to develop internationally accepted conceptual and operational definitions of social prescribing with an international, multidisciplinary panel of experts. It is anticipated that this study will involve approximately 40 participants (range = 20–60 participants) and consist of 3–5 rounds. Consensus will be defined a priori as ≥80% agreement.
Discussion:
Not only will these definitions serve to unite the social prescribing community, but they will also inform research, policy, and practice. By laying the groundwork for the formation of a robust evidence base, this foundational work will support the advancement of social prescribing and help to unlock the full potential of the social prescribing movement.
Conclusion:
This important work will be foundational and timely, given the rapid spread of the social prescribing movement around the world.
Keywords: definition, Delphi, protocol, social prescribing
Introduction
There is growing interest in social prescribing around the world, with initiatives in Asia (e.g., China, Japan, Singapore, South Korea), Europe (e.g., Denmark, Finland, Germany, Ireland, the Netherlands, Portugal, Spain, Sweden, the United Kingdom (UK)), North America (e.g., Canada, the United States (US)), and Oceania (e.g., Australia, New Zealand) [1]. While the term originates from the UK and dates back to almost a century ago [1,2], it is mostly within the past decade that social prescribing has entered into the lexicons of health systems across the globe [1]. During this time, there have been remarkable developments in the social prescribing movement. In 2015, the Social Prescribing Network was established [3], which was followed by the launch of the National Academy for Social Prescribing in 2019 [4]. In 2021, as a mark of the unprecedented momentum around the social prescribing movement at the global level, the National Academy for Social Prescribing partnered with the World Health Organization, the World Health Innovation Summit, and the United Global SDG Index Institute Foundation to establish the Global Social Prescribing Alliance, which is a global working group that is dedicated to the advancement of social prescribing through promotion, collaboration, and innovation [5]. Alongside these exciting developments, a global network of student champions has emerged to build the social prescribing student movement, with student groups in Australia, Canada, Japan, Portugal, Singapore, the UK, and the US, which collectively developed the Social Prescribing International Student Movement Framework in 2021 [2] and subsequently launched the Global Social Prescribing Student Council in 2022. Finally, as a way to foster the implementation of social prescribing, the World Health Organization recently released a social prescribing toolkit [6] and online training module [7], both of which highlight the importance of social prescribing as a vital tool in achieving integrated care. This sentiment is echoed by social prescribing experts around the world, who emphasize that social prescribing is a holistic approach to health care that fosters collaboration across health, social, and community sectors to provide integrated care [1].
While it is impressive to witness the explosion of interest in social prescribing across the globe, the fact of the matter is that there is currently no agreed definition of social prescribing [1,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27]. The reality is that the concept is considered nebulous and open to different interpretations [8]. There is little consistency in how social prescribing is defined across studies, with many studies not even providing a definition, meaning efforts to generate robust evidence on social prescribing are significantly hindered by the lack of consensus around the definition [9,10]. This is problematic for research, policy, and practice, as the use of common language is the crux of establishing a common understanding, hence the need to establish an agreed definition of social prescribing [9,10,11,12].
Both conceptual and operational definitions of social prescribing are needed to address this gap. Richards [28] emphasizes the importance of having both types of definitions – a conceptual definition outlines what a concept means but it does not explain how to measure it, whereas an operational definition outlines how to measure a concept but it does not explain what it means. Therefore, the aim of the study that is outlined in this protocol is to establish internationally accepted conceptual and operational definitions of social prescribing.
Methodology
Study Design
A Delphi study will be conducted to develop internationally accepted conceptual and operational definitions of social prescribing with an international, multidisciplinary panel of experts. The Delphi technique is a method of gaining consensus on a particular topic through multiple rounds of questioning of experts in the field, who remain anonymous and receive feedback between each round [29,30]. This consensus method is widely used by health science researchers to achieve expert consensus [31], particularly to establish agreed definitions [32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68]. Thus, the Delphi technique is an appropriate method to achieve the aims of this study.
Study Overview
The research team will conduct an online Delphi study. The Delphi surveys will be administered through Welphi (www.welphi.com), which is an online survey platform that is specifically designed for Delphi studies. Given the importance of the quality of the online survey platform to the success of this study, the research team carefully reviewed and tested several different options prior to selection.
At the start of each round, participants will receive an email with a link to the online survey. Important information about conceptual and operational definitions will be provided to ensure that there is common understanding of these terms. Each round will take approximately one month to complete. Participants will have two weeks to complete each round. Reminder emails will be sent out one week, two days, and one day before the closure of each round, as well as the day of the closure of each round. After two weeks, an email will be sent out to non-responders to give them an additional three days to complete the survey – if they do not respond by this time, they will be removed from the study. The research team will have the remaining two weeks of each round to complete data analysis, survey development, and pilot testing.
Participants
There is a lack of standard guidelines and agreement in the literature as to what constitutes an expert for Delphi studies [29,30]. According to Niederberger and Spranger [31], the definition of an expert is either based on expertise or lived experience. This is reflected in the conceptualization of experts in this study, as experts will be defined according to the following criteria: (1) Person involved with the Social Prescribing Network; or (2) Person involved with the Social Prescribing Youth Network; or (3) Person involved with the Global Social Prescribing Alliance; or (4) Person involved with the National Academy for Social Prescribing; or (5) Person involved with the Canadian Institute for Social Prescribing; or (6) Student involved with any national social prescribing student group; or (7) Author of academic or grey literature on social prescribing, even if not labelled as “social prescribing”; or (8) Researcher involved in social prescribing, even if not labelled as “social prescribing”; or (9) Health care provider involved in social prescribing, even if not labelled as “social prescribing”; or (10) Link worker involved in social prescribing, even if not labelled as “link worker” or “social prescribing”; or (11) Patient involved in social prescribing, even if not labelled as “social prescribing”; or (12) Health care administrator or manager tasked with overseeing the use of social prescribing, even if not labelled as “social prescribing”. It is anticipated that this criteria will result in a heterogeneous group of experts for this study, which will increase the validity of the findings [29]. Furthermore, since this study will be in English, only those who are able to speak, read, and write English will be eligible to participate in this study. Determination of eligibility will be at the discretion of the interested party.
There is also a lack of standard guidelines and agreement in the literature as to the appropriate size of the expert panel for Delphi studies [29,30,31]. However, the average number of experts that health science researchers include in Delphi studies is 40 [31], which aligns with the fact that most Delphi studies that have been conducted by health science researchers to establish agreed definitions have had between 20–60 people on the expert panel [34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54]. Therefore, it is anticipated that approximately 40 participants (range = 20–60 participants) will be involved in this study.
Recruitment
During targeted recruitment, the research team will develop a comprehensive list with the names and contact information of experts (e.g., authors of academic or grey literature on social prescribing) and send out invitations to this group via email. This information will be obtained from public sources (e.g., corresponding author details). During open recruitment, the research team will send out invitations to experts via relevant communication channels (e.g., Social Prescribing Network newsletter) and advertise invitations to experts via relevant social media platforms (e.g., Twitter). Consistent with snowball sampling, experts will be asked to disseminate the call to relevant contacts. Throughout the recruitment process, the acquisition of an international, multidisciplinary panel of experts will be prioritized. The research team will use a matrix to ensure that there is diversity amongst experts in terms of country, job title, expertise in social prescribing, and years of experience with social prescribing. The registration survey will be closely monitored, and invitations will be sent out to experts from groups that are underrepresented.
Experts will receive an email with a link to the registration survey, which will be administered through Qualtrics (www.qualtrics.com). The first page of the survey will be the Letter of Information. Experts will be informed that by proceeding to the next page, they are consenting to participate in the study. Sociodemographic data, including name, email address, country, job title, organization, expertise in social prescribing, and years of experience with social prescribing, will be collected through the registration survey. Once enough experts have registered for the study, the first round will begin. Participants will receive an email one week prior to the launch of the first round to notify them and provide them with necessary information.
Consensus
While there is a lack of standard guidelines and agreement in the literature as to what constitutes consensus for Delphi studies [29,30,31,69], most Delphi studies define consensus as a certain percentage of participants being in agreement, with the level of agreement ranging from 51–100% [29]. Consensus will be defined a priori as ≥80% agreement for this study. This threshold was chosen as an appropriate cutoff given that most health science researchers that have conducted Delphi studies to establish agreed definitions have defined consensus as either ≥70%, ≥75%, or ≥80% agreement [33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,59,60,61,62,63,64,65,66,67], with ≥80% being the most stringent level of agreement.
Rounds
While Delphi studies can consist of any number of rounds, they commonly involve 3–5 rounds [30]. In fact, most Delphi studies that have been conducted by health science researchers to establish agreed definitions have consisted of 3–5 rounds [32,33,34,35,36,40,41,42,43,44,45,46,47,48,49,50,51,52,53,63,64,65,66,67]. Based on this trend, and the format of each round in this study, it is anticipated that there will be 3–5 rounds.
Data Collection
Round 1
For the first round of this study, participants will complete an open-ended survey to gather information through open-ended questions, which aligns with the original Delphi technique [29,30]. The first question will ask participants to list key elements that are essential to the conceptual definition of social prescribing. The second question will build on this by asking participants to provide corresponding statements that operationalize each of the key elements.
Round 2
For the second round of this study, participants will rate items from the first round through a structured survey, which will consist of two sections. In the first section, findings from the first question will be presented, and participants will be asked to rate their agreement with these items for inclusion in the conceptual definition of social prescribing. In the second section, findings from the second question will be presented, and participants will be asked to rate their agreement with these items for inclusion in the operational definition of social prescribing. Ratings will be given on a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree). The Likert scale was selected because it is commonly used to collect rated responses [29], and more specifically, the 5-point Likert scale was chosen because it is one of the most common scales that health science researchers use in Delphi studies [31]. A free-text box will also be provided to add comments, and participants will be encouraged to do so.
Round 3 and Beyond
For the third round and beyond, participants will complete a structured survey to rate their agreement with the conceptual and operational definitions of social prescribing. Consistent with the second round, ratings will be given on a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree). A free-text box will also be provided to add comments, and participants will be encouraged to do so. If consensus is not reached for both definitions by the end of the fifth round, a meeting will be held with participants via teleconferencing software to achieve consensus through discussion. While meetings are a common modification to the original Delphi technique, it is important to acknowledge that this gathering would violate one of the fundamental characteristics of this consensus method, in that participants would no longer be anonymous to the group [29], so this meeting will only take place if it is necessary. Once consensus is reached for both definitions in the third round or beyond, the study will be terminated, which will signify the successful development of internationally accepted conceptual and operational definitions of social prescribing.
Data Analysis
The data that is collected from the registration survey will be analyzed with the latest version of Microsoft Excel (www.microsoft.com). With respect to the Delphi surveys, the online survey platform will generate the pooled results for each round. Quantitative data will be expressed in percentages as statistical group response. Qualitative data will be presented for each survey item, and it will be analyzed through qualitative content analysis, which is “an approach to text analysis which combines strict rulebound interpretive category assignments with quantifications of category occurrences” [70 p. 209]. This will be done with the latest version of QCAmap (www.qcamap.org), which is an online software program that is specifically designed for qualitative content analysis. QCAmap interactively guides users through the steps of qualitative content analysis [70], which will ensure that each step is carried out in this study.
Round 1
The research team will conduct qualitative content analysis to analyze participants’ responses. The findings will be used to create a structured survey for the next round. At the start of the next round, participants will receive feedback from this round, which will consist of a summary of participants’ responses. All responses will be anonymized.
Round 2
Since consensus will be defined a priori as ≥80% agreement, items where ≥80% of participants rate their agreement to include them as 4 (agree) or 5 (strongly agree) on the Likert scale will be included in the preliminary conceptual and operational definitions of social prescribing. The research team will conduct qualitative content analysis to analyze participants’ responses. Based on the findings, preliminary conceptual and operational definitions of social prescribing will be developed. At the start of the next round, participants will receive feedback from this round. Quantitative feedback will consist of the percentage of agreement and the individual response of each participant in relation to the group response, and qualitative feedback will consist of a summary of participants’ responses. All responses will be anonymized.
Round 3 and Beyond
For consensus to be reached, ≥80% of participants will need to rate their agreement with the conceptual and operational definitions of social prescribing as 4 (agree) or 5 (strongly agree) on the Likert scale. If this threshold is not met for both definitions, another round will be completed. The research team will conduct qualitative content analysis to analyze participants’ responses. Based on the findings, modifications will be made to the conceptual and operational definitions of social prescribing. At the start of each round, participants will receive feedback from the previous round. Quantitative feedback will consist of the percentage of agreement and the individual response of each participant in relation to the group response, and qualitative feedback will consist of a summary of participants’ responses. All responses will be anonymized.
Quality and Transparency
This study will be conducted and reported in accordance with the Conducting and REporting DElphi Studies (CREDES) guideline [71]. This protocol has been registered on Open Science Framework (osf.io/pfyqg) and published in this journal to promote transparency.
Ethics
This study has been reviewed for ethical compliance by the Queen’s University Health Sciences and Affiliated Teaching Hospitals Research Ethics Board (NURS-540-22).
Discussion
The Delphi study that is outlined in this protocol will be conducted to develop internationally accepted conceptual and operational definitions of social prescribing with an international, multidisciplinary panel of experts. This protocol has been published in this journal to promote transparency. It adds to the growing number of Delphi study protocols.
There are several strengths of the Delphi technique, such as its flexible approach to gathering the opinions of experts on a particular topic [29,69]. The most significant strengths of this consensus method are the three characteristics that make it unique, namely participant anonymity, multiple rounds of questioning, and provision of feedback between each round, which reduce bias in the process of gaining consensus [29,31,69]. Another important strength is that the online format is well suited to gathering the opinions of experts who are geographically dispersed [29,30], which is particularly relevant given the challenges imposed by the COVID-19 pandemic and the desire to acquire an international panel of experts to be able to develop internationally accepted definitions. There are also some limitations of the Delphi technique, such as its reputation for being complex and time consuming [30,31,69]. The most significant limitation of this consensus method is the lack of standard guidelines and agreement in the literature [29,30,31,69]. This issue was mitigated in this protocol by making key decisions based on the literature, and it will be addressed in this study, as this research will be conducted and reported in accordance with the CREDES guideline [71]. Another important limitation is the risk of panel attrition [29,30,69]. This issue was mitigated in this protocol by taking this risk into account when determining the size of the expert panel, and it will be addressed in this study, as this research will include various retention strategies (e.g., sending out reminder emails to non-responders, providing quick turnaround times between rounds of questioning).
It is anticipated that this research will have extensive and enduring impacts on the social prescribing community. Not only will these internationally accepted conceptual and operational definitions of social prescribing serve to unite the social prescribing community, but they will also inform research, policy, and practice. By laying the groundwork for the formation of a robust evidence base, this foundational work will support the advancement of social prescribing and help to unlock the full potential of the social prescribing movement, not to mention that it will answer the call that was made by the Social Prescribing Network in 2016 for an agreed definition of social prescribing [16].
Conclusion
This paper outlines the protocol for a Delphi study that will be conducted to develop internationally accepted conceptual and operational definitions of social prescribing with an international, multidisciplinary panel of experts. This important work will be foundational and timely, given the rapid spread of the social prescribing movement around the world.
Reviewers
Chris Drinkwater, Emeritus Professor of Primary Care Development, Northumbria University, UK.
David Price, Professor, Department of Family Medicine, McMaster University, Canada.
Competing Interests
The authors have no competing interests to declare.
References
- 1.Morse DF, Sandhu S, Mulligan K, Tierney S, Polley M, Chiva Giurca B, et al. Global developments in social prescribing. BMJ Glob Health. 2022. May; 7: e008524. DOI: 10.1136/bmjgh-2022-008524 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Global Social Prescribing Alliance. Social prescribing: International student movement framework [Internet]. Global Social Prescribing Alliance; 2021. Available from: https://www.gspalliance.com/student-movement-framework.
- 3.Social Prescribing Network. About [Internet]. Social Prescribing Network; 2022. [cited 2022 Apr 22]. Available from: https://www.socialprescribingnetwork.com/about. [Google Scholar]
- 4.National Academy for Social Prescribing. About us [Internet]. National Academy for Social Prescribing; 2021. [cited 2022 Apr 22]. Available from: https://socialprescribingacademy.org.uk/about-us/. [Google Scholar]
- 5.Global Social Prescribing Alliance. Global Social Prescribing Alliance: International playbook [Internet]. Global Social Prescribing Alliance; 2021. Available from: https://www.gspalliance.com/gspa-playbook. [Google Scholar]
- 6.World Health Organization. A toolkit on how to implement social prescribing [Internet]. World Health Organization; 2022. Available from: https://www.who.int/publications/i/item/9789290619765. [Google Scholar]
- 7.World Health Organization. Social prescribing [Internet]. World Health Organization. [cited 2022 May 30]. Available from: https://openwho.org/courses/social-prescribing-WPRO. [Google Scholar]
- 8.Malby B, Boyle D, Wildman J, Smith S, Omar S. The asset based health inquiry: How best to develop social prescribing [Internet]. London South Bank University; 2019. Available from: https://www.lsbu.ac.uk/__data/assets/pdf_file/0018/251190/lsbu_asset-based_health_inquiry.pdf. [Google Scholar]
- 9.Costa A, Sousa CJ, Seabra PRC, Virgolino A, Santos O, Lopes J, et al. Effectiveness of social prescribing programs in the primary health-care context: A systematic literature review. Sustainability. 2021. Mar 3; 13(5): 2731. DOI: 10.3390/su13052731 [DOI] [Google Scholar]
- 10.Zurynski Y, Vedovi A, Smith K-l. Social prescribing: A rapid literature review to inform primary care policy in Australia [Internet]. NHMRC Partnership Centre for Health System Sustainability Australian Institute of Health Innovation, Macquarie University; 2020. Available from: https://chf.org.au/sites/default/files/sprapidreview_3-2-20_final.pdf. [Google Scholar]
- 11.Rempel ES, Wilson EN, Durrant H, Barnett J. Preparing the prescription: A review of the aim and measurement of social referral programmes. BMJ Open. 2017. Oct; 7(10): e017734. DOI: 10.1136/bmjopen-2017-017734 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Islam MM. Social prescribing—An effort to apply a common knowledge: Impelling forces and challenges. Front Public Health. 2020. Nov 27; 8: 515469. DOI: 10.3389/fpubh.2020.515469 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Bickerdike L, Booth A, Wilson PM, Farley K, Wright K. Social prescribing: Less rhetoric and more reality. A systematic review of the evidence. BMJ Open. 2017. Apr; 7(4): e013384. DOI: 10.1136/bmjopen-2016-013384 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Booth A, Wilson P, Bickerdike L. Evidence to inform the commissioning of social prescribing [Internet]. Centre for Reviews and Dissemination, University of York; 2015. Available from: https://www.york.ac.uk/media/crd/Ev%20briefing_social_prescribing.pdf. [Google Scholar]
- 15.Carnes D, Sohanpal R, Frostick C, Hull S, Mathur R, Netuveli G, et al. The impact of a social prescribing service on patients in primary care: A mixed methods evaluation. BMC Health Serv Res. 2017. Dec; 17(835): 1–9. DOI: 10.1186/s12913-017-2778-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Dixon M, Polley M. Report of the annual Social Prescribing Network Conference [Internet]. Social Prescribing Network; 2016. Available from: https://42b7de07-529d-4774-b3e1-225090d531bd.filesusr.com/ugd/14f499_9ba1233600eb454ab836b1c6424feed3.pdf. [Google Scholar]
- 17.Elston J, Gradinger F, Asthana S, Lilley-Woolnough C, Wroe S, Harman H, et al. Does a social prescribing ‘holistic’ link-worker for older people with complex, multimorbidity improve well-being and frailty and reduce health and social care use and costs? A 12-month before-and-after evaluation. Prim Health Care Res Dev. 2019; 20(e135): 1–10. DOI: 10.1017/S1463423619000598 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Gibbons A, Howarth M, Lythgoe A. Social prescribing in Greater Manchester [Internet]. University of Salford; 2019. Available from: https://usir.salford.ac.uk/id/eprint/52968/1/SocialPrescribingGMFullReport2019.pdf. [Google Scholar]
- 19.Kimberlee R. What is social prescribing? Adv Soc Sci Res J. 2015. Jan 25; 2(1): 102–110. DOI: 10.14738/assrj.21.808 [DOI] [Google Scholar]
- 20.Kinsella S. Social prescribing: A review of the evidence [Internet]; 2015. Available from: https://www.scie-socialcareonline.org.uk/social-prescribing-a-review-of-the-evidence/r/a11G000000CTdhKIAT.
- 21.Lejac B. A desk review of social prescribing: From origins to opportunities [Internet]. Support in Mind Scotland; 2021. Available from: https://www.rsecovidcommission.org.uk/a-desk-review-of-social-prescribing-from-origins-to-opportunities/. [Google Scholar]
- 22.Moffatt S, Steer M, Lawson S, Penn L, O’Brien N. Link worker social prescribing to improve health and well-being for people with long-term conditions: Qualitative study of service user perceptions. BMJ Open. 2017. Jul; 7(7): e015203. DOI: 10.1136/bmjopen-2016-015203 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Pescheny J, Pappas Y, Randhawa G. Facilitators and barriers of implementing and delivering social prescribing services: A systematic review. BMC Health Serv Res. 2018. Dec; 18(86). DOI: 10.1186/s12913-018-2893-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Pescheny J, Gunn LH, Randhawa G, Pappas Y. The impact of the Luton social prescribing programme on energy expenditure: A quantitative before-and-after study. BMJ Open. 2019. Jun; 9(6): e026862. DOI: 10.1136/bmjopen-2018-026862 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Pilkington K, Loef M, Polley M. Searching for real-world effectiveness of health care innovations: Scoping study of social prescribing for diabetes. J Med Internet Res. 2017. Feb 2; 19(2): e20. DOI: 10.2196/jmir.6431 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Polley M, Fleming J, Anfilogoff T, Carpenter A. Making sense of social prescribing [Internet]. University of Westminster; 2017. Available from: https://waystowellness.org.uk/site/assets/files/1317/making-sense-of-social-prescribing_2017_2.pdf. [Google Scholar]
- 27.Westlake D, Elston J, Gude A, Gradinger F, Husk K, Asthana S. Impact of COVID-19 on social prescribing across an Integrated Care System: A Researcher in Residence study. Health Social Care Comm. 2022. Mar 30; 1–9. DOI: 10.1111/hsc.13802 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Richards W. The zen of empirical research [Internet]. Vancouver, BC: Empirical Press; 1998. Part 1, Scientific research. Available from: https://www.sfu.ca/personal/archives/richards/Zen/Pages/zentoc.html. [Google Scholar]
- 29.Chalmers J, Armour M. The Delphi technique. In: Liamputtong, P (ed.), Handbook of Research Methods in Health Social Sciences [Internet]. Singapore: Springer Singapore; 2019. [cited 2021 Aug 30]. p. 715–35. Available from: http://link.springer.com/10.1007/978-981-10-5251-4_99. DOI: 10.1007/978-981-10-5251-4_99 [DOI] [Google Scholar]
- 30.Puig A, Adams C. Delphi technique. In: The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation [Internet]. 2455 Teller Road, Thousand Oaks, California 91320: SAGE Publications, Inc.; 2018. [cited 2021 Aug 30]. Available from: https://methods.sagepub.com/reference/the-sage-encyclopedia-of-educational-research-measurement-and-evaluation. [Google Scholar]
- 31.Niederberger M, Spranger J. Delphi technique in health sciences: A map. Front Public Health. 2020. Sep 22; 8(457). DOI: 10.3389/fpubh.2020.00457 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Metzelthin SF, Rostgaard T, Parsons M, Burton E. Development of an internationally accepted definition of reablement: A Delphi study. Ageing and Society. 2020. Sep 4; 1–16. DOI: 10.1017/S0144686X20000999 [DOI] [Google Scholar]
- 33.Tait AR, Geisser ME. Development of a consensus operational definition of child assent for research. BMC Med Ethics. 2017. Dec; 18(41). DOI: 10.1186/s12910-017-0199-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Allen J, Brenner M, Hauer J, Molloy E, McDonald D. Severe Neurological Impairment: A Delphi consensus-based definition. European Journal of Paediatric Neurology. 2020. Nov; 29: 81–6. DOI: 10.1016/j.ejpn.2020.09.001 [DOI] [PubMed] [Google Scholar]
- 35.Avdagic E, Wade C, McDonald M, McCormack D, Dakin P, Macvean M, et al. Resilience in young children: A Delphi study to reach consensus on definitions, measurement and interventions to build resilience. Early Child Development and Care. 2020. Oct 2; 190(13): 2066–77. DOI: 10.1080/03004430.2018.1556211 [DOI] [Google Scholar]
- 36.Campbell KA, Wood JN, Lindberg DM, Berger RP. A standardized definition of near-fatal child maltreatment: Results of a multidisciplinary Delphi process. Child Abuse & Neglect. 2021. Feb; 112: 104893. DOI: 10.1016/j.chiabu.2020.104893 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Beune IM, Damhuis SE, Ganzevoort W, Hutchinson JC, Khong TY, Mooney EE, et al. Consensus definition of fetal growth restriction in intrauterine fetal death: A Delphi procedure. Archives of Pathology & Laboratory Medicine. 2021. Apr 1; 145(4): 428–36. DOI: 10.5858/arpa.2020-0027-OA [DOI] [PubMed] [Google Scholar]
- 38.Ben-Chetrit E, Gattorno M, Gul A, Kastner DL, Lachmann HJ, Touitou I, et al. Consensus proposal for taxonomy and definition of the autoinflammatory diseases (AIDs): A Delphi study. Ann Rheum Dis. 2018. Nov; 77(11): 1558–65. DOI: 10.1136/annrheumdis-2017-212515 [DOI] [PubMed] [Google Scholar]
- 39.Guseva Canu I, Marca SC, Dell’Oro F, Balázs Á, Bergamaschi E, Besse C, et al. Harmonized definition of occupational burnout: A systematic review, semantic analysis, and Delphi consensus in 29 countries. Scand J Work Environ Health. 2021. Mar 1; 47(2): 95–107. DOI: 10.5271/sjweh.3935 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Dekker J, Amitami M, Berman AH, Brown H, Cleal B, Figueiras MJ, et al. Definition and characteristics of behavioral medicine, and main tasks and goals of the International Society of Behavioral Medicine—An international Delphi study. IntJ Behav Med. 2021. Jun; 28(3): 268–76. DOI: 10.1007/s12529-020-09928-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Khalil A, Beune I, Hecher K, Wynia K, Ganzevoort W, Reed K, et al. Consensus definition and essential reporting parameters of selective fetal growth restriction in twin pregnancy: A Delphi procedure. Ultrasound Obstet Gynecol. 2019. Jan; 53(1): 47–54. DOI: 10.1002/uog.19013 [DOI] [PubMed] [Google Scholar]
- 42.Luquin MR, Kulisevsky J, Martinez-Martin P, Mir P, Tolosa ES. Consensus on the definition of advanced Parkinson’s disease: A neurologists-based Delphi study (CEPA study). Parkinson’s Disease. 2017; 2017: 1–8. DOI: 10.1155/2017/4047392 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Monnier AA, Eisenstein BI, Hulscher ME, Gyssens IC, the DRIVE-AB WP1 group, Adriaenssens N, et al. Towards a global definition of responsible antibiotic use: Results of an international multidisciplinary consensus procedure. Journal of Antimicrobial Chemotherapy. 2018. Jun 1; 73(6): vi3–16. DOI: 10.1093/jac/dky114 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Schols AMR, Dinant GJ, Hopstaken R, Price CP, Kusters R, Cals JWL. International definition of a point-of-care test in family practice: A modified e-Delphi procedure. Family Practice. 2018. Jul 23; 35(4): 475–80. DOI: 10.1093/fampra/cmx134 [DOI] [PubMed] [Google Scholar]
- 45.Van den Steene H, van West D, Glazemakers I. Towards a definition of multiple and complex needs in children and youth: Delphi study in Flanders and international survey. Scandinavian Journal of Child and Adolescent Psychiatry and Psychology. 2019; 7: 60–7. DOI: 10.21307/sjcapp-2019-009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.van der Horst N, Backx F, Goedhart EA, Huisstede BM. Return to play after hamstring injuries in football (soccer): A worldwide Delphi procedure regarding definition, medical criteria and decision-making. Br J Sports Med. 2017. Nov; 51(22): 1583–91. DOI: 10.1136/bjsports-2016-097206 [DOI] [PubMed] [Google Scholar]
- 47.Wong HS, Curry NS, Davenport RA, Yu L, Stanworth SJ. A Delphi study to establish consensus on a definition of major bleeding in adult trauma. Transfusion. 2020. Dec; 60(12): 3028–38. DOI: 10.1111/trf.16055 [DOI] [PubMed] [Google Scholar]
- 48.Zanker J, Scott D, Reijnierse EM, Brennan-Olsen SL, Daly RM, Girgis CM, et al. Establishing an operational definition of sarcopenia in Australia and New Zealand: Delphi method based consensus statement. J Nutr Health Aging. 2019; 23(1): 105–10. DOI: 10.1007/s12603-018-1113-6 [DOI] [PubMed] [Google Scholar]
- 49.Adams B, Sereda M, Casey A, Byrom P, Stockdale D, Hoare DJ. A Delphi survey to determine a definition and description of hyperacusis by clinician consensus. International Journal of Audiology. 2020. Dec 11; 1–7. DOI: 10.1080/14992027.2020.1855370 [DOI] [PubMed] [Google Scholar]
- 50.Keller S, Salinas A, Williams D, McGoldrick M, Gorski L, Alexander M, et al. Reaching consensus on a home infusion central line-associated bloodstream infection surveillance definition via a modified Delphi approach. American Journal of Infection Control. 2020. Sep; 48(9): 993–1000. DOI: 10.1016/j.ajic.2019.12.015 [DOI] [PubMed] [Google Scholar]
- 51.Lakhani BK, Giannouladis K, Leighton P, King AJ. Seeking a practical definition of stable glaucoma: A Delphi consensus survey of UK glaucoma consultants. Eye. 2020. Feb; 34(2): 335–43. DOI: 10.1038/s41433-019-0540-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.van Helsdingen CP van, Jongen AC, Jonge WJ de, Bouvy ND, Derikx JP. Consensus on the definition of colorectal anastomotic leakage: A modified Delphi study. WJG. 2020. Jun 21; 26(23): 3293–303. DOI: 10.3748/wjg.v26.i23.3293 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Lagacé-Legendre C, Boucher V, Robert S, Tardif PA, Ouellet MC, de Guise E, et al. Persistent postconcussion symptoms: An expert consensus-based definition using the Delphi method. Journal of Head Trauma Rehabilitation. 2021. Mar; 36(2): 96–102. DOI: 10.1097/HTR.0000000000000613 [DOI] [PubMed] [Google Scholar]
- 54.Sudore RL, Lum HD, You JJ, Hanson LC, Meier DE, Pantilat SZ, et al. Defining advance care planning for adults: A consensus definition from a multidisciplinary Delphi panel. Journal of Pain and Symptom Management. 2017. May; 53(5): 821–832.e1. DOI: 10.1016/j.jpainsymman.2016.12.331 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Bell DR, Snedden T, Biese K, Nelson E, Watson A, McGuine T, et al. Consensus definition of sport specialization in youth athletes using a Delphi approach. J Athl Train; 2021. Mar 31. DOI: 10.4085/1062-6050-0725.20 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Kaur N, Pluye P. Delineating and operationalizing the definition of patient-oriented research: A modified e-Delphi study. Journal of Patient-Centered Research and Reviews. 2019. Jan 28; 6(1): 7–16. DOI: 10.17294/2330-0698.1655 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Julià-Torras J, Cuervo-Pinna MÁ, Cabezón-Gutiérrez L, Lara PC, Prats M, Margarit C, et al. Definition of episodic dyspnea in cancer patients: A Delphi-based consensus among Spanish experts: The INSPIRA study. Journal of Palliative Medicine. 2019. Apr; 22(4): 413–9. DOI: 10.1089/jpm.2018.0273 [DOI] [PubMed] [Google Scholar]
- 58.Feo R, Conroy T, Jangland E, Muntlin Athlin Å, Brovall M, Parr J, et al. Towards a standardised definition for fundamental care: A modified Delphi study. J Clin Nurs. 2018. Jun; 27(11–12): 2285–99. DOI: 10.1111/jocn.14247 [DOI] [PubMed] [Google Scholar]
- 59.D’Souza N, de Neree tot Babberich MPM, d’Hoore A, Tiret E, Xynos E, Beets-Tan RGH, et al. Definition of the rectum: An international, expert-based Delphi consensus. Annals of Surgery. 2019. Dec; 270(6): 955–9. DOI: 10.1097/SLA.0000000000003251 [DOI] [PubMed] [Google Scholar]
- 60.Sacco S, Braschinsky M, Ducros A, Lampl C, Little P, van den Brink AM, et al. European headache federation consensus on the definition of resistant and refractory migraine: Developed with the endorsement of the European Migraine & Headache Alliance (EMHA). J Headache Pain. 2020. Dec; 21(76). DOI: 10.1186/s10194-020-01130-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 61.Schaap T, Bloemenkamp K, Deneux-Tharaux C, Knight M, Langhoff-Roos J, Sullivan E, et al. Defining definitions: A Delphi study to develop a core outcome set for conditions of severe maternal morbidity. BJOG: Int J Obstet Gy. 2019. Feb; 126(3): 394–401. DOI: 10.1111/1471-0528.14833 [DOI] [PubMed] [Google Scholar]
- 62.Lawrence M, Asaba E, Duncan E, Elf M, Eriksson G, Faulkner J, et al. Stroke secondary prevention, a non-surgical and non-pharmacological consensus definition: Results of a Delphi study. BMC Res Notes. 2019. Dec; 12(823). DOI: 10.1186/s13104-019-4857-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 63.Casellas-Grau A, Jordán de Luna C, Maté J, Ochoa C, Sumalla EC, Gil F. Developing a consensus definition of psychosocial complexity in cancer patients using Delphi methods. Pall Supp Care. 2021. Feb; 19(1): 17–27. DOI: 10.1017/S1478951520000784 [DOI] [PubMed] [Google Scholar]
- 64.Katz RV, Dearing BA, Ryan JM, Ryan LK, Zubi MK, Sokhal GK. Development of a tongue-tie case definition in newborns using a Delphi survey: The NYU–tongue-tie case definition. Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology. 2020. Jan; 129(1): 21–6. DOI: 10.1016/j.oooo.2019.01.012 [DOI] [PubMed] [Google Scholar]
- 65.Smith ME, Hardman JC, Mehta N, Jones GH, Mandavia R, Anderson C, et al. Acute otitis externa: Consensus definition, diagnostic criteria and core outcome set development. PLoS One. 2021; 16(5): e0251395. DOI: 10.1371/journal.pone.0251395 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 66.Spinelli A, Anania G, Arezzo A, Berti S, Bianco F, Bianchi PP, et al. Italian multi-society modified Delphi consensus on the definition and management of anastomotic leakage in colorectal surgery. Updates Surg. 2020. Sep; 72(3): 781–92. DOI: 10.1007/s13304-020-00837-z [DOI] [PubMed] [Google Scholar]
- 67.Yousif A, Forget A, Beauchesne MF, Lemière C, Dugré N, Fénélon-Dimanche R, et al. Development of an operational definition of treatment escalation in adults with asthma adapted to healthcare administrative databases: A Delphi study. Respiratory Medicine. 2021. Aug; 185: 106510. DOI: 10.1016/j.rmed.2021.106510 [DOI] [PubMed] [Google Scholar]
- 68.Tack J, Boardman H, Layer P, Schiefke I, Jayne D, Scarpignato C, et al. An expert consensus definition of failure of a treatment to provide adequate relief (F-PAR) for chronic constipation – An international Delphi survey. Aliment Pharmacol Ther. 2017. Feb; 45(3): 434–42. DOI: 10.1111/apt.13874 [DOI] [PubMed] [Google Scholar]
- 69.Barrett D, Heale R. What are Delphi studies? Evid Based Nurs. 2020. Jul; 23(3): 68–9. DOI: 10.1136/ebnurs-2020-103303 [DOI] [PubMed] [Google Scholar]
- 70.Mayring P. Qualitative content analysis: A step-by-step guide. SAGE; 2021. Available from: https://uk.sagepub.com/en-gb/eur/qualitative-content-analysis/book269922. [Google Scholar]
- 71.Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017. Sep; 31(8): 684–706. DOI: 10.1177/0269216317690685 [DOI] [PubMed] [Google Scholar]