Abstract
Background
The rapidly increasing number of older drivers is accentuating the challenges in concurrently identifying older drivers posing an unacceptable risk if they continue to drive, while not discriminating against those capable of safely driving. Attendees of an invitational meeting about the assessment of older drivers were asked to participate in a modified Delphi process designed to develop consensus statements on the assessment of older drivers.
Methods
Forty-one non-student symposium attendees were invited to participate in two rounds of a survey, in which they were asked to indicate their level of agreement (or disagreement) on a five-point Likert scale to a series of statements about the assessment of older drivers. Consensus was defined as 80% + of respondents either agreeing or disagreeing with a statement.
Results
More than one-half (n = 23) completed the first round of the survey and 12 participated in the second. There was consensus on the need for a modifiable, fair, rational, and widely accessible multi-step approach to the assessment of older drivers. This would require the engagement and support of physicians and other health-care practitioners in identifying and reporting medically at-risk drivers of any age. At a societal level, alternatives to driving a personal motor vehicle should be developed.
Conclusions
An on-going dialogue about this complex issue is required. Decisions should be based on explicitly stated principles and informed by the best available evidence.
Keywords: older driver, driving ability, assessment, consensus statements
INTRODUCTION
Driving a personal motor vehicle is the primary method of community transportation for older adults (i.e., those 65 years of age or greater) in Canada.(1) The aging of Canadian society, coupled with the higher-than-historical rates of driving among middle-aged women soon to become seniors, will lead to an unprecedented number of older drivers on our roads over the coming decades.
For many older adults, retaining a driver’s licence is important in maintaining both independence and well-being. Driving cessation is associated with depressive symptoms,(2,3) declines in social and physical functioning,(4,5) greater risk of long-term care placement,(6) and a higher likelihood of death.(7) At the same time there are concerns about the safety of older drivers. Many of the disorders that can adversely affect driving abilities become more common with increasing age.(8) When corrected for kilometers driven, older drivers have a relatively high rate of motor vehicle crashes (MVCs),(9) but this may be due to what has been termed low-mileage bias. Older drivers tend to drive less, and independent of age, low-mileage drivers typically have higher crash rates.(9,10) It is clear, though, that older drivers in MVCs are more likely to suffer serious injuries or die because of their increased vulnerability to trauma.(11) Fortunately, the fatality and injury risk for older Canadian drivers has recently dropped—for example, the number of fatalities among older drivers declined from 469 to 406 between 2003 and 2010, even though the number of older drivers increased by over 700,000 during this time span.(12,13)
A significant challenge for a driving regulatory system is to concurrently identify older drivers who pose an unacceptable risk to themselves and/or other road users if they continue to drive, while not discriminating against similarly aged individuals capable of safely driving a motor vehicle. An effective system for traffic safety entails delineating the roles of various groups including older drivers, health-care providers (e.g., physicians), and licensing authorities, and coordinating their activities. Older drivers frequently self-regulate(14) and in some jurisdictions, like the United Kingdom, are personally required to declare if they are suffering from a medical condition that could prevent them from holding a licence.(15) Physicians are expected to know what conditions (alone or in combination) may lead to concerns about driving abilities, detect them and assess their functional impact, discuss with patients the implications of their health status on driving abilities, and report those who they believe are medically unfit to drive, according to the rules of the jurisdiction in which they practise.(8,16) While licensing authorities have the final responsibility for determining eligibility for a driver’s licence,(8) physicians and their older patients routinely talking about driving could “normalize” these difficult conversations and allow time both to adapt to changing abilities and to cope with cessation.(17)
In June of 2012, the Alberta Motor Association Foundation for Traffic Safety, the Brenda Strafford Centre on Aging, and the Institute of Public Health of the University of Calgary hosted an international invitational symposium on the assessment of older drivers. Recognized content experts from around the world, health-care practitioners, those responsible for regulating older drivers, and representatives from senior organizations were brought together to examine the assessment of older drivers, deliberate about the principles that should underlie our approach to this contentious area, and discuss future directions for research, practice, and policy. Presentations summarizing the current state of the field were followed by small group breakout sessions and a final plenary session.
To capitalize on this diverse gathering of knowledgeable individuals, attendees were asked to participate in a modified Delphi process designed to develop consensus statements on the assessment of older drivers. Our objective was to clarify areas of broad agreement and identify those where there was no consensus. This paper reviews the methods used, and presents the results of the survey. It is our hope that these consensus statements will help guide policy development in Canada.
METHODS
Participants were recruited from attendees of the Global Perspectives of Assessing Older Drivers symposium held in Calgary, Alberta from June 21–June 23, 2012. These attendees included researchers (n = 8), physicians and representatives of physician organizations (n = 4), occupational therapists (n = 2), government officials (including members of advisory boards) (n = 16), independent consultants and representatives of private companies (n = 4), staff and students affiliated with the funding agencies (n = 13), and a representative of a seniors’ organization (n = 1). Attendees were selected for their knowledge and/or involvement in the assessment of older drivers. Invited speakers were chosen by the organizers of the symposium based on both reputation and a desire to ensure diversity in the opinions expressed, geographic distribution, and the disciplines represented. Organizations and government agencies who agreed to participate selected their own delegates, but were advised to choose knowledgeable individuals in decision-making positions. The Alberta Motor Association Foundation for Traffic Safety, the Driver Education Branch of the Alberta Motor Association, the Brenda Strafford Centre on Aging, and the Institute for Public Health of the University of Calgary provided financial support. The funders had no role in the design or conduct of this study.
The Delphi technique is a widely used methodology for achieving convergence of opinion among a group of knowledgeable individuals concerning a complex, real-world topic.(18,19) Over the years it has been modified in a number of ways.(20) The in-depth deliberations of the breakout groups of the Global Perspectives of Assessing Older Drivers symposium were used to create an initial series of statements about the assessment of older drivers that were utilized in the two-round, email-based voluntary survey of symposium participants described in the following paragraph.
An email invitation was sent to 41 non-student attendees of the symposium who had given permission to be contacted after the symposium. The email informed them of the study and its objectives, described the methodology that would be used, and asked for their voluntary participation. The steps taken to ensure the anonymity of their responses were described in the invitation. Those choosing to participate were asked to indicate their level of agreement (or disagreement) with the provided statements on a five-point Likert scale (i.e., agree completely, agree somewhat, neutral, disagree somewhat, disagree completely). Respondents had the option of not responding to a particular statement. During the first round, participants could suggest modifications to the provided statements or propose additional ones for inclusion in the second round. Our intent was to achieve consensus on as many statements as possible. In the second round, respondents were informed of which statements from the first round achieved consensus as originally worded. They were then asked to indicate their level of agreement, using the same five-point Likert scale, with the modified or new statements developed in response to the feedback obtained during the first round, comment on why they thought certain statements or recommendations did not achieve consensus, and provide any other feedback.
Respondents completed the survey on their own computers using a link and password provided to them in the email invitation. Respondents could only complete the survey once during each of the two iterations. As noted previously, all responses were anonymous. One email reminder to complete the survey was sent to potential participants one to two weeks before the specified deadlines for receipt of responses. The survey was done using Qualtrics survey software (www.qualtrics.com).
Consensus was pre-defined as occurring if 80% or more of the respondents for that iteration of the survey indicated agreement (agreeing completely or somewhat) or disagreement (disagreeing completely or somewhat) with a specific statement or recommendation. The 80% threshold for consensus is commonly used and generally perceived as fair, as it allows a strong minority dissenting opinion to prevent the achievement of consensus. To minimize non-response bias, the desired response rate for the survey was 50% or higher.(21)
The University of Calgary Conjoint Faculties Research Ethics Board approved the study. Responding to the survey was taken as evidence of consent by participants. No financial compensation was offered for participation. Participants were asked to declare any direct or indirect financial interest in a company whose concerns were in the area covered by the listed statements or recommendations. One participant responded in the affirmative. Their responses were retained in our analyses.
RESULTS
Of the 41 participants invited to take part, 23 (56.1% of those approached) completed the first round of the survey where they rated their level of agreement to 31 statements grouped by the following categories that were based on the themes of the symposium breakout sessions: detection of potential risk, determination of driving fitness, ethico-legal issues, and research recommendations. Twenty-five achieved consensus support (please see Table 1 for the specific statements that achieved consensus and Table 2 for those that didn’t). Collectively, these statements emphasized the need for a fair, evidence-based, and widely accessible multi-step approach to the assessment of older drivers that could be modified as improvements became available. Respondents recognized that this required the engagement and support of physicians and other health-care practitioners who would be expected to identify and report medically at-risk drivers of any age. At a societal level, alternatives to driving a personal motor vehicle should be developed to meet community mobility needs. General comments provided by participants indicated that the statements “made sense.” One referred to them as being “motherhood statements,” while another felt that it was “hard to imagine they wouldn’t get unqualified support”. It was anticipated, though, that challenges in getting them “accepted and implemented” would arise.
TABLE 1.
|
TABLE 2.
The option of modified licensing as an alternative to either full renewal or rescinding a driver’s licence should be utilized more often (18/23, 78.3%). |
There should be national harmonization on standards but regional flexibility in the delivery of programs (18/23, 78.3%). |
We need sophisticated modeling approaches to both inform policy development and project resource implications of proposed changes to the driving assessment and licensing system (18/23, 78.3%). |
Research support should be targeted to assessing the role of in-car technology to both assess drivers and deal with identified problems (e.g., blind spot detection) (18/23, 78.3%). |
Age-based testing is discriminatory (17/23, 73.9%). |
We currently do not have a sufficiently accurate, practical, and acceptable office-based approach to the detection of potential risk for continued driving that would trigger a further in-depth evaluation of driving fitness (16/23, 69.6%). |
A total of six statements did not achieve consensus agreement (Table 2). Respondents’ comments gave an indication of the concerns about these statements. A respondent not agreeing with “Age-based testing is discriminatory” noted that increasing age is a risk factor for many of the conditions that can impair driving abilities, and argued for age-based screening followed by “more specific testing for those identified as demonstrating some impairment”. While most (69.6%) concurred that we did not have “… a sufficiently accurate, practical and acceptable office-based approach to the detection of potential risk for continued driving”, there was a comment that we should rally behind whatever appeared to be the most reasonable current approach, even if less than ideal, as something was needed now. Reservations about the recommendation that “… modified licensing as an alternative to either full renewal or rescinding a driver’s license should be utilized more often” centered on the needs to more fully develop this option, evaluate it, and determine whether it could be effectively implemented. Those disagreeing with the statement advocating for “… sophisticated modeling approaches to both inform policy development and project resource implications of proposed changes to the driving assessment and licensing system” felt the appropriateness of this methodology would depend on the specific research or policy question being addressed.
Four modified and four new statements were included in the second iteration of the survey. As there were only 12 participants (29.3% of those initially approached—much less than the desired survey response rate of 50%+) in the second round, none of these statements were considered for consensus agreement. The new statements (with degree of support received) were as follows: “Older drivers and persons with disabilities cannot be held to a higher standard than the general driving population” (agreed to by 12/12); “Advocacy groups for seniors or specific health conditions should have a formal place in the development of regulations and policy” (10/12); “Contesting a government prohibition on driving should be a cost borne by the individual” (7/12); and, “Driving reassessments required by government for renewal of a driving licence should be paid for by government” (5/12).
DISCUSSION
These survey results can serve as a basis for the development of policy about the assessment and regulation of older drivers in Canada. There was a great deal of agreement among this diverse group of knowledgeable individuals on the key elements of a desired system and current knowledge gaps. We are aware of only one other recent national effort dealing with older drivers that involved a similar range of stakeholders. In 2009, the Canadian Association of Occupational Therapists, with input from a broadly based 21-member National Advisory Committee, produced the National Blueprint for Injury Prevention in Older Drivers.(22) Our consensus statements align and add to the guiding principles contained in this document.
A number of statements were endorsed by most respondents, but did not achieve the pre-determined threshold for consensus. An example was the statement that age-based testing was discriminatory (17/23 or 73.9% supported it). This result contrasts with a 2000 Canadian Gallup poll where over 80% of respondents favoured mandatory testing of all older drivers,(23) but is in accord with research indicating more stringent licensing requirements for older drivers do not lead to societal benefit.(24–32) Greater use of modified (or restricted) licensing also obtained a high level of support (18/23, 78.3%), but did not achieve consensus. Though editorials in the CMAJ have advocated for reverse graduated licensing of older drivers(33,34) and there is qualified research support for it,(35–37) a sufficient number of respondents felt this approach required additional study before becoming more widely used. While most respondents agreed that we do not have a sufficiently accurate, practical, and acceptable office-based approach to identifying potentially dangerous drivers, this statement did not achieve consensus support. Opposition to this statement was partially driven by the sense that we had to endorse something, even if not ideal. Aside from specific issues that may arise with particular instruments, such as the American Medical Association Assessment of Driving Related Skills (ADReS) battery,(38) there are theoretical and methodological challenges to any such attempt.(32,39) Notwithstanding growing interest in the role of advanced in-car safety technology,(40,41) a number of respondents raised reservations about targeting research funding to studying its use in assessing drivers and/or mitigating driving risk.
A strength of the study was the scope and expertise of the participants, though the anonymous nature of the survey prevents us from knowing the actual diversity of those who responded. A further strength was the attempt to deal with the practical and not just the ideal, which was made possible by involving practitioners and those responsible for implementing driving policy, as well as researchers.
A number of limitations should be noted. There is the possibility of both selection (in deciding who was invited to the symposium) and self-selection (in completing the survey) bias. While the meeting brought together a diverse group of knowledgeable participants, it did not include all stakeholder groups, and no attempt was made to ensure numerical balance. Survey participation was made as simple as possible, with reminders sent to encourage completion, but our response rates, especially for the second iteration, raise the possibility of non-response bias. As well, the anonymous nature of the survey didn’t allow us to compare the characteristics of respondents and non-respondents. While replication of our results is needed, carefully selected groups of knowledgeable individuals using Delphi methodology have been shown to produce reliable results.(42) Some of the consensus statements were broad, and moving to more specific, implementable recommendations would likely lead to disagreement. We feel this was demonstrated by the universal agreement that sustainable and broadly-based financing is required, but the lack of agreement on who should pay for two specific elements (mandatory driving reassessments and appeals), during the second iteration of the survey. There was redundancy in a number of the statements (e.g., statements I.d and II.d in Table 1). We felt, though, this allowed us to check on the consistency of the opinions expressed. Finally, the definition of consensus used was arguably arbitrary (though commonly used), with a number of statements just meeting or falling below the pre-determined threshold.
An on-going and inclusive dialogue is needed about this complex issue. These deliberations should be based on explicitly stated principles and informed by the best available evidence, with policy implemented only after careful consideration of the implications of the actions potentially being taken.
Acknowledgments
We would like to thank Mr. David Borkenhagen for his assistance with the online survey, our funders, the attendees of the symposium, and all those who responded to the survey.
Footnotes
CONFLICT OF INTEREST DISCLOSURES
The authors declare that no conflicts of interest exist.
REFERENCES
- 1.Turcotte M. Profile of seniors’ transportation habits. Can Soc Trends. 2012 Summer;93:1–16. [Google Scholar]
- 2.Marottoli RA, Mendes de Leon CF, Glass TA, et al. Driving cessation and increased depressive symptoms: prospective evidence from the New Haven EPESE. J Am Geriatr Soc. 1997;45(2):202–06. doi: 10.1111/j.1532-5415.1997.tb04508.x. [DOI] [PubMed] [Google Scholar]
- 3.Ragland DR, Satariano WA, MacLeod KE. Driving cessation and increased depressive symptoms. J Gerontol A Biol Sci Med Sci. 2005;60(3):399–403. doi: 10.1093/gerona/60.3.399. [DOI] [PubMed] [Google Scholar]
- 4.Mezuk B, Rebok GW. Social integration and social support among older adults following driving cessation. J Gerontol B Soc Sci. 2008;63B(5):S298–S303. doi: 10.1093/geronb/63.5.S298. [DOI] [PubMed] [Google Scholar]
- 5.Edwards JD, Lunsman M, Perkins M, et al. Driving cessation and health trajectories in older adults. J Gerontol A Biol Sci Med Sci. 2009;64(12):1290–95. doi: 10.1093/gerona/glp114. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Freeman EE, Gange SJ, Munoz B, et al. Driving status and risk of entry into long-term care in older adults. Am J Pub Health. 2006;96(7):1254–59. doi: 10.2105/AJPH.2005.069146. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Edwards JD, Perkins M, Ross LA, et al. Driving status and three-year mortality among community-dwelling older adults. J Gerontol A Biol Sci Med Sci. 2009;64(2):300–05. doi: 10.1093/gerona/gln019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Canadian Medical Association . CMA driver’s guide – determining medical fitness to operate motor vehicles. 8th edition. Ottawa: Canadian Medical Association; 2012. [Google Scholar]
- 9.Hakamies-Blomqvist L, Raitanen T, O’Neill D. Driver ageing does not cause higher accident rates per km. Transport Res F. 2002;5(4):271–74. doi: 10.1016/S1369-8478(03)00005-6. [DOI] [Google Scholar]
- 10.Langford J, Methorst R, Hakamies-Blomqvist L. Older drivers do not have a high crash rate. A replication of low mileage bias. Accid Anal Prev. 2006;38(3):574–78. doi: 10.1016/j.aap.2005.12.002. [DOI] [PubMed] [Google Scholar]
- 11.Li G, Braver ER, Chen L-H. Fragility versus excessive crash involvement as determinants of high death rates per vehicle-mile of travel among older drivers. Accid Anal Prev. 2003;35(2):227–35. doi: 10.1016/S0001-4575(01)00107-5. [DOI] [PubMed] [Google Scholar]
- 12.Transport Canada . Canadian motor vehicle traffic collision statistics 2003. Ottawa: Minister of Transport; 2004. [Google Scholar]
- 13.Transport Canada . Canadian motor vehicle traffic collision statistics 2010. Ottawa: Minister of Transport; 2012. [Google Scholar]
- 14.Braitman KA, McCartt AT. Characteristics of older drivers who self-limit their driving. Ann Adv Automot Med. 2008;52:245–54. [PMC free article] [PubMed] [Google Scholar]
- 15.Mitchell CGB. The licensing and safety of older drivers in Britain. Accid Anal Prev. 2013;50:732–41. doi: 10.1016/j.aap.2012.06.027. [DOI] [PubMed] [Google Scholar]
- 16.Coopersmith HG, Korner-Bitensky NA, Mayo NE. Determining medical fitness to drive: physicians’ responsibilities in Canada. CMAJ. 1989;140(4):375–78. [PMC free article] [PubMed] [Google Scholar]
- 17.Betz ME, Jones J, Petroff E, et al. “I wish we could normalize driving health”: a qualitative study of clinician discussions with older drivers. J Gen Intern Med. 2013;28(12):1573–80. doi: 10.1007/s11606-013-2498-x. [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Dalkey NC, Helmer O. An experimental application of the Delphi method to the use of experts. Manage Sci. 1963;9(3):458–67. doi: 10.1287/mnsc.9.3.458. [DOI] [Google Scholar]
- 19.Hsu C-C, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess Res Eval. 2007;12(10) Accessed 2013 March 12. Available from: http://pareonline.net/pdf/v12n10.pdf. [Google Scholar]
- 20.Marriott J, Stehlik P. A critical analysis of methods used to develop explicit clinical criteria for use in older people. Age Ageing. 2012;41(4):441–50. doi: 10.1093/ageing/afs064. [DOI] [PubMed] [Google Scholar]
- 21.Draugalis JR, Coons SJ, Plaza CM. Best practices for survey research reports: a synopsis for authors and reviewers. Am J Pharm Educ. 2008;72(1):11. doi: 10.5688/aj720111. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Canadian Association of Occupational Therapists . National blueprint for injury prevention in older drivers. Ottawa: CAOT Publications; 2009. [Google Scholar]
- 23.Buske L. Driving: how old is too old? CMAJ. 2000;163(10):1334. [Google Scholar]
- 24.Hakamies-Blomqvist L, Johansson K, Lundberg C. Medical screening of older drivers as a traffic safety measure: a comparative Finnish-Swedish evaluation study. J Am Geriatr Soc. 1996;44(6):650–53. doi: 10.1111/j.1532-5415.1996.tb01826.x. [DOI] [PubMed] [Google Scholar]
- 25.Langford J, Fitzharris Koppel S, Newstead S. Effectiveness of mandatory license testing for older drivers in reducing crash risk among urban older Australian drivers. Traffic Inj Prev. 2004;5(4):326–35. doi: 10.1080/15389580490509464. [DOI] [PubMed] [Google Scholar]
- 26.Langford J, Fitzharris M, Newstead S, et al. Some consequences of different older driver licensing procedures in Australia. Accid Anal Prev. 2004;36(6):993–1001. doi: 10.1016/j.aap.2003.11.003. [DOI] [PubMed] [Google Scholar]
- 27.Mitchell CGB. The licensing of older drivers in Europe – a case study. Traffic Inj Prev. 2008;9(4):360–66. doi: 10.1080/15389580801895160. [DOI] [PubMed] [Google Scholar]
- 28.Langford J, Bohensky M, Koppel S, et al. Do older drivers pose a risk to other road users? Traffic Inj Prev. 2008;9(3):181–89. doi: 10.1080/15389580801975632. [DOI] [PubMed] [Google Scholar]
- 29.Caragata Nasvadi G, Wister A. Do restricted driver’s licenses lower crash risk among older drivers? A survival analysis of insurance data from British Columbia. Gerontologist. 2009;49(4):474–84. doi: 10.1093/geront/gnp039. [DOI] [PubMed] [Google Scholar]
- 30.Ross LA, Browning C, Luszcz MA, et al. Age-based testing for driver’s license renewal: potential implications for older Australians. J Am Geriatr Soc. 2011;59(2):281–85. doi: 10.1111/j.1532-5415.2010.03241.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Tay R. Ageing driver licensing requirements and traffic safety. Ageing Soc. 2012;32:655–72. doi: 10.1017/S0144686X11000535. [DOI] [Google Scholar]
- 32.Siren A, Meng A. Cognitive screening of older drivers does not produce safety benefits. Accid Anal Prev. 2012;45:634–38. doi: 10.1016/j.aap.2011.09.032. [DOI] [PubMed] [Google Scholar]
- 33.MacDonald N, Hebert PC. Driving retirement program for seniors: long overdue. CMAJ. 2010;182(7):645. doi: 10.1503/cmaj.100273. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Redelmeier DA, Stanbrook MB. Graduated drivers’ licenses for seniors: reclaiming one benefit of being young [editorial] CMAJ. 2012;184(10):1123. doi: 10.1503/cmaj.120521. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Marshall SC, Spasoff R, Nair R, et al. Restricted driver licensing for medical impairments: does it work? CMAJ. 2002;167(7):747–51. [PMC free article] [PubMed] [Google Scholar]
- 36.Braitman KA, Chaudhary NK, McCartt AT. Restricted licensing among older drivers in Iowa. J Safety Res. 2010;41(6):481–86. doi: 10.1016/j.jsr.2010.10.001. [DOI] [PubMed] [Google Scholar]
- 37.Langford J, Koppel S. License restrictions as an under-used strategy in managing older driver safety. Accid Anal Prev. 2011;43(1):487–93. doi: 10.1016/j.aap.2010.09.005. [DOI] [PubMed] [Google Scholar]
- 38.Ott BR, Davis J, Papandonatos GD, et al. Assessment of driving-related skills prediction of unsafe driving in older adults in the office setting. J Am Geriatr Soc. 2013;61(7):1164–69. doi: 10.1111/jgs.12306. [DOI] [PubMed] [Google Scholar]
- 39.Hakamies-Blomqvist L. Are there safe and unsafe drivers? Transport Res F. 2006;9(5):347–52. doi: 10.1016/j.trf.2006.06.004. [DOI] [Google Scholar]
- 40.KPMG LLP and Center for Automotive Research . Self-driving cars: the next revolution. KPMG; 2012. Accessed 2013 March 18. Available from: https://www.kpmg.com/US/en/IssuesAndInsights/ArticlesPublications/Documents/self-driving-cars-next-revolution.pdf. [Google Scholar]
- 41.Burns LD. Sustainable mobility: a vision of our transport future. Nature. 2013;497:181–82. doi: 10.1038/497181a. [DOI] [PubMed] [Google Scholar]
- 42.Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Method. 2005;5:37. doi: 10.1186/1471-2288-5-37. [DOI] [PMC free article] [PubMed] [Google Scholar]