An international study of the quality of national-level guidelines on driving with medical illness

Abstract

Background: Medical illnesses are associated with a modest increase in crash risk, although many individuals with acute or chronic conditions may remain safe to drive, or pose only temporary risks. Despite the extensive use of national guidelines about driving with medical illness, the quality of these guidelines has not been formally appraised.

Aim: To systematically evaluate the quality of selected national guidelines about driving with medical illness.

Design: A literature search of bibliographic databases and Internet resources was conducted to identify the guidelines, each of which was formally appraised.

Methods: Eighteen physicians or researchers from Canada, Australia, Ireland, USA and UK appraised nine national guidelines, applying the Appraisal of Guidelines for Research and Evaluation (AGREE II) instrument.

Results: Relative strengths were found in AGREE II scores for the domains of scope and purpose, stakeholder involvement and clarity of presentation. However, all guidelines were given low ratings on rigour of development, applicability and documentation of editorial independence. Overall quality ratings ranged from 2.25 to 5.00 out of 7.00, with modifications recommended for 7 of the guidelines. Intra-class coefficients demonstrated fair to excellent appraiser agreement (0.57–0.79).

Conclusions: This study represents the first systematic evaluation of national-level guidelines for determining medical fitness to drive. There is substantive variability in the quality of these guidelines, and rigour of development was a relative weakness. There is a need for rigorous, empirically derived guidance for physicians and licensing authorities when assessing driving in the medically ill.

Introduction

Motor vehicle collisions were reported as the cause of 1.2 million deaths and 20–50 million injuries or disabilities world-wide in 2002 across all ages,¹ and are predicted to be the third leading cause of disability-adjusted life years lost and fifth leading cause of death by 2030.^1,2 There is limited research into the associations between medical illness and crash risk. A number of illnesses are associated with a modest increase in driving risk, with some health conditions associated with a transient risk to road safety.³ Assessment of driving risk becomes more complex in the presence of multiple medical illnesses and multiple medications.⁴

Well-developed guidelines can enhance clinical practice and positively influence patient outcomes.⁵ Although there are many guidelines that support clinical practice regarding driving with medical illness, their quality has not been examined. In the last decade, rigorous standards have been increasingly applied for the evaluation of clinical practice guidelines. The ‘Appraisal of Guidelines Research and Evaluation (AGREE II)’ tool⁶ sets out explicit expectations of guidelines. This tool has been found to have acceptable reliability and validity,⁷ and has been used to assess a myriad of guidelines across numerous clinician specialties. However, most guidelines focus on a single disease, in contrast with guidelines on driving with medical illness, which have the added complexity of addressing dozens of medical conditions.

Objective

The objective of this study was to systematically evaluate the quality and accessibility of existing national-level guidelines on driving with medical illness. The secondary aim was to examine the inter-rater reliability, internal validity and construct validity of the AGREE II tool when applied to the appraisal of these guidelines.

Materials and methods

Search strategy

A systematic literature search (1993–2013) was undertaken to identify national-level guidelines that included description of driving restrictions for various medical conditions and/or provided recommendations to evaluate medical fitness to drive. The search was conducted using bibliographic databases (MEDLINE, EMBASE, PsycINFO, PubMed, CINAHL and the Cochrane Library), guideline search engines (e.g. National Guidelines Clearinghouse) and a general Internet search engine (Google). Searches of all relevant government transportation department or ministry websites were also conducted for each country where English was identified as the primary spoken language. State-, province- or jurisdiction-level guidelines were not evaluated in this study. The detailed search strategy with exclusion criteria appears in Supplementary Appendix S1.

Appraisal instrument

The AGREE II instrument⁸ is composed of 23 items, organized into six domains: (i) scope and purpose, which pertains to the overall aim, health questions and target population; (ii) stakeholder involvement, which provides a rating of the extent to which the guideline represents the views of knowledge users and stakeholders; (iii) rigour of development, which is concerned with the process of data synthesis and recommendation formulation; (iv) clarity of presentation, which refers to clarity in language, structure and format; (v) applicability, which pertains to barriers and facilitators to implementation and (vi) editorial independence, which describes the freedom from bias based on competing interests. Individual items are ranked on a 7-point Likert scale ranging from 1 to 7, with 1 indicating ‘strongly disagree’ and 7 indicating ‘strongly agree’. For each domain, standardized quality scores are calculated by summing the individual item scores in a domain and standardizing the total as a percentage of the maximum possible score for that domain. Raters are also asked to rank the overall quality of the guideline on a 7-point Likert scale ranging from the lowest (1) to the highest possible quality (7), and then to indicate whether they would recommend the guideline for use in practice as is, with modifications, or not at all. Three AGREE II items were modified prior to appraisal in order to better apply to the broad spectrum of illnesses covered by the guidelines (see Supplementary Appendix S2).

Appraisers

The guideline appraisal team included 18 physicians and/or researchers recruited from Canada, Australia, Ireland, USA and UK, representing a broad range of relevant disciplines each with at least 5 years of experience in the field of medical fitness to drive (see Supplementary Appendix S3). Eight different appraisers from the team rated each identified national guideline. All appraisers participated in a 2-h online webinar-based training session on the use of the AGREE II instrument prior to completing their ratings. Conflicts of interest were declared, and appraisers did not evaluate guideline(s) they created or edited.

After completing their ratings, appraisers participated in an online survey where they ranked the AGREE II domains in relative order of priority for guidelines on driving with medical illness, and then rated the significance of each of the 23 individual items on a scale from 0 to 3, from ‘not important’ to ‘very important’ respectively. Five research associates and one researcher (C.H.) from five different countries (Ireland, Australia, USA, UK and Canada) described the online and print accessibility of the guidelines; each guideline’s accessibility was assessed by raters from at least two countries.

Statistical analysis

Summary data and descriptive statistics were calculated for AGREE II domain scores and items for the driving guidelines, and the accessibility findings were described. Internal consistency of each domain was assessed with Cronbach’s alpha using the mean item scores per domain. Inter-rater reliability was assessed using intra-class correlation coefficients (two-way random model). Construct validity was assessed using Spearman’s rank correlation coefficients between appraisers’ domain scores and the overall assessment scores.

Results

Guideline search

Twenty-five guideline documents were identified by the systematic search (Figure 1), and nine national-level guidelines were included, with one each from Australia (Austroads and National Transport Commission of Australia [Austroads]⁹), Ireland (Royal College of Physicians of Ireland and Road Safety Authority [RCPI/RSA]¹⁰), New Zealand (New Zealand Transport Agency [NZ]¹¹), Singapore (Singapore Medical Association [SMA]¹²), and UK (Drivers Medical Group, Driver and Vehicle Licensing Agency, Swansea [DVLA]¹³). Canada and USA each had guidelines for physicians and for administrators (Canadian Medical Association [CMA]¹⁴, Canadian Council of Motor Transport Administrators [CCMTA]¹⁵, American Medical Association [AMA]¹⁶, and American Association of Motor Vehicle Administrators [AAMVA]¹⁷). The AMA and AAMVA guidelines were co-developed by the US National Highway Traffic Safety Administration (NHTSA) (Table 1).

Figure 1.

Flow chart for selection of national-level guidelines. ^aReasons for exclusion: provincial/state-level guideline (n = 5); guideline for commercial drivers only (n = 2); only addressed assessment of older adults (n = 1); only addressed assessment of a specific condition (n = 5); did not address assessment of medical fitness to drive (n = 1); evaluation tool, not a guideline (n = 1); screening measure, not a guideline (n = 1).

Table 1.

Overall quality scores and recommendations in descending order of overall quality

National-level guideline		Overall quality (/7)	Overall Recommendation (%)
			Recommend	Recommend with modifications	Do not recommend
1.	Physician’s guide to assessing and counseling older drivers, NHTSA/AMA, 20101,6	5.00	25	75	0
2.	Assessing fitness to drive for commercial and private vehicle drivers: medical standards for licensing and clinical management guidelines, Austroads, 20129	4.88	62.5	37.5	0
3.	Driver fitness medical guidelines, NHTSA/AAMVA, 20091,7	4.88	12.5	75	12.5
4.	CMA driver’s guide: determining medical fitness to operate motor vehicles, CMA, 20121,4	4.13	12.5	75	12.5
5.	For medical practitioners: at a glance guide to the current medical standards of fitness to drive, DVLA, 20131,3	3.88	12.5	50	37.5
6.	Determining driver fitness in Canada: CCMTA medical standards for drivers, CCMTA, 201315	3.63	0	87.5	12.5
7.	Medical fitness to drive guidelines (Group 1 Drivers), RCPI/RSA, 2013]10	3.50	0	75	25
8.	Medical aspects of fitness to drive: A guide for medical practitioners, NZ, 200911	3.13	0	50	50
9.	Medical guidelines on fitness to Drive, SMA, 20111,2	2.25	0	12.5	87.5

AGREE II domain scores

Low scores were evident for three of the domains. Editorial independence scores ranged from 8% to 46% (median = 14%; interquartile range [IQR] = 12%), with seven of the guidelines scoring below 20%. Rigour of development scores ranged from 9% to 37% (median = 22%; IQR = 21%) across the individual guidelines and five scored below 30%. Applicability scores ranged from 11% to 55% (median = 28%; IQR = 13%), with eight guidelines scoring <40%. Higher scores were found for the remaining domains. Stakeholder involvement scores ranged from 36% to 83% (median = 58%; IQR = 29%), with three of the guidelines scoring above 70%. Clarity of presentation scores ranged from 42% to 76% (median = 63%; IQR = 10%), although only one of the guidelines scored above 70%. Scope and purpose was rated highest, with scores ranging from 59% to 89% (median = 78%; IQR = 29%), and seven of the guidelines scoring above 70% (Figure 2).

Figure 2.

AGREE II standardized domain scores. (a) Domain 1: Scope and purpose; (b) Domain 2: Stakeholder involvement; (c) Domain 3: Rigour of development; (d) Domain 4: Clarity and presentation; (e) Domain 5: applicability; (f) Domain 6: Editorial independence. Scores presented as a percentage of the maximum possible score for that domain.

AGREE II item scores

The majority of guidelines were judged to be strong in their description of their overall objectives, clinical questions addressed and targeted users. As well, the clarity of recommendations and ease with which key recommendations could be identified were reported as strong by the appraisers. In contrast, most of the guidelines were rated as weak for items pertaining to conflicts of interest, cost implications and barriers, as well as for all individual items evaluating rigour of development (Table 2).

Table 2.

Proportion of appraisers who indicated high ratings for each of the 23 AGREE II items

National-level guideline		Frequency (%) of appraisers agreeing or strongly agreeing (≥5 out of 7) with the item
		Austroads	CCMTA	CMA	DVLA	NHTSA/ AAMVA	NHTSA/ AMA	NZ	RCPI/ RSA	SMA
Domain 1: Scope and purpose.
1.	The overall objective(s) of the guideline is (are) specifically described	100	87.5	87.5	100	87.5	87.5	87.5	100	50
2.	The question(s) covered by the guideline is (are) specifically described.a	87.5	75	75	75	87.5	87.5	87.5	62.5	62.5
3.	The population (patients, public, etc.) to whom the guideline is meant to apply are specifically described.	75	62.5	75	87.5	62.5	87.5	87.5	87.5	50
Domain 2: Stakeholder Involvement.
4.	The guideline development group includes individuals from all of the relevant professional groups.	75	12.5	50	12.5	25	87.5	50	75	25
5.	The views and preferences of the target population (patients, public, etc.) have been sought.	62.5	0	37.5	25	37.5	62.5	25	75	0
6.	The target users of the guideline are clearly defined.	100	75	87.5	100	75	75	100	100	75
Domain 3: Rigour of development.
7.	Systematic methods were used to search for evidence.	25	12.5	12.5	12.5	12.5	0	0	12.5	0
8.	The criteria for selecting the evidence are clearly described.	0	0	0	12.5	12.5	0	0	0	0
9.	The strengths and limitations of the body of evidence are clearly described.	50	12.5	12.5	12.5	75	12.5	0	0	0
10.	The methods used for formulating the recommendations are clearly described.	12.5	12.5	25	25	25	0	0	25	0
11.	The health benefits, side effects, and risks (of the assessment and its results) have been considered in formulating the recommendations.a	50	25	62.5	0	37.5	87.5	12.5	37.5	0
12.	There is an explicit link between the recommendations and the supporting evidence.	12.5	12.5	12.5	0	75	50	0	0	0
13.	The guideline has been externally reviewed by experts prior to its publication.	37.5	0	12.5	0	0	62.5	37.5	0	12.5
14.	A procedure for updating the guideline is provided.	0	0	75	50	0	12.5	0	12.5	0
Domain 4: Clarity and presentation.
15.	The recommendations are specific and unambiguous.	62.5	87.5	37.5	100	87.5	62.5	50	75	37.5
16.	The different management options are clearly presented.a	75	50	62.5	25	50	75	37.5	62.5	12.5
17.	Key recommendations are easily identifiable.	100	75	50	87.5	100	37.5	75	75	25
Domain 5: Applicability.
18.	The guideline describes facilitators and barriers to its application.	37.5	12.5	62.5	12.5	37.5	50	12.5	12.5	0
19.	The guideline provides advice and/or tools on how the recommendations can be put into practice.	62.5	25	50	37.5	37.5	100	25	25	12.5
20.	The potential resource implications of applying the recommendations have been considered.	12.5	12.5	0	0	0	50	0	0	0
21.	The guideline presents monitoring and/or auditing criteria.	12.5	50	0	0	12.5	25	12.5	12.5	0
Domain 6. Editorial independence.
22.	The views of the funding body have not influenced the content of the guideline.	12.5	12.5	0	0	37.5	75	0	0	12.5
23.	Competing interests of guideline development group members have been recorded and addressed.	0	0	0	12.5	12.5	12.5	0	0	0

^aThe project team made adaptations to this item (see Supplementary Appendix S2).

AGREE II global ratings and recommendations

Mean overall quality ratings ranged from 2.25 to 5.00 out of 7 (SD = 0.92), and the majority of guidelines received recommendations for modifications (Table 1). The SMA guideline was not recommended for use by the majority of its appraisers. Half of its appraisers did not recommend the NZ guideline, and the other half recommended modifications. Only the Austroads guideline was recommended without modification by more than half of its raters.

Prioritization of AGREE II items and domains

Appraisers ranked the AGREE II domains from most to least important as follows (mean rank score out of 6 ± SD in parentheses, where 1 is most important/highest priority and 6 is least important/lowest priority): Rigour of development (1.24 ± 0.56), stakeholder involvement (3.65 ± 1.66), clarity and presentation (3.65 ± 1.37), scope and purpose (3.82 ± 1.74), editorial independence (4.29 ± 1.57) and applicability (4.35 ± 1.06). The three most important individual AGREE II items for evaluating the development of guidelines on driving with medical illness were identified as the use of systematic methods to search for evidence, description of methods used to formulate the recommendations and description of the criteria used for selecting the evidence.

Evaluation of the AGREE II tool

Inter-rater reliability

There was excellent appraiser agreement (intra-class correlation coefficients) for the NHTSA/AAMVA (0.75), NZ (0.79) and RCPI/RSA (0.75) guidelines, good agreement for the Austroads (0.72), CCMTA (0.62), CMA (0.72), NHTSA/AMA (0.67) and SMA (0.64) guidelines, and fair agreement for the DVLA guideline (0.57).

Internal consistency

Internal consistency (Cronbach’s alpha) was high for applicability (0.88), stakeholder involvement (0.86) and scope and purpose (0.83), moderate for rigour of development (0.78) and editorial independence (0.59) and low for clarity of presentation (0.23). Internal consistency was similar for the three unmodified domains (mean = 0.78, SE = 0.09) and the three domains with items that were modified by the project team (mean = 0.61, SE = 0.19, t (4) = 0.76, P = 0.49).

Construct validity

Moderately strong correlation coefficients (Spearman’s rank) were found between appraisers’ domain scores and their overall assessment scores for editorial independence (0.43), applicability (0.53) and rigour of development (0.66), all significant at the P < 0.001 level. Weak coefficients were found for scope and purpose (0.31, P = 0.008), stakeholder involvement (0.36, P = 0.002) and clarity of presentation (0.39, p = 0.001). Construct validity scores were comparable for the modified (mean = 0.45, SE = 0.11) and unmodified domains (mean = 0.44, SE = 0.05, t (4) = 0.114, P = 0.915).

Accessibility results

Three of the national guidelines (Austroads, CMA and RCPI/RSA) were located within the first two web pages in the general web search conducted in each of the five countries that were included. Two guidelines (NZ and SMA) did not appear within the first 2 web pages in any country, with some variability for the remaining guidelines. The local guideline appeared first in the search for its respective country for Australia, Canada, England and Ireland, with some variability with the other guidelines. Complete electronic versions of all of the guidelines were freely available for downloading. Hard copies were readily available for only the NHTSA/AMA, Austroads and CMA guidelines, and costs ranged from free to $25.00 US. At the time of the current manuscript submission, the AMA no longer supports its national driving guidelines.

Discussion

When evaluated using the standards of the AGREE II tool, the national-level guidelines on driving with medical illness were rated as moderate in terms of their quality ratings, and two-thirds of the guidelines were recommended for modifications by at least half of the raters. While guidelines are key resources for supporting clinical decisions concerning fitness to drive,¹⁸ as well as assisting malpractice-related cases¹⁹ and policy development,²⁰ we have identified relative weaknesses in the current guidelines on driving with medical illness in the domains of applicability, rigour of development and editorial independence. Appraisals of guidelines for medical illnesses that do not pertain to driving report similar weaknesses in rigour and applicability.^21,22 Stakeholder involvement, which was a relative strength in this study, tends to be poorly rated in appraisals of medical guidelines not focusing on driving.^22,23 We found largely good-to-excellent inter-rater agreement for all but one of the guidelines, and acceptable internal consistency, except in the realm of clarity. Content validity, the association between the domain scores and the global scores, was only moderate, which was similar to studies that examined content validity of the AGREE II tool with more focused topics such as traumatic brain injury²⁴ and asthma.²⁵

None of the driving guidelines was found through a search of the peer-reviewed databases, and ease of locating these resources varied according to the country in which the search was conducted. Accessibility barriers can impact the likelihood that a guideline is adopted in busy clinical settings, and evidence to date would suggest that there is limited adherence to guideline recommendations even in jurisdictions where there is mandatory reporting of medical conditions by physicians.²⁶

While applicability was not a highly rated priority in this study, there are many barriers to physicians’ engagement in assessing driving safety.^27,28 Delineating the functional aspects of the risks posed by medical conditions on driving presents a special challenge to clinical practice guidelines on driving with medical illness. Additionally, medical conditions account only for a minority of causes of crashes internationally,²⁹ and clinicians are wary of unfairly restricting their patients’ driving because of negative impacts on their autonomy, independent functioning and on the doctor-patient relationship.^{27,28,30–32} Studies have been published that show clinicians are interested and willing to learn and adopt guidelines with measurable changes in practice behaviour, knowledge, confidence and attitudes.³³ Creating better processes and methodology to build better guidelines is important, but these efforts fall short unless there is adequate infrastructure and financial support to inform and educate clinicians on the use of these tools, addressing the barriers to their application. We also acknowledge that there is limited evidence-based information on driving risk and management on the myriad of medical conditions that are addressed in these guides.

The strengths of this study include the international nature of both the guidelines and the raters; the exploration of the inter-rater reliability, internal consistency and content validity; the delineation of accessibility challenges, and the training process to maximize agreement. A limitation of our study is that many of the raters were involved in development of guidelines on driving, and guideline developers tend to give lower AGREE II ratings than clinicians or policy-makers.⁷ Additionally, there are guidelines on driving with medical illness that have been developed in specific states, provinces or jurisdictions that were outside the scope of the present analysis and were excluded. These may have unique strengths and suitability for local contexts, and it would be worthwhile for a future study to systematically examine their quality.

While the national guidelines on medical fitness to drive were initially developed prior to the introduction of the AGREE II tool, subsequent editions have generally not been modified to reflect the expectations of a clinical practice guideline, although the CPI/RSA guidelines were updated in a more rigorous fashion on the month of submission of the present manuscript.³⁴ Efficiencies can be introduced with international collaboration, similar to that in this study, in order to improve the rigour and applicability in guidelines.²³ Given the economic impact and personal anguish associated with death and injury from road trauma,³⁵ credible and rigorously developed guidelines need to be made accessible, and their use facilitated in clinical practice, in order to overcome the many barriers to their adoption.

Supplementary material

Supplementary material is available at QJMED online.

Funding

This work was supported by a Planning Grant from the Canadian Institutes of Health Research (CIHR) entitled ‘Canadian Guidelines for Determining Medical Fitness to Operate Motor Vehicles’ (grant application No. 304408; funding reference No. KPE 129596; URL: http://www.cihr-irsc.gc.ca/e/193.html). CIHR funds were used for the online webinar-based training sessions on the use of the AGREE II instrument, as well as teleconferences amongst all authors to discuss interpretation of the data and the manuscript. The authors would also like to thank the CIHR Team in Driving in Older Persons (Candrive II) for their in-kind support. Dr. Naglie is supported by the George, Margaret and Gary Hunt Family Chair in Geriatric Medicine, University of Toronto. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Conflict of Interest: The authors of this manuscript have the following competing interests: S.M. has received funding in the past for consulting with the Ministry of Transportation of Ontario; J.C. has received funding in the past for manuscript writing/reviewing from Monash University (her current employer) and for lectures with Medicare Local, Bayside on content similar to the current article (i.e., general information regarding medical review for driver licensing); D.C. has received research funding in the past from the National Institute of Health and Missouri Department of Transportation; however, none of the funders for these individual authors were involved in the development of the current manuscript. S.M., J.D., S.S., D.C. and D.O. have also been involved in the development of guidelines on driving with medical illness that were evaluated in the current article, although none of them evaluated guideline(s) they edited or were involved with in any capacity. There were no financial relationships with any organizations that might have an interest in the submitted work. There were no other relationships or activities that could appear to have influenced the submitted work.