Abstract
Background
The American Academy of Orthopaedic Surgeons (AAOS) provides clinical guidelines and frequently updates and expands on the recommendations. High quality, up-to-date, and applicable clinical guidelines are increasingly important tools for physicians to have. Assessing these continuously updating guidelines’ overall quality has most recently been done in 2013 when there were 14 guidelines. Since then, nearly all guidelines have been updated and now 28 guidelines are provided. Our goal was to evaluate the updated 2023 AAOS guidelines using the well-established Advancing Guideline Development, Reporting and Evaluation in Health Care (AGREE II) tool.
Methods
On April 23, 2023, all 28 guidelines set forth by AAOS were compiled. Using the AGREE II tool, a well-recognized and established tool for evaluation of guideline quality used across all disciplines, three independent reviewers appraised the current guidelines. These data were used to run descriptive statistics and assessed accordingly.
Results
Across the 28 guidelines, the average AGREE II domain scores were as follows: Scope and Purpose (median score, 85%), Stakeholder Involvement (median score, 87%), Rigor of Development (median score, 87%), Clarity of Presentation (median score, 87%), Applicability (median score, 48%), Editorial Independence (median score, 87%), and Overall Assessment (median score, 80s%).
Conclusions
Five of the six AGREE II domains, including the overall assessment, tested achieved a median score above the 50% mark. These results indicate that the AAOS guidelines met many criteria for high quality based on a critical appraisal.
1. Introduction
With everchanging improvements to the scope of evidenced-based medical knowledge, the use of clinical practice guidelines (CPGs) remains a valuable tool for physicians to use in weighing treatment, management, and diagnosis options for patients. A PubMed query of “orthopedic surgery” and “guideline” in the abstract/title resulted in 184 results for 2022 alone. The number of guidelines put forth by the American Academy of Orthopaedic Surgeons (AAOS) has increased from 14 in 2014 to 28 in 2023.1 With more actively being in development (https://www.aaos.org/quality/quality-programs/). Proven benefits of rigorous CPGs in applications to medical practice have been confirmed but the overall practicality as well as quality of CPGs remains challenged.2, 3, 4, 5, 6, 7
Individual AAOS guidelines are often the subject of evaluation, but they have not been aggregated as a whole and had their quality assessed on a macro scale since 2014, which showed that the overall quality of the guidelines were high but had poor applicability.1 The guidelines and recommendations are rated on the site itself based on strength of evidence to support them, from Inconclusive (0 stars), Consensus Recommendation (1 star), Limited Recommendation (2 stars), Moderate Recommendation (3 stars), and Strong Recommendation (4 stars). The different guidelines are grouped into four major areas: Upper Extremity Programs, Lower Extremity Programs, Pediatric Programs, and Tumor, Infection, and Military Medicine Programs.8, 9, 10
To methodically assess these guidelines, the updated 2017 AGREE II (Advancing Guideline Development, Reporting, and Evaluation in Health Care) tool by the Canadian Institutes of Health Research was utilized. AGREE II is a widely utilized and validated method (Brouwers2010, Brouwers2016) that has been previously used in assessing guidelines ranging from pediatric fever management, TBI management, to atopic dermatitis management. The AGREE II assessment questionnaire spans six domains and is composed of 23 different items with an additional overall assessment item.11, 12, 13, 14, 15 Our goal was to evaluate the updated 2023 AAOS clinical practice guidelines using the well-established Advancing Guideline Development, Reporting and Evaluation in Health Care (AGREE II) tool.
2. Methods
2.1. Information sources and eligibility criteria
The guidelines of interest for this study were the AAOS CPGs, which were accessed for free through the AAOS website (https://www.aaos.org/quality/quality-programs/) on a search done on April 23, 2023. Inclusion criteria for the study required the CPG be updated at least 10 years ago, that the focus of the guideline was clearly listed (i.e., diagnosis, prevention, management, treatment, detection), and that the entire guideline document was fully and freely accessible from the website.
2.2. Data extraction and guideline evaluation procedures
Using the AAOS website, each CPG's title was extracted along with the most recent year of update, the number of recommendations per guideline, the focus of guideline, the total size of the accompanying guideline document, the endorsements and/or collaborations for each guideline, and the star rating for each guideline recommendation as determined by the AAOS.
The 2017 AGREE II (Advancing Guideline Development, Reporting, and Evaluation in Health Care) tool by the Canadian Institutes of Health Research was utilized to evaluate the online AAOS guidelines. Three fellowship-trained orthopedic surgeons (JD, SB, DH) independently assessed (https://www.agreetrust.org/) the 28 guidelines. Each was provided with the Overview Tutorial and Practice Exercise for AGREE II available for free on the website, as well as a reference summary sheet of the 23 AGREE II items and a breakdown of how the rating system works. Briefly, each item asks reviewers to rate the guideline at hand on a 7-point Likert scale. The lowest rating of 1 (strongly disagree) indicates either no relevant information to the AGREE II item, a very poorly reported concept, or if stated criteria were obviously not met. The highest rating of 7 (strongly agree) indicates exceptional quality of reporting, stated criteria are fully met, and all relevant considerations are addressed and met. Domain 1 (Scope and Purpose) contains 3 items and addresses the general purpose of the guideline and the target population. Domain 2 (Stakeholder Involvement) contains 3 items and addresses the developers of the guidelines and if they are representative of intended users and their views. Domain 3 (Rigor of Development) contains 8 items, the most of any domain, and focuses on the processes used to gather evidence and make or update the recommendations. Domain 4 (Clarity of Presentation) contains 3 items and addresses the ease of understanding, language, and formatting of the guidelines. Domain 5 (Applicability) contains 4 items, and it addresses barriers and facilitators to implementation of the guidelines. Domain 6 (Editorial Independence) contains 2 items and addresses potential biases and competing user interests with regard to the guideline developments. Finally, a global Overall Assessment score, room for overall comments, and a “Yes/No” choice if the reviewer is to recommend the guideline are available at the end of each 23-item guideline assessment.
2.3. Data analysis
Data were analyzed using SPSS V.29.0 software. Descriptive, statistical, and comparative statistics of the data were performed. Statistics done were all derived from mean scores from each AGREE II item across evaluators. Mean, median, SD, and variance were calculated for each individual item as well as an overall score that encompassed all six domains for all 28 guidelines. Percentages were reported out of total possible score.
Interrater reliability between the three evaluators was calculated using the intraclass correlation coefficient (ICC) for a two-way random effects model. Using the ICC scores, level of agreement was classified using the following commonly used cut-offs: poor (<0.40), fair (0.40–0.59), good (0.60–0.74), and excellent (0.75–1.00).16
3. Results
A compilation of the descriptive characteristics for the 28 current AAOS CPGs was created, including the most recent year of update, the number of recommendations per guideline, the stated focus of the guideline, the size of the guideline document itself, and the endorsements from various associations for each guideline (Table 1). 16 CPGs have a supplementary “Appropriate Use Criteria” (AUC) tool associated with them that is freely accessible on the guideline page itself. The CPG with the most recommendations (35) as well as the longest document (986 pages) was found to be “Carpal Tunnel Syndrome.” The CPG with the fewest recommendations (1) was “Evaluation of Psychosocial Factors Influencing Recovery from Adult Orthopaedic Trauma.” The CPG with the smallest document size (21 pages) was “Tranexamic Acid in Total Joint Arthroplasty.” Five CPGs and their corresponding AUC tools were developed in collaboration with the Major Extremity Trauma Research Consortium (METRC) and funded through a United States Department of Defense research grant.
Table 1.
Select characteristics of updated AAOS guidelines.
| AAOS Guideline Title | Year | Number of Recommendations | Focus of Guideline | Size | Endorsed By |
|---|---|---|---|---|---|
| Acute Compartment Syndrome | 2018 | 15a | Diagnosis, Management | 134 pp. | ACS, AOFAS, SOMOS Collaboration with METRC |
| Anesthesia and Analgesia in Total Joint Arthroplasty (2020) | 2020 | 20 | Treatment | 76 pp. | Developed by AAHKS, ASRPMA, AAOS, THS, TKS |
| Anesthesia and Analgesia in Total Joint Arthroplasty (2021) | 2021 | 35 | Treatment | 95 pp. | Developed by AAHKS, ASRAPM, AAOS, THS, TKS |
| Carpal Tunnel Syndrome (update pending) | 2016 | 35a | Management | 986 pp. | ASSH, ASPS, ACR, ACS, ASA |
| Clinical Practice Guideline for the Treatment of Clavicle Fractures | 2022 | 14 | Treatment | 55 pp. | ASES, ASSET, OTA |
| Detection and Nonoperative Management of Pediatric Developmental Dysplasia of the Hip in Infants Up to 6 Months of Age | 2022 | 9a | Detection, Management | 43 pp. | None listed |
| Diagnosis and Prevention of Periprosthetic Joint Infections | 2019 | 25 | Diagnosis, Prevention | 69 pp. | ACR, IDSA, SNMMI |
| Diagnosis and Treatment of Osteochondritis Dissecans | 2010 | 16a | Diagnosis, Treatment | 182 pp. | None listed |
| Evaluation of Psychosocial Factors Influencing Recovery from Adult Orthopaedic Trauma | 2019 | 1a | Management | 36 pp. | OTA, SCCM, AAPM&R Collaboration with METRC |
| Limb Salvage or Early Amputation | 2019 | 14a | Treatment | 54 pp. | AOFAS, OTA Collaboration with METRC |
| Management of Anterior Cruciate Ligament Injuries | 2022 | 15 | Management | 74 pp. | AMSSM, AAPM&R |
| Management of Distal Radius Fractures | 2021 | 7a | Treatment | 38 pp. | AAHS, ASHT Collaboration with ASSH |
| Management of Glenohumeral Joint Osteoarthritis | 2020 | 29a | Treatment | 76 pp. | ASES, AOSSM, ASSET, ACR |
| Management of Hip Fractures in Older Adults | 2021 | 19a | Management | 68 pp. | AAHKS, OTA, APTA, AAPM&R |
| Management of Osteoarthritis of the Hip (update pending) | 2017 | 23a | Management | 854 pp. | POSNA, APTA, ACR, ASA |
| Management of Osteoarthritis of the Knee (Non-Arthroplasty) | 2021 | 29a | Management | 126 pp. | AAHKS, APTA |
| Management of Rotator Cuff Injuries | 2019 | 33a | Treatment | 79 pp. | AANA, ASES, AOSSM, ASSET, APTA |
| Management of Surgical Site Infections | 2018 | 15a | Management | 54 pp. | POSNA, AANA, APTA, MIS, OTA |
| Pharmacologic, Physical, and Cognitive Pain Alleviation for Musculoskeletal Extremity/Pelvis Surgery | 2021 | 31a | Management | 103 pp. | SOMOS, OTA Collaboration with METRC |
| Prevention of Orthopaedic Implant Infection in Patients Undergoing Dental Procedures | 2012 | 3 | Prevention | 325 pp. | Collaboration with ADA |
| Prevention of Surgical Site Infection After Major Extremity Trauma | 2022 | 20a | Prevention | 70 pp. | ASES, POSNA, AOFAS, IDSA Collaboration with METRC |
| Surgical Management of Osteoarthritis of the Knee | 2022 | 23 | Treatment | 108 pp. | AANA, AAHKS |
| Tranexamic Acid in Total Joint Arthroplasty | 2018 | 8 | Management | 21 pp. | Developed by AAHKS |
| Treatment of Metastatic Carcinoma and Myeloma of the Femur | 2020 | 13 | Treatment | 54 pp. | Joint guideline with MSTS, ASTRO, ASCO |
| Treatment of Pediatric Diaphyseal Femur Fractures | 2020 | 7 | Treatment | 36 pp. | None listed |
| Treatment of Pediatric Supracondylar Humerus Fractures | 2011 | 14a | Treatment | 249 pp. | None listed |
| Use of Imaging Prior to Referral to a Musculoskeletal Oncologist | 2018 | 25 | Management | 249 pp. | Developed by MSTS |
| Venous Thromboembolic Disease in Elective TKA and THA | 2019 | 7 (orthopaedic surgery specific) | Prevention | 47 pp. | Developed by ASH |
AAOS = American Academy of Orthopaedic Surgeons ⎟ ACS = American College of Surgeons ⎟ AOFAS = American Orthopaedic Foot and Ankle Society ⎟ SOMOS = Society of Military Orthopaedic Surgeons ⎟ METRC = Major Extremity Trauma Research Consortium ⎟ AAHKS = American Association of Hip and Knee Surgeons ⎟ ASRAPM = American Society of Regional Anesthesia and Pain Medicine ⎟ THS = The Hip Society ⎟ TKS = The Knee Society ⎟ ASSH = American Society for Surgery of the Hand ⎟ ASPS = American Society of Plastic Surgeons ⎟ ACR = American College of Radiology ⎟ ASA = American Society of Anesthesiologists ⎟ ASES = American Shoulder and Elbow Surgeons ⎟ ASSET = American Society of Shoulder and Elbow Therapists ⎟ OTA = Orthopaedic Trauma Association ⎟ IDSA = Infectious Diseases Society of America ⎟ SNMMI = Society of Nuclear Medicine and Molecular Imaging ⎟ SCCM = Society of Critical Care Medicine ⎟ AAPM&R = The American Academy of Physical Medicine and Rehabilitation ⎟ AMSSM = American Medical Society for Sports Medicine ⎟ AAHS = American Association for Hand Surgery ⎟ ASHT = American Society of Hand Therapists ⎟ AOSSM = American Orthopaedic Society for Sports Medicine ⎟ APTA = American Physical Therapy Association ⎟ POSNA = Pediatric Orthopaedic Society of North America ⎟ AANA = Arthroscopy Association of North America ⎟ MIS = Medical Intelligence Society ⎟ ADA = American Dental Association ⎟ MSTS = Musculoskeletal Tumor Society ⎟ ASTRO = American Society for Radiation Oncology ⎟ ASCO = American Society of Clinical Oncology ⎟ TKA = Total Knee Arthroplasty ⎟ THA = Total Hip Arthroplasty ⎟ AHS = American Society of Hematology.
Includes an additional Appropriate Use Criteria tool accessible on guideline page.
Each guideline's recommendations had star ratings associated with them on the AAOS website as well (Table 2). The total number of individual recommendations was 505 across 28 CPGs. The CPGs with the highest percentage of recommendations falling in the “Strong Recommendation” category were Tranexamic Acid in Total Joint Arthroplasty (62%), Management of Hip Fractures in Older Adults (58%), and Surgical Management of Osteoarthritis of the Knee (43%). The CPGs with the highest percentage of recommendations falling in the “Inconclusive” category were Diagnosis and Treatment of Osteochondritis Dissecans (63%), Treatment of Pediatric Supracondylar Humerus Fractures (57%), and Prevention of Orthopaedic Implant Infection in Patients Undergoing Dental Procedures (33%). Overall, of the 505 total given recommendations, 4% (19) were Inconclusive, 20% (103) were Consensus Recommendations, 25% (126) were Limited Recommendations, 28% (140) were Moderate Recommendations, and 23% (117) were Strong Recommendations.
Table 2.
Breakdown of recommendation strengths for updated AAOS guidelines.
| AAOS Guideline Title | Total Recommendations | Inconclusive (% of Total) | Consensus (% of Total) | Limited (% of Total) | Moderate (% of Total) | Strong (% of Total) |
|---|---|---|---|---|---|---|
| Acute Compartment Syndrome | 15 | 0 (0) | 8 (53) | 4 (27) | 3 (20) | 0 (0) |
| Anesthesia and Analgesia in Total Joint Arthroplasty (2020) | 20 | 0 (0) | 2 (10) | 2 (10) | 10 (50) | 6 (30) |
| Anesthesia and Analgesia in Total Joint Arthroplasty (2021) | 35 | 0 (0) | 6 (17) | 5 (14) | 12 (34) | 12 (34) |
| Carpal Tunnel Syndrome (update pending) | 35 | 0 (0) | 0 (0) | 13 (37) | 13 (37) | 9 (26) |
| Clinical Practice Guideline for the Treatment of Clavicle Fractures | 14 | 0 (0) | 2 (14) | 8 (57) | 3 (21) | 1 (7) |
| Detection and Nonoperative Management of Pediatric Developmental Dysplasia of the Hip in Infants Up to 6 Months of Age | 9 | 0 (0) | 0 (0) | 5 (56) | 3 (33) | 1 (11) |
| Diagnosis and Prevention of Periprosthetic Joint Infections | 25 | 0 (0) | 6 (24) | 10 (40) | 6 (24) | 3 (12) |
| Diagnosis and Treatment of Osteochondritis Dissecans | 16 | 10 (63) | 4 (25) | 2 (13) | 0 (0) | 0 (0) |
| Evaluation of Psychosocial Factors Influencing Recovery from Adult Orthopaedic Trauma | 1 | 0 (0) | 0 (0) | 0 (0) | 1 (100) | 0 (0) |
| Limb Salvage or Early Amputation | 14 | 0 (0) | 3 (21) | 4 (29) | 6 (43) | 1 (7) |
| Management of Anterior Cruciate Ligament Injuries | 15 | 0 (0) | 1 (7) | 6 (40) | 3 (20) | 5 (33) |
| Management of Distal Radius Fractures | 7 | 0 (0) | 1 (14) | 2 (29) | 2 (29) | 2 (29) |
| Management of Glenohumeral Joint Osteoarthritis | 29 | 0 (0) | 16 (55) | 1 (3) | 6 (21) | 6 (21) |
| Management of Hip Fractures in Older Adults | 19 | 0 (0) | 0 (0) | 3 (16) | 5 (26) | 11 (58) |
| Management of Osteoarthritis of the Hip (update pending) | 23 | 0 (0) | 5 (22) | 6 (26) | 8 (35) | 4 (17) |
| Management of Osteoarthritis of the Knee (Non-Arthroplasty) | 29 | 0 (0) | 2 (7) | 11 (38) | 8 (28) | 8 (28) |
| Management of Rotator Cuff Injuries | 33 | 0 (0) | 7 (21) | 5 (15) | 9 (27) | 12 (36) |
| Management of Surgical Site Infections | 15 | 0 (0) | 5 (33) | 3 (20) | 4 (27) | 3 (20) |
| Pharmacologic, Physical, and Cognitive Pain Alleviation for Musculoskeletal Extremity/Pelvis Surgery | 31 | 0 (0) | 1 (3) | 16 (52) | 6 (19) | 8 (26) |
| Prevention of Orthopaedic Implant Infection in Patients Undergoing Dental Procedures | 3 | 1 (33) | 1 (33) | 1 (33) | 0 (0) | 0 (0) |
| Prevention of Surgical Site Infection After Major Extremity Trauma | 20 | 0 (0) | 1 (5) | 5 (25) | 9 (45) | 5 (25) |
| Surgical Management of Osteoarthritis of the Knee | 23 | 0 (0) | 2 (9) | 5 (22) | 6 (26) | 10 (43) |
| Tranexamic Acid in Total Joint Arthroplasty | 8 | 0 (0) | 0 (0) | 0 (0) | 3 (38) | 5 (62) |
| Treatment of Metastatic Carcinoma and Myeloma of the Femur | 13 | 0 (0) | 9 (69) | 1 (8) | 2 (15) | 1 (8) |
| Treatment of Pediatric Diaphyseal Femur Fractures | 7 | 0 (0) | 0 (0) | 4 (57) | 1 (14) | 2 (29) |
| Treatment of Pediatric Supracondylar Humerus Fractures | 14 | 8 (57) | 2 (14) | 2 (14) | 2 (14) | 0 (0) |
| Use of Imaging Prior to Referral to a Musculoskeletal Oncologist | 25 | 0 (0) | 15 (60) | 1 (4) | 7 (28) | 2 (8) |
| Venous Thromboembolic Disease in Elective TKA and THA |
7 (orthopaedic surgery specific) |
0 (0) |
4 (57) |
1 (14) |
2 (29) |
0 (0) |
| Total | 505 | 19 (4) | 103 (20) | 126 (25) | 140 (28) | 117 (23) |
AAOS = American Academy of Orthopaedic Surgeons ⎟ TKA = Total Knee Arthroplasty ⎟ THA = Total Hip Arthroplasty.
The quality of the AAOS CPGs was found to be moderate on average across the board, with guideline scores varying but consistently being rated within 25%–75% (Table 3). The guideline assessed to be of overall highest quality was “Management of Hip Fractures in Older Adults” (73%), and the guideline assessed the poorest was “Prevention of Orthopaedic Implant Infection in Patients Undergoing Dental Procedures” (40%). A more global analysis of the AGREE II domain scores across the 28 guidelines was performed and interrater agreeance was determined using the ICC (Table 4). This analysis found no significant difference between all six AGREE II domain results and the Overall Assessment item as well. The mean and median results relative to total possible score were variable, but ultimately all final scores fell between 48% and 87%: scope and purpose (mean 85%, median 85%), stakeholder involvement (mean 87%, median 85%), rigor of development (mean 87%, median 86%), clarity of presentation (mean 87%, median 86%), applicability (mean 48%, median 46%), editorial independence (mean 87%, median 87%), overall assessment (mean 80%, median 84%). ICC was calculated to assess the general agreeance of the evaluators’ AGREE II scores, and they were found to be between poor (<0.40) and fair (0.40–0.59).
Table 3.
Individual AAOS guidelines AGREE II domain scores organized in ascending order of average AGREE II score.
| AAOS Guideline Title | Scope and Purpose |
Stakeholder Involvement |
Rigor of Development |
Clarity of Presentation |
Applicability |
Editorial Independence |
Average AGREE II |
|---|---|---|---|---|---|---|---|
| Avg. AGREE II Score ±95% CI (Avg. %) | Avg. AGREE II Score ±95% CI (Avg. %) | Avg. AGREE II Score ±95% CI (Avg. %) | Avg. AGREE II Score ±95% CI (Avg. %) | Avg. AGREE II Score ±95% CI (Avg. %) | Avg. AGREE II Score ±95% CI (Avg. %) | Avg. AGREE II Score ±95% CI (Avg. %) | |
| Prevention of Orthopaedic Implant Infection in Patients Undergoing Dental Procedures | 5.30 ± 0.91 (82) | 5.32 ± 1.24 (83) | 5.31 ± 1.22 (84) | 5.35 ± 1.12 (85) | 2.32 ± 0.47 (41) | 5.34 ± 1.10 (85) | 5.36 ± 1.11 (85) |
| Acute Compartment Syndrome | 5.15 ± 1.25 (80) | 5.19 ± 0.93 (81) | 5.20 ± 0.96 (82) | 5.22 ± 0.97 (83) | 2.48 ± 0.39 (45) | 5.21 ± 0.98 (82) | 5.20 ± 0.99 (82) |
| Management of Osteoarthritis of the Knee (Non-Arthroplasty) | 5.89 ± 1.08 (95) | 5.88 ± 1.03 (94) | 5.86 ± 1.11 (93) | 5.87 ± 1.09 (94) | 2.94 ± 0.34 (54) | 5.85 ± 1.05 (94) | 5.84 ± 1.04 (93) |
| Use of Imaging Prior to Referral to a Musculoskeletal Oncologist | 5.26 ± 1.21 (84) | 5.23 ± 1.29 (82) | 5.24 ± 1.13 (83) | 5.27 ± 1.15 (84) | 2.51 ± 0.45 (46) | 5.28 ± 1.14 (84) | 5.29 ± 1.13 (84) |
| Treatment of Pediatric Supracondylar Humerus Fractures | 5.17 ± 0.88 (81) | 5.18 ± 0.85 (81) | 5.17 ± 0.89 (81) | 5.18 ± 0.83 (81) | 2.17 ± 0.38 (40) | 5.19 ± 0.81 (82) | 5.16 ± 0.82 (80) |
| Treatment of Metastatic Carcinoma and Myeloma of the Femur | 5.21 ± 1.14 (83) | 5.25 ± 1.02 (83) | 5.28 ± 1.01 (84) | 5.30 ± 1.03 (84) | 2.41 ± 0.37 (43) | 5.29 ± 1.02 (84) | 5.28 ± 1.00 (83) |
| Clinical Practice Guideline for the Treatment of Clavicle Fractures | 5.41 ± 0.97 (85) | 5.42 ± 1.11 (85) | 5.46 ± 1.15 (86) | 5.44 ± 1.11 (86) | 2.58 ± 0.35 (49) | 5.45 ± 1.13 (86) | 5.43 ± 1.14 (86) |
| Evaluation of Psychosocial Factors Influencing Recovery from Adult Orthopaedic Trauma | 5.48 ± 1.09 (86) | 5.47 ± 0.92 (86) | 5.50 ± 0.94 (87) | 5.49 ± 0.90 (87) | 2.89 ± 0.33 (53) | 5.48 ± 0.89 (87) | 5.47 ± 0.87 (87) |
| Diagnosis and Treatment of Osteochondritis Dissecans | 5.33 ± 0.94 (84) | 5.34 ± 1.05 (85) | 5.36 ± 1.06 (85) | 5.38 ± 1.07 (86) | 2.52 ± 0.36 (46) | 5.37 ± 1.08 (85) | 5.38 ± 1.09 (86) |
| Pharmacologic, Physical, and Cognitive Pain Alleviation for Musculoskeletal Extremity/Pelvis Surgery | 5.22 ± 1.11 (83) | 5.21 ± 0.97 (82) | 5.23 ± 1.04 (83) | 5.26 ± 1.06 (84) | 2.46 ± 0.40 (44) | 5.24 ± 1.04 (83) | 5.25 ± 1.03 (83) |
| Diagnosis and Prevention of Periprosthetic Joint Infections | 5.64 ± 0.96 (89) | 5.62 ± 1.19 (89) | 5.61 ± 1.12 (88) | 5.63 ± 1.10 (88) | 2.72 ± 0.31 (51) | 5.62 ± 1.11 (88) | 5.60 ± 1.12 (88) |
| Management of Distal Radius Fractures | 5.28 ± 1.00 (84) | 5.27 ± 0.90 (83) | 5.29 ± 0.92 (84) | 5.32 ± 0.94 (85) | 2.35 ± 0.38 (42) | 5.31 ± 0.95 (84) | 5.33 ± 0.96 (85) |
| Treatment of Pediatric Diaphyseal Femur Fractures | 5.50 ± 1.19 (87) | 5.51 ± 1.13 (87) | 5.53 ± 1.19 (88) | 5.55 ± 1.18 (88) | 2.67 ± 0.34 (50) | 5.54 ± 1.15 (88) | 5.53 ± 1.16 (88) |
| Management of Surgical Site Infections | 5.53 ± 0.95 (88) | 5.54 ± 0.99 (88) | 5.52 ± 0.98 (88) | 5.53 ± 0.89 (88) | 2.62 ± 0.33 (49) | 5.52 ± 0.87 (87) | 5.50 ± 0.86 (87) |
| Management of Osteoarthritis of the Hip (update pending) | 5.40 ± 1.02 (85) | 5.37 ± 1.20 (85) | 5.40 ± 1.16 (85) | 5.42 ± 1.14 (86) | 2.53 ± 0.39 (47) | 5.41 ± 1.16 (86) | 5.40 ± 1.15 (85) |
| Management of Glenohumeral Joint Osteoarthritis | 5.11 ± 1.31 (80) | 5.12 ± 1.28 (80) | 5.13 ± 1.26 (80) | 5.14 ± 1.24 (80) | 2.19 ± 0.37 (40) | 5.15 ± 1.22 (81) | 5.13 ± 1.20 (80) |
| Limb Salvage or Early Amputation | 5.70 ± 0.86 (91) | 5.69 ± 0.88 (91) | 5.71 ± 0.90 (92) | 5.72 ± 0.88 (92) | 2.92 ± 0.32 (53) | 5.73 ± 0.84 (92) | 5.74 ± 0.83 (92) |
| Carpal Tunnel Syndrome (update pending) | 5.56 ± 1.05 (89) | 5.57 ± 1.04 (88) | 5.56 ± 1.06 (88) | 5.59 ± 1.07 (89) | 2.64 ± 0.34 (49) | 5.58 ± 1.08 (89) | 5.57 ± 1.09 (89) |
| Venous Thromboembolic Disease in Elective TKA and THA | 5.60 ± 1.22 (90) | 5.59 ± 1.10 (89) | 5.58 ± 1.14 (89) | 5.57 ± 1.12 (89) | 2.56 ± 0.36 (48) | 5.56 ± 1.13 (89) | 5.55 ± 1.10 (88) |
| Detection and Nonoperative Management of Pediatric Developmental Dysplasia of the Hip in Infants Up to 6 Months of Age | 5.08 ± 1.17 (79) | 5.09 ± 1.15 (79) | 5.10 ± 1.18 (80) | 5.11 ± 1.16 (80) | 2.16 ± 0.35 (39) | 5.12 ± 1.17 (81) | 5.14 ± 1.18 (80) |
| Management of Anterior Cruciate Ligament Injuries | 5.75 ± 1.16 (92) | 5.76 ± 1.07 (92) | 5.77 ± 1.09 (93) | 5.78 ± 1.05 (93) | 2.87 ± 0.30 (52) | 5.79 ± 1.03 (93) | 5.80 ± 1.02 (93) |
| Management of Rotator Cuff Injuries | 5.16 ± 1.03 (81) | 5.17 ± 0.95 (81) | 5.18 ± 0.91 (81) | 5.19 ± 0.92 (82) | 2.24 ± 0.39 (41) | 5.20 ± 0.90 (82) | 5.19 ± 0.88 (82) |
| Surgical Management of Osteoarthritis of the Knee | 5.35 ± 0.99 (84) | 5.36 ± 1.01 (84) | 5.38 ± 1.03 (86) | 5.37 ± 1.01 (85) | 2.44 ± 0.37 (44) | 5.39 ± 0.99 (86) | 5.39 ± 0.98 (85) |
| Prevention of Surgical Site Infection After Major Extremity Trauma | 5.20 ± 1.26 (82) | 5.22 ± 1.16 (82) | 5.21 ± 1.10 (82) | 5.23 ± 1.08 (83) | 2.50 ± 0.38 (45) | 5.22 ± 1.09 (83) | 5.23 ± 1.07 (82) |
| Anesthesia and Analgesia in Total Joint Arthroplasty (2021) | 5.13 ± 1.15 (80) | 5.14 ± 1.12 (80) | 5.15 ± 1.20 (80) | 5.16 ± 1.18 (81) | 2.27 ± 0.36 (42) | 5.17 ± 1.19 (82) | 5.18 ± 1.17 (81) |
| Tranexamic Acid in Total Joint Arthroplasty | 5.19 ± 1.30 (82) | 5.20 ± 1.14 (82) | 5.19 ± 1.05 (81) | 5.21 ± 1.04 (82) | 2.42 ± 0.35 (43) | 5.23 ± 1.06 (83) | 5.24 ± 1.05 (83) |
| Anesthesia and Analgesia in Total Joint Arthroplasty (2020) | 5.67 ± 0.89 (90) | 5.65 ± 0.91 (90) | 5.66 ± 0.87 (90) | 5.68 ± 0.86 (91) | 2.76 ± 0.31 (51) | 5.67 ± 0.85 (90) | 5.66 ± 0.84 (90) |
| Management of Hip Fractures in Older Adults | 5.58 ± 1.11 (89) | 5.55 ± 1.08 (88) | 5.54 ± 1.02 (87) | 5.56 ± 1.00 (88) | 2.60 ± 0.33 (48) | 5.55 ± 1.01 (87) | 5.54 ± 0.99 (88) |
AAOS = American Academy of Orthopaedic Surgeons ⎟ TKA = Total Knee Arthroplasty ⎟ THA = Total Hip Arthroplasty.
Table 4.
AAOS guidelines summary of overall results.
| AGREE II Domain Area | Mean (%) | Median (%) | Variance (%) | ICC (95% CI) |
|---|---|---|---|---|
| Scope and Purpose | 85 | 85 | 5.8 | 0.39 (0.12–0.59) |
| Stakeholder Involvement | 87 | 85 | 6.3 | 0.17 (−0.19 to 0.44) |
| Rigor of Development | 87 | 86 | 6.5 | 0.26 (0.08–0.42) |
| Clarity of Presentation | 87 | 86 | 6.2 | 0.20 (−0.15 to 0.46) |
| Applicability | 48 | 46 | 8.5 | −0.03 (−0.41 to 0.26) |
| Editorial Independence | 87 | 87 | 6.0 | −0.12 (−0.74 to 0.31) |
| Overall Assessment | 80 | 84 | 6.5 | −0.16 (−1.17 to 0.43) |
AAOS = American Academy of Orthopaedic Surgeons ⎟ AGREE II = Advancing Guideline Development, Reporting and Evaluation in Health Care tool II ⎟ ICC = Interclass correlation coefficient.
4. Discussion
The utility and potential benefit to patients and physicians of well-made clinical practice guidelines has been previously established.3 Well-made guidelines generally intend to encourage certain interventions that have an evidence-based benefit, discourage ineffective interventions that may be potentially harmful, discourage unnecessary procedures, and reduce health disparities by providing quick access to the highest quality information. Our study demonstrated no significant difference between evaluators’ assessment of the different components of the AAOS guidelines. Five of the six AGREE II domains, including the overall assessment, tested achieved a median score above the 50% mark.
The AAOS guidelines are held to high standards and frequently scrutinized with rigor when they appear to fall short of providing the best guidance.8,10,17,18 And while increasing numbers of guidelines in theory may be hoped to be positive movement, an increase in quantity without improving quality of existing guidelines is problematic. If clinicians become distrusting of one guideline put forth by the corresponding association, a higher likelihood exists their overall trust and confidence in the guidelines as a whole will decrease. And so it is certainly a fine line to walk for those who put out guidelines to maintain trust while providing up-to-date, relevant, applicable, accessible, clear, evidence-based information for clinicians to reference and incorporate into their practice. The ultimate goal of this all being to improve healthcare practices as a whole. In our study, we aimed to assess the 28 current, updated, and official CPGs provided for free by the AAOS and went about doing so using the updated 2017 AGREE II tool. We found that overall quality of guidelines across the board was moderate across all of the AGREE II domains. In contrast to previous appraisals of the AAOS guidelines,2 we found the applicability domain score to be statistically the same as the scores of the other five domains. We also found that our three credible appraisers sometimes differed significantly in their AGREE II items scoring at times. This fact underscores that these guidelines remain far from overarchingly unifying, and they should continue to be updated to reduce significant professional dissonance in their assessments.
Limitations to this study that warrant consideration are described following. Firstly, the AGREE II tool manual suggests four evaluators for comprehensive guideline assessments. A minimum of two is required, and while we have three appraisers in our study, having below the recommended number of appraisers is a potential limitation. Secondly, our interrater reliability was low, despite the AGREE II manual's instructions being used, and as such perhaps additional evaluators may have been beneficial. This low interrater reliability is a potential limitation to this study, even though differences in assessment of the guidelines were individually justified.
Our study did not identify clear or significant differences across AGREE II domain areas. Many of the domain item scores across the 28 guidelines indicated disagreement, sometimes to a more noteworthy degrees. Our descriptive statistics indicated the median scores rested slightly above the 50% point, which we assessed as being of moderate quality. While neither exceptional nor abysmal, the hovering descriptive statistics around 50% indicated sufficient quality of the guideline to warrant recommendation while still needing improvement – as the AAOS consistently works on. AGREE II scores should not be used as sole determinants of application of CPGs, but rather they are intended to serve a synergistic role in conjunction with clinical decision making, specific clinical context, and comfort level of both physician and patient depending on the situation at hand. However, the AGREE II framework allows for a standardized and easily approachable avenue for assessing guidelines and an increase in AGREE II domain scores will presumably correlate to an increase in the inherent quality of that guideline.
A trend identified was for those CPGs that had a higher percentage of recommendations being Moderate Recommendations or Strong Recommendations, generally the overall assessment of the guidelines was higher. As evidenced by the guidelines: Management of Rotator Cuff Injuries, Management of Hip Fractures in Older Adults, Anesthesia and Analgesia in Total Joint Arthroplasty (2020), Anesthesia and Analgesia in Total Joint Arthroplasty (2021), Tranexamic Acid in Total Joint Arthroplasty, and Prevention of Surgical Site Infection After Major Extremity Trauma among others. Therefore, despite data spread in the individual item scores persisting in those evaluations, the overall quality of the guidelines with stronger conviction of recommendation by the AAOS itself translated into higher appraisals. This fact highlights the need for continuing research and refinement of the guidelines contained more Inconclusive, Consensus Recommendations, or Limited Recommendations. Certainly, there are areas of medicine better studied and therefore with greater bodies of evidence and data to back up recommendations.6 These higher performing guidelines particularly followed trends of higher scores for rigor of development, stakeholder involvement, editorial independence, and scope and purpose. While increases in applicability scores and clarity of presentation did occur, they were not as pronounced as a whole. As previously described,2 no matter the rigor of development and theoretical benefit of the guideline, without the appropriate means of applying the guideline in clinical practice the guideline loses much of its general quality. Notably, despite fluctuations across domain scoring between our appraisers, every guideline ultimately received at least 2/3 “Yes” responses to the final AGREE II question of if they would recommend the guideline. Very few guidelines received a “No” response to this, therefore this indicates despite criticism of the individual guideline AGREE II domains, in the end the benefit of the guideline outweighed the harm in having it available. This fact serves as a reminder that when critically appraising AAOS guidelines, even when certain areas may not meet exceptional standards, the overall guideline development process of the AAOS consistently generates useable CPGs. With cycles of improvement, modification, and new evidence, even the lowest scoring guidelines can be refined and more favorably critiqued.
In order to improve the somewhat frequently tailing AGREE II domain of applicability, some suggestions are to ensure the longitudinal inclusion a public health representative to offer a less technical yet more wholistic view of the guideline. Another is to ensure the theme of the guideline is truly pertinent to practice. Common alternative approaches to the guideline recommendation at hand should also be well-discussed and addressed in order to ensure those clinicians with differing baseline views of that theme have a clearer path to applying the official guideline if so deemed appropriate. Including specific, targeted questions that steer focus group discussions and meetings where guidelines are developed/updated can be beneficial to improving clarity and applicability as well. The continued effort to include more practicing clinicians of diverse backgrounds in these discussions should also help applicability, particularly to communities that may encounter more barriers to having the guidelines followed.19 Continually working towards transparency in the development and updating of CPGs will likely translate to higher domain scores for applicability, clarity of presentation, and editorial independence with expected improvement in scope and purpose, stakeholder involvement, and rigor of development as well, consequently.
This study demonstrated that the overall quality of the AAOS CPGs I high, with disagreement between qualified evaluators on certain domain areas. The guidelines put forth by the AAOS are worth continuing to recommend, even if they may include modifications before use. To most efficiently improve on the overall quality of guidelines, the AGREE II domains of applicability and clarity of presentation should be targeted areas. Including recommendations that have strong evidence as opposed to consensus or limited will also positively influence the assessment of guideline quality. Further increasing diverse involvement in guideline creation, mindfully incorporating interdisciplinary focus group members, and ensuring clear guideline themes with well-thought out considerations are recommended. This will enable improved applicability, clarity of presentation, while maintaining the overall quality of the AAOS guidelines and, in turn, healthcare for orthopaedic patients.
Funding
None.
Acknowledgements
None.
References
- 1.Meiyappan K.P., Cote M.P., Bozic K.J., Halawi M.J. Adherence to the American Academy of Orthopaedic Surgeons clinical practice guidelines for nonoperative management of knee osteoarthritis. J Arthroplasty. 2020;35(2):347–352. doi: 10.1016/j.arth.2019.08.051. [DOI] [PubMed] [Google Scholar]
- 2.Sabharwal S., Patel N.K., Gauher S., Holloway I., Athansiou T. High methodologic quality but poor applicability: assessment of the AAOS guidelines using the AGREE II instrument. Clin Orthop. 2014;472(6):1982–1988. doi: 10.1007/s11999-014-3530-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Grimshaw J.M., Russell I.T. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993;342(8883):1317–1322. doi: 10.1016/0140-6736(93)92244-N. [DOI] [PubMed] [Google Scholar]
- 4.Woolf S.H., Grol R., Hutchinson A., Eccles M., Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ. 1999;318(7182):527–530. doi: 10.1136/bmj.318.7182.527. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Cook D. The trials and tribulations of clinical practice guidelines. JAMA. 1999;281(20):1950. doi: 10.1001/jama.281.20.1950. [DOI] [PubMed] [Google Scholar]
- 6.Ariel Franco J.V., Arancibia M., Meza N., Madrid E., Kopitowski K. Clinical practice guidelines: concepts, limitations and challenges. Medwave. 2020 doi: 10.5867/medwave.2020.03.7887. Published online April 30. [DOI] [PubMed] [Google Scholar]
- 7.Andrade R., Pereira R., Van Cingel R., Staal J.B., Espregueira-Mendes J. How should clinicians rehabilitate patients after ACL reconstruction? A systematic review of clinical practice guidelines (CPGs) with a focus on quality appraisal (AGREE II) Br J Sports Med. 2020;54(9):512–519. doi: 10.1136/bjsports-2018-100310. [DOI] [PubMed] [Google Scholar]
- 8.Leopold S.S. Editorial: the new AAOS guidelines on knee arthroscopy for degenerative meniscus tears are a step in the wrong direction. Clin Orthop. 2022;480(1):1–3. doi: 10.1097/CORR.0000000000002068. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Okoroafor UC, Cannada LK. Do Orthopedic Trauma Surgeons Adhere To Aaos Guidelines When Treating Distal Radius Fractures?. [PMC free article] [PubMed]
- 10.Lubowitz J.H., Provencher M.T., Poehling G.G. Congratulations and condemnations: level I evidence prize for femoral Tunnel position in ACL reconstruction, and AAOS clinical practice guidelines miss the mark—again. Arthrosc J Arthrosc Relat Surg. 2014;30(1):2–5. doi: 10.1016/j.arthro.2013.11.008. [DOI] [PubMed] [Google Scholar]
- 11.Brouwers M.C., Kho M.E., Browman G.P., et al. Agree II: advancing guideline development, reporting and evaluation in health care. Can Med Assoc J. 2010;182(18):E839–E842. doi: 10.1503/cmaj.090449. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Brouwers M.C., Kerkvliet K., Spithoff K., AGREE Next Steps Consortium The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines. BMJ. 2016:i1152. doi: 10.1136/bmj.i1152. Published online March 8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Chiappini E., Bortone B., Galli L., Martino M.D. Guidelines for the symptomatic management of fever in children: systematic review of the literature and quality appraisal with AGREE II. BMJ Open. 2017;7(7) doi: 10.1136/bmjopen-2016-015404. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Lee S.Y., Amatya B., Judson R., et al. Clinical practice guidelines for rehabilitation in traumatic brain injury: a critical appraisal. Brain Inj. 2019;33(10):1263–1271. doi: 10.1080/02699052.2019.1641747. [DOI] [PubMed] [Google Scholar]
- 15.Ghazal S., Ridha Z., D'Aguanno K., et al. Treatment guidelines for atopic dermatitis since the approval of dupilumab: a systematic review and quality appraisal using AGREE-II. Front Med. 2022;9 doi: 10.3389/fmed.2022.821871. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Cicchetti DV. Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology.
- 17.Murad M.H. Clinical practice guidelines. Mayo Clin Proc. 2017;92(3):423–433. doi: 10.1016/j.mayocp.2017.01.001. [DOI] [PubMed] [Google Scholar]
- 18.Enikeev D., Misrai V., Rijo E., et al. EAU, AUA and NICE guidelines on surgical and minimally invasive treatment of benign prostate hyperplasia: a critical appraisal of the guidelines using the AGREE-II tool. Urol Int. 2022;106(1):1–10. doi: 10.1159/000517675. [DOI] [PubMed] [Google Scholar]
- 19.Fervers B., Carretier J., Bataillard A. Clinical practice guidelines. J Vis Surg. 2010;147(6):e341–e349. doi: 10.1016/j.jviscsurg.2010.10.010. [DOI] [PubMed] [Google Scholar]