Description of the Problem
Since the 1980s, there has been a growing emphasis on the use of evidence-based practice guidelines in medicine.1 However, national guidelines are based on evidence of variable quality, including non-evidence-based common medical practice. While attempts are made to ensure impartial recommendations during guideline development, if there is lack of sufficient clinical evidence, some recommendations could be the result of inadvertent bias.2 At times, guidelines from different organizations on the same clinical entity can also contradict each other.3
Even if guidelines are based on well-conducted clinical trials or other unbiased high quality evidence, repeated studies have also shown that physician practice is often not concordant with existing guidelines.4 There is similar lack of radiologists’ adherence to evidence-based imaging guidelines.5 Several explanations have been proposed for the discrepancy between clinical practice and evidence-based guidelines, including lack of awareness of the guidelines, lengthy and unclear guidelines, decreased willingness to change existing practice patterns, absence of embedded guidelines in clinical workflow, and patient preference.6
Lack of physician agreement with portions of clinical guidelines may be an important contributor to why guidelines are often not broadly followed in clinical practice. We therefore conducted a survey of a multispecialty group of expert physicians in our institution to determine their agreement with the 2017 Fleischner Society Guidelines for Management of Incidental Pulmonary Nodules (FSG), the most frequently referenced imaging guidelines for pulmonary nodules.7
What We Did
Like most guidelines, the FSG are a collection of recommendations based on multiple different clinical scenarios. The 2017 FSG were comprised of 18 unique recommendations, which for the purpose of this study were each codified into a “Clinical Evidence Logic Statement”, or CELS, defined as a statement regarding the appropriateness of diagnostic imaging for a certain indication. CELS were formatted as “IF… THEN” statements, a previously-described guideline representation format.8 Representation of recommendations in such a format may simplify the creation of clinical decision support tools to enable and accelerate broad adoption.
For each recommendation, the FSG includes ratings for the strength of evidence supporting the recommendation based on the American Society of Chest Physicians grading system. However, before we assessed radiologist agreement with specific recommendations, two medical librarians from the Harvard Library of Evidence analyzed each CELS independently and graded the recommendation based on the supporting clinical studies using the Oxford Centre for Evidence-based Medicine (OCEBM) Levels of Evidence and the U.S. Preventative Services Task Force I-Scores, an objective and broadly accepted methodology.8 The purpose of internally grading the strength of evidence behind each recommendation, as opposed to relying solely on the FSG’s own rating, was to assess whether physician agreement with each guideline recommendation varied according to the internally- or FSG-ascertained level of evidence supporting a specific recommendation.
A total of 9 physicians were then surveyed via SurveyMonkey to assess agreement with each of the 18 recommendations, or CELS, contained in the FSG. Physicians were invited to participate in the study based on clinical expertise in pulmonary nodule management and/or experience with FSG recommendations, in order to represent a broad range of specialties that would manage patients with incidental pulmonary nodules. Among the 9 survey participants, there were 3 radiologists with subspecialties in thoracic or cancer imaging, 3 thoracic surgeons, and 3 internal medicine and pulmonology physicians. The physicians were given 18 different clinical vignettes, the associated FSG recommendation, the Harvard Library of Evidence grading of evidence, and the FSGs’ own grading of evidence (Figure 1). Survey participants were asked to agree with the FSG recommendation without modification, agree with modification, or disagree. A free text comment box was also provided.
Figure 1.
Example Survey Vignette and Question
The primary outcome was the percent of physicians who agreed with each FSG recommendation and the corresponding CELS. Physicians who chose to modify the FSG recommendation were considered to have disagreed with the recommendation for our analysis. We compared the proportion of physicians who agreed and disagreed with each of the 18 CELS using a chi-square analysis. We also performed subgroup analysis comparing percent agreement for the different nodule types and risk categories outlined in the FSG.
Outcomes and Limitations
All participants completed the survey. The percent agreement between physicians differed considerably across each of the recommendations, or CELS, ranging from 0% agreement to 100% agreement, with an average agreement rate of 48.2% (Table 1). There was significant variation in the proportion of physicians who agreed with each CELS in the FSG (p<0.01). Rates of physician agreement were not associated with FSG grading of the strength of underlying evidence. For instance, 88.9% of physicians agreed with FSG that a single low risk nodule >8 mm should be managed by a follow-up chest CT at 3 months, PET/CT, or tissue sampling; a recommendation that had an FSG grade of 1A (high quality evidence). In contrast, the rate of physician agreement for follow-up management of multiple subsolid pulmonary nodules with the most suspicious being equal to or larger than 6 mm was also 88.9%, although that recommendation had an FSG grade of 1C (low or very low-quality evidence) (p=0.21). Similar analysis was not performed for the OCEBM grading given that all but one recommendation had the same strength of evidence (5 I, based on expert opinion with insufficient evidence).
Table 1.
Physician Agreement with Individual Recommendations/ Clinical Evidence Logic Statements (CELS) from 2017 Fleischer Society Guidelines for the Management of Incidental Pulmonary Nodules
| CELS | Fleischner Guideline | Fleischner Society Recommendation | Oxford Grade | Fleischner Society Grade | Physicians agreeing with statement (%) | P Value |
|---|---|---|---|---|---|---|
| 1 | Single nodule < 6mm, low risk | No routine CT follow-up. | 5-I | 1C | 22.2 | <0.01 |
| 2 | Single nodule 6 –8 mm, low risk | Chest CT at 6–12 months, then consider CT at 18–24 months | 5-I | 1C | 66.7 | |
| 3 | Single nodule >8 mm, low risk | Consider Chest CT at 3 months or PET/CT or tissue sampling | 5-I | 1A | 88.9 | |
| 4 | Single nodule < 6 mm, high risk | Optional Chest CT at 12 months | 5-I | 2A | 0.00 | |
| 5 | Single nodule 6 – 8 mm, high risk | Chest CT at 6–12 months, then CT at 18–24 months | 5-I | 1B | 33.3 | |
| 6 | Single nodule > 8mm, high risk | Consider Chest CT at 3 months or PET/CT or tissue sampling | 5-I | 1A | 100.0 | |
| 7 | Multiple nodules, most suspicious nodule < 6 mm, low risk | No routine CT follow-up | 5-I | 2B | 11.1 | |
| 8 | Multiple nodules, most suspicious nodule 6 – 8 mm, low risk | Chest CT at 3–6 months, then consider CT at 18–24 months | 5-I | 1B | 66.7 | |
| 9 | Multiple nodules, most suspicious nodule > 8 mm, low risk | Chest CT at 3–6 months, then consider CT at 18–24 months | 5-I | 2A | 66.7 | |
| 10 | Multiple nodules, most suspicious nodule < 6 mm, high risk | Optional Chest CT at 12 months | 5-I | Not stated | 0.0 | |
| 11 | Multiple nodules, most suspicious nodule 6 – 8 mm, high risk | Chest CT at 3–6 months, then CT at 18–24 months | 5-I | Not stated | 66.7 | |
| 12 | Multiple nodules, most suspicious nodule > 8 mm, high risk | Chest CT at 3–6 months, then CT at 18–24 months | 5-I | Not stated | 22.2 | |
| 13 | Single ground glass nodule < 6 mm | No routine CT follow-up | 5-NS-contra-dicts | 1B | 22.2 | |
| 14 | Single ground glass nodule >= 6 mm | Chest CT at 6–12 months to confirm persistence, then CT every 2 years until 5 years | 5-I | 1B | 88.9 | |
| 15 | Single part solid nodule < 6 mm | No routine CT follow-up | 5-I | 1C | 0.0 | |
| 16 | Single part solid nodule >= 6 mm | Chest CT at 3–6 months to confirm persistence. If unchanged and solid component remains <6mm, annual CT should be performed for 5 years | 5-I | 1B | 77.8 | |
| 17 | Multiple subsolid nodules, most suspicious < 6 mm | Chest CT at 3–6 months. If stable, consider CT at 2 and 4 years | 5-I | 1C | 44.4 | |
| 18 | Multiple subsolid nodules, most suspicious >= 6 mm | CT at 3–6 months | 5-I | 1C | 88.9 |
Subgroup analysis was performed to determine if physicians showed higher rates of agreement with FSG recommendations for specific subgroups of nodules (Table 2). CELS recommendations were grouped according to their corresponding nodule description provided in the FSG. CELS regarding the management of large nodules had the highest physician-FSG agreement at 76.2%, compared to only 30.3% agreement for small and medium nodules (p<0.01). There was no significant difference in rates of physician agreement with FSG recommendations between high- versus low-risk solid nodules, single versus multiple nodules, or solid versus subsolid nodules. While a few specific FSG recommendations had 100% agreement, when FSG recommendations were grouped according to their broader nodule description, there was no unanimous acceptance of the FSG guidelines by our local expert panel.
Table 2.
Physician Agreement by Pulmonary Nodule Subgroup
| Subgroup | Clinical Evidence Logic Statements (CELS) categories | Agree (%) | P Value |
|---|---|---|---|
| Patient risk | |||
| High Risk Solid Nodules | 1–3 | 59.3 | 0.14 |
| Low Risk Solid Nodules | 4–6 | 44.4 | |
| Number of nodules | |||
| Single Nodule | 1–6, 13–16 | 50.0 | 0.66 |
| Multiple Nodule | 7–12, 17–18 | 45.8 | |
| Solid Appearance | |||
| Solid Nodules | 1–12 | 45.4 | 0.40 |
| Subsolid Nodules | 13–18 | 53.7 | |
| Nodule Size | |||
| Small and Medium Nodules | 1, 2, 4, 5, 7, 8, 10, 11, 13, 15, 17 | 30.3 | <0.01 |
| Large Nodules | 3, 6, 9, 12, 14, 16, 18 | 76.2 |
Our work has several limitations. First, the study surveyed only 9 physicians in a single large academic institution. However, the study was designed to be exploratory and to highlight and test the hypothesis that guideline non-adherence may be partly affected by lack of physician agreement with guideline component recommedations. Despite this important limitation, the survey participants represented a wide variety of specialties that would manage patients with pulmonary nodules, and many of them had extensive experience with the content of the 2017 Fleischner Society Guidelines.
Guidelines are composed of multiple unique recommendations and the strength of the underlying evidence supporting each recommendation can vary considerably. The factors that underlie rates of adherence to guidelines are multiple and include the strength of the underlying evidence, the risks and benefits associated with the disease, and as we discussed in this paper, physician agreement with the underlying clinical recommendations embedded within those guidelines. The issue of local experts disagreeing with ‘national’ guidelines, especially those based primarily on professional society consensus rather than high quality scientific evidence, may be a widespread issue that would benefit from further research.
Funding Source(s) Statement:
This study was partly funded by the Association of University Radiologists (AUR) GE Radiology Research Academic Fellowship (GERRAF), and grant R01HS024722 from the Agency for Healthcare Research and Quality (AHRQ).
Footnotes
Conflict(s) of Interest Statement: The authors declare no conflicts of interest.
The authors declare that they had full access to all of the data in this study and the authors take complete responsibility for the integrity of the data and the accuracy of the data analysis.
References
- 1.Guiberteau MJ, Wilcox PA & Bjork S Practice guidelines: the radiology perspective. J. Am. Coll. Radiol 1, 92–97 (2004). [DOI] [PubMed] [Google Scholar]
- 2.Neuman J, Korenstein D, Ross JS & Keyhani S Prevalence of financial conflicts of interest among panel members producing clinical practice guidelines in Canada and United States: cross sectional study. BMJ 343, (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Kachalia A & Mello MM Breast Cancer Screening: Conflicting Guidelines and Medicolegal Risk. JAMA 309, 2555–2556 (2013). [DOI] [PubMed] [Google Scholar]
- 4.Cabana MD Why Don’t Physicians Follow Clinical Practice Guidelines?: A Framework for Improvement. JAMA J. Am. Med. Assoc 282, 1458–1465 (1999). [DOI] [PubMed] [Google Scholar]
- 5.Bobbin MD, Ip IK, Sahni VA, Shinagare AB & Khorasani R Focal Cystic Pancreatic Lesion Follow-up Recommendations After Publication of ACR White Paper on Managing Incidental Findings. J. Am. Coll. Radiol. JACR 14, 757–764 (2017). [DOI] [PubMed] [Google Scholar]
- 6.Timmermans S & Mauck A The Promises And Pitfalls Of Evidence-Based Medicine. Health Aff. (Millwood) 24, 18–28 (2005). [DOI] [PubMed] [Google Scholar]
- 7.Bueno J, Landeras L & Chung JH Updated Fleischner Society Guidelines for Managing Incidental Pulmonary Nodules: Common Questions and Challenging Scenarios. RadioGraphics 38, 1337–1350 (2018). [DOI] [PubMed] [Google Scholar]
- 8.Lacson R et al. Assessing Strength of Evidence of Appropriate Use Criteria for Diagnostic Imaging Examinations. J. Am. Med. Inform. Assoc 23, 649–653 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]