Tips for Teachers of Evidence-based Medicine: Making Sense of Diagnostic Test Results Using Likelihood Ratios

INTRODUCTION

Now is an exciting time to be or become a diagnostician. More diagnostic tests, including portions of the medical interview and physical examination, are being studied rigorously for their accuracy, precision, and usefulness in practice,^1,2 and this research is increasingly being systematically reviewed and synthesized.^3,4 Diagnosticians are gaining increasing access to this research evidence, raising hope that this knowledge will inform their diagnostic decisions and improve their patients’ clinical outcomes.⁵ For patients to benefit fully from this accumulating knowledge, the diagnosticians serving them must be able to reason probabilistically, to understand how test results can revise disease probability to confirm or exclude disorders, and to integrate this reasoning with other types of knowledge and diagnostic thinking.^6–8

Yet, clinicians encounter several barriers when trying to integrate research evidence into clinical diagnosis.⁹ Some barriers involve difficulties in understanding and using the quantitative measures of tests’ accuracy and discriminatory power, including sensitivity, specificity, and likelihood ratios (LRs).^9,10 We have noticed that LRs are particularly troubling to many learners at first, and we have wondered if this is because of the way they have been taught. Stumbling blocks can arise in several places when learning LRs: the names and formulae themselves can be intimidating; the arithmetic functions can be mystifying when attempted all at once; if two levels of test results are taught first, learners can have difficulty ‘stretching’ to multiple levels; and if disease probability is framed in odds terms (to directly multiply the odds by the likelihood ratio), learners can misunderstand why and how this conversion is done. Other stumbling blocks may occur as well.

Other authors have described various approaches to helping clinicians understand LRs.^11–16 In this article, we describe two additional approaches to help clinical learners understand how LRs describe the discriminatory power of test results. Whereas we mention other concepts such as pretest and posttest probability, full treatment of those subjects is beyond the scope of this article. These approaches were developed by experienced teachers of evidence-based medicine (EBM) and were refined over years of teaching practice. These tips have also been field-tested to double-check the clarity and practicality of these descriptions, as explained in the introductory article of this series.¹⁷

To help the reader envision these teaching approaches, we present sequenced advice for teachers in plain text, coupled with sample words to speak, in italics. These scripts are meant to be interactive, which means that teachers should periodically check in with the learners for their understanding and that teachers should try other ways to explain the ideas if the words we have suggested do not “click.” We present them in order from shorter to longer; however, because these 2 scripts cover the same general content, we encourage teachers to use either or both in an order that best fits their setting and learners.

TEACHING TIP 1: STARTING WITH THE NAME, USING THE ‘CONCEPTUAL ALGEBRA’ APPROACH

When to Use This Tip

This script can work for learners at all levels of experience with both clinical medicine and EBM. Although groups vary in their pace of understanding, teachers can anticipate spending 5–10 minutes on this tip. The general objective is to introduce learners to the use of LRs as measures of diagnostic test accuracy, with the following specific objectives:

• Learn that LRs simply compare two likelihoods—the frequency of a test result in those with the target disorder compared to the frequency of the same test result in those without the disease

• Learn to interpret the meaning of three ‘zones’ of LR values—above 1, below 1, and equal to 1

• Learn to express the meaning of a given LR using words, rather than numbers and formulae

This script begins with the potentially intimidating name itself, parses it into less threatening words that can be understood more readily, and skips the arithmetic and formula altogether by concentrating on understanding how to interpret LRs and communicate them in language.

Preparing to Teach

This script does not require preparation of materials ahead of time. We present this approach scripted for use on a white board, but we find it works equally well in other media, such as flip charts or overhead transparencies.

The Script

Start with an empty white board: This article (or other source) mentions the use of a LR for making sense of the results of this diagnostic test. What is a likelihood ratio? Write “Likelihood Ratio” on the board and wait patiently while the learners ponder this, but be prepared for silence: What kind of arithmetic problem is a ratio? Again, wait patiently: Right, it’s a division problem. Write on the board an equal sign to the right of “Likelihood Ratio” followed by a short horizontal line. A LR compares two likelihoods by dividing one by the other. Write “L1” above your horizontal line and “L2” below the line.

To the right of this, write another equal sign and draw a much longer horizontal line. During clinical diagnosis, we retrieve the results of diagnostic tests. For each result, the LR compares how likely that result is to occur in patients with disease (write “ ‘test result’ in disease” above the line) to how likely that same result is to occur in patients without disease (write “ ‘test result’ in no disease” below the line). Your board should now show:

When the group is ready, move on: There are three possible results of this ratio. Write ‘LR > 1’ on the board. Ask: When the ratio is greater than one, is the numerator or denominator larger? Right, this test result will occur more often in patients with the disease. To the right of ‘LR > 1’, write ‘ ↑ in disease’. Underneath ‘LR > 1’, write ‘LR < 1’. Ask: When the LR is less than one, is the numerator or the denominator smaller? Right, so this test result will occur more often in patients without disease. To the right of this inequality, write ‘ ↑ in no disease’. Underneath ‘LR < 1’, write ‘LR = 1’. Ask: When the LR equals one, how do the numerator and denominator compare? Right, they are equal, so this test result occurs just as often in those with disease as those without. For clinical diagnosis, a LR at or near one means this test result would not change the probability of disease. The farther a LR is from one, in either direction, the more it changes the probability of disease. Clarify as needed.

When your learners are ready, move on: Write on the board ‘LR = 5’. Let’s practice putting this ratio into English. If a LR is 5, what does that mean? Be prepared to wait: Well, is the numerator or the denominator larger? Right, by how much is it larger? So, this would mean that the test result occursfive times more often in those with disease than in those without. Repeat a few times with other LR values and clarify as needed.

Additional Comments for Tip #1

By using this ‘conceptual algebra’ technique, we de-emphasize the arithmetic and emphasize the concepts, and we find most groups of learners can understand the idea well enough to use LRs. We particularly select this approach when the name ‘likelihood ratio’ has already been introduced before learners have had the opportunity to work through the ‘split column’ approach of Script #2.

The Bottom Line

• LRs compare two likelihoods—the frequency of a test result in those with the target disorder compared with the frequency of the same test result in those without the disease.

• LR values above 1 mean the test result occurs more often in those with disease, whereas values below 1 mean the result occurs more often in those without disease. Values equal to 1 mean the result occurs equally often in those with and without the disorder.

• Expressing a LR in words can help us understand its meaning and communicate clearly to others.

See Appendix 1 for the summary card for this tip.

TEACHING TIP 2: STARTING FROM A RESULTS TABLE, USING THE ‘SPLIT COLUMN’ APPROACH

When to Use This Tip

This script can work for learners at all levels of experience with both clinical medicine and EBM. It is particularly well suited for fresh beginners, as it aims to build their understanding before fear of the arithmetic, the formulae, or the jargon terms can block their learning. Whereas learners vary in their pace of understanding, teachers should anticipate spending 15–20 minutes for the lessons of ‘Page 1’ and another 5–10 minutes for ‘Page 2’. The general objective is to introduce learners to the use of LRs as measures of diagnostic test accuracy, with the following specific objectives:

• Learn how to calculate the LRs for test results directly from the data table, using two steps of simple division and without using formulae

• Learn how to use the LRs to revise the probability of disease

• Learn how the LRs of a given test, i.e., its discriminatory power, change when multiple levels of test result are ‘collapsed down’ to two levels

This ‘Split Column’ approach starts with multilevel test result data, separates the two steps of simple division, emphasizes the concepts and comparisons, and then shows how the same method can be used for two-level test result data.

Preparing to Teach

Select for your teaching example a high quality study of the accuracy of a diagnostic test whose results can be portrayed in three or more levels, whether as intervals of a continuous variable or as ordered categories (such as “high,” “medium,” and “low”). We will illustrate using a study of the accuracy of the serum ferritin, when compared with bone marrow iron stores, for confirming or excluding iron deficiency as the cause of anemia in older persons.¹⁸ We present this approach scripted for use on a flip chart, but we find it also works equally well in other media, such as a white board or overhead transparencies.

Start with a blank flip chart and leave the topmost cover page blank and cut it vertically all the way up, allowing you to flip either side over the top and out of the way independent of the other side. On the first data page, labeled “Page 1,” place the line dividing the two columns of results data to lie directly underneath the vertical cut of the covering page, so that flipping down one vertical half of the covering sheet conceals neatly the corresponding half of the data table. Have at least three different colors of marker to help learners follow along.

Arrange the results levels so that the LRs you will eventually calculate will be arrayed from the largest number on the top through smallest number (less than one) on the bottom. In our example, this means extracting data from one of the article’s tables and reversing the order (see Table 1). For each cell of the table, write the number of patients found in the study to have that test result in the upper left-hand corner of the cell, allowing room for calculations below and to the right. When preparing “Page 2,” select ahead of time the results ‘cut point’ you will use to divide the data into only two levels. Rename the two levels using the raw results from “Page 1,” now dichotomized at the cut point (see Table 6). We find it useful to have the LR nomogram handy, either in its original version¹⁹ or subsequent adaptations including electronic forms (for example, the Oxford Centre for Evidence-Based Medicine, http://www.cebm.net/).

Table 1.

Shows “Page 1” at the Beginning

	Bone marrow: iron deficient	Bone marrow: normal iron
Test results
≤18	47	2
19–45	23	13
46–100	7	27
>100	8	108
Totals	85	150

Table 6.

Shows “Page 2” at the Start

	Bone marrow: iron deficient	Bone marrow: normal iron
Test results
≤45	70	15
≥46	15	135
Totals	85	150

Teaching Page 1

Start with Page 1 showing fully. Orient your learners to the task ahead and to the data table with its columns, rows, and data points (see Table 1): Let’s make sense of these results. Flip the right-hand half of the cover sheet down, hiding the right-hand column of the data on Page 1, leaving the left-hand column visible: The left side of this table shows the test results for all the patients found by the bone marrow to have iron deficiency. In all patients with iron deficiency, how often does the ferritin result of “less than or equal to 18” occur? Facing your learners, wait patiently until one suggests the correct answer: Right, this result occurs 47 times out of 85. Using a second color, write this fraction in the cell under the 47, and ask someone to divide the numbers: Yes, 0.553 is the proportion of patients with the iron deficiency who have ferritin results <18. With the second color, write an equal sign after the fraction and enter the proportion. Repeat the same step of division for each level of test result, so Page 1 looks like Table 2: Using simple division, we’ve found a proportion for each test result that describes quantitatively how likely that result is to occur in these patients with iron deficiency. Clarify as needed.

Table 2.

Shows “Page 1” When Left-hand Column is Completed

	Bone marrow: iron deficient	Bone marrow: normal iron
Test results
≤18	47 47/85 = 0.553
19–45	23 23/85 = 0.271
46–100	7 7/85 = 0.082
>100	8 8/85 = 0.094
Totals	85

When ready, move on: flip down the left-hand side of the covering sheet, obscuring the column you have just worked on, then flip up the right-hand side of the covering sheet, uncovering the right-hand column of the table: Let’s use the same approach to describe how likely these test results are to occur in those without iron deficiency. By now, most groups of learners will understand what to do and will divide the numbers. Record the fractions and the proportions in each data cell using the second color. When you finish, Page 1 should look like Table 3.

Table 3.

Shows “Page 1” When Right-hand Column is Completed

	Bone marrow: iron deficient	Bone marrow: normal iron
Test results
≤18		2 2/150 = 0.013
19–45		13 13/150 = 0.087
46–100		27 27/150 = 0.18
>100		108 108/150 = 0.72
Totals		150

Next, flip back the left-hand side of the cover sheet to reveal the entire table on Page 1 (see Table 4). We have calculated proportions that describe quantitatively how likely each of these test results are to occur in those with iron deficiency, then we calculated proportions that describe how often the same results occur in those without iron deficiency. For each level of test result (point to top level), how can we compare these two proportions (point back and forth between the two numbers, 0.553 and 0.013)? Again, face your learners and wait patiently while they ponder: Let’s try division. Let’s make a ratio of these two proportions, 0.553 to 0.013. Ask learners to calculate, and using a third color, write the arithmetic to the right of the table at each row level: Right, 0.553:0.013 = 42.5. Repeat this step until the group has computed ratios for all levels of test results in the table rows (see Table 5). Clarify as needed: Using another step of simple division, we’ve calculated ratios for each level of test result. These ratios compare how likely this test result is to occur in those with iron deficiency to how likely the result is in those not iron deficient.

Table 4.

Shows “Page 1” When Both Completed Columns are Shown Together

	Bone marrow: iron deficient	Bone marrow: normal iron
Test results
≤18	47 47/85 = 0.553	2 2/150 = 0.013
19–45	23 23/85 = 0.271	13 13/150 = 0.087
46–100	7 7/85 = 0.082	27 27/150 = 0.18
>100	8 8/85 = 0.094	108 108/150 = 0.72
Totals	85	150

Table 5.

Shows “Page 1” When Likelihood Ratios are Calculated

	Bone marrow: iron deficient	Bone marrow: normal iron	Likelihood ratios
Test results:
≤18	47 47/85 = 0.553	2 2/150 = 0.013	0.553:0.013 = 42.5
19–45	23 23/85 = 0.271	13 13/150 = 0.087	0.271:0.087 = 3.11
46–100	7 7/85 = 0.082	27 27/150 = 0.18	0.082:0.18 = 0.456
>100	8 8/85 = 0.094	108 108/150 = 0.72	0.094:0.72 = 0.131
Totals	85	150

Next, explain at least one of the ratios in words: This ratio of 42.5 means that a ferritin result of less than or equal to 18 is over 42 times more likely to occur in those with iron deficiency than in those without. Clarify as needed, and then name the ratios: These ratios are called ‘likelihood ratios’, abbreviated as ‘LR’. In general, the LR for each test result compares how likely that result is to occur in those with the condition to how likely the result is in those without the disorder. When a ratio equals 1, its numerator and denominator are equal, so a test result with a LR of 1 means it occurs equally in those with and without the disorder. For clinical diagnosis, such a test result with a LR of 1 neither raises nor lowers the probability of the disorder being considered. When a ratio is larger than 1, the numerator is larger than the denominator; here that means the test result is more likely in those with the disease than those without. For clinical diagnosis, a LR above 1 raises the probability of the disorder. When a ratio is smaller than 1, the numerator is smaller than the denominator, which here means the test result is less likely in those with the disease than those without. For clinical diagnosis, a LR less than 1 lowers the probability of the disorder. The farther a LR is from 1, in either direction, the more it changes the probability of disease. Clarify as needed.

Direct your learners to the LR nomogram:¹⁹This nomogramshows us this idea visually. Orient your learners to the nomogram’s three components, pretest probability, LR and posttest probability, and to the nomogram’s use. In our study, 85 of 235, or 36.2% of patients, had iron deficiency by bone marrow, so we can use 36% as our starting pretest probability. Now, for each level of ferritin test result, what is the resulting posttest probability? Walk through the steps for all four levels, showing how much the probability of disease is changed by the LR of each test result. Give your learners a general sense of the scale of LRs, e.g., In clinical diagnosis, LRs of 10 or higher, or 0.1 or lower, can result in large changes in disease probability. Clarify as needed.

Whereas learners vary, we find most groups can work through the lessons of Page 1 in 15–20 minutes using this ‘split column’ technique. Note the arithmetic has been simple division and no formulae have been introduced.

Teaching Page 2

We find it desirable to work through both pages consecutively with our learners because the calculations and the insights are cumulative. Most groups of learners no longer need the split covering sheet for Page 2.

Show Page 2 fully (see Table 6). We often encounter clinical situations where test results are portrayed in only two levels instead of multiple levels. Orient your learners to the columns, rows, and data points in the table on Page 2, emphasizing that they are the same patients seen on Page 1. Start with the left-hand column in the upper row: Of all those with iron deficiency, how likely are they to have ferritin values of 45 or less?Right, 70 of 85, or 0.824. Record the fractions and proportions in the same second color you used on Page 1. With only four data cells and your learners’ growing confidence, you should be able to quickly fill in the cells of the table: Just as before, these proportions describe how likely each test result is to occur in the patient groups. Now let’s calculate the LRs for the two levels of test result. Using the same third color you used on Page 1, write the raw ratios and the resulting LRs to the right side of the table at each level (see Table 7). Clarify as needed.

Table 7.

Shows “Page 2” with Calculations Completed

	Bone marrow: iron deficient	Bone marrow: normal iron	Likelihood ratios
Test results
≤45	70 70/85 = 0.824	15 15/150 = 0.1	0.824:0.1 = 8.24
≥46	15 15/85 = 0.176	135 135/150 = 0.9	0.176:0.9 = 0.196
Totals	85	150

When the group is ready, move on: When we grouped the test results into two levels instead of four, we can still calculate LRs. Notice that low ferritin test results still have a LR above one, actually 8.24, so test results in this range would raise the probability of iron deficiency. But notice also that this LR is smaller than the 42.5 we got on Page 1 for those with ferritin ≤18. The same is true for ferritin results 46 or higher. Repeat the exercise, comparing the LR of about 0.2 on Page 2 to the lowest from Page 1. In general terms, collapsing multilevel test results down to two levels moves LRs closer to one, so results at the extremes will appear less powerful in changing disease probability. Direct the learners’ attention to the LR nomogram again to demonstrate how much less these LRs can change the probability of disease. Clarify as needed.

Whereas learners vary, we find most groups can work through the lessons of Page 2 in 5–10 minutes, if they have done Page 1 just beforehand.

Additional Comments for Tip 2

By using this ‘split column’ technique, we separate the arithmetic steps involved in calculating LRs, so that most learners can follow the concepts without much difficulty. By starting with multilevel test results, then showing the same approach for two levels, we find we can show the greater discriminatory power of multiple levels better than when we have tried it in reverse order. We also find that working through the arithmetic first helps most learners build an understanding of the concept, before the name ‘likelihood ratio’ can confuse them or the formulae can scare them. Whereas groups vary, we find most learners need to work through this script only one to three times to demystify likelihood ratios. Once they are comfortable with the concepts and how to use them, most clinical learners no longer need to do the arithmetic shown here.

The Bottom Line

• LRs for test results can be calculated directly from the data table, using two steps of simple division and without using formulae.

• LRs equal to 1 do not change the probability of disease, whereas the farther the LR is from 1 in either direction, the more powerfully the test result can revise the probability of disease.

• When test results in multiple levels are ‘collapsed down’ to two levels, the LRs for these collapsed levels move closer to one.

See the Appendix 1 for the summary card for this tip.

Report on Field Testing

One of us (SAK), who, while an experienced teacher of evidence-based medicine, had not seen either tip demonstrated by the developers, tested the feasibility of teaching these scripts based on the above instructions. The field testing occurred during a 90-minute session in August, 2005 as part of a regular conference schedule. Of the 16 residents in Internal Medicine, 12 were naive to EBM, whereas 4 had had prior exposure. After introductory goal setting and a concise summary of the ferritin study, the group worked through the ‘conceptual algebra’ of Tip 1 in 20 minutes and the ‘split column’ approach of Tip 2 in 40 minutes. The use of color and the use of the flip chart exactly as described were critical to the clarity of the progressive creation of the tables for the LR calculations in Tip 2. Some learners reported discomfort from not reviewing sensitivity, specificity, and predictive values, reflecting their greater familiarity with those concepts.

Appendix 1

Summary cards for 2 teaching tips on the use of likelihood ratios.

This Appendix has been designed so that it can be printed on a single sheet of 8 1/2 × 11 in. paper. The individual summary cards can then be cut out, if desired, for use during teaching sessions.

Teaching Tip 1: Starting with the name, using the ‘Conceptual Algebra’ approach.

Scenario: The phrase “likelihood ratio” has already come up before the clinicians have learned about them.

1. Write “likelihood ratio = ” and ask the learners what sort of arithmetic function is a ratio, helping learners to recognize it as division.

2. Write “L1/L2 = ” and explain that a likelihood ratio is a ratio of likelihoods.

3. Write “ = ‘test result’ with disease/‘test result’ no disease” and explain the sources of these two proportions.

4. Work through the meaning of the three ‘zones’ of LR values—greater than 1, less than 1, and equal to 1.

Summary points:

• LRs compare two likelihoods—the frequency of a test result in those with the target disorder compared with the frequency of the same test result in those without the disease.

• LR values above 1 mean the test result occurs more often in those with disease, whereas values below 1 mean the result occurs less often in those with disease. Values equal to 1 mean the result occurs equally often in those with and without the target disorder.

• Expressing a LR using words in a sentence can help us understand its meaning and communicate clearly to others.

Teaching Tip 2: Starting from a results table, using the ‘Split Column’ approach.

Scenario: Select a study that reports the test results in 3 or more levels.

1. Draw 2 copies of the data table, the first with 3 or more rows for each test result level, whereas the second with the results dichotomized to 2 levels of results.

2. Using Page 1, hide the right-hand column of data. Ask the learners to describe the likelihood or frequency of each test result in those with the target disorder.

3. Repeat the process after covering the left-hand data column and showing the right side.

4. Uncover both columns, then ask the learners to form ratios for each result level that compare the frequency of that test result in those with the target disorder to the frequency of the test result in those without the condition.

5. Name the likelihood ratio and explain its use in revising probability of disease.

6. Using Page 2, repeat the process to find likelihood ratios when same test result data are displayed in only two levels.

Summary points:

• LRs for test results can be calculated directly from the data table, using two steps of simple division and without using formulae.

• LRs equal to 1 don’t change the probability of disease, whereas the farther from one in either direction, the more powerfully the test result can revise the probability of disease.

• When test results in multiple levels are ‘collapsed down’ to two levels, the LRs for these collapsed levels move closer to one.