How Useful Are Physical Examination Procedures? Understanding and Applying Likelihood Ratios

Abstract

Objective: To describe the calculation and interpretation of likelihood ratios for examination procedures performed by certified athletic trainers.

Background: Physical examination procedures or “special tests” are commonly taught to athletic training students and performed by certified athletic trainers. Likelihood ratios offer an approach to assessing test performance that incorporates estimates of sensitivity and specificity into a clinically useful value. We describe the calculation of likelihood ratios and the application of likelihood ratios to clinical decision making.

Recommendations: The performance characteristics of physical examination procedures taught and practiced in athletic training should be considered in the planning of course materials as well as test interpretation after a physical examination. Research is needed to better understand how well physical examination procedures, when performed by certified athletic trainers, identify those athletes with and without specific musculoskeletal injuries.

Musculoskeletal injury “recognition, evaluation, and assessment” is one of the 6 central domains of athletic training education. ¹ The certified athletic trainer (AT) is often the first health care provider encountered by athletes and people engaged in physical activity after such injuries. The AT, therefore, is often called on to evaluate the injured person and make decisions with regard to return to participation or nonemergent referral. These decisions require that the diagnostic probabilities be considered.

The educational model in athletic training calls for students to demonstrate an ability to complete an evaluation of numerous musculoskeletal structures. Although specific physical examination procedures are not identified in the educational standards, the instruction of numerous “special tests” is a longstanding practice. Moreover, the texts used to support athletic training education also reference many special tests. Magee, ² for example, described more than 2 dozen examination procedures and modifications for evaluation of the ligaments and menisci of the knee alone. Such instruction is commonplace, but we believe it may be incomplete. How valuable to the clinical decision-making process are the examination procedures we teach? Which tests should we teach, and which should be omitted? What does the AT need to know to apply research on diagnostic procedures to clinical practice? What should we really be teaching students about the physical examination process?

Clinical examination procedures have emerged as individual clinicians attempted to improve on their ability to accurately diagnose illnesses and injuries. The clinical examination findings influence treatment and referral decisions and permit prognostication regarding outcome after one or more courses of treatment. Before we allow the results of a particular examination procedure to influence a clinical judgment, however, we should know the performance characteristics of the test. Test performance characteristics can be estimated by comparing the results of a clinical examination procedure with an established diagnostic standard. For musculoskeletal conditions, operative findings and the results of advanced diagnostic imaging (eg, magnetic resonance imaging [MRI]) provide such reference standards. Applying research related to the performance characteristics of musculoskeletal physical examination procedures requires an understanding of terminology and statistical procedures unfamiliar to some clinicians. One method that we believe has great utility in athletic training involves the calculation and interpretation of likelihood ratios (LRs).

Our purpose, therefore, is to describe the calculation and interpretation of likelihood ratios for examination procedures performed by ATs. Likelihood ratios are not the only means of describing test performance characteristics, but they are attractive for the reasons we describe, especially for dichotomous clinical examination procedures (eg, Lachman procedure ³) in which the intended result is either positive (anterior cruciate ligament [ACL] is torn) or negative (ACL is intact). We first define key terms and review issues of validity and test reliability. The derivation of values needed to understand and apply the results of investigations of diagnostic tests is then provided.

For the purpose of this article, we consider the assessment of an injured knee. More specifically, we present and discuss the performance characteristics of the Lachman ³ and McMurray ⁴ tests for ACL and meniscal injuries, respectively. These patient examples link an understanding of likelihood ratios, clinical research, and individual clinical practice and teaching. These concepts can be applied across a spectrum of examination procedures common to athletic training education and practice.

METHODOLOGIC QUALITY AND STUDIES OF DIAGNOSTIC PROCEDURES

Assessing the performance of diagnostic tests and applying this knowledge in clinical practice requires familiarity with the associated terminology. The clinician must be able to evaluate the methodologic quality of studies of diagnostic testing and have a basic understanding of how a few numeric values are derived. This section provides an introduction into the assessment of methodologic quality of studies of diagnostic procedures.

Validity

Validity is a complex subject, and our purpose is ultimately to demonstrate the calculation and application of likelihood ratios in clinical practice. These apparently foreign issues are, however, closely linked. Validity has been defined as “the extent to which a test measures what it is intended to measure,” ⁵ or, stated differently, “a term used to characterize the overall correctness of a test.” ⁶ Messick, ⁷ however, clarified the issue when he stated that “validity is not a property of a test or measurement as such but rather the meaning of the test scores.” Data are the results of measurement. Thus, in the context of physical examination procedures, validity really boils down to whether the data collected (positive or negative results) reflect the reality of the condition of each patient. We shall see that data derived from physical examination procedures with high positive likelihood ratios (+LR) and low negative likelihood ratios (−LR) are more likely to reflect the presence or absence of a condition, respectively, than those with low +LR or high −LR.

Likelihood ratios for physical examination procedures are derived by comparing the results of a procedure of interest (eg, Lachman test ³) with the results of a previously validated examination (eg, arthroscopy) often referred to as the “gold standard.” Because an entire population of patients cannot be studied, investigators can only estimate the true performance characteristics (including +LR and −LR) of a diagnostic test. Likelihood ratios are, in fact, data that provide estimates of how well physical examination procedures measure what they are intended to measure, namely the presence or absence of a medical condition.

Likelihood ratios (the data derived from studies of diagnostic testing) can be influenced, or biased, by the design and study methods employed to assess the performance of physical examination procedures and other diagnostic tests. Therefore, before we address the interpretation and application of LRs, it is important to identify methodologic issues that may threaten the validity of LR estimates.

Biases in Studies of Physical Examination Procedure Performance

When we review studies of diagnostic procedures, our first consideration is what was the gold standard used for comparison? Authors of a report should provide evidence that the gold standard is adequately accurate for comparison purposes within the context of a study. ^{8 9} Direct observation during surgery is often the most accurate method of confirming musculoskeletal injury. Diagnostic imaging, while highly useful, is not perfectly accurate. Thus, when MRI, for example, is employed as a gold standard for comparison in a study of clinical examination procedures, the values of specificity, sensitivity, and LRs (described in detail shortly) for the MRI should be provided to the research consumer. The consumer must then decide if the gold standard is sufficiently accurate for comparison purposes.

The comparison with an established standard, however, is not the sole concern when evaluating the methods of a study of diagnostic procedures. Bias can be introduced through subject selection ^{8 9} and other methodologic issues. A good study includes a spectrum of patients to whom the test in question would typically be applied in a clinical setting. Spectrum bias is introduced when only patients very likely to have a condition (based on history or other criteria) are studied or when patients who clearly do not have a condition are included. ^{6 8}

The results of a study can also be influenced by “work-up” bias. Work-up bias may exist if the gold standard is not applied to everyone in the study. ⁸ For example, if MRI is ordered only for those thought most likely to have sustained an ACL tear, the number of false-negative results on the criterion measure (eg, Lachman test) may be underreported. Bias may also be introduced when examiners are not blinded to the gold standard test results. We are all human, and we tend to find what we expect, such as a positive Lachman test in a patient with an MRI identifying a torn ACL. Furthermore, and for the same reason, the gold-standard interpreters must be blinded to the clinical results. ^{8 9} For example, if 20 people were tested clinically for ACL deficiency with a Lachman test and 10 were found to be positive, the radiologist reading the MRI should not be aware of which subjects had laxity in the clinical tests.

One final consideration exists when reading research related to diagnostic testing: the generalizability (also referred to as external validity) of the investigators' conclusions. ¹⁰ Readers need to judge how closely the study setting and patient sample reflect their own environment. Furthermore, readers must be sensitive to the training and expertise of the health care professionals performing, administering, and interpreting the tests of interest. This issue is, in our opinion, of particular concern in athletic training and is discussed further at the end of this paper.

Reliability

Reliability, in the context of physical examination procedures, is the extent to which the results of a test can be replicated when the same phenomenon is measured under the same conditions multiple times. ⁶ The reliability of physical examination procedures can be divided into 2 components: intratester (same tester blindly repeating an examination) and intertester (level of agreement between 2 or more testers). ¹¹ Reliability is a requisite of validity; however, reliability alone does not establish validity. ¹¹ For example, 2 clinicians might agree on the results of a diagnostic test in 18 of 20 patients (high reliability) but in fact correctly categorize only 20% (low validity) according to a gold standard. In some cases, such as with goniometric measurements ¹² and assessment of sacroiliac ¹³ and shoulder dysfunction, ¹⁴ reliability estimates have been reported. In other cases, such as with examination procedures to detect a damaged meniscus, less is known about the intratester or intertester reliability. Unless reliability data are available, the clinician must be cautious in believing reports suggesting that a particular examination procedure is good for detecting and ruling out a particular condition and must recognize that the procedure may not perform as well in other settings or when administered by clinicians with different training and skills.

From the above discussion, it becomes apparent that a careful consideration of the research methods is necessary before the results from a study and the conclusions drawn by investigators are reviewed. The research consumer must consider the potential that study methods have biased the data. Furthermore, test reliability, setting, study population, and training of the clinicians involved in the diagnostic process must be considered before generalizing an investigator's conclusions to individual clinical practices. With this foundation, it is time to interpret and apply study results and conclusions. This process requires an understanding of how sensitivity, specificity, and LRs are derived.

SENSITIVITY, SPECIFICITY, AND LIKELIHOOD RATIOS

Sensitivity and specificity are central concepts to understanding test performance characteristics. Sensitivity and specificity are related to the ability of a test to identify those with and without a condition and are needed to calculate LRs. Table 1 illustrates how test results can be categorized. ⁶ The inserted values correspond to the following example.

Table 1. Contingency Table of Test Results Used to Calculate Sensitivity and Specificity*.

Sensitivity is the number of illnesses or injuries that are correctly diagnosed by the clinical examination procedure being investigated (cell A) divided by the true number of illnesses or injuries (cells A + C) (based on the criterion or gold standard measure) and is calculated as follows ^{8 15}:

For example, consider 35 patients with knee injuries. After assessment via an anterior drawer test, 18 are correctly diagnosed as being ACL deficient. All 35 undergo arthroscopic surgery and 20 are found to have torn their ACL. From the formula above, 18/20 yields a sensitivity of .90.

Specificity is the number of individuals correctly classified as not having the condition of concern based on the test being investigated (cell D) divided by the true number of negative cases (cells B + D) (based on the criterion or gold standard measure). Is it possible to have a positive test in someone without the target condition? Absolutely: these are the cases contained in cell B of the contingency table.

From the previous example, 15 patients did not have an ACL tear. Let's assume, however, that 3 of these patients were judged to have positive anterior drawer tests; thus, only 12 were correctly classified as not having injured their ACLs. Therefore, the specificity = 12/15 or .80.

Diagnostic procedures may have high sensitivity and low specificity or vice versa. It is often difficult to determine the effect of the estimates of sensitivity and specificity on the usefulness of a procedure in clinical practice. Ideally, a test would have high sensitivity and specificity, but this is often not the case. Furthermore, even for tests with high sensitivity and specificity, the effect of test results on the probability that a condition either is or is not present cannot be calculated directly from these values. To better understand how test performance affects clinical decisions, predictive values or LRs can be calculated. Predictive values are affected by the incidence of the condition being assessed in the population. ^{15 16} Likelihood ratios, sensitivity, and specificity do not vary with changes in incidence. ^15–17 Thus, LRs offer an approach to assessing test performance that is unaffected by the incidence of the condition being assessed in the population and that incorporates estimates of sensitivity and specificity into a clinically useful value.

A +LR indicates the effect of a positive examination finding on the probability that the condition in question exists. For tests with dichotomous results, a +LR is calculated as follows ¹¹:

+LR = Sensitivity/(1 − Specificity)

Using the example above, the +LR equals .90/(1 − .8) = 4.5. This means that, based on our hypothetical numbers, a positive anterior drawer is 4.5 times more likely to occur in a patient with a torn ACL than one with an intact ligament. ¹¹ The application of LR values in the context of clinical practice will be addressed in the next section.

A −LR addresses the effect of a negative examination on the probability that the condition in question is present. A negative result from a diagnostic test with a small −LR suggests that the chance of the condition of concern existing is very low. Negative likelihood is calculated as follows ^{15 17}:

−LR = (1 − Sensitivity)/Specificity

Again using the same values, the −LR equals (1 − .90)/.8 = 0.13. In this case, the examiner would find a positive anterior drawer 13/100 times as often in uninjured knees as in injured knees. Jaeschke et al ¹⁸ summarized LRs (positive and negative) into broader categories of clinical value ( Table 2). From our examples above, one could conclude that for an anterior drawer test, the +LR of 4.5 suggests that a positive test results in a small to moderate but likely important shift in the probability of a torn ACL. The −LR of 0.13 suggests that a negative test results in a moderate and likely important shift in probability favoring an intact ACL.

Table 2. Clinical Values of Likelihood Ratios from Jaeschke et al ¹⁸ .

APPLICATIONS

Now that we have survived this crash course in the generation of LRs, the issues become why this statistic is of value and how it can be applied in practice and teaching? We will illustrate the applications of LRs by discussing 2 examination procedures of the knee. Before doing so, however, let us consider the realities and uncertainties of the examination process to better appreciate the role diagnostic tests, including physical examination procedures, play in the management of patients.

In clinical practice, physical examination procedures are performed in the context of a comprehensive evaluation of a patient. The examination procedures are performed based on some level of suspicion that the condition exists. This pretest probability estimate varies with clinicians and the circumstances of the individual patient. The key is to recognize that a level of suspicion regarding diagnostic possibilities exists before the examination procedure of interest (eg, McMurray test) is performed.

Fritz and Wainner ¹⁵ described the relationship between pretest probability and LRs. Understanding this link is prerequisite to understanding the effect of diagnostic test results on posttest probability, or the degree of certainty a condition does or does not exist after a clinical examination is completed. The issue of pretest estimate raises 2 issues. First, do clinicians really go through this process? Second, what is a “good” estimate? At some level, clinicians must, and we mean must, make a judgment regarding the probability of one or more diagnoses. This process starts before any clinical examination procedures are performed or diagnostic tests ordered. In fact, it is the process by which the clinician selects the components of the remainder of the evaluation. In many cases, one could argue that a clinician's pretest probability is too conservative or too liberal. This is a reminder that clinicians vary in their probability estimates based on experience and specialization, as well as the subtle findings gleaned from history and observation. The effect of various pretest probabilities is discussed later.

Probability implies uncertainty. Consider how often you are absolutely, positively certain of a diagnosis at the end of a physical examination. These events happen, of course, but not as often as most of us would like. Uncertainty is inherent in the clinical practice of athletic training. However, decisions regarding referral, plans of treatment, and a physician's use of additional diagnostic studies revolve around the level of certainty (probability) that a condition does or does not exist. The value of specific examination procedures may be best viewed in the context of their effect on patient care decisions based on a positive or negative result.

Now we move to the application of LRs on posttest probability of a condition being present or absent. The assessment of the performance characteristics of a McMurray test ⁴ for meniscal injury and a Lachman test ³ for ACL lesions provides a spectrum of LRs. Thus, we will apply the results from a comprehensive analysis of each test to patient examples to integrate and illustrate the concepts introduced in this paper.

Assessing the Medial Meniscus: Patient Example

A 33-year-old male recreational soccer player presents with a 2-month history of knee pain resulting from an episode in which the knee was twisted while he was being tackled during a game. He localizes the pain to the medial aspect of the knee and reports that after activity the knee feels swollen and that he experiences a painful catching sensation intermittently. Corea et al ¹⁹ noted that in a series of patients with meniscal injuries, all had pain, 61% reported painful clicking, and 55% experienced recurrent effusion. Thus, based on this patient's history, a clinician may establish a pretest probability of a meniscus tear of 60%. This level of probability can be transformed into pretest odds using the following formula: Probability/(1 − Probability). ^{15 16} In this case, .6/(1 − .6) = 1.5, indicating that the patient is 1.5 times more likely to have a meniscus tear than someone without these signs and symptoms. To assess the effect of a positive McMurray test, ⁴ the pretest odds are multiplied by the +LR. Turning to the literature, we find that Scholten et al ²⁰ calculated a summary +LR for the McMurray test ⁴ of 3.4. Thus, 1.5 × 3.4 yields posttest odds of 5.1. Posttest probability is calculated by the formula (Posttest odds)/[posttest odds + 1]). ¹³ Thus, the posttest probability of a meniscus tear can be calculated as such: 5.1/6.1 = 0.84. Therefore, using the +LR estimate reported ²¹ and the pretest odds provided in this exercise, a positive McMurray test ⁴ has shifted the estimate of probability of meniscus tear from 60% to 84%.

The same process may be used to assess the effect of a negative McMurray test ⁴ on the same patient. Again, a pretest probability of 60% (pretest odds = 1.5) is assumed. Using the mean −LR of 0.6 estimated by Scholten et al, ²⁰ a posttest odds estimate of (1.5 × 0.6) = 0.9. Dividing 0.9 by 1.9 yields a posttest probability of 0.47. Thus, the effect of a negative McMurray test has resulted in a very small (13%) change in the probability a meniscus tear does not exist and will likely have little effect on treatment decisions.

Assessing the Anterior Cruciate Ligament: Patient Example

A 21-year-old female recreational basketball player presents to an outpatient sports medicine clinic on referral from her primary care physician. She reports injuring her left knee last week while playing. She states that when she turned to cut to the left, her shoe stuck to the court, causing her to fall. She notes that the knee was immediately painful and she discontinued playing, went home, and applied ice. She reports that she had substantial swelling the next day and tried to “stay off her feet.” She was evaluated by her family physician, given analgesic medication, crutches, and a knee immobilizer, and was referred for further evaluation. Upon presentation, she complains of pain on the medial portion of her knee, as well as a deep pain that cannot be touched. She is unsure of what she heard or felt at the time of injury. Moderate swelling is evident.

The clinician estimates the pretest probability of ACL rupture at 50%. An illustration of the effect of various pretest estimates on posttest probability follows.

Given the 50% pretest probability estimate, the pretest odds = 0.5/(1 − 0.5) = 1. Upon physical examination, a positive Lachman test is demonstrated. Applying the data from Scholten et al, ²¹ the sensitivity and specificity of the Lachman test ³ were estimated to be 0.86 and 0.91, respectively. These values yield a +LR = 9.6 and a − LR = 0.15. Applying these values to the scenario above, the posttest probability is 91% (9.6/ 10.6) after a positive test and 13% (0.15/1.15) after a negative test. Thus, the physical examination results have had a large effect on the probability of an ACL lesion and, thus, treatment decisions.

Linking History and Physical Examination Findings

If the clinician had estimated a 20% pretest probability of an ACL tear, a positive Lachman test ³ would result in a posttest probability of 71%, and a negative Lachman test would result in a posttest probability of 4%. Conversely, an 80% pretest probability estimate combined with a positive Lachman test ³ results in a posttest probability of 97%, whereas a negative Lachman test ³ yields a 38% posttest probability. These results still represent generally large shifts in probability but illustrate the links among history, observation, and clinical examination procedure results on diagnostic certainty.

DISCUSSION

Our intent was for this paper to introduce one approach to assessing the value of the physical examination procedures used in athletic training and other health care specialties. Likelihood ratios are appealing in that the values are relatively easy to calculate from published reports and readily applied in clinical practice. Other techniques also permit improved clinical decision making. For example, receiver operator characteristic curves permit interpretation of test results that are points on a continuum (such as systolic blood pressure) rather than dichotomous values and summarize estimates provided from multiple reports. ⁸ Athletic trainers, however, use numerous physical examination procedures that have an intended positive or negative result. Likelihood ratios are the most easily calculated and applied estimates of test performance for these examination procedures.

How should the process we describe be applied in athletic training practice, education, and research? First, practicing clinicians should consider how they interpret the examination procedures learned and practiced over the years. Evaluating examination procedures places the uncertainties associated with the examination process in perspective and permits selection of the tests most likely to help make better clinical decisions.

We must also recognize that the examination procedures we discuss are but a small portion of those taught throughout the educational experience in entry-level education programs. Which examination procedures should the student be taught? We believe that those procedures with the best performance characteristics should be emphasized. For example, Scholten et al ²¹ compiled and analyzed the results from 10 studies on the performance characteristics of the Lachman, ³ anterior drawer, and pivot shift tests ( Table 3). Because the +LR and −LR for a Lachman test ³ indicate that the test is of value in identifying ACL lesions as well as ruling the injury out in those with intact ACLs, the educator may elect to instruct students in the Lachman test ³ only. Doing so might foster greater mastery of this test because students are relieved of the responsibility of practicing multiple tests of similar purpose.

Table 3. Likelihood Ratio Calculations from the Summary Estimates of Sensitivity and Specificity Values from Scholten et al ²¹ .

Regardless of the selection of particular examination procedures included in instruction, an understanding of LRs will improve students' understanding of physical examination procedures in the context of a comprehensive patient evaluation. We believe that the ability to search the literature and calculate LRs and other indicators of test performance should be corequisite to learning and practicing these procedures.

It should also be appreciated that few if any reports have been published related to examination procedures when performed by ATs. Simply because an examination procedure is reported to be valuable when performed by an orthopaedic surgeon does not mean other clinicians will achieve similar results. For example, Cooperman et al ²² found that orthopaedists were more skilled at the Lachman test ³ and accurately identifying ACL ruptures than physical therapists. Hurley ²³ reported a generally poor level of agreement between ATs and an orthopaedic surgeon in the assessment of a sample of subjects with and without known ACL deficiency. Of particular interest was that fact that only 4 of 22 ATs were found to perform the test in a manner consistent with the original report of a Lachman test by Torg et al. ³ These 4 ATs achieved 67% agreement with the orthopaedic surgeon, whereas the other 18 ATs performed what Hurley ²³ defined as a generalized anterior tibial translation test and achieved only a 19% level of agreement. We believe there is a need to study examination procedure performance characteristics of examinations performed by ATs.

CONCLUSIONS

The special tests performed in the physical examination of patients with musculoskeletal injuries are learned skills. Proficient performance requires formal instruction in and practice of the proper technique. Skilled performance, however, does not assure accuracy. Athletic trainers should understand the value and limitations of these special tests, so that test results are interpreted in the context of the full examination. Furthermore, examination procedure performance characteristics should be considered in the development of the athletic training curriculum. Procedures that are poor discriminators may be best left out of instruction so that students may focus their attention on mastering those skills most useful in clinical practice. Lastly, research into the various aspects of injury evaluation by ATs is needed. Little in the athletic training literature describes how well these clinicians recognize, evaluate, and assess ill and injured athletes despite the prominence of these responsibilities in the professional and educational standards.