Application of the Stockholm Hierarchy to Defining the Quality of Reference Intervals and Clinical Decision Limits

Abstract

The Stockholm Hierarchy is a professional consensus created to define the preferred approaches to defining analytical quality. The quality of a laboratory measurement can also be classified by the quality of the limits that the value is compared with, namely reference interval limits and clinical decision limits. At the highest level in the hierarchy would be placed clinical decision limits based on clinical outcome studies. The second level would include both formal reference interval studies (studies of intra and inter-individual variations) and clinical decision limits based on clinician survey. While these approaches are commonly used, they require a lot of resources to define accurately. Placing laboratory experts on the third level would suggest that although they can also define reference intervals by consensus, theirs aren’t as well regarded as clinician defined limits which drive clinical behaviour. Ideally both analytical and clinical considerations should be made, with clinicians and laboratorians both having important information to consider. The fourth level of reference intervals would be for those defined by survey or by regulatory authorities because of the focus on what is commonly achieved rather than what is necessarily correct. Finally, laboratorians know that adopting reference limits from kit inserts or textbook publications is problematic because both methodological issues and reference populations are often not the same as their own. This approach would rank fifth and last. When considering which so called ‘common’ or ‘harmonised reference intervals’ to adopt, both these characteristics and the quality of individual studies need to be assessed. Finally, we should also be aware that reference intervals describe health and physiology while clinical decision limits focus on disease and pathology, and unless we understand and consider the two corresponding issues of test specificity and test sensitivity, we cannot assure the quality of the limits that we report.

Introduction

I was fortunate to be amongst the 100 participants from 27 countries that met at the Nobel Institute in Stockholm in April 1999 to consider the many strategies that can be used to set analytical goals for clinical laboratory tests. The meeting was sponsored by the International Federation of Clinical Chemistry, the International Union of Pure and Applied Chemistry, and the World Health Organization, and included 22 formal presentations from the world’s leaders in analytical quality including Anders Kallner,¹ Callum Fraser,² Per Hyltoft Petersen,³ George Klee,⁴ Linda Thienpont,⁵ Sverre Sandberg,⁶ Carmen Ricós,⁷ Sharon Ehrmeyer⁸ and James Westgard.⁹

The meeting ended with a critical discussion that included all participants and the formulation of a consensus statement regarding a hierarchical approach to the definition of analytical quality specifications (Table 1).¹⁰ Working Subgroup 3 of ISO Technical Committee 212 (TC212), the committee that developed the internationally accepted Medical Laboratory Quality Standard (ISO 15189), proposed that this hierarchy should be used for the ongoing revision of ISO 15196 (the analytical quality standard) which still remains uncompleted. Nevertheless, the ‘Stockholm Hierarchy’ represents the last word by TC212 on this issue.¹¹

Table 1.

The Stockholm Hierarchy for analytical quality goals. Adapted from Kenny D, et al.¹⁰

Evaluation of the effect of analytical performance on clinical outcomes in specific clinical settings. Evaluation of the effect of analytical performance on clinical decisions in general with Data derived from biological variation. Data based on clinicians’ opinions. Published professional recommendations National or international expert bodies. Expert local groups or individuals. Performance goals set by Regulatory bodies. Organisers of external quality assurance (EQA) schemes. Analytical goals based on the current state of the art As demonstrated by data from External Quality Assurance. As found in current publications.

The hierarchy outlined in Table 1 shows that the highest level of defining analytical quality specifications is based on studies that can evaluate the effect of analytical performance on clinical outcome but rarely available in practice. An imprecise or biased method would logically misclassify patients and potentially lead to suboptimal clinical management and poorer clinical outcome. At the other extreme, ‘state of the art’ analytical quality specifications, such as the expected imprecision of a particular method listed in a kit insert, do not guarantee that the method is fit for the variety of clinical purposes that may exist for that analyte. Analytical goals are essentially required to assure diagnostic classification.^12–14 Therefore, the hierarchy is a classification of our confidence in the clinical purpose of the assay rather than an isolated study of analytical quality.

ISO 15189 literally requires medical laboratories to include biological reference intervals or clinical decision limits, whichever is applicable, because a single value has little relevance in isolation. While subsequent values can be compared with the initial value when monitoring and trying to judge if a patient is getting better or worse, the initial result is usually compared with a confidence interval that describes apparently healthy reference individuals (reference interval) or with a limit that defines the likelihood of its indicating a particular disease state (clinical decision limit). The quality of reference intervals and decision limits will affect an individual patient’s classification just as much as the analytical quality of that individual patient’s value(s).

While the laboratory profession has already taken significant steps in the traceability or standardisation of methods, more recently we have become concerned with what appears to be a large variation in reference intervals between laboratories. Professional dialogue and co-operation is now required to reduce the unnecessary variation in reference intervals, especially since the move to electronic health records will otherwise entail a plethora of idiosyncratic laboratory-based reference intervals. These professional efforts in harmonisation of reference intervals are aimed at finding the best available evidence and encouraging their use.

Considering that the Stockholm Hierarchy can be viewed as a quality based hierarchy for judging the clinical worth of laboratory measurements, we should be able to use it in ranking the quality of candidate reference intervals (and decision limits) for common use. The remainder of this article is devoted to illustrating the application of each level of the Stockholm Hierarchy to the derivation and quality of reference intervals and clinical decision limits.

Level 1. Clinical Decision Limits Based on Clinical Outcomes in Specific Clinical Settings

Evidence based medicine is about making clinical decisions based on the evidence of proven clinical outcomes. In the Diabetes Control and Complications Trial (DCCT), HbA_1c levels over 8.0% (64 mmol/mol) were associated with worse outcomes than levels below 7.0% (53 mmol/mol) and therefore these were proven clinical decision levels for intervention when monitoring diabetes mellitus.¹⁵ More recently the global acceptance of HbA_1c as a diagnostic test for diabetes mellitus is based on the understanding that the specific adverse clinical outcomes characteristic of that illness (e.g. retinopathy) start to rise at a DCCT level of 6.5% (48 mmol/mol).¹⁶ There are some clinical outcomes present in both diabetes and prediabetes (e.g. coronary vascular disease and stroke) that begin to be noticed at DCCT levels of 5.5% (37 mmol/mol).¹⁷ Therefore there are several clinical decision limits that we can apply to HbA_1c; (i) 7.0% vs. 8.0% for altering diabetic management, (ii) ≥6.5% for diabetes diagnosis and (iii) ≥5.5% for assessing cardiovascular risk.

The strengths of this approach are the link to clinical outcome and the promise of improving those outcomes with targeted intervention. The weakness is that these clinical decision limits are usually specific to a particular clinical outcome, which may or may not be the foremost clinical question for each individual patient. Another major problem is the amount of effort required to define these limits for each possible clinical outcome.

Similarly, while outcome based clinical decision limits may be defined for a specific population, that clinical outcome may have less relevance in other populations. For example, cholesterol cut-offs based on the risk of cardiovascular disease in adults may not be relevant in paediatrics or obstetrics. Similarly with urate, the upper ‘apparently healthy’ reference limit may be different from the clinical decision limit for predicting gout and from the concentrations associated with metabolic syndrome and pre-eclampsia in pregnancy.

Level 2. Evaluation of the Effect of Reference Limits on Clinical Decisions in General

When addressing analytical quality, this level had two subcategories, one based on data derived from biological variation and another based on clinicians’ opinions. Biological variation involves the study of both intra- and inter-individual variation, and population based reference intervals include both of these components as well as the measurement uncertainty of the assay used to generate them. Clinician opinions are based on surveys of what clinicians believe, and this should be distinguished from clinical decision limits which are based on what clinicians have proven through formal clinical outcome studies. As we lack such high level evidence for all clinical decisions, each clinician often has to weigh up the available evidence when deciding on each individual patient’s management. While this process is celebrated as ‘the art of medicine’ and requires years of experience, the task can be made more objective by expert clinicians creating consensus guidelines.

2a. Reference Intervals Defined by Intra- and Inter-Individual Biological Variation

The advantage of reference intervals is that they represent our best effort at defining the spread of results in health, and results outside a reference interval are assumed to indicate a high relative risk of disease. As it is almost impossible to find individuals in a perfect state of health, a practical alternative suggestion is that reference individuals are in a reasonable state of health and not suffering from significant illness.¹⁸ The weakness of this approach is that it is hard to define and find such healthy individuals that equally represent the many physiological variations that can exist in health. Some of those important healthy variations include pregnancy, gender and age (as in infants, children, puberty, young adults, menopausal women and elderly adults). There are also appreciable diurnal variations with fasting, eating, drinking and exercise, as well as monthly variations (menstrual cycle) and circannual variations including diet, sunlight and ambient temperature.

Fortunately, an internationally accepted standard for defining reference intervals exists (CLSI C28-A3).¹⁹ In the direct approach, healthy individuals are recruited, re-assessed for health, their samples analysed and the results described using parametric or non-parametric statistics. It takes a major effort to define these populations and analyse them in a consistent or traceable manner. Excellent examples include the NORIP study defining adult reference intervals in Scandinavia,²⁰ the six Asian cities study,²¹ the CALIPER studies defining paediatric intervals in Canada²² and the paediatric health survey in Germany (KiGGS).²³ While it might be assumed that these populations are homogenous, major differences do exist,²⁴ diminishing their general applicability. Furthermore, the methods used in the studies define both an accuracy base and expected imprecision for field methods.²⁵

Indirect methods seek to distinguish reference populations within the mixture of populations that exist in a laboratory database.^26,27 They usually depend on an assumption of a Gaussian or log-Gaussian distribution of reference data. Importantly, they do not represent healthy reference intervals because most patients tested in clinical laboratories have a clinical complaint. Nevertheless, since an ideal reference population is one that exactly mirrors the patient except for the disease in question, it can be argued that reference populations defined within a laboratory database are better candidates for comparison: they have had their samples collected, transported and analysed exactly as patients do, and they have similar complaints (e.g. abdominal pain) without the disease (e.g. pancreatitis). In another example, reference populations for phaeochromocytoma should be patients with hypertension (but no phaeochromocytoma)²⁸ rather than completely healthy normotensive populations. The issue of whether the reference population should be in an ideal state of health or more simply not have the disease in question addresses a weakness of ‘healthy’ reference intervals, namely that they are defined by specificity for health rather than sensitivity for disease. A disadvantage of the indirect method is that the reference intervals derived may vary with long-term variations in analytical quality.²⁹ Studies have been conducted that show that the main differences between directly and indirectly defined reference intervals are due to assay bias.³⁰

Both parametric and non-parametric approaches usually describe 95% confidence intervals for the reference population. However, this convention can be modified to 99% confidence intervals where greater specificity is required (e.g. troponin, CA125, alpha fetoprotein) or 90% confidence intervals when greater sensitivity has been accepted as a convention (e.g. testosterone). In other words, healthy reference intervals are by definition expressions of specificity for health; they don’t directly transpose to the likelihood of disease, the more typical clinical question.

2b. Decision Limits Based on Clinicians’ Opinions

While clinicians’ opinions can be used to decide whether a reference interval should have 95% or 99% specificity, this category refers more to the setting of a clinical decision limit based on clinical survey or consensus rather than a limit derived through a clinical outcome study as described initially above. An example of a clinical decision limit based on consensus is an upper reference limit for thyroid-stimulating hormone (TSH) of 2.5 mIU/L by the National Academy of Clinical Biochemistry (NACB).³¹ The strength of the approach is that clinicians can balance the need for specificity, sensitivity and management. The weakness of this approach is that clinicians may not be aware of the variations among assays that exist despite claims to traceability to a common reference standard.³² Other examples include the definition of testosterone deficiency (<8.5 nmol/L) by specialists advising the Australian pharmaceutical benefits schedule,³³ and the recent definition of vitamin D sufficiency at the end of winter in Australia as ≥60 nmol/L.³⁴

Ideally, laboratorians should be involved in the discussions of clinical decision limits as they may be more aware than the clinicians of differences in methodologies and they can also help to ensure that laboratories adopt the recommendations of the group.^35–38

3. Published Professional Recommendations

Recommendations can come from national or international expert bodies or from local groups or individuals. They usually refer to laboratory experts defining reference intervals rather than clinicians involved in clinical decision limits. However laboratory experts can promote the adoption of clinical decision limits in preference to reference intervals and include further recommendations on how laboratories should structure their reports.³⁹

Examples of reference intervals determined by laboratory professionals include those promoted by the Auckland Regional Quality Assurance Group (ARQAG),^40,41 and those of the Sonic biochemistry reference interval network.^42–45 This is not to say that these groups create reference intervals without evidence such as contained in the aforementioned approaches, but that these groups provide a laboratory perspective to the data that is available by pragmatically agreeing on reference intervals that are most appropriate for the populations and methodologies used in their network. Therein lies both the strength and weakness of this approach. Apart from harnessing the combined experience and qualifications of experts in the network, the strength is that these experts can gather all available evidence that applies to their populations and consider any method biases that exist in their methods, including pre-analytical factors. The weaknesses are that the reference intervals will need to be redefined when there is a change to network methods and that they may not be applicable to other networks with differing methods or populations (e.g. ambulant outpatient populations versus recumbent inpatient populations).

4. Reference Limits Set by Regulatory Bodies or Organisers of External Quality Assurance Schemes

When setting analytical quality goals, it is possible to consider the existing achievable performance benchmarks in external quality assurance surveys to establish a standard that can be achieved. It is also possible to compare analytical variation with variation in reference intervals to investigate whether reference interval variation is greater than the variation between methods.^46,47 Reference interval surveys have been conducted in the USA⁴⁸ as well as the UK where the surveys form part of the information for harmonising reference intervals.^49,50 A proposed benchmark from such surveys might be the most common interval used or the median reference interval. Postanalytical external quality assurance (EQA) can also serve to improve both the quality of laboratory based interpretation and the reference intervals that laboratories use to make those interpretations.⁵¹

Regulatory authorities can also play a large role in determining reference intervals and clinical decision limits. When an assay is approved for use by local regulatory authorities, the approval often includes the expected patient distributions or reference limit for that assay. Subsequent approval of similar tests may be based on their similarity to predicate devices including having the same interpretive limits. In 1994, prostate-specific antigen (PSA) was approved by the US Food and Drug Administration (FDA) as an aid for early prostate cancer detection using a 4.0 μg/L cut-off. Subsequent approvals for PSA also claimed a 4.0 μg/L cut-off for all men even though clinical practice had progressed to using lower reference limits in younger men to define men with more favourable prognosis.⁵² This issue has also complicated the approval of cardiac troponin I assays when the FDA has a predicate device with a cut-off defined by receiver operating characteristic (ROC) analysis, while cut-offs for new troponin I assays are now being defined by evaluating diagnostic performance at the 99^th percentile of the reference population.⁵³

5. Reference Limits Based on the Current State of the Art or Current Publications

This was defined as the lowest level of assessing analytical performance goals because laboratories that compared their analytical performance with published data could not be sure that the published analytical performance fulfilled any quality goals itself. Taking a reference interval from a publication usually does not provide the confidence that the published reference interval was established using the same method as yours let alone that the reference population would be the same as yours. Comparable methods and comparable reference populations are the two essential considerations according to the CLSI C28 guideline when laboratory directors subjectively accept published reference intervals. While a reference interval (or ‘expected values’) from a kit insert that you are using might be assumed to be the same method, this usually requires method validation. Similarly, while a reference interval derived in the USA or Europe might be assumed to be similar to one in Australia, this also requires validation. This is why almost all kit inserts suggest that the laboratory performs their own reference interval studies. Validation of the published reference interval is sometimes impossible when the kit insert does not include a description of the reference population including their age and gender, and too often it fails to recognise important morbidities such as obesity. It may have once been acceptable to extract reference intervals from common textbooks, but this is (by this definition) the worst way to do things. Of course, sometimes there are no reference intervals available for your particular method in children or pregnancy so published values such as provided by reference books by Soldin⁵⁴ and Gronowski⁵⁵ can be useful guides. However, the methods used in those publications and their reference populations (e.g. hospitalised children) must be critically evaluated.

Discussion

It should be noted that while outlining the levels of the Stockholm Hierarchy as it would apply to reference intervals and clinical decision limits, I have alternated between these different concepts. For example, level 1 is to do with clinical decision limits, level 2 is either to do with reference intervals or clinical decision limits, and level 3 is mainly to do with reference intervals but could be to do with clinical decision limits, as is also true for levels 4 and 5. In reality, there are two hierarchies, one set that applies to reference intervals and another set that applies to clinical decision limits. The only distinguishing feature is whether we are focusing on limiting false positives in health (specificity) or limiting false negatives in disease (sensitivity). For example, level 1 does not necessarily have to address the clinical outcomes of disease; it could apply to the expected clinical outcome when disease is absent (health), e.g. survival or longevity. Finally, the two aspects, health and disease, can be combined statistically though ROC analysis where cut-offs that balance sensitivity and specificity can be defined. This approach is useful for all the situations where we cannot decide whether test sensitivity or specificity is the more important issue.

I hope I have demonstrated that the Stockholm Hierarchy, initially proposed as a tool to define analytical quality, is just as useful a tool for evaluating reference intervals and clinical decision limits. The recent literature indicates that the inter-relationship between analytical goals, reference intervals, decision limits and their impact on patient classification (or misclassification)^56–58 is the current frontier that brings together everything that underlies the value of our clinical laboratory tests.

Table 2.

The Stockholm Hierarchy applied to reference intervals and clinical decision limits.

Clinical decision limit based on clinical outcome study e.g. HbA1ccut-off based on the presence of diabetes outcome (retinopathy).16 Other methods of determining reference interval or clinical decision limit Reference intervals derived from apparently healthy populations e.g. NORIP,20 CALIPER.22 Clinical decision limits based on clinicians’ opinions of disease e.g. thyroid-stimulating hormone (TSH) upper reference limit (2.5 mIU/L) from NACB.31 Published professional recommendations National or international expert bodies e.g. national urine protein cut-offs.38 Expert local groups or individuals e.g. ARQAG,41 SONIC.43 Reference limits set by Regulatory bodies e.g. prostate-specific antigen (PSA) cut-offs.52 Formal Reference Interval Survey e.g. UK Harmony Survey.49 Referance interval based on the current state of the art Reference interval used in postanalytical external quality assurance e.g. pathology interpretation exercises.51 Current publications on methodology e.g. textbooks or kit inserts.54,55