Measuring Success: Disease Intervention Specialists Performance Metrics and Outcome Assessments

We critically examine the benefits and limitations of existing measures used to assess the impact of disease intervention specialists work on disease prevention.

Abstract

Disease intervention specialists (DIS) are the cornerstone of public health. However, the incremental gains of DIS-led interventions are difficult to detect at the population level. Health departments attempt to quantify the impact of key DIS activities through performance measures that assess how many and how quickly both patients are interviewed, and contacts are notified, tested, and treated. However, DIS work encompasses more than case finding and existing performance measures may not capture the full value DIS provide to health departments. In this article, we first describe how DIS investigations and contact tracing are conducted for sexually transmitted diseases and other communicable diseases to understand how the definition of effectiveness may vary by disease. Then, we examine the benefits and limitations of traditional performance measures using syphilis investigations as an example. Recognizing the limits of existing measures will improve our understanding of DIS impact and assist in the development of new measures of effectiveness that better represent the totality of DIS work.

The disease intervention specialist (DIS) workforce has long been considered critical to the identification and disruption of communicable disease transmission networks in the United States. Originally introduced in the 1930s to prevent syphilis transmission by notifying and assuring the treatment of sexual partners,¹ DIS are called upon to provide expertise for a variety of investigations conducted by state and local health departments. Disease intervention specialists not only control disease through partner notification but also collect valuable information about disease networks that help health departments understand transmission and direct resources. Their unique skill set includes the ability to find hard-to-reach individuals, translate sometimes complex test results and prevention guidance to their clients, and elicit the names of contacts who might have been exposed to a communicable disease, such as syphilis, human immunodeficiency virus (HIV), or tuberculosis.² Disease intervention specialists often are among the first people deployed to assist with emerging threats, including COVID-19 and mpox.^3–5 Disease intervention specialists also have been called upon to refer patients and their close contacts to wraparound support service that extend beyond the notification and treatment of communicable diseases.^6–8

To understand public health impact and quantify the return on investment of scarce public health resources into the DIS workforce, accurate measurement and regular monitoring of key activities and outcomes are necessary. Most DIS activities are conducted with the goal of decreasing incidence and prevalence of disease in the populations they serve. These goals and priorities should directly align with what we are measuring. For most DIS work, particularly related to sexually transmitted disease (STD) and HIV, health departments have prioritized case-finding and treatment. However, published data from the past 40 years suggests DIS consistently cannot find and treat all new cases of disease through STD/HIV partner services. The average number of diagnosed partners treated for syphilis because of DIS intervention per patient (ie, brought-to-treat index) has ranged from a high of 0.46 in rural Texas in the early 1990s to a low of 0.07 in Philadelphia in 2010.^9–13 Both the populations served by DIS and the interventions provided have changed over the past several decades.^2,14–18 Yet, our standard set of performance measures has largely remained the same over time. In this article, we sought to describe aspects of effective DIS and contact tracing work for a variety of communicable diseases to understand what can and should be measured. Then, using syphilis as a model, we aimed to identify the aspects of DIS work that are measured well and the aspects that are not measured well by our traditional metrics. Through this critical examination of outcomes and performance measures, we attempt to improve our ability both to assess the effectiveness of DIS-led interventions and recognize the value provided by the DIS workforce.

QUANTIFYING EFFECTIVENESS

The threshold for determining effective DIS work depends on the transmission dynamics, disease severity, and likelihood of treatment and prevention of the disease being investigated. Syphilis is transmitted sexually, limiting the number of people exposed to the infection. In addition, the time between infection and infectiousness for syphilis is about 21 days.¹⁹ It is theoretically possible for DIS to reach all exposed partners for a syphilis patient (if DIS are aware of them) in time to assure the provision of an effective and easily administered cure before transmission can occur. Disease intervention specialist work most rapidly to find and treat persons diagnosed with primary and secondary syphilis who are at highest risk of transmitting disease and pregnant women diagnosed with any stage of syphilis to prevent congenital syphilis, with its negative associated sequelae. Similarly, for HIV investigations, DIS strive to quickly reach persons diagnosed with acute HIV who are highly infectious. However, it should be noted that stigma surrounding STDs, anonymous partnerships, and privacy concerns, among others, contribute to the underreporting of partners to DIS and subsequently reduce the impact DIS can have on preventing transmission.^2,20,21

Although the goals of contact tracing for other communicable diseases include finding new cases of disease and interrupting transmission, the approach and expectations may differ from STDs. For example, the outcomes for Ebola are severe and include death. For contact tracing to be effective at containing Ebola, all contacts must be found quickly and should be quarantined and monitored for symptoms until they are outside the incubation period. Contact tracing for tuberculosis is also intensive and attempts to identify, quarantine, and monitor exposed contacts and treat secondary cases of active and latent tuberculosis quickly to prevent transmission, particularly in immunocompromised and other vulnerable populations. To prioritize work, tuberculosis contacts are typically assigned a risk level based on proximity and duration of exposure to infected persons. Higher risk contacts are followed up until no secondary cases are identified. Health departments then expand contact tracing to lower risk persons as they deem necessary. Measures of effectiveness for tuberculosis contact tracing include case finding and completion of monitoring among prioritized contacts. For respiratory diseases like COVID-19 with short window periods between exposure and infectiousness, contact tracing may be less successful because timely notification and assurance that all exposed individuals quarantine and/or isolate is nearly impossible. Furthermore, the number of cases makes public health-provided case investigation and contact tracing difficult. Therefore, health departments emphasized the timely notification of case-patients and their contacts via more passive methods such as text messaging. Case interviews and contact tracing efforts for respiratory diseases like COVID-19 increasingly were prioritized to the populations that are most vulnerable and to the settings where disease has a better chance of being contained through public health efforts.⁷ Contact tracing for many other reportable communicable diseases, such as shigella, focus on case-finding outcomes in a specific setting of exposure (eg, childcare facilities). The goal of these exposure setting contact tracing efforts is to contain an outbreak in the setting before it can spread to the community. Once all cases have been found and treated appropriately and no new cases are found among contacts within the specified incubation period, these settings can resume normal operating procedures.

The incremental gains from contact tracing and other DIS work may be difficult to detect as reductions in incidence and prevalence at the population level. Multiple factors influence disease incidence, including changes in partner mixing patterns, condom use (for STDs), availability of treatment, care-seeking behavior, and screening coverage. Unless these factors are held constant, the impact of DIS work can easily be underestimated. Disease intervention specialist efforts are often focused on providing safety-net interventions in specific populations because health departments often do not have enough DIS to provide the level of intervention needed to impact the entire population. As a result, population-level outcomes, such as incidence and prevalence, are not good markers of whether DIS-led interventions are successful or what they contribute to disease intervention efforts.

WHAT IS MEASURED WELL

To assess performance, health departments measure many steps along the path between the DIS-led intervention and population-level outcomes. Most of these performance measures assess how quickly and how often specific tasks are being done.^9,11,13,22 For syphilis partner services, we typically measure DIS work through a series of process activities and clinical outcomes (Fig. 1). Process metrics evaluate the actions DIS take to accomplish their jobs: interviewing patients, eliciting contacts, and finding, notifying, and testing contacts. These process metrics also have been used to assess contact tracing for other communicable diseases.⁷ Measures of clinical outcomes assess the diagnosis and treatment of patients and their contacts. Health departments use the data collected by DIS to calculate contact, treatment, and disease intervention indices to measure performance (Table 1). These indices attempt to relate outcomes to the intervention provided by the DIS (e.g., number of contacts named per patient interviewed or the number of contacts brought to treatment by DIS per patient interviewed).

Figure 1.

Overview of DIS case-finding process for sexually transmitted diseases.

TABLE 1.

Strengths and Limitations of Commonly Used DIS Performance Metrics for STD and HIV Investigations

Commonly Used Metric	Definition	Strengths	Limitations
Contact Index	No. contacts/no. case patients*	• Measure important steps of a DIS investigation • Straightforward • Easy to calculate • DIS routinely collect these data during investigations • Input data are readily available in program databases • Based on longstanding disposition codes widely used by DIS	• Does not consider unnamed contacts • Does not consider unreported contacts • Does not capture the number of people notified or treated due to the indirect work of DIS • Misclassification of disposition codes used to inform performance metrics is possible • Diagnosis and treatment of comorbid conditions not always (well) captured or reported • Referrals to other public health services not always (well) captured or reported
Notification Index	No. contacts notified/no. case patients*
Case-finding Index	No. contacts found to be infected/no. case-patients*
Treatment Index	No. contacts treated/no. case-patients*
Brought-to-Treatment Index	No. infected contacts who were treated because of DIS efforts/no. case-patients*

*Denominator can be all reported, assigned, or interviewed case patient.

Traditional performance measures for syphilis partner services have many useful qualities. They are simple and straightforward. These metrics are based on a few key activities that are routinely collected during DIS investigations (eg, number of partners, number of partners notified, number of partners treated) and well documented in most data systems. This allows the metrics to be calculated easily for different periods and subpopulations, facilitating comparison. Similarly, these metrics are often used to review the work done by individual DIS to identify well-performing DIS as well as training gaps. Many of these metrics rely upon longstanding disposition codes that are assigned by DIS to categorize syphilis notification, diagnosis, and treatment outcomes for cases and their contacts. If applied correctly, programs can use disposition codes to detect changes in DIS effectiveness across populations. However, these codes may not be uniformly applied in practice.

Finally, existing performance measures consider the direct impact of DIS activities on finding and treating new cases of disease. We know roughly how many patients DIS interview and how many partners DIS notify, test, and treat. This allows programs to assess more than just test positivity among named partners. Health departments can precisely understand where along their continuum of work that DIS are more and less successful by measuring each step between the reporting of a new case of disease and the testing of their partners. Furthermore, these metrics allow health departments to quantify the number and proportion of partners that were treated because of DIS efforts. By examining which patient populations yield better case finding and treatment outcomes, health departments can intentionally prioritize DIS investigations to the groups where their services may be most impactful and identify populations where standard DIS practices are less successful in interrupting transmission. Test positivity among all contacts does not necessarily reflect all the work of the DIS to find, diagnose, and treat exposed persons in priority populations.

WHAT IS NOT MEASURED WELL

Despite their benefits, traditional performance measures for DIS have several limitations. First, traditional performance measures are missing a lot of information. Most programs only require DIS to capture information about partners they have the potential to find (because the original patient has provided some locating information). Syphilis and HIV partner services call these “named partners.” Sometimes, patients report the existence of additional partners to DIS, but either cannot or do not provide enough locating information for the DIS to initiate an investigation (unnamed partners), and some partners go completely unreported. The sum of unnamed and unreported partners represents the portion of exposed partners that DIS do not interview or contact. For syphilis, unnamed and unreported partners have been estimated to represent between 75% and 82% of exposed partners.²⁰ The proportion of unreached partners may be higher or lower for other diseases depending on the ability to know who is exposed, disease-related stigma, and the ability of DIS to follow up with patients. If the goal of DIS is to find and treat new cases to control disease, we need to recognize that our current indices do not assess outcomes from most exposed partners. However, if we prioritize our DIS workforce and performance measures to populations where DIS work is most effective or to populations most in need (either by design or out of necessity because of staffing issues), the proportion of exposed partners not reached is less meaningful. Regardless of the approach, health departments should assess and recognize this missing data, particularly among the populations we in public health need to assure disease control (eg, pregnant women).

If we think of DIS work as being linear, then traditional performance metrics do a decent job measuring the direct impact DIS have on finding, testing, and treating new cases of disease. However, the impact of DIS work to prevent disease extends beyond this direct pathway. The conversations that DIS have with patients who do not report any partners (even a source for their infection) could result in the patient self-notifying their partners and/or contacts. The work DIS do to engage and educate providers and other community stakeholders often leads to the testing and treatment of both named and unnamed partners via an indirect path (Fig. 2). The partners treated via these indirect pathways are not easily measurable by health departments and as a result, are not included in traditional performance metrics. Yet, these indirect pathways no doubt lead to the notification, testing, treatment, and linkage to care of partners.

Figure 2.

Pathways to intervene on disease transmission: direct DIS intervention, indirect DIS intervention, and no DIS intervention.

In practice, indirect effects are difficult to count and measure. Advanced epidemiologic methods have been developed to quantify the indirect effect of interventions on outcomes (eg, DIS interventions to treat partners for syphilis).^23,24 However, these methods assume that there is no unmeasured confounding, which may not be the case when applied to real-world settings. Furthermore, most health departments do not have the capacity to apply these methods and communicate them to leadership and other key stakeholders. Capturing the number of patients who report that they will self-notify their partners will give programs an estimate for the number of additional partners notified by patients after conversations with DIS. However, not all patients who say they will self-notify their partners will do so. It is difficult for public health to know if these partners were notified, tested, or treated if we do not know who the partners are. The number and percent of case patients who report they were diagnosed because of DIS efforts could be captured during case investigation interviews and may fill in some of the gaps about the indirect pathways of DIS work. However, this metric is imperfect if the patient did not know they were notified because of conversations between the original patient and DIS. Additional simple and straightforward methods need to continue to be developed and evaluated to estimate indirect pathways for a fuller understanding of DIS impact.

In addition to the indirect intervention pathways, DIS often are expected to offer prevention services that extend beyond the assigned disease. For syphilis partner services, DIS provide a variety of HIV services that include HIV testing, linkage (or relinkage) to care if the person tests positive, and referrals to preexposure prophylaxis (PrEP) if the person tests negative. Because people with a new syphilis diagnosis are at a higher risk of acquiring and/or transmitting HIV,²⁵ incorporating HIV services into syphilis investigations is an excellent use of resources. Disease intervention specialist efforts have led to the diagnosis of HIV in 6% to 7% of syphilis patients and up to 3% of partners.^9,12,26–29 Disease intervention specialist have been successful with referring syphilis patients and their partners for PrEP services in Seattle, San Francisco, Iowa, and Chicago, but there is wide variation in how these outcomes are documented and reported across populations and jurisdictions.^26,30–32s Because the provision of HIV services during syphilis investigations is already a prioritized activity, particularly for men who have sex with men (MSM), we need to have better and standardized measurement of this work. Excluding these HIV prevention activities from measures of DIS effectiveness will artificially diminish the perceived public health impact of DIS work in the populations they serve.

Contact tracing and partner services work have also been used as an opportunity refer patients and their partners for supportive services. This has been a particular focus for HIV partner services, and more recently the COVID-19 response. Contact tracers for COVID-19 were often the first point of contact for support services offered by public health (eg, food, shelter, and financial assistance) and demonstrated the utility of offering these services during interviews.^6–8,33s Because the outcomes from referrals are historically not the primary purpose of contact tracing, they are not as well documented as the activities performed by DIS to investigate the disease of interest. The guidance for calculating useful metrics related to referrals and the utility of these metrics is less well understood than traditional performance measures for DIS work. Evaluation of and research on the referrals DIS are currently making and their downstream consequences will help identify which referral actions are important to patient support and disease control.^2,6,8 This will allow programs to efficiently track these services and assure complete data capture over time and across populations.

Traditional performance measures for syphilis and HIV are based on a set of disposition codes assigned by DIS. CDC provides guidance about how these codes should be applied,^34s but not every case of syphilis is clear-cut and misclassification of disposition codes is possible. Many health departments have developed quality control checks for DIS-entered data to assure codes are being used appropriately. However, in times of case surges and staff shortages, it may be difficult for programs to thoroughly check the accuracy of assigned codes in a timely manner. In an investigation of disposition codes assigned to partners of syphilis case-patients in 7 jurisdictions in the United States, 43% of partners assigned as being infected and brought to treatment because of patient interview and partner notification (eg, assigned a C disposition code), were treated before the original patient who named the partner was interviewed.⁹ Similar misclassification of disposition codes has been observed in HIV investigations.^35s,36s Disease intervention specialist often interpret complex scenarios differently and therefore do not assign disposition codes uniformly.^37s The same measures used to assess program effectiveness are used to assess DIS performance. Although this practice assures DIS are rated on the most important aspects of DIS work, it can also lead to misrepresentation in the assigned disposition codes to achieve more favorable scores during performance reviews. Standard data points captured in surveillance systems (eg, dates) could be used to automatically classify the process and clinical outcomes of DIS investigations and enhance correctly assigned DIS disposition codes. These system-generated classifications would need to be validated against DIS classifications given the nuances of DIS work.^36s

Regular review of the data collected by DIS and subsequent conversations between field staff and epidemiologists may help focus DIS efforts for maximum impact. For instance, assessment of syphilis partner services data has demonstrated that MSM provide locating information for a smaller proportion of their partners on average compared with women and men who have sex with women only.²⁰ Consequently, DIS-led interventions for syphilis may be more successful in women where they can find more partners and potentially prevent congenital syphilis, one of the most serious outcomes of syphilis. Epidemiologists may also help DIS find the right balance between speed and effectiveness. Although studies have demonstrated that investigational outcomes are better when DIS reach patients and their contacts quickly,⁹ holding DIS to strict timeframes could be a detriment to the overall goals of DIS in some instances. Focusing on quick outreach also increases costs and possibly decreases efficiency and effectiveness within programs because DIS are more concerned with closing cases than gathering information needed to intervene on transmission. Reducing the value speed plays in our performance measures for certain diseases or in some populations may allow DIS to better prioritize their caseload and have time to conduct a full and thorough investigation, thereby making their intervention more impactful finding and treating more cases.

CONCLUSION

Disease intervention specialist work hard. Their efforts are undeniably valuable to health departments, but quantifying that value is difficult. Without the counterfactual scenario of what would happen in the absence of DIS, we will not be able to precisely measure the magnitude of their impact to reduce disease incidence and prevalence. Public health has invested in the DIS workforce at the federal, state, and local level; accurate measurement of DIS activities will help public health agencies better distribute and direct funds for optimum impact on disease incidence and prevalence. We can and do measure many of the direct outcomes of DIS work related to finding, testing, and treating new cases of disease. However, the activities we measure do not represent the totality of DIS work. We can improve our understanding the effectiveness (and cost-effectiveness) of DIS-led interventions by first recognizing and naming the limitations of these existing metrics and the data that informs them. Second, we know that traditional measures capture only a subset of DIS work. A better understanding of useful and meaningful metrics for evaluating the “indirect” work DIS do to stop disease transmission and counting referrals made for other public health services would be beneficial. The potential for STD investigations to link patients to PrEP and HIV treatment may mean prioritizing the collection and measurement of HIV outcomes. Third, we need to develop approaches to limit error and misrepresentation in the data used to calculate these metrics. Public health data systems may allow or be developed to enhance more automated data capture, reducing data input errors. This information can supplement DIS-assigned disposition codes and allow for a better understanding of program effectiveness. Finally, because of limited time and resources, the impact of DIS work is often not visible at the population level. However, strong partnerships between DIS and epidemiologists might facilitate evidence-based decision-making to prioritize DIS work to populations where it is most needed and can be most impactful. Disease intervention specialist work is foundational to public health, particularly for diseases like HIV and syphilis. Using our traditional measures of effectiveness allows for comparability over time and space. However, as DIS work evolves, so too must our measurement. Critically thinking about what and how we should be measuring on a routine basis will only improve our ability to both assess and make an impact.

For further references, please see “Supplemental References,” http://links.lww.com/OLQ/A875.

Supplementary Material

std-50-s18-s001.docx ^{(14.1KB, docx)}

For further references, please see “Supplemental References,” http://links.lww.com/OLQ/A875.