Accuracy of Physician Reporting in Routine Public Health Surveillance for Hepatitis C Virus Infection

Abstract

Objective

From January 2007 to December 2008, the Montréal Public Health Department sent postal questionnaires to physicians and conducted patient interviews for all those newly diagnosed with hepatitis C virus (HCV) infection. We evaluated physician responses to risk factor questions for non-acute HCV cases.

Methods

We compared physician and patient responses with each of nine risk factor questions, determined the sensitivity and specificity of physician responses compared with patient responses, and evaluated agreement using Gwet's agreement coefficient (AC₁). We ranked risk factors and compared the distributions by principal exposure category according to physician reporting vs. patient interview using the Chi-square test.

Results

The completeness of physicians' responses (yes, no, or unknown) varied by risk factor question from 90.8% to 96.7%. For risk factors present among more than 5% of cases, sensitivity of physician responses ranged from 26.9% to 87.7% and specificity ranged from 93.0% to 98.6%. The AC₁ coefficients for agreement between physician and patient responses to lifetime risk factors considered most important in HCV acquisition were 0.80 for injection drug use, 0.95 for blood transfusion before 1990, and 0.86 for birth in a country with high HCV prevalence. Risk distributions by principal exposure category according to physician reporting vs. patient interview were not statistically different (χ²[4] = 2.17, p=0.704).

Conclusion

Postal questionnaires completed by physicians appear valid for determining the principal exposure category among non-acute HCV cases. Physician reporting can be a useful and low-cost component of routine HCV surveillance.

Hepatitis C virus (HCV) infection is an important health problem affecting all countries, with an estimated global prevalence of 2.35%, representing 160 million chronically infected individuals.¹ Initial infection, most commonly by parenteral exposure to infected blood, is usually asymptomatic. Three out of four infections become chronic and may not be diagnosed until years later, when the patient develops abnormal liver function or presents with complications such as cirrhosis, liver failure, or hepatocellular carcinoma. A modeling study of the incidence and prevalence of HCV infection and sequelae in Canada estimated that about 242,500 individuals, or 0.78% of the population, were living with HCV infection at the end of 2007.² According to the modeling results, 11% of prevalent cases had been infected through blood transfusion, 36% were past injection drug users (IDUs), 22% were current IDUs, and 31% had been infected through other routes.

Risk factors for HCV infection have changed over time, both in North America and globally.^3–5 Before 1990, when the first serological test for HCV became available for donor screening, HCV-contaminated blood was a well-recognized cause of post-transfusion hepatitis. Since the introduction of nucleic acid testing for HCV in 1999, the per-unit risk of HCV contamination of blood or blood products in Canada has fallen to approximately one in three million.^6,7 Among people born outside of Canada, the likelihood of HCV infection prior to immigration depends on age, risk behaviors, and the extent of unscreened blood donation or inadequate sterilization of reusable medical devices in the country of origin.^4,8,9

Sharing drug injection equipment is the predominant risk factor for new HCV infections in Canada,^10,11 and the importance of screening current IDUs for HCV is understood. In contrast, people who injected drugs decades ago may not be questioned about past risk behaviors, and immigrants who acquired infection through other routes may also remain undiagnosed until complications arise. Because of the late emergence of sequelae, it is expected that an increasing number of people will require treatment for HCV-related liver disease in the coming years.² A recent cost-effectiveness simulation study in the United States found that one-time HCV screening among adults born between 1945 and 1965 could potentially prevent 82,000 HCV-related deaths compared with the status quo.¹² Characterizing the demographic and risk profiles of people infected in the past but diagnosed today is, therefore, integral to evaluating HCV screening policies and practices and planning appropriate treatment services for different populations.

Surveillance context

HCV infection became a notifiable disease in the province of Québec in 2002, requiring diagnostic laboratories to report all positive HCV test results and physicians to report each newly detected case to the regional public health department. In practice, laboratories report regularly and physicians report rarely. Given the high number of HCV case reports at the time (about 1,000 annually),¹³ the Montréal Public Health Department (MPHD) opted in 2003 to implement a surveillance system whereby physicians who had ordered the HCV diagnostic test were asked to complete a short postal questionnaire.

In 2005, the MPHD began participating in the Enhanced Hepatitis Strain Surveillance System (EHSSS) funded by the Public Health Agency of Canada.¹⁰ Under the EHSSS protocol, systematic efforts were made to contact all newly diagnosed HCV cases to conduct a telephone interview using a standardized questionnaire and, when required for the purpose of case classification, to obtain information from the clinic chart.

This study was conducted to assess the usefulness of physician reporting in determining the principle exposure category for non-acute cases in the context of routine HCV surveillance. The study objective was to evaluate physician responses to risk factor questions on a postal questionnaire when used for non-acute HCV cases. We compared physician and patient responses to the same risk factor question, determined the accuracy of physician responses to the responses obtained through patient interview, and evaluated agreement between the two. After ranking risk factors and assigning cases to a principal exposure category, we compared the risk distributions obtained by each surveillance method.

METHODS

Surveillance questionnaires and procedures

During the study period, two questionnaires were used to collect information on HCV case risk factors. A two-page questionnaire was mailed to physicians who had ordered the first confirmed positive HCV test reported to the MPHD. If postal questionnaires were not returned after the first mailing, they were followed up by a second mailing and then a telephone reminder at approximately three-week intervals. The response options were yes, no, and unknown.

A second questionnaire was based on the standardized EHSSS patient questionnaire and administered by telephone to the patient or through a family member or interpreter. The interview questionnaire was more detailed but included all of the risk factor questions found on the postal questionnaire.¹⁰ The interview took approximately 30 minutes and was conducted by a single experienced public health nurse. The possibility of responding “unknown” for the presence of a risk factor was not included in the standardized EHSSS questionnaire, but unknown responses were documented and considered in our analysis.

To minimize interviewer bias in evaluating risk factors, the nurse interviewing patients did not have access to the information provided by physicians. However, for public health protection purposes, it was mandatory to share certain information with the nurse: a clinical presentation consistent with acute viral hepatitis; a documented seroconversion within the last 12 months; a blood, organ, or tissue donation; a transfusion or transplant; or possible transmission during a health-care or cosmetic procedure. Sharing specific physician-provided information with the interview nurse occurred in fewer than 10% of the selected cases.

Case definition and classification

Cases were classified as “acute” (recent infection) or “non-acute” (past or present infection) according to the EHSSS case definition.¹⁴ An anti-HCV antibody confirmed or HCV ribonucleic acid-positive case was classified as acute when the infected person had either (1) symptoms of acute viral hepatitis in the six months preceding a positive test in the absence of markers for acute hepatitis A or B infection or (2) a negative anti-HCV antibody test result in the preceding 12 months. A laboratory-confirmed case without clinical or laboratory evidence to meet the case definition of acute was classified as non-acute. For the purpose of case classification, information on symptoms obtained during telephone interview, or clinical and laboratory data obtained by chart review, superseded information reported on the physician questionnaire.

Selection criteria and study population

The case selection process for this analysis is illustrated in Figure 1. From January 2007 to December 2008, the MPHD received 1,696 new case reports of HCV infection. Case investigation identified 782 non-acute newly diagnosed cases (diagnosed on the date of the first positive test report received by the MPHD or within the 12 preceding months). Of these 782 cases, 391 (50%) met the case investigation criteria for study inclusion (i.e., having at least one question answered on any section of the physician questionnaire and having completed the interview questionnaire).

Figure 1.

Hepatitis C virus case selection flow chart: MPHD, 2007–2008

^aDiagnosed within the 12 months preceding the first case report received by the MPHD

^bDiagnosed more than 12 months prior to the first case report received by the MPHD

MPHD = Montréal Public Health Department

Risk factors and exposure categories

Risk factors considered in this analysis are listed in Table 1. We used the country of birth provided by physicians and patients to construct the risk factor “birth in a country with high HCV prevalence.” Countries were grouped into three categories according to prevalence estimates: (1) countries for which prevalence estimates have remained <2.5%, (2) countries for which prevalence is estimated as ≥2.5%, and (3) European countries for which prevalence was estimated as ≥2.5% in birth cohorts before 1950.^15–19 Country of birth was considered a risk factor for all people born in category 2 countries and for those born before 1950 in category 3 countries.

Table 1.

Lifetime risk factors for HCV infection reported in physician questionnaires vs. patient interviews (n=391): MPHD, 2007–2008

^aRisk factors are not mutually exclusive.

^bBecause of rounding, percent distributions may not add up to 100.

^cSnorting or smoking drugs such as cocaine, heroin, and crystal methamphetamine

HCV = hepatitis C virus

MPHD = Montréal Public Health Department

To compare the risk factor profiles obtained through physician questionnaire and patient interview, the three risk factors considered most important for HCV infection among non-acute cases were ranked in order of likelihood (IDU, transfusion in Canada or the U.S. before 1990, and birth in a country with high HCV prevalence), and any remaining risk factors were grouped as “other.” Cases were attributed to mutually exclusive exposure categories according to the risk factor with the highest rank.

Analyses

We evaluated the accuracy of physician responses to each risk factor question by calculating sensitivity (i.e., correctly identifying the presence of a risk factor) and specificity (i.e., correctly identifying the absence of a risk factor), using patient responses obtained by interview as the reference. To assess agreement between physician and patient responses, we calculated Gwet's agreement coefficient for unordered categorical ratings (AC₁) for each risk factor question. Gwet's agreement statistic adjusts for chance agreement but is not susceptible to distortions that occur with the more familiar Cohen's Kappa when trait prevalence approaches 0 or 1.^20–22 We compared the risk distributions by principal exposure category according to physician reporting vs. patient interview using the Chi-square test. Unclassified cases were excluded from the proportions and the Chi-square test.

Analyses were first performed considering unknown and missing responses as negative responses because physicians often recorded only “yes” responses, leaving other responses blank. To evaluate whether the sensitivity and specificity estimates were distorted, we conducted additional analyses restricted to yes or no responses. Because we obtained similar results, we only report the analyses combining unknown, missing, and negative responses.

We performed all analyses using SAS® version 9.1. Data as of November 9, 2010, were used.

RESULTS

As shown in Figure 2, most cases meeting the selection criteria were ≥40 years of age (n=311, 79.5%) and male (n=240, 62.4%). The risk factors most commonly reported by both patients and their physicians were injection drug use, birth in a country with high HCV prevalence, and snorting or smoking drugs (Table 1). Patients were more likely than physicians to report a history of incarceration (27.6% vs. 10.2%) or having tattoos or piercing (57.0% vs. 17.1%).

Figure 2.

Distribution of hepatitis C virus cases in study sample (n=391) stratified by sex and age group: MPHD, 2007–2008

MPHD = Montréal Public Health Department

The proportion of categorical yes or no responses by physicians varied by risk factor, from 70.1% to 87.7%, with the exception of “sex with an HCV-infected partner,” for which the proportion of yes or no responses was 49.9%. For this risk factor, 44.8% of physician responses were unknown. For other risk factors, unknown physician responses varied from 3.1% to 23.3%. Missing responses varied from 3.3% to 9.2% for physicians, but represented <1% for patients.

For risk factors present in more than 5% of cases on interview, sensitivity ranged from 26.9% to 87.7% and specificity ranged from 93.0% to 98.6% (Table 2). Agreement was substantial for transfusion in Canada or the U.S. before 1990 (AC₁=0.95), sex with an HCV-infected partner (AC₁=0.87), birth in a country with high HCV prevalence (AC₁=0.86), and injection drug use (AC₁=0.80). Agreement was lower for other risk factors.

Table 2.

Sensitivity, specificity, and agreement of physician responses vs. patient responses on lifetime risk factors for HCV infection (n=391): MPHD, 2007–2008

^a“No,” “unknown,” and missing responses were considered as negative.

^bSnorting or smoking drugs such as cocaine, heroin, and crystal methamphetamine

^cNo case of hemodialysis was identified.

^dOnly one transplant case was identified.

HCV = hepatitis C virus

MPHD = Montréal Public Health Department

AC₁ = Gwet's agreement coefficient for unordered categorical ratings

NA = not applicable

When cases were attributed to ranked exposure categories (Table 3), the most common lifetime risk factors in the study sample were birth in a country with high HCV prevalence (44.9% by physician questionnaire vs. 39.9% by patient interview) and injection drug use (36.4% by physician questionnaire vs. 38.1% by patient interview). When using the information provided by physicians, 59 cases could not be assigned to one exposure category; however, only three cases could not be assigned when using information from patient interviews. The risk distributions by principal exposure category according to physician vs. patient reporting were not statistically different (χ²[4] = 2.17, p=0.704) (Table 3).

Table 3.

Risk distribution by principal exposure category for HCV infection according to physician questionnaire vs. patient interview (n=391): MPHD, 2007–2008

^aNumber and percent of affirmative responses (yes). Percentage calculations exclude unclassified cases, representing 59 (15%) physician questionnaires and three (0.8%) case interviews.

^b“Other risk factors” include non-injection drug use, sex with an HCV-infected partner, hemodialysis, organ or tissue transplant in Canada, tattooing and/or piercing, or incarceration.

^cThese cases were excluded as unclassified because the response to at least one risk factor was unknown and responses to the other risk factors compared were either negative or missing.

HCV = hepatitis C virus

MPHD = Montréal Public Health Department

NA = not applicable

DISCUSSION

We evaluated physician responses to risk factor questions on a surveillance questionnaire to assess the usefulness of physician reporting in identifying the principle exposure category for non-acute HCV cases. Ongoing characterization of reported HCV cases is important for evaluating and updating screening policies and practices. With more than 800 newly reported cases per year in Montréal, efficient ways of obtaining this information have to be identified. Data from the HCV surveillance system in Montréal during 2007 and 2008 allowed for an evaluation of physician responses to risk factor questions.

The experience of physician reporting as part of routine HCV surveillance has recently been described.²³ Although several components of the surveillance program reported are similar to our own, that study focused on operational aspects of improving provider-based reporting. To our knowledge, ours is the first published study to evaluate the accuracy of physician reporting of risk factors for HCV in the context of routine surveillance.

Physician participation

The estimated rate of return for physician questionnaires was quite good (624/782), due in part to the postal and telephone reminders. This rate of return reflects only what is known for newly diagnosed non-acute cases. Actually, the rate of return for the total number of newly reported cases would be lower. We did not assess the reasons why some physicians refused to complete questionnaires. Exclusion of cases due to anonymous and non-nominal reporting might indicate concerns about confidentiality. Qualitative analyses could help in investigating physician motives and improving their participation.

Accuracy of physician responses

The variation in the proportion of categorical yes or no responses indicates that physicians as a group evaluate some risk factors less consistently than others. For the question regarding sex with an HCV-infected partner, a substantial proportion of physicians (44.8%) responded “unknown,” which is consistent with the difficulty in ascertaining this risk and congruent with the high proportion of unknown patient responses (24.3%). A U.S. study reported that physicians consider sex with an HCV-infected partner to be an important risk factor when recommending screening.²⁴ In our sample, physicians provided a response to this question in more than 95% of cases, leaving the reason for the low sensitivity of physician responses unclear.

The sensitivity of physician responses was also low for the presence of tattoos and piercings, which are not difficult to ascertain. We ascribe the low sensitivity to different interpretations of the question. During patient interviews, the patient was asked about all body piercings, including ear piercings, whereas some physicians may have marked “yes” for this risk factor only if the patient had piercings other than in the ears.²⁵ Specificity was high for all questions, indicating that physicians accurately ascertained the absence of a risk factor.

Agreement between physician and patient responses

The agreement between physician and patient responses for each risk factor question varied from 0.19 to 0.95, reflecting the variation in accuracy and risk factor prevalence. For instance, although the sensitivity and specificity for transfusion before 1990 and other drug use were similar, the lower prevalence of transfusion resulted in a higher value for agreement. Similarly, the higher agreement for incarceration as compared with tattooing or piercing, for which sensitivity and specificity were similar, resulted from the lower prevalence of incarceration.

Risk distribution by principal exposure category

The study population comprised newly diagnosed HCV non-acute cases, as defined previously. Exposure categories in this population do not reflect the current risks of HCV transmission; rather, they reflect the lifetime risks among non-acute cases being tested. The similar risk distributions obtained by each surveillance method suggest that a questionnaire completed by physicians could be a reliable instrument for determining the principal exposure category among non-acute cases in our setting.

One might have expected a proportion of cases reporting lifetime injection drug use to be closer to what is estimated at the national level among prevalent cases (58%).² The lower proportion of IDUs may be explained in part by the selection criterion requiring cases to have completed a telephone interview. Active IDUs are more difficult to reach by telephone, as they may have no permanent address or telephone number.²⁶ Indeed, the proportions of IDUs were 66.8% and 33.8%, respectively, for excluded cases with only a physician questionnaire or only a telephone interview.

Grouping countries according to the prevalence of HCV infection is approximate at best, as data for many countries are limited or absent. Even when estimates are available, prevalence is variable within countries and can be different among those who leave their country as refugees or as applicant migrants. While our risk attribution of cases to infection in their country of origin is inexact, the substantial immigrant population in Montréal (approximately 30%),²⁷ coupled with the higher prevalence of HCV among immigrants and refugees to Canada,²⁸ suggest that HCV infection among the foreign-born is a feature of HCV epidemiology in Montréal.

Limitations

This study had a number of limitations. First, no risk factor information was obtained for 488 (28.8%) newly reported cases. For 302 of these cases, no case investigation could be undertaken, either because the case had been tested anonymously or had been seen in a clinic that does not disclose information about its clients. For an additional 186 cases, no information was obtained even though case investigation steps were undertaken. Another 391 (23.1%) newly diagnosed and non-acute cases were excluded because only one of the two sources of information was available. These 391 excluded cases differed in age, sex, and risk factor distribution from those included. Specifically, they were younger (67.5% vs. 79.5% were ≥40 years of age), and the proportions of males (67.5% vs. 62.4%) and IDUs (52.8% vs. 38.1%) were higher. Also, we do not know the effect on the risk factor ordering of excluding the 62 unclassified cases.

Second, we used patient responses obtained during telephone interviews as the reference for the sensitivity and specificity estimates of physician responses. We considered that a detailed telephone interview conducted by an experienced public health nurse was sufficiently reliable to ascertain the presence or absence of a risk factor. While some patients may have denied risks at interview (e.g., IDUs) or reported exposures that had not occurred (e.g., transfusion), we assumed that nondisclosure or inaccuracy by the patient was as likely with the physician as with the public health nurse. Given that this study focused on a limited number of lifetime risk factors among cases of non-acute HCV infection, we consider that misclassification at interview was not an important source of error.

Third, it is important to note that considering unknown and missing as negative responses is equivalent to crediting physicians as accurately ascertaining the absence of a risk factor even when the answer to the risk factor question was unknown or missing. Because specificity values were similar and sensitivity values, although higher for all questions, showed the same relative ordering, we conclude that the accuracy of physician responses was not misrepresented by this classification.

Fourth, our study was restricted to cases classified as non-acute. However, given that the majority of HCV infections are asymptomatic, it follows that a proportion of cases classified as non-acute may in fact have been recent infections, including cases with previous negative HCV test results. Because negative laboratory tests are not notifiable and HCV testing in Montréal is not centralized, it was not possible to exclude documented cases of seroconversion or new-onset viremia unless previous negative HCV test results were reported by the physician or the patient. This limitation highlights the need for information from physicians and patients to determine infection status. When infection is recent, questions about lifetime risks can lead to an over-determination of risk factors and result in a misclassification of the primary exposure category. While it was not possible to address this limitation in our study, it underlines the imprecision that is inherent in assessing risk factors when the approximate time of infection cannot be determined.

CONCLUSIONS

Until the majority of prevalent HCV cases are detected, newly reported cases will mostly represent chronic infections of variable duration and some cases of recent asymptomatic infection. The results show that a postal questionnaire completed by physicians can be an adequate surveillance method for characterizing the principal exposure categories. Case interviews by public health staff should focus on acute symptomatic cases or documented cases of recent seroconversion or new-onset viremia, where the approximate time of infection can be determined and the identification of the probable exposure factor is more likely, including potential transmission in health-care or cosmetic settings.

In our experience, physician reporting can be a useful component of routine HCV surveillance. While the lifetime risk factor information on non-acute cases obtained through physician questionnaires may be limited, when combined with demographic data, the information can be of considerable use to local health centers in adapting HCV screening and care services to their populations.²⁹ Direct mailings to physicians are also an opportunity to provide tools for counseling and clinical management. As the efficacy and acceptability of drug regimens for HCV improve, treatment services will need to expand to meet increased demand. Particularly in large urban centers, where care is often fragmented, public health follow-up with the diagnosing physician can also facilitate patient referral to treatment and community support.