Results of a Quality Assurance Program for Detection of Cytomegalovirus Infection in the Pediatric Pulmonary and Cardiovascular Complications of Vertically Transmitted Human Immunodeficiency Virus Infection Study

Abstract

A quality assurance program was established by the Pediatric Pulmonary and Cardiovascular Complications of Vertically Transmitted Human Immunodeficiency Virus Type 1 Infection Study Group for monitoring cytomegalovirus (CMV) antibody and culture results obtained from nine different participating laboratories. Over a 3-year period, every 6 months, each laboratory was sent by the designated reference laboratory six coded samples: three urine samples for CMV detection and three serum samples for CMV immunoglobulin G (IgG) and IgM antibody determination. Overall, the participating laboratories exhibited the following composite performance statistics, relative to the reference laboratory (sensitivity and specificity, respectively): 100 and 97.4% for CMV cultures, 95.5 and 94.4% for CMV IgG antibody assays, and 92.6 and 90.2% for CMV IgM assays. The practice of having individual laboratories use different commercial methods and reagents for CMV detection and antibody determination was successfully monitored and provided useful information on the comparable performance of different assays.

The Pediatric Pulmonary and Cardiovascular Complications of Vertically Transmitted Human Immunodeficiency Virus (P²C² HIV) Study was initiated in 1990 to determine the prevalence, incidence, and types of cardiovascular and pulmonary complications in the fetus, newborn, and young child with vertically transmitted HIV infection and to describe the course and outcome of these disorders (10). The relative role of immunologic dysfunction, as well as coinfections with Epstein-Barr virus and cytomegalovirus (CMV), in both the pathogenesis of cardiovascular and pulmonary complications and the progression of HIV disease in these patients was an important objective of the study (5, 6).

Since all participating centers were members of the National Institute of Allergy and Infectious Diseases AIDS Clinical Trials Group, all centers had established quality control procedures for HIV testing and immunologic tests (4). A central laboratory for Epstein-Barr virus culture and serology was established (5), but CMV culture and serology testing were performed locally at each participating institution, and the results were reported on a standardized data collection form to the Clinical Coordinating Center. To assure standardized performance of CMV testing performed at individual participating centers, a quality assurance program was initiated to validate each center's performance and to collect data on the comparable performance and reliability of different methods of CMV testing. The results of this CMV quality assurance program for the P²C² HIV multicenter study are presented. The study not only provides valuable information for data analysis specific for the P²C² HIV Study but also provides information useful to other multicenter studies that may wish to implement a proficiency program, as well as for laboratories who seek information on the comparable performance of different methods for detection of CMV infection.

MATERIALS AND METHODS

Participating laboratories.

Nine laboratories from six clinical centers participated in the CMV quality assurance program. In addition to participating in the quality assurance program, one laboratory in Houston, Tex., was designated as the reference laboratory for this program. The duties of the reference laboratory included design of the quality assurance program, assembly and shipping of the coded survey samples to all participating laboratories, receipt of the results forms from the participating laboratories, data entry and analysis for each individual survey as well as cumulative analysis, and preparation of reports and recommendations to all laboratories and appropriate committees.

Quality assurance program procedures.

Every six months, from 1994 through 1996, six coded samples were prepared by a representative (A.I.) of the reference laboratory and mailed, by overnight express mail, to each of the nine laboratories participating in the study. The reference laboratory also participated in the program by receiving its own set of coded samples which were prepared, packaged, and mailed in the same manner as the samples sent to the other eight laboratories and processed by technicians who did not participate in the assembly of the coded samples. Included in each survey package were six specimens, three urine samples and three serum samples, as well as a form for reporting sample conditions on arrival, CMV testing methodology, and CMV test results. Kool Packs were used to keep samples cool, but not frozen, during overnight transport. Urine samples coded as negative for CMV consisted of human urine, determined by standard virologic technique on human foreskin fibroblast cell lines to be virus-free, spiked with sterile cell culture medium. Urine samples coded as positive for CMV were spiked with live CMV, either CMV strain AD169 or clinical strains, reclaimed from the cryopreserved stock stores of the reference laboratory. Both relatively weak (approximate 50% tissue culture infective dose, 10⁻³) and strong (approximate 50% tissue culture infective dose, 10⁻⁵ to 10⁻⁷) titers of virus were used in different samples and surveys. On one occasion, a virus other than CMV (adenovirus) was included in a urine sample coded as negative for CMV. Serum samples consisted of human serum from cryopreserved stock stores from the reference laboratory, on which CMV immunoglobulin G (IgG) and IgM antibody testing previously had been performed using more than one method on blood samples obtained from persons experiencing a primary or recurrent infection with CMV or on persons consistently CMV seronegative and therefore determined to have never been infected with CMV (2, 10). Each laboratory was instructed to receive and process the coded samples as if they were obtained from a P²C² HIV Study subject and report the final results to the reference laboratory within a 4-week period. An agreement of results by over 67% of participating laboratories was required to certify the coded results as valid, and participating laboratories were expected to achieve correct results on the majority of valid coded samples in each series. If consensus agreement did not occur, the sample was sent to an independent laboratory for analysis. If discrepant or nonresponsive results were obtained, or if samples were received in unsatisfactory condition, the laboratory was offered the opportunity to receive, process, and test repeat samples provided by the reference laboratory. After responses from all the laboratories were received and tabulated, the results for the coded samples, as well an analysis of the survey, were made available to all participating laboratories, as well as the Clinical Coordinating Center and the Chair of the Immunology/Infectious Diseases Subcommittee.

Statistical analysis.

To determine methodologic and reagent-specific differences, the composite performance (sensitivity and specificity and 95% confidence intervals plus predictive values) statistics of all nine laboratories, as well as all vendors, to detect CMV in urine and CMV IgG and IgM antibody in serum over the 3-year period were calculated using the reference laboratory results as the reference standard. Comparisons of sensitivity and specificity estimates between laboratories and between vendors were made with McNemar's test. In addition, to determine laboratory-specific differences the performance of each laboratory for each test for the duration of the program was calculated and compared against the reference standard results by using McNemar's test.

RESULTS

CMV detection in urine.

All nine participating laboratories performed CMV detection in urine. Two laboratories used cell monolayer culture only, two laboratories used shell vial assay only, and five laboratories used a combination of shell vial assay with cell monolayer culture backup (3). Of the seven laboratories that performed cell monolayer culture (alone or in combination with a shell vial assay), six inoculated samples on the same day as receipt and one laboratory inoculated samples within the same week of receipt. Six laboratories used standard tube cell cultures, and one laboratory used a microtiter plate cell culture format. Four laboratories used MRC-5 cells, while one each used HFF, HEL, or WI38 cells. All laboratories used a commercial vendor as the source of cell lines, listing a variety of sources, including Baxter/Bartels, Issaquah, Wash.; Whittaker M.A. Bioproducts, Walkersville, Md.; Ortho Diagnostics, Rochester, N.Y.; or Viromed Laboratories, Minneapolis, Minn. The cell cultures were visually inspected under light microscopy for evidence of viral cytopathic effect (CPE) daily by one laboratory, every other day by two laboratories, twice weekly by three laboratories, and weekly by one laboratory. Five laboratories used confirmatory immunofluorescence, while two laboratories relied solely on CPE for virus identification. All seven laboratories performing shell vial assay (alone or in combination with cell monolayer culture) used MRC-5 cells obtained from commercial vendors (Baxter/Bartels, Whittaker M.A. Bioproducts, or Viromed Laboratories) as well as commercial antibody sources (Baxter/Bartels; Chemicon Co., Temecula, Calif.; or Dupont Specialty Diagnostics, Wilmington, Del.). All laboratories performing shell vial assays inoculated samples the same day of receipt; however, the time of incubation that the antibody was reacted varied from 18 to 48 h. All laboratories maintained the same methodology and reagents for CMV detection in urine during the quality assurance program period. However, different lot numbers of the reagents were used by participating laboratories over the 3-year period of the survey. All six surveys had majority consensus for urine CMV detection among the laboratories.

The composite performance of all nine laboratories to detect CMV in the urine was 100% sensitivity, 97.4% specificity, 97.5% positive predictive value, and 100% negative predictive values (Table 1). Only two false-positive results were encountered. The first false-positive result occurred during the first survey in a urine sample that had a coded result of negative for CMV (in fact, it was negative for all viruses) and was reported by a laboratory using a combination of shell vial assay with cell culture backup. The second false-positive result occurred during the sixth survey in a urine sample that had a coded result of negative for CMV but positive for adenovirus and was reported by a laboratory that used only cell culture CPE for virus detection and identification. No false-negative results were encountered.

TABLE 1.

Sensitivity and specificity of composite performance compared to the reference laboratory results

Test	No. of true positives	No. of true negatives	No. of false positives	No. of false negatives	Total	Sensitivity		Specificity
						Frequency	Estimate (95% CIb)	Frequency	Estimate (95% CI)
Urine culture	79	75	2	0	159	79/79	100.0 (95.4–100.0)	75/77	97.4 (90.9–99.7)
Serology (IgG)	63	51	3	3	120	63/66	95.5 (87.3–99.1)	51/54	94.4 (84.6–98.8)
Serology (IgM)	25	46	5a	2	78a	25/27	92.6 (75.7–99.1)	46/51	90.2 (78.6–96.7)

^aTwo equivocal tests were included in the analysis. 

^bCI, confidence interval. 

CMV IgG antibody detection in serum.

Seven of the nine laboratories performed CMV IgG antibody detection on serum specimens. All seven laboratories used commercially available enzyme immunoassay (EIA) methodology, but five different vendors supplied the reagents to these laboratories: Whittaker M.A. Bioproducts; Abbott Laboratories, Abbott Park, Ill.; Baxter/Bartels; Zeus Scientific, Raritan, N.J.; and Gull Laboratories, Salt Lake City, Utah. One laboratory changed the vendor that supplied CMV IgG antibody reagents during the quality assurance program period. Five surveys had majority consensus for CMV IgG detection among the laboratories. One survey contained a serum specimen that had only 50% agreement among the laboratories. An aliquot of this sample was analyzed by an independent laboratory for CMV IgG antibody detection, with agreement of results with the reference laboratory, and was therefore counted as a valid sample for analysis.

The composite performance statistics of the laboratories to detect CMV IgG antibody when compared to the reference laboratory were as follows: sensitivity, 95.5%; specificity, 94.4%; positive predictive value, 95.5%; and negative predictive value, 94.4%. Sensitivity and specificity did not significantly differ between laboratories. The average performance statistics did not differ from the composite statistics. There were six discrepant results (three false positive and three false negative) that were different from the reference laboratory results. These discrepant results occurred during different surveys and were from four different laboratories, all using different reagents.

Analysis of results comparing the EIA reagent vendor used in each laboratory against the reference laboratory, which used Whittaker M.A. Bioproducts, revealed 100% sensitivity and 100% specificity for Whittaker M.A. Bioproducts and for Gull Laboratories, 87.5% sensitivity and 100% specificity for Baxter/Bartels, and for Zeus Scientific, and 93.8% sensitivity and 78.6% specificity for Abbott Laboratories. No statistically significant differences were detected between vendors in sensitivity and specificity.

CMV IgM antibody detection in serum.

Five of the laboratories performed CMV IgM antibody detection on serum specimens. All five of these laboratories used commercially available EIA methodology, but different vendors were used: Abbott Laboratories, Baxter/Bartels, Whittaker M.A. Bioproducts, and Zeus Scientific. All laboratories maintained the same CMV IgM antibody detection reagents during the quality assurance program. All surveys had majority consensus for CMV IgM antibody detection among the laboratories.

The composite performance statistics of the laboratories to detect CMV IgM antibody when compared to the reference laboratory were as follows: sensitivity, 92.6%; specificity, 90.2%; positive predictive value, 83.3%; negative predictive value, 95.8%. Sensitivity and specificity did not significantly differ between laboratories. There were seven discrepant results (three false positive, two false negative, and two equivocal) that were different from the reference laboratory results. These discrepant results occurred during different surveys and were from four different laboratories, all using different reagents.

Analysis of results comparing the EIA reagent vendor used in each laboratory against the reference laboratory, which used Whittaker M.A. Bioproducts, revealed 100% sensitivity and 100% specificity for Baxter/Bartels, 100% sensitivity and 92.3% specificity for Whittaker M.A. Bioproducts, and for Zeus Scientific, and 83.3% sensitivity and 75% specificity for Abbott Laboratories. Estimates of sensitivity and specificity did not significantly differ between vendors.

DISCUSSION

This report describes the development, implementation, and evaluation of a multilaboratory, real-time quality assurance program for detection of CMV in urine and detection of CMV IgG and IgM antibody in serum. This program successfully monitored the performance of nine participating laboratories and provided information on the comparative performance of commonly used CMV detection methods.

Detection of CMV in urine was highly reproducible, producing only two false-positive results and no false-negative results, despite the different reagents and methodologies used among the participating laboratories. One false-positive result was obtained from a laboratory that used CPE only on cell monolayer culture to identify virus. Since this particular urine sample did not contain CMV but did contain adenovirus, a virus that also produces focal CPE in cell culture somewhat similar to CMV, it is possible the false-positive report was due to misidentification of the viral CPE. The use of immunofluorescence reagents to confirm the identity of the virus detected by CPE may have helped the laboratory correctly identify the virus. It is unclear why another urine sample, containing no virus, was reported as positive for CMV from a laboratory using shell vial assay with cell monolayer culture backup, but this finding does show that such false-positive results can occur.

Detection of CMV IgG antibody in serum specimens also provided consistent results, with performance statistics between 94 and 100% for all laboratories and with an equal number of false-positive and false-negative results relative to the reference standard. CMV IgG antibody may be detected by a variety of different methods, including neutralization, radioimmunoassay, immunofluorescence, complement fixation, indirect hemagglutination, latex agglutination, and EIA (1, 8). Comparable performance has been observed with all of these different methods, but most studies reveal discrepant results in a small number of samples (1, 8). While all the laboratories in this quality assurance program used the same type of methodology, EIA, most of the laboratories obtained their reagents from different vendors. The reasons for the three false-positive and three false-negative results therefore may be related to the abilities of different reagents to detect CMV IgG antibody. Another possible reason for laboratory variability is a difference in laboratory technical expertise or experience. However, since the variability did not appear to be laboratory specific, it is less likely to be an explanation. Detection of CMV IgM antibody in serum specimens was more likely to yield false-positive results than false-negative results relative to the reference standard. CMV IgM antibody also may be detected by a variety of different methods, but wide variability of results between IgM detection methodologies, as well as variability in the ability of these methods to detect primary, recurrent, or congenital CMV infection in different patient populations, has been clearly and consistently documented (2, 7–9, 11). Similar to CMV IgG antibody detection, the reasons for the false-positive and false-negative results observed in this study for CMV IgM detection were more likely due to reagent-specific rather than laboratory-specific differences. This report confirms that discrepant results for CMV IgG and IgM antibody detection can be obtained from aliquots from the same serum specimen that is tested in different laboratories using the same methodology, EIA, but employing different reagents, and participants in multicenter studies should be aware of this phenomenon.

Clinicians without formal laboratory training or expertise may assume there are uniform procedures when the same test is performed in different laboratories. However, this study illustrates the diversity of methodologic approaches and reagent choices available to laboratories and that this diversity may impact test performance. Therefore, investigators and data managers who participate in a multicenter study in which a central laboratory is not feasible or desirable for certain types of testing should consider a quality assurance program that parallels in time the execution of the study. Such a program will provide important information on comparative performance of methodologies, reagents, and laboratories; allow a timely correction of discrepant results or laboratory-based errors or differences; and facilitate data analysis.

Appendix

A complete list of P²C² HIV Study Group Members can be found in reference 10. Study group members at the National Heart, Lung, and Blood Institute included Hannah Peavy, Anthony Kalica, Elaine Sloand, George Sopko, and Margaret Wu. The chairman of the steering committee was Robert Mellins. Study group members at clinical centers included William Shearer, Howard M. Rosenblatt, and Linda Davis (Baylor College of Medicine, Houston, Tex.); Debra Mooneyham and Teresa Tonsberg (University of Texas School of Medicine, Houston); Steven Lipshultz, Kenneth McIntosh, Janice Hunter, and Ellen McAuliffe (The Children's Hospital, Boston/Harvard Medical School, Boston, Mass.); Suzanne Steinbach, Ellen Cooper, and Karen Lewis (Boston Medical Center, Boston, Mass.); Meyer Kattan, David Hodes, and Diane Carp (Mount Sinai School of Medicine, New York, N.Y.); Stephen Heaton and Mary Ann Worth (Beth Israel Medical Center, New York, N.Y.); Robert Mellins, Philip LaRussa, Jane Pitt, and Kim Geromanos (Presbyterian Hospital in the City of New York/Columbia University, New York, N.Y.); Samuel Kaplan, Yvonne Bryson, and Helene Cohen (UCLA School of Medicine, Los Angeles, Calif.); Joseph Church, Arnold Platzker, Lucy Kunzman, and Toni Ziolkowski (Children's Hospital, Los Angeles, Calif.); and Andrea Kovacs and Lynn Fukushima (University of Southern California, L.A.C.). Study group members of the clinical coordinating center included Michael Kutner, Mark Schluchter (through April 1998), Johanna Goldfarb, Douglas Moodie, Cindy Chen, Kirk Easley, Scott Husak, Victoria Konig, Sonil Rao, Paul Sartori, Amrik Shah, Susan Sunkle, and Weihong Zhang (The Cleveland Clinic Foundation, Cleveland, Ohio) and Richard Martin (Case Western Reserve University). Members of the study group's policy, data, and safety monitoring board included Henrique Rigatto, Edward B. Clark, Robert B. Cotton, Vijay V. Joshi, Paul S. Levy, Norman S. Talner, Patricia Taylor, Robert Tepper, Janet Wittes, Robert H. Yolken, and Peter E. Vink.