Skip to main content
Journal of Clinical Microbiology logoLink to Journal of Clinical Microbiology
. 2007 Jan 31;45(4):1211–1218. doi: 10.1128/JCM.00010-06

Evaluation of Different PCR Assays for Early Detection of Acute and Relapsing Brucellosis in Humans in Comparison with Conventional Methods

Stella Mitka 1,*, Constantine Anetakis 1, Efimia Souliou 2, Eudoxia Diza 2, Athina Kansouzidou 1
PMCID: PMC1865811  PMID: 17267626

Abstract

Human brucellosis is a significant public health problem in many Mediterranean countries including Greece. The conventional serological methods, as well as blood cultures, have serious limitations, especially in chronic, relapsing, and focal forms of the disease. Four different PCR assays were applied in 4,926 buffy coat, whole-blood, and serum samples received from 200 patients admitted with brucellosis to the Infectious Diseases Hospital, Thessaloniki, Greece, for the rapid diagnosis of acute infection and relapses and compared to blood culture and serological tests (i.e., Wright's seroagglutination test, Coombs' antibrucella test, and the complement fixation test). The four PCR assays had excellent sensitivity and specificity and were able to detect all of the cases of acute disease. Buffy coat and whole blood were the optimal specimens. All four PCR assays were negative in all follow-up samples from 183 patients who had completed a successful treatment and were positive in all follow-up samples from 17 patients who had relapses in the first year after therapy, including the times of the relapses. In conclusion, PCR is a very useful tool for the rapid diagnosis of acute brucellosis and a good marker for the posttreatment follow-up and the early detection of relapses.


Brucellosis is a zoonosis distributed worldwide, especially in the Mediterranean basin, the Middle East, India, and South America, constituting an important public health problem in these areas (3, 12, 37). In Greece, Brucella melitensis is isolated in most cases of human disease, whereas the isolation of B. abortus is very rare. From 1981 to 1992 a significant reduction in the human brucellosis cases reported was observed in Greece due to the eradication program implemented in livestock. From 1994 until 2004 a new increase in the number of human brucellosis cases has been recorded, and more than 350 new cases have been reported each year to the Ministry of Health, Greece, over the last 10 years (17). The figures reported worldwide may underestimate the magnitude of the problem (29, 32), which has been estimated to be 10 to 25 times higher than reported (13). The clinical symptoms of human brucellosis are nonspecific, and several other febrile diseases may be simulated. The infection is characterized by protean manifestations and prolonged recurrent febrile episodes so that the disease is referred to as “undulant fever” (14). The features of acute disease are varied and may be insidious, whereas the features of chronic disease, which may persist or recur for years, are often vague (36). The disease, therefore, cannot be diagnosed on clinical grounds alone, and microbiological confirmation is required through the isolation of the microorganism or the detection of specific antibodies in the patient's serum. However, the established methods for laboratory diagnosis are often unreliable in several respects. Culture of blood or marrow samples commonly takes 5 days or longer, is often unsuccessful in the case of chronic brucellosis (20), and increases the hazards of handling the organism in the laboratory. Conventional serological methods have important limitations. Such methods display poor sensitivity in the early stages of the disease, during which the levels of antibodies may be low. Furthermore, there can be cross-reactions with certain other gram-negative bacteria (7, 11, 30), and in areas of endemicity a significant proportion of the population may be seropositive with no evidence of the disease (18). Serological methods are also of limited value in assessing individuals who have been treated for brucellosis and are suspected of relapse (8). The development of specific PCR assays is a recent advance; however, standardization of the methods is lacking, and a better understanding of the clinical significance of the results is still needed (28).

In the present study the value of PCR assays for the rapid diagnosis of human brucellosis in acute and relapsing forms of the disease versus the use of conventional diagnostic methods has been investigated. The methods used in the present study included four different PCR assays, blood culture, and the following serological tests: Wright's seroagglutination test (SAT), Coombs' antibrucella test (CT), and the complement fixation test (CF).

MATERIALS AND METHODS

Clinical specimens.

From 1998 to 2004, a total of 4,926 samples—1,642 peripheral whole-blood samples, 1,642 buffy coat samples, and 1,642 serum samples—were collected from 200 patients diagnosed with brucellosis in the Infectious Diseases Hospital, Thessaloniki, Greece. The duration of symptoms prior to first admittance to the hospital ranged from 1 to 8 weeks. The samples of each patient were obtained (i) at admittance before starting an antibiotic treatment, (ii) at the end of the treatment period, (iii) monthly for the first 3 months after treatment, and (iv) every third month thereafter for the next 9 months. In the case of clinical relapses, an additional sample was obtained. Blood samples were collected in EDTA, and buffy coat was isolated by using a pipette after centrifugation of the blood samples.

The diagnosis of brucellosis was established according to one of the following criteria: (i) isolation of Brucella spp. in blood culture and/or (ii) the presence of compatible clinical signs or symptoms, together with the presence of specific antibodies at significantly high titers or a seroconversion, or also a fourfold increase in titer between two sequential samples from the same patient (4, 5, 24). Significant titers were considered to be a SAT titer of ≥1/160, a CT titer of ≥1/320, or a CF titer of ≥1/16. Relapse was defined as the reappearance of the signs or symptoms of the disease and/or a new positive blood culture during the 12-month period after therapy (5). In addition, a seroconversion or a fourfold increase in titer in serological tests could be considered a sign of relapse; however, in our patients serological criteria were not used. Of the 200 patients, 164 (82%) had typical symptoms, and 36 (18%) had variable atypical signs of the disease. The antibiotic regimens used for 128 of the 200 patients included doxycycline and rifampin for 6 weeks and in the remainder included doxycycline and rifampin for 6 weeks and streptomycin for 3 weeks.

The patients were separated into two groups according to the outcome of the treatment. A total of 183 (91.5%) of them (group 1, “effective treatment”) did not have any clinical signs or relapses of brucellosis for 12 months after therapy, thus establishing a successful treatment, and 17 (8.5%) patients (group 2, “ineffective treatment”) had persistent clinical signs of brucellosis and/or consecutive clinically confirmed relapses, as ascertained by the reappearance of signs or symptoms of the disease. All 17 patients in this group had two relapses, and 8 of them had a third relapse, the first within 10 to 30 days after therapy and the second and third within 6 to 8 months after therapy. Therefore, the treatment of these patients was not successful, and their infections were considered to be persistent. It seems more likely that these 17 patients had a true relapse rather than a reinfection because the first episode occurred less than 1 month after the treatment was concluded. Furthermore, no new exposure could be detected for these patients at the time of the relapses. Of the 183 patients with effective treatment and the 17 patients with ineffective treatment, 8 and 6 patients, respectively, had variable focal complications such as arthritis, orchitis, spondylitis, and pericarditis at admittance. The signs and symptoms, as well as the results of the diagnostic tests, of the two groups of patients at admittance are shown in Table 1.

TABLE 1.

Clinical manifestations and laboratory tests of 200 patients with brucellosis at admittancea

Parameter Effective treatment
Ineffective treatment
No. of patients (n = 183) % No. of patients (n = 17) %
Signs and symptoms
    High fever (>38°C) 168 91.8 15 88.2
    Sweat 161 88 14 82.3
    Arthromyalgia 155 84.7 14 82.3
    Headache 146 79.8 12 70.6
    Debility 137 74.9 11 64.7
    Anorexia 128 69.9 13 76.4
    Chills 126 68.8 10 58.8
    Weight loss 76 41.5 8 47
    Hepatomegaly 75 41 7 41.2
    Splenomegaly 66 36 6 35.2
    Lymphadenopathy 27 14.8 3 17.6
Focal complicationsb
    Spondylitis 3 1.64 1 5.9
    Orchitis 3 1.64 2 11.8
    Arthritis 2 1.1 2 11.8
    Pericarditis 0 0 1 5.9
Laboratory tests
    Positive blood culture 135 73.8 13 76.5
    SAT 1:160c 28 15.3 3 17.6
    SAT 1:320 41 22.4 4 23.5
    SAT 1:400 29 15.8 3 17.6
    SAT 1:800 21 11.5 2 11.8
    SAT 1:1,600 12 6.5 1 5.9
    SAT 1:3,200 14 7.6 1 5.9
    SAT >1:3,200 20 10.9 2 11.8
    CT 1:320 14 7.6 2 11.8
    CT 1:640 76 41.5 9 52.9
    CT ≥1:1,280 27 14.8 3 17.6
    CF 1:16 13 7.1 2 11.8
    CF 1:32 41 22.4 5 29.4
    CF 1:64 51 27.9 5 29.4
    CF ≥1:128 34 18.6 4 23.5
    PCR A in whole blood 182 99.4 16 94.1
    PCR B in whole blood 180 98.4 16 94.1
    PCR C in whole blood 181 98.9 17 100
    PCR D in whole blood 181 98.9 16 94.1
a

%, Proportion of positive results or of signs and symptoms. n, Total number of patients examined.

b

Significant difference between the two groups (P < 0.005).

c

For SAT, CT, and CF the titer result is indicated.

Control specimens.

As positive controls for PCR, DNA samples extracted from six reference Brucella strains (B. abortus biovar 1 strain 544, B. melitensis biovar 1 strain 16M, B. melitensis biovar 2 strain 63/9, B. melitensis biovar 3 strain Ether, B. suis biovar 1 strain 1330, and B. canis strain RM6/66) and from one vaccine strain (B. melitensis biotype 1 Rev-1), all supplied by the Central Veterinary Laboratory in Weybridge (Surrey, England), were tested. As negative controls for PCR, whole-blood and serum samples obtained from 50 healthy individuals and from 50 patients with infections caused by other bacteria (i.e., Salmonella enterica serovar Typhi, S. enterica serovar Typhimurium, S. enterica serovar Enteritidis, Yersinia enterocolitica O:3, Yersinia enterocolitica O:9, Enterobacter aerogenes, Escherichia coli, Klebsiella pneumoniae, Haemophilus influenzae, and Pseudomonas aeruginosa) were tested by both PCR and serological methods. Blood cultures were processed only in samples obtained from the group of 50 patients with other infections. In addition, DNA samples from 13 strains of gram-negative bacteria (i.e., Salmonella enterica serovar Enteritidis, Yersinia enterocolitica O:3, Yersinia enterocolitica O:9, Enterobacter aerogenes, Escherichia coli Ο157, Haemophilus influenzae, Pseudomonas aeruginosa, Campylobacter jejuni, Neisseria meningitidis, Legionella pneumophila, Vibrio cholerae O1, Ochrobactrum anthropi, and Mycobacterium tuberculosis; all isolated by blood culture from patients at the Infectious Diseases Hospital, Thessaloniki, Greece) giving cross-reactions with Brucella spp. in conventional serological tests were also tested as negative controls for PCR.

Bacteriological and serological methods.

Blood cultures were processed with the BACTEC 9050 system (Becton Dickinson, Towson, MD) according to standard techniques (27) and were incubated for 30 days. Three blood culture specimens were obtained from each of the 200 patients at admittance. All of the Brucella strains isolated from the patients and all of the reference Brucella strains were identified and biotyped by standard techniques, including the CO2 requirement, H2S production, susceptibility to dyes (thionin, thionin blue, and basic fuchsin), agglutination in monospecific sera A and M, and lysis by phage Tblisi (1, 27). The monospecific sera and the phage were supplied by the OIE Brucellosis Reference Centre, Veterinary Laboratories Agency, Weybridge, Surrey, United Kingdom. The conventional serological methods SAT, CT, and CF were performed according to standard techniques (1) using commercially available antigens (Laboratory Diagnostics Co., Inc., Morganville, NJ; Serion Immundiagnostica GmbH, Wurzburg, Germany; and bioMerieux SA, Marcy l'Etoile, France, respectively).

Extraction of DNA.

DNA from all clinical samples and bacterial strains was extracted by using a commercial purification system with columns (QIAamp Blood Midi; QIAGEN GmbH, Hilden, Germany) according to the manufacturer's instructions. The concentration and the purity of the DNA were determined spectrophotometrically by determining the A260 and A280 values.

DNA amplification.

Four different PCR assays that target different gene regions of Brucella spp. were performed for the detection of Brucella DNA (PCRs A, B, C, and D). All specific primers for the PCR assays were supplied by Invitrogen Ltd., Paisley, United Kingdom.

(i) PCR A.

This PCR assay amplifies a 223-bp sequence of the gene bcsp31 encoding an immunogenic outer membrane protein of 31 kDa of B. abortus, which is conserved in all Brucella species, using the specific primers B4 and B5 for this PCR assay (8). PCR was performed in a total volume of 50 μl containing template DNA (5 μl), Tris-HCl (pH 8.4, 20 mM), KCl (50 mM), MgCl2 (1 mM), a 200 μM concentration of each deoxynucleoside triphosphate (Promega, Madison, WI), a 0.5 μM concentration of each of the primers B4 and B5 (25 pmol per 50 μl), and 2 IU of Taq polymerase (Promega). This PCR consisted of an initial 5-min incubation step at 93°C, followed by 40 cycles of denaturation at 90°C for 1 min, annealing at 60°C for 1 min, and extension at 72°C for 1 min, with a final incubation step at 72°C for 10 min.

(ii) PCR B.

This PCR assay amplifies a 193-bp sequence of the gene omp2 encoding an outer membrane protein of 26 kDa of B. abortus in all Brucella species, using the specific primers JPF and JPR (19). These primers, recognizing the B. abortus sequence, detect all strains of B. melitensis and B. abortus, excluding the B. suis biovars 2, 3, and 4; B. canis; and B. ovis (19). PCR was performed in a total volume of 50 μl containing template DNA (5 μl), Tris-HCl (pH 8.4, 20 mM), KCl (50 mM), MgCl2 (3 mM), a 200 μM concentration of each deoxynucleoside triphosphate, a 1 μM concentration of each of the primers JPF and JPR (50 pmol per 50 μl), and 2 IU of Taq polymerase. This PCR consisted of an initial 4-min incubation step at 94°C, followed by 35 cycles of denaturation at 94°C for 1 min, annealing at 58°C for 1 min, and extension at 72°C for 1 min, with a final incubation step at 72°C for 10 min.

(iii) PCR C.

This PCR assay uses a different region of the gene omp2 of B. abortus as a target, producing a 282-bp sequence in all Brucella species, using the specific primers P1 and P2 (9). The PCR was performed in a total volume of 50 μl containing template DNA (5 μl), Tris-HCl (pH 8.4, 20 mM), KCl (50 mM), MgCl2 (2 mM), a 200 μM concentration of each deoxynucleoside triphosphate, a 1 μM concentration of each of the primers P1 and P2 (50 pmol per 50 μl), and 2 IU of Taq polymerase. This PCR consisted of an initial 3-min incubation step at 94°C, followed by 30 cycles with denaturation at 90°C for 40 s, annealing at 50°C for 1 min, and extension at 72°C for 1 min, with a final incubation step at 72°C for 10 min.

(iv) PCR D.

This PCR assay amplifies the entire bp26 gene of B. melitensis 16M, encoding the BP26 protein, also named Omp28, which is identified as an immunodominant antigen (10). This PCR assay produces a 1,029-bp sequence in all Brucella species, except isolates from marine mammals, in which it produces a sequence of 1,900 bp, using the specific primers 26A and 26B (10). PCR was performed in a total volume of 50 μl containing template DNA (5 μl), Tris-HCl (pH 9.0, 10 mM), KCl (50 mM), MgCl2 (1.5 mM), a 200 μM concentration of each deoxynucleoside triphosphate, a 1 μM concentration of each of the primers 26A and 26B (50 pmol per 50 μl), and 2.5 IU of Taq polymerase. This PCR consisted of an initial 5-min incubation step at 94°C, followed by 30 cycles of denaturation at 94°C for 1 min, annealing at 58°C for 1 min, and extension at 72°C for 1 min, with a final incubation step at 72°C for 10 min.

The oligonucleotide sequences of all primers used in the present study, along with the most relevant information about them, are presented in Table 2. The reactions for all four PCR assays were performed in a programmable thermocycler without mineral oil (Gene Amp PCR System 2400; Roche Diagnostics). Amplicons were detected by fluorescence after electrophoresis in 2% agarose gel in the presence of ethidium bromide (1 μg/ml) under UV light. Positive controls derived from B. abortus biovar 1 strain 544 and B. melitensis biovar 1 strain 16M were included in all of the tests, as were negative controls that contained all of the elements of the reaction mixture except for template DNA. In order to guarantee the reliability of the results, all samples were processed in duplicate, and they produced identical positive or negative results.

TABLE 2.

Oligonucleotide primers used in PCR assays

PCR Primer Sequence (5′-3′) Target gene PCR product (bp) Annealing temp (°C) No. of cycles
A B4 (8) TGGCTCGGTTGCCAATATCAA bcsp31a 223 60 40
B5 (8) CGCGCTTGCCTTTCAGGTCTG
B JPF (19) GCGCTCAGGCTGCCGACGCAA omp2b 193 58 35
JPR (19) ACCAGCCATTGCGGTCGGTA
C P1 (9) TGGAGGTCAGAAATGAAC omp2 282 50 30
P2 (9) GAGTGCGAAACGAGCGC
D 26A (10) GCCCCTGACATAACCCGCTT bp26c 1029 58 30
26B (10) GAGCGTGACATTTGCCGATA
a

Encoding an immunogenic 31-kDa outer membrane protein of B. abortus.

b

Encoding a 26-kDa outer membrane protein of B. abortus.

c

Encoding the BP26 protein of B. melitensis, also named Omp28.

Statistical analysis.

Differences in serological tests and PCR results were assessed by the χ2 test. Significant differences were considered when the probability (P) was <0.05. The sensitivity, specificity, positive predictive value, and negative predictive value for each of the conventional tests and PCR assays were also calculated as described by Armitage and Berry (6).

RESULTS

Results of clinical specimens.

(i) A total of 148 (74%) of the 200 patients included in the present study had positive blood cultures at admittance (Table 3). All of the 148 strains isolated were identified and biotyped by standard techniques. A total of 18 (12.2%) of these strains were identified as B. melitensis biovar 1, 72 (48.6%) were identified as B. melitensis biovar 2, and 58 (39.2%) were identified as B. melitensis biovar 3. In 6 of the 148 patients, the blood cultures were initially negative were determined to be positive after we obtained and tested a second sample 7 days later. In all of these cases B. melitensis biovar 2 was isolated. (ii) With regard to serology, 181 (90.5%) of the 200 patients had a SAT titer of ≥1/160, 131 (65.5%) had a CT titer of ≥1/320, and 155 (77.5%) had a CF titer of ≥1/16 at admittance. Of 200 patients, 8 (4%) were negative with blood cultures, and all of the serological methods (combined sensitivity of 96%), even though they had all of the compatible signs or symptoms of acute brucellosis. The most relevant data about the serology results in the total treatment surveillance period are presented in Table 3. The positive and negative predictive values of the serological methods are given in Table 4. (iii) All four PCR assays were carried out at admittance, and the results were positive in the buffy coat samples of the 200 (100%) patients. PCRs A and C were positive in 198 (99%) blood samples, whereas PCR D was positive in 197 (98.5%) and PCR B was positive in 196 (98%). Slightly lower sensitivities were found when serum samples were used: 97% for PCRs A and C, 96.5% for PCR D, and 95.5% for PCR B. The eight patients who were negative with all of the conventional techniques, as well as the six patients with initially negative blood cultures, were PCR positive at admittance with all four PCR assays. The positive and negative predictive values of the molecular assays were calculated as shown in Table 4. (iv) With regard to the clinical manifestations at admittance (Table 1), the proportion of focal presentations was significantly higher (P < 0.005) in the patients of group 2 (ineffective treatment) than in the patients of group 1 (effective treatment).

TABLE 3.

Results of conventional methods and PCR in 200 patients with brucellosis during treatment surveillance

Methoda No. (%)b
Antibiotic treatmentc
Follow-up after the end of treatmentd
Pre Post Mo 1 Mo 2 Mo 3 Mo 6 Mo 9 Mo 12
Blood culture 148 (74) ND ND ND ND ND ND ND
SAT ≥1:160 181 (90.5) 158 (79) 132 (66) 119 (59.5) 101 (50.5) 52 (26) 53 (26.5) 46 (23)
CT ≥1:320 131 (65.5) 115 (57.5) 112 (56.0) 109 (54.5) 108 (54.0) 92 (46) 86 (43) 81 (40.5)
CF ≥1:16 155 (77.5) 131 (65.5) 129 (64.5) 126 (63.0) 125 (62.5) 120 (60) 114 (57) 92 (46)
PCR A, bc 200 (100) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5)
PCR A, wb 198 (99) 17 (8.5) 16 (8) 16 (8) 17 (8.5) 15 (7.5) 16 (8) 17 (8.5)
PCR A, se 194 (97) 9 (4.5) 8 (4) 9 (4.5) 9 (4.5) 10 (5) 8 (4) 9 (4.5)
PCR B, bc 200 (100) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5)
PCR B, wb 196 (98) 17 (8.5) 15 (7.5) 15 (7.5) 16 (8) 15 (7.5) 17 (8.5) 16 (8)
PCR B, se 191 (95.5) 9 (4.5) 7 (3.5) 7 (3.5) 8 (4) 9 (4.5) 9 (4.5) 7 (3.5)
PCR C, bc 200 (100) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5)
PCR C, wb 198 (99) 17 (8.5) 17 (8.5) 16 (8) 17 (8.5) 16 (8) 17 (8.5) 16 (8)
PCR C, se 194 (97) 9 (4.5) 8 (4) 10 (5) 9 (4.5) 7 (3.5) 8 (4) 9 (4.5)
PCR D, bc 200 (100) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5) 17 (8.5)
PCR D, wb 197 (98.5) 17 (8.5) 16 (8) 17 (8.5) 16 (8) 16 (8) 17 (8.5) 15 (7.5)
PCR D, se 193 (96.5) 9 (4.5) 9 (4.5) 8 (4) 9 (4.5) 6 (3) 8 (4) 7 (3.5)
a

For SAT, CT, and CF the titer result is indicated. bc, buffy coat; wb, whole blood; se, serum.

b

No., number of patients with positive result; %, proportion of positive results. ND, not done.

c

Pre, at admittance (before antibiotic treatment); Post, end of antibiotic treatment.

d

The time point in months after the end of therapy is indicated.

TABLE 4.

Efficiency of blood culture, serological testing, and PCR assays

Methoda Value (%)b
Sensitivity Specificity PPN NPN
Blood culture 74 100 100 65.8
SAT ≥1:160 90.5 96.1 98 88.5
CT ≥1:320 65.5 98 99 59.2
CF ≥1:16 77.5 98 99 69
PCR A, bc 100 100 100 100
PCR A, wb 100 100 100 98
PCR A, se 97 100 100 94.3
PCR B, bc 100 100 100 100
PCR B, wb 98 100 100 96.1
PCR B, se 95.5 100 100 91.7
PCR C, bc 100 100 100 100
PCR C, wb 99 100 100 98
PCR C, se 97 100 100 94.3
PCR D, bc 100 100 100 100
PCR D, wb 98.5 100 100 97.1
PCR D, se 96.5 100 100 93.5
a

For SAT, CT, and CF the titer result is indicated. bc, buffy coat; wb, whole blood; se, serum.

b

PPN, positive predictive number; NPN, negative predictive number.

The results during the follow-up period were as follows (Table 3). At the end of therapy, the SAT result was positive in 158 (79%), the CT result was positive in 115 (57.5%), and the CF result was positive in 131 (65.5%) of the 200 patients. The eight patients who were determined to be negative by all of the conventional methods at admittance produced specific anti-brucella antibodies in significant CT and CF titers and low SAT titers (1:80) at this time point, affirming the initial clinically based diagnosis of brucellosis. At 12 months after therapy the SAT results still remained positive in 46 (23%), the CT results remained positive in 81 (40.5%), and the CF results remained positive in 92 (46%) of the 200 patients (Table 3). At the end of the therapeutic schedules all four PCR assays were negative in 183 (91.5%) of the 200 patients in all clinical specimen types tested, and the assays remained negative for 1 year thereafter. In all of these 183 patients (group 1), neither relapses nor complications of brucellosis appeared in the same 1-year period. In 17 (8.5%) of the 200 patients the four PCR assays remained positive in buffy coat and in whole blood, and in 9 (4.5%) of the 200 patients the four PCR assays remained positive in serum. In the next months of follow-up, all 17 patients continued to be PCR positive in buffy coat with all four PCR assays. During the same time, the number of positive results in whole blood ranged from 15 to 17 and in serum from 6 to 10 depending on the PCR assay and the time of sampling (Table 3). All 17 patients (group 2) with positive results by the PCR assay at the end of the therapy and during the follow-up period had consecutive clinically confirmed relapses. As shown in Table 5, the serological tests in the 17 patients of group 2 were positive in proportions similar to those for the patients of group 1 (P ≥ 0.70) at the end of therapy, while the differences in PCR results between groups 1 and 2 were significant (P < 0.005) at this time. Consequently, PCR assay proved to be useful for establishing the success of the therapy, whereas the conventional serological methods remained positive in many cases even in intermediate range titers. At the time of the relapses, 7 of these 17 patients had a positive blood culture, and only 4 of the 17 patients demonstrated a seroconversion or a fourfold increase in titer in the serological tests (Table 6), whereas all 17 patients were determined to be PCR positive in the buffy coat samples with all four PCR assays at all relapses. The results of the PCR assays with all kinds of clinical specimens at the time of the relapses are presented in Table 7.

TABLE 5.

Comparison of serological tests and PCR results between patients with effective and ineffective antibiotic treatment at the conclusion of therapy

Method and findinga No. of patients with:
P
Effective antibiotic treatment (n = 183) Ineffective antibiotic treatment (n = 17)
SAT
    Positive 145 13 >0.70
    Negative 38 4
CT
    Positive 105 10 >0.90
    Negative 78 7
CF
    Positive 120 11 >0.90
    Negative 63 6
PCR
    Positive 0 17 <0.0050
    Negative 183 0
a

SAT, Wright's SAT positive ≥1:160; CT, CT positive ≥1:320; CF, CF positive ≥1:16; PCR, PCRs A, B, C, and D in buffy coat or in whole blood.

TABLE 6.

Results of blood culture and serological tests for 17 patients with relapsing brucellosis at the prerelapse and relapse periods

Patient Reciprocal titera
First relapse
Second relapse
Blood culture
SAT
CT
CF
Blood culture
SAT
CT
CF
Pre Rel Pre Rel Pre Rel Pre Rel Pre Rel Pre Rel Pre Rel Pre Rel
1 + 160 160 640 640 32 32 + 160 160 640 640 32 32
2 160 320 640 >1,280 16 32 + 160 160 640 >1,280 16 32
3 + 80 160 320 640 16 64 160 160 640 640 16 64
4 + 80 80 320 320 32 64 80 160 320 640 32 64
5 320 320 >1,280 >1,280 64 256 + 160 160 640 >1,280 64 >256
6 160 160 640 640 64 128 160 400 >1,280 >1,280 64 >256
7 + 160 160 320 1,280 16 128 160 160 640 640 16 128
8 400 320 >1,280 >1,280 16 32 400 320 >1,280 >1,280 16 32
9 + 160 160 640 640 32 128 + 80 160 320 640 32 128
10 80 160 160 160 16 64 80 160 320 640 16 64
11 160 80 640 1,280 32 128 160 160 640 640 32 128
12 80 160 320 320 32 16 160 160 640 640 32 16
13 + 40 80 160 320 32 256 + 40 320 160 320 32 256
14 160 80 640 320 64 128 160 80 640 320 64 128
15 160 400 640 >1,280 64 64 + 320 160 640 >1,280 64 64
16 160 320 160 320 64 128 + 160 320 160 320 64 128
17 + 320 400 >1,280 1,280 32 >256 160 160 640 640 32 256
a

Pre, prerelapse period; Rel, during relapse. The first relapse occurred within 10 to 30 days after the end of treatment; the second relapse occurred within 6 to 8 months after treatment. SAT, Wright's SAT positive ≥1:160; CT, CT positive ≥1:320; CF, CF positive ≥1:16.

TABLE 7.

Results of PCR assays at the time of relapses

Methoda PCR assay resultb
First relapse (n = 17)
Second relapse (n = 17)
Third relapse (n = 8)
No. % No. % No. %
PCR A, bc 17 100 17 100 8 100
PCR A, wb 17 100 16 94.1 8 100
PCR A, se 14 82.4 14 82.4 5 62.5
PCR B, bc 17 100 17 100 8 100
PCR B, wb 15 88.2 15 88.2 7 87.5
PCR B, se 11 64.7 10 58.8 5 62.5
PCR C, bc 17 100 17 100 8 100
PCR C, wb 16 94.1 16 94.1 8 100
PCR C, se 14 82.4 14 82.4 5 62.5
PCR D, bc 17 100 17 100 8 100
PCR D, wb 16 94.1 16 94.1 7 87.5
PCR D, se 13 76.4 11 64.7 5 62.5
a

bc, buffy coat; wb, whole blood; se, serum.

b

The first relapse occurred 10 to 30 days after therapy, and the second and third relapses occurred 6 to 8 months after therapy. No., number of patients with positive PCR results at relapses; %, proportion of positive results. n, Total number of patients examined.

Specificity of the conventional techniques and the PCR assays.

All samples obtained from the 50 healthy individuals were determined to be negative by all of the conventional serological methods, as were all samples from the 50 patients with infections other than brucellosis, using blood culture, whereas 4 of the samples were determined to be positive by the SAT, 2 were determined to be positive by the CT, and 2 were determined to be positive by the CF, so that the specificity of the conventional methods was calculated as shown in Table 4. All of the reference Brucella strains used here produced positive results with all four PCR assays except for B. canis strain RM6/66, which yielded negative results with PCR B, whereas all samples obtained from the 50 healthy individuals and from the 50 patients with other types of infections produced negative results in all four PCR assays, as did all DNA samples from the 13 different gram-negative bacteria giving cross-reactions with Brucella spp. in conventional serological tests, conferring an assay specificity of 100% on all four PCR assays (Table 4).

Analytical sensitivity of the PCR assays.

Tenfold serial dilutions in steps from 100 ng to 1 fg of isolated DNA from all of the aforementioned reference Brucella strains were used to determine the analytical sensitivity of the four PCR assays. The detection limits of the molecular methods were estimated to be 10 to 100 fg for PCR A, 25 to 250 fg for PCR B, 12.5 to 125 fg for PCR C, and 20 to 200 fg for PCR D. The sensitivity of the methods remained unchanged after the addition of 4 μg of human DNA to the samples.

DISCUSSION

The clinical picture of brucellosis alone cannot always lead to diagnosis since the symptoms are nonspecific and often atypical; therefore, diagnosis needs to be supported by laboratory tests. Although many serological tests and new automated blood culture techniques have been developed to diagnose brucellosis, there are still significant problems in the diagnosis of the disease. Blood cultures are time-consuming, since the average time required for growth has been reported to be 6.65 days (16). Most recent studies report that incubation periods for the isolation of Brucella from BACTEC bottles range from 2.5 to 5 days, with 94 to 97% of Brucella strains growing within this time period (34, 35), whereas in the present study the average incubation time for a positive blood culture was 7 days. In addition, handling the organism poses a high risk of contagion for laboratory personnel, so that brucellosis is one of the most commonly recognized causes of laboratory-transmitted infections, and 2% of all brucellosis cases are laboratory acquired (33, 38). The sensitivity of blood cultures ranges from 53 to 90% (4), whereas it is significantly reduced in focal and chronic forms of the disease (2, 20, 26). In the present study, we found the sensitivity to be 74% in the acute phase. Serological methods lack specificity, and titers often remain positive for a protracted period after therapy even in cases of complete recovery (5, 15, 30). The results presented here confirm this fact. Moreover, one of the main characteristics of brucellosis is that the risk of relapse remains high even after appropriate antibiotic treatment (4). For this reason, patients who have completed a full antimicrobial therapy should be strictly followed up for 1 year in order to detect any relapse and to provide adequate treatment.

In the present study, four different PCR assays were performed for the diagnosis of brucellosis in acute and relapsing forms; simple commercial methods for DNA extraction that provide a very good DNA purification were used. All four assays had an excellent specificity (100%). They also had an excellent diagnostic sensitivity of from 95.5 to 100% in acute infection, depending on the PCR assay and the type of specimen (Table 3). We found PCRs A and C to be slightly more sensitive than PCRs D and B. The diagnostic sensitivity for PCR A is similar to results reported previously (21, 23, 24, 25, 31, 39). PCR B was previously used mostly for the detection of Brucella in animals (19), and only one previously published study used this PCR assay for the detection of Brucella in human patients, with unsatisfactory results (39), compared to the sensitivity of 95.5% found in the present study. In accordance with previous results (8, 19, 31), the analytical sensitivities for PCRs A and B were estimated in the present study to be 10 to 100 fg and 25 to 250 fg, respectively. PCRs C and D have been previously used for strain differentiation of Brucella (9, 10) and, to our knowledge, no results concerning clinical specimens have been reported to date. These two PCR assays appear in the present study to be useful in the early diagnosis of acute brucellosis since they are highly sensitive, and their detection limits, estimated here for the first time, are also excellent.

The results of the present study show that the optimal clinical specimen for PCR in acute brucellosis is buffy coat. The simple method for leukocyte isolation used in the present study instead of the complicated Ficoll-Hypaque method, which has been previously used by others (21), allows the use of buffy coat as a preferable clinical specimen in clinical laboratory practice. The presence of high concentrations of leukocyte DNA, which has been previously reported to inhibit PCR (23), did not affect the amplification in any of the four PCR assays. Whole blood also proved to be a very good clinical sample for the detection of Brucella DNA. The sensitivity of the four PCR assays with whole blood was slightly less than with buffy coat; nevertheless, a sensitivity reaching 98 to 99% is still excellent. Heme compounds and other factors such as anticoagulants that have been previously reported as PCR inhibitors (23), in combination with the very small sample volume in each PCR mixture, which has been reported as a major limitation for PCR-based assays (24), could account for the few false-negative results (2 to 4 out of 200 patients, depending on the PCR assay) in acute brucellosis. The sensitivity of the four PCR assays in serum is similar to the sensitivity reported in a previous study on patients with acute brucellosis (39) using for Brucella DNA amplification the B4 and B5 primers that were used here in PCR A. Serum could be also considered a good clinical sample, with an associated sensitivity of 95.5 to 97%; however, the results of the present study show that the buffy coat and the whole-blood samples are the preferred clinical samples for the diagnosis of acute brucellosis. The false-negative results in serum (6 to 9 out of 200 patients depending on the PCR assay) might be due to the intracellular residence of Brucella in leukocytes, which results in low numbers of bacteria in serum, and also to the small size of the inoculum.

The conventional serological tests and blood cultures in acute brucellosis displayed a combined sensitivity (96%), which is comparable to the sensitivity of each of the PCR assays. However, these conventional techniques are time-consuming and difficult, they have to be carried out in combination, and in many cases a second sample has to be examined after 2 to 3 weeks. Since laboratories use various and often nonstandardized serological methods that frequently lead to false-negative and false-positive results (39), PCR appears to be a very useful test in the clinical laboratory practice and might be established as a diagnostic criterion for the diagnosis of acute brucellosis in the future. One of the main characteristics of the PCR assays that enhances their value, as the results of the present study confirm, is the ability to establish the diagnosis of brucellosis earlier than the conventional methods. This is very significant since starting the antibiotic treatment earlier may reduce the rates of the focal disease, since a correlation with the duration of illness before hospitalization has been reported (4, 5).

After the conclusion of the antibiotic treatment, all patients were strictly followed up for 1 year. A total of 183 of the 200 patients were considered completely cured (group 1), and 17 of the 200 patients had two or three clinically confirmed relapses during this period (group 2). At the conclusion of the therapeutic regimens, the differences in the proportions of positive serological results between the patients of groups 1 and 2 were not statistically significant (P ≥ 0.70, Table 4). On the other hand, PCR assays were able to clearly distinguish the patients of the two groups (P < 0.005) at that time. At the time of the relapses, only 7 of 17 patients in group 2 had a new positive blood culture indicating a relapse, and only 4 of 17 demonstrated a distinct increase in serological titer, whereas all 17 patients were PCR positive not only at the time of relapses but also during the whole follow-up period. Again, buffy coat and whole blood were the optimal clinical samples, whereas serum does not appear to be a useful clinical sample for a posttreatment follow-up. The very small number of circulating bacteria after antibiotic treatment and the small inoculum could result in the absence of the target DNA in serum and therefore account for the false-negative results. The increase in the proportion of PCR-positive samples in serum at the time of the relapses may indicate an increased bacteremia compared to the prerelapse periods. To date, no definite criteria to establish a successful therapy of brucellosis exist, and the question of whether the patients have completely recovered or may relapse is unpredictable (5). The results of the present study show that the PCR assays could become the method of choice for the follow-up of patients with brucellosis, as other authors have previously suggested (31). The use of a PCR assay at the conclusion of the therapy could provide a very useful tool, with high negative and positive predictive values for the diagnosis of an unsuccessful treatment. In the case of a positive PCR result at the end of therapy, the treatment could be continued, thus preventing a relapse. Given the high sensitivity of the PCR assays, there is always a possibility to amplify nonviable bacteria or phagocytosed microorganisms. This possibility should be considered in the interpretation of a positive result; however, it has been previously reported that this is an uncommon event (25). On the other hand, the continuance of antibiotic therapy, with a more effective therapeutic regimen if necessary, could lead to a reduction of the relapse rate. Moreover, PCR is valuable for the detection of relapses, whereas conventional methods have proved to be inadequate.

The four PCR assays evaluated here were selected because they are able to amplify genomic DNA from B. melitensis, which is the dominant species of Brucella in Greece, in order to decide which of them is more suitable for use in clinical laboratory practice. PCRs A and C, which were slightly more sensitive and had lower detection limits than PCRs B and D, were chosen for use in the routine of the clinical laboratory of the Infectious Diseases Hospital of Thessaloniki (22).

In conclusion, the results of the present study suggest that the four single-step, in-house PCR assays used here are simple, highly sensitive, specific, and relatively inexpensive, allowing the use of a PCR assay as a routine test in clinical laboratory practice. PCR assays established the diagnosis of acute brucellosis earlier than did the conventional methods and were able to detect all cases of patients whose treatments were unsuccessful, thus making PCR a reliable means for predicting an unfavorable course of the disease, whereas conventional tests could not provide such information. Furthermore, PCR assays were able to detect all relapses, whereas the conventional methods were inefficient. Therefore, PCR proved to be a very useful tool not only for the diagnosis of acute brucellosis but also as a predictive marker for the course of the disease and the posttreatment follow-up, which is valuable for the early detection of relapses.

Acknowledgments

We thank B. D. Danielides for critically reviewing the manuscript and P. Karakoltsides for helpful advice and discussions. We thank P. Mamasi for help with the serological methods.

Footnotes

Published ahead of print on 31 January 2007.

REFERENCES

  • 1.Alton, G. G., L. M. Jones, R. D. Angus, and J. M. Verger. 1988. Techniques for the brucellosis laboratory. Institut National de la Recherche Agronomique, Paris, France.
  • 2.Ariza, J. 1996. Brucellosis. Curr. Opin. Infect. Dis. 9:126-131. [Google Scholar]
  • 3.Ariza, J. 1999. Brucellosis: an update. The perspective from the Mediterranean basin. Rev. Med. Microbiol. 10:125-135. [Google Scholar]
  • 4.Ariza, J., J. Corredoira, R. Pallares, P. F. Viladrich, G. Rufi, M. Pujol, and F. Gudiol. 1995. Characteristics of and risk factors for relapse of brucellosis in humans. Clin. Infect. Dis. 20:1241-1249. [DOI] [PubMed] [Google Scholar]
  • 5.Ariza, J., T. Pellicer, R. Pallares, A. Foz, and F. Gudiol. 1992. Specific antibody profile in human brucellosis. Clin. Infect. Dis. 14:131-140. [DOI] [PubMed] [Google Scholar]
  • 6.Armitage, P., and G. Berry. 1994. Statistical methods in medical research, 3rd ed. Blackwell Scientific Publications, London, United Kingdom.
  • 7.Avhoven, P., E. Jansson, and K. Aho. 1969. Marked cross-agglutination between brucellae and a subtype of Yersinia enterocolitica. Acta Pathol. Microbiol. Scand. 75:291-295. [PubMed] [Google Scholar]
  • 8.Baily, G. G., J. B. Krahn, B. S. Drasar, and N. G. Stoker. 1992. Detection of Brucella melitensis and Brucella abortus by DNA amplification. J. Trop. Med. Hyg. 95:271-275. [PubMed] [Google Scholar]
  • 9.Bardenstein, S., M. Mandelboim, T. A. Ficht, M. Baum, and M. Banai. 2002. Identification of the Brucella melitensis vaccine strain Rev 1 in animals and humans in Israel by PCR analysis of the PstI site polymorphism of its omp2 gene. J. Clin. Microbiol. 40:1475-1480. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Cloeckaert, A., M. Grayon, and O. Grepinet. 2000. An IS711 element downstream of the bp26 gene is a specific marker of Brucella spp. isolated from marine mammals. Clin. Diagn. Lab. Immunol. 7:835-839. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Corbel, M. J. 1979. The relationship between the protective and cross-reacting antigenes of Brucella spp., Yersinia enterocolitica O:9, and Salmonella serotypes of Kauffmann-White group. N. Contr. Microbiol. Immunol. 5:50-63. [PubMed] [Google Scholar]
  • 12.Corbel, M. J. 1997. Recent advances in brucellosis. J. Med. Microbiol. 46:101-103. [DOI] [PubMed] [Google Scholar]
  • 13.Corbel, M. J. 1997. Brucellosis: an overview. Emerg. Infect. Dis. 3:213-221. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Cutler, S. J., A. M. Whatmore, and N. J. Commander. 2005. Brucellosis - new aspects of an old disease. J. Appl. Microbiol. 98:1270-1281. [DOI] [PubMed] [Google Scholar]
  • 15.Gazapo, E., J. Gonzalez-Lahoz, J. L. Subiza, M. Baquero, J. Gil, and E. G. de la Concha. 1989. Changes in IgM and IgG antibodies concentrations in brucellosis over time: importance for diagnosis and follow-up. J. Infect. Dis. 159:219-225. [DOI] [PubMed] [Google Scholar]
  • 16.Gotuzo, E., C. Carrillo, J. Guerra, and L. Llosa. 1986. An evaluation of diagnostic methods for brucellosis-the value of blood marrow culture. J. Infect. Dis. 153:122-125. [DOI] [PubMed] [Google Scholar]
  • 17.Helenic Center for Infectious Diseases Control. 2004. Epidemiological data for the human brucellosis in Greece: 1994-2004. [Online.] http://www.keel.org.gr.
  • 18.Khalil, I. A., S. Phkrykian, and A. D. Farr. 1988. Brucella antibodies in Sudanese blood donors. Med. Lab. Sci. 45:312. [PubMed] [Google Scholar]
  • 19.Leal-Klevezas, D. S., I. O. Martinez-Vasquez, A. Lopez-Merino, and J. P. Martinez-Soriano. 1995. Single-step PCR for detection of Brucella spp. from blood and milk of infected animals. J. Clin. Microbiol. 33:3087-3090. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Lulu, A. R., G. F. Araj, M. L. Khateed, et al. 1988. Human brucellosis in Kuwait: a prospective study of 400 cases. Q. J. Med. 66:39-54. [PubMed] [Google Scholar]
  • 21.Matar, G. M., I. A. Khneisser, and A. M. Abdelnoor. 1996. Rapid laboratory confirmation of human brucellosis by PCR analysis of a target sequence on the 31-kilodalton Brucella antigen DNA. J. Clin. Microbiol. 34:477-478. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Mitka, S. 2005. Evaluation of the newest molecular techniques for the diagnosis of human brucellosis. Ph.D. thesis. Aristotle University of Thessaloniki, Thessaloniki, Greece.
  • 23.Morata, P., M. I. Queipo-Ortuno, and J. de Dios Colmenero. 1998. Strategy for optimizing DNA amplification in a peripheral blood PCR assay used for diagnosis of human brucellosis. J. Clin. Microbiol. 36:2443-2446. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Morata, P., M. I. Queipo-Ortuno, J. M. Reguera, M. A. Garcia-Ordonez, A. Carrdena, and J. D. Colmenero. 2003. Development and evaluation of a PCR-enzyme-linked immunosorbent assay for diagnosis of human brucellosis. J. Clin. Microbiol. 41:144-148. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Morata, P., M. I. Queipo-Ortuno, J. M. Reguera, M. A. Garcia-Ordonez, C. Pichardo, and J. de Dios Colmenero. 1999. Posttreatment follow-up of brucellosis by PCR assay. J. Clin. Microbiol. 37:4163-4166. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Mousa, A. M., K. M. Elhag, M. Khogali, and T. N. Sugathan. 1987. Brucellosis in Kuwait: a clinicoepidemiological study. Trans. R. Soc. Trop. Med. Hyg. 81:1820-1921. [DOI] [PubMed] [Google Scholar]
  • 27.Moyer, N. P., and P. L. A. Holcomb. 1995. Brucella, p. 549-555. In P. R. Murray, E. J. O. Baron, M. A. Pfaler, F. C. Tenover, and R. H. Yolken (ed.), Manual of clinical microbiology, 6th ed. American Society for Microbiology, Washington, DC.
  • 28.Navarro, E., M. A. Cassao, and J. Solera. 2004. Diagnosis of human brucellosis using PCR. Expert Rev. Mol. Diagn. 4:115-123. [DOI] [PubMed] [Google Scholar]
  • 29.Pappas, G., N. Akritidis, M. Bosilkovski, and E. Tsianos. 2005. Brucellosis. N. Engl. J. Med. 352:2325-2336. [DOI] [PubMed] [Google Scholar]
  • 30.Pellicer, T., J. Ariza, A. Foz, R. Pallares, and F. Gudiol. 1988. Specific antibodies detected during relapses of human brucellosis. J. Infect. Dis. 157:918-924. [DOI] [PubMed] [Google Scholar]
  • 31.Queipo-Ortuno, M. I., P. Morata, P. Ocon, P. Machado, and J. D. Colmenero. 1997. Rapid diagnosis of human brucellosis by peripheral-blood PCR assay. J. Clin. Microbiol. 35:2927-2930. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.World Health Organization. 1997. Fact sheet N173, July 1997. World Health Organization, Geneva, Switzerland.
  • 33.Yagupsky, P., and E. J. Baron. 2005. Laboratory exposures to Brucellae and implications for bioterrorism. Emerg. Infect. Dis. 11:1180-1185. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Yagupsky, P., N. Peled, J. Press, M. Abu-Rashid, and O. Abramson. 1997. Rapid detection of Brucella melitensis from blood cultures by a commercial system. Eur. J. Clin. Microbiol. Infect. Dis. 16:605-607. [DOI] [PubMed] [Google Scholar]
  • 35.Yagupsky, P., N. Peled, J. Press, O. Abramson, and M. Abu-Rashid. 1997. Comparison of BACTEC 9240 Peds Plus medium and Isolator 1.5 microbial tube for detection of Brucella melitensis from blood cultures. J. Clin. Microbiol. 35:1382-1384. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Young, E. J. 1995. Brucellosis: current epidemiology, diagnosis, and management. Curr. Top. Infect. Dis. 15:115-128. [PubMed] [Google Scholar]
  • 37.Young, E. J. 1997. Brucellosis, p. 447-451. In D. H. Connor, F. W. Chandler, H. J. Manz, et al. (ed.), Pathology of infectious diseases. Appleton & Lange, Stanford, CT.
  • 38.Young, E. J. 2005. Brucella species, p. 2669-2674. In G. L. Mandell, J. E. Bennett, and R. Dolin (ed.), Mandell, Douglas, and Bennett's principles and practice of infectious disease. Elsevier/Churchill-Livingstone, Philadelphia, PA.
  • 39.Zerva, L., K. Bourantas, S. Mitka, A. Kansouzidou, and N. J. Legakis. 2001. Serum is the preferred clinical specimen for diagnosis of human brucellosis by PCR. J. Clin. Microbiol. 39:1661-1664. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Journal of Clinical Microbiology are provided here courtesy of American Society for Microbiology (ASM)

RESOURCES