Performance of the ImmuView and BinaxNOW assays for the detection of urine and cerebrospinal fluid Streptococcus pneumoniae and Legionella pneumophila serogroup 1 antigen in patients with Legionnaires’ disease or pneumococcal pneumonia and meningitis

Paul H Edelstein; Charlotte Sværke Jørgensen; Leslie A Wolf

doi:10.1371/journal.pone.0238479

. 2020 Aug 31;15(8):e0238479. doi: 10.1371/journal.pone.0238479

Performance of the ImmuView and BinaxNOW assays for the detection of urine and cerebrospinal fluid Streptococcus pneumoniae and Legionella pneumophila serogroup 1 antigen in patients with Legionnaires’ disease or pneumococcal pneumonia and meningitis

Paul H Edelstein ^1,^2,^*, Charlotte Sværke Jørgensen ³, Leslie A Wolf ⁴

Editor: Ray Borrow⁵

¹Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States of America

²Department of Medicine, University of Cambridge, Cambridge, United Kingdom

³Department of Virus & Microbiological Special Diagnostics, Statens Serum Institut, Copenhagen, Denmark

⁴Department of Pathology and Laboratory Medicine, University of Louisville, Louisville, Kentucky, United States of America

⁵Public Health England, UNITED KINGDOM

Competing Interests: The authors have read the journal’s policy and have the following competing interests: SSI Diagnostica (https://www.ssidiagnostica.com/) provided a salary to PHE. This salary support, which was required and administered by the University of Pennsylvania, amounted to 1% of his annual salary, and was used for study planning, study supervision, data analysis from all three study sites and writing the manuscript. He received no other support from SSI Diagnostica, prior to or after this study. Pernille E. Landsbo and Sanne Otte of SSI Diagnostica wrote the clinical trial protocols, provided logistic support and trained site technologists in the performance of the assays. Ida T. Andersen of SSI Diagnostica collated data from the study sites. Christopher Bentsen, an independent consultant hired by SSI Diagnostica, assisted in clinical trial protocol development and human subject committee applications. SSI Diagnostica reviewed the manuscript for accuracy of the reported data. This does not alter our adherence to PLOS ONE policies on sharing data and materials. There are no patents, products in development or marketed products associated with this research to declare.

^✉

* E-mail: paul.edelstein@pennmedicine.upenn.edu

Roles

Paul H Edelstein: Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Writing – original draft, Writing – review & editing

Charlotte Sværke Jørgensen: Data curation, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Writing – review & editing

Leslie A Wolf: Data curation, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Writing – review & editing

Ray Borrow: Editor

PMCID: PMC7458278 PMID: 32866217

Abstract

The performances of the ImmuView Streptococcus pneumoniae (Sp) and Legionella pneumophila (Lp) urinary antigen test were compared to that of the BinaxNOW Sp and Lp assays, using frozen urine from 166 patients with Legionnaires' disease (LD) and 59 patients with pneumococcal pneumonia. Thirty Sp-positive or contrived cerebrospinal fluids (CSF) were also tested. Test specimens were collected and tested at different sites, with each site testing unique specimens by technologists blinded to expected results. No significant differences in test concordances were detected for the ImmuView and BinaxNOW assays for the Sp or Lp targets for urine from patients with pneumococcal pneumonia or LD when performance from both sites were combined. At one of two test sites the ImmuView Lp assay was more sensitive than the BinaxNOW assay, with no correlation between test performance and Lp serogroup 1 monoclonal type. Urines from six of seven patients with LD caused by Legionella spp. bacteria other than Lp serogroup 1 were negative in both assays. Both tests had equivalent performance for Sp-positive CSF. The clinical sensitivities for pneumococcal pneumonia were 88.1 and 94.4% for the ImmuView and Binax assays, and 87.6 and 84.2% for the Lp assays, respectively. Test specificities for pneumococcal pneumonia were 96.2 and 97.0% for the ImmuView and Binax assays, and 99.6 and 99.1% for the Lp assays. Both assays were highly specific for Sp in pediatric urines from children with nasopharyngeal colonization by the bacterium. ImmuView and BinaxNOW assay performance was equivalent in these studies.

Introduction

Detection of antigenuria is a commonly used test for pneumococcal pneumonia and Legionnaires' disease, and in the case of the latter is the most common method of diagnosis of the disease[1, 2]. A number of commercial assays are used for these tests, with generally comparable performances, using both conventional enzyme immunoassays and lateral flow immunoassays. The tests for Legionnaires' disease are highly specific and are most sensitive for the detection of L. pneumophila serogroup 1, with only rare positive results for disease caused by L. pneumophila serogroups other than serogroup 1 or for other Legionella spp., except for a combination L. pneumophila-L. longbeachae lateral flow assay. In addition, some of the L. pneumophila serogroup 1 antigenuria tests are more sensitive for the Pontiac L. pneumophila serogroup 1 monoclonal group than for non-Pontiac strains[3]. Apart from the use of these assays to assist in the etiologic diagnosis of pneumonia, some of these assays can be used to detect pneumococcal antigen in cerebrospinal fluid to assist in the detection of pneumococcal meningitis[4].

The SSI Diagnostica ImmuView assay is a CE-marked and U.S. FDA-cleared lateral flow assay that detects both L. pneumophila and S. pneumoniae antigen using a single test strip (www.immuview.com). The Abbott BinaxNOW card assays for L. pneumophila and S. pneumoniae are U.S. FDA cleared devices that use separate test kits and test strips for each bacterium (www.alere.com/en/home/products-services/brands/binaxnow.html). We performed a comparison of these two test types to serve as data for a U.S. FDA 510k application for the ImmuView assay. These assays were carried out at one diagnostic laboratory in the USA and one in Denmark using a large number of archived frozen urine specimens specific to each laboratory, from patients with pneumococcal pneumonia or Legionnaires' disease, or neither disease. In addition, two U.S. laboratories and one Danish laboratory tested both test kits for the presence of pneumococcal antigen in CSF specimens. We show that both the ImmuView and BinaxNOW assays have equivalent performance.

Materials and methods

Testing was performed at three study site laboratories, the Infectious Diseases Laboratory of the University of Louisville, the Clinical Microbiology Laboratory of the University of Pennsylvania Hospital (UPenn) and the serological laboratory of the Statens Serum Institut in Copenhagen (SSI). The SSI is a different entity, and independent from, SSI Diagnostica, the manufacturer of the ImmuView urine antigen test kit. Urines from patients with Legionnaires' disease and bacteremic pneumococcal pneumonia were tested at UPenn and SSI, with the specimens being different at the two sites. The 55 positive urine specimens from patients with Legionnaires' disease tested by SSI were novel and not previously used in a prior published evaluation of the ImmuView or any other test[5]. All patients with a diagnosis of Legionnaires' disease or bacteremic pneumococcal pneumonia had clinical findings consistent with acute pneumonia including pulmonary infiltrates on chest roentgenography. All patients with previously negative tests for Legionnaires' disease or pneumococcal pneumonia were suspected of having bacterial pneumonia by their ordering clinicians. The urine specimens tested were not the result of consecutive sampling during infection, were not selectively collected because of specific clinical findings, and were collected during the initial presentation of the pneumonia with only one specimen collected per patient.

Urines from 56 children with lower respiratory infection not due to pneumococcal pneumonia and known S. pneumoniae nasopharyngeal colonization status, were tested at SSI only. The criteria for pneumonia were clinical signs and symptoms of pneumonia. All children had respiratory specimens submitted for culture; none grew S. pneumoniae, while 12 grew Staphylococcus aureus, three Branhamella catarrhalis, one Acinetobacter spp., and the rest normal upper respiratory flora. Seven of the children had positive prior testing on their urines for pneumococcal antigen using the Binax NOW S. pneumoniae antigen card, with the remainder being negative. All but four of the children had nasopharyngeal cultures obtained to determine S. pneumoniae colonization on the same day that urines were collected for antigen testing, with 13 children found to be colonized. Culture and prior pneumococcal antigen testing results were not used to select these specimens, with the exception that no child was included if the respiratory tract culture grew S. pneumoniae. The age of the children ranged from <1 to 19 years, with a median of 5.5 years and mean of 6.6 years. All three laboratories tested negative control cerebrospinal fluid (CSF), CSF from five patients with culture-confirmed pneumococcal meningitis and five contrived S. pneumoniae-containing CSF. One hundred CSF specimens were distributed for testing to each study site by SSI Diagnostica, with each site being sent different CSF specimens (Table 1). All specimens tested at each site were randomized and identified only by a number code, with the technologists performing testing being unaware of the true identity or expected result for each specimen. The urine specimens that were tested at each site had been stored frozen (-20 to -80 C) for up to 39 years after initial collection, and were all collected from adult patients with suspected pneumonia and known results for either L. pneumophila urine antigen tests, respiratory cultures for conventional bacterial pathogens, respiratory cultures for Legionella spp., blood cultures, or a combination of one or more of these tests (Table 2).

Table 1. Specimens tested at each testing site.

Test site	Specimen type
	CSF	Urine
	Pneumococcal meningitis^a	Legionnaires’ disease^b	Pneumococcal pneumonia^c	LRI children^d
UPenn	10/100^e; 5/94^f	111/49^e; 111/49^f	7/153^e; 7/153^f	not tested
Univ of Louisville	10/100^e; 0/0^f	not tested	not tested	not tested
SSI	10/100^e; 8/88^f	55/183^e; 54/178^f	52/186^e; 47/184^f	0/56^g; 0/55^h
All sites	30/300^e; 13/182^f	166/232^e; 165/227^f	59/339^e; 54/337^f	0/56^g; 0/55^h

Open in a new tab

^a CSF from five patients with pneumococcal meningitis or five normal CSFs seeded with S. pneumoniae. All three study sites tested different CSF specimens, both positive and negative

^b positive Legionella spp. sputum culture, prior positive L. pneumophila urine antigen assay, or both; negative controls were patients with prior negative L. pneumophila urine antigen tests, positive blood cultures for another pathogen (including S. pneumoniae), sputum cultures yielding another pathogen, or negative blood cultures and no respiratory pathogen isolated

^c pneumococcal pneumonia with S. pneumoniae bacteremia; negative controls were from patients with other infections, including Legionnaires’ disease

^d children suspected of bacterial pneumonia, 13 of whom had positive nasopharyngeal cultures for S. pneumoniae. This group was included to study tests for S. pneumoniae and L. pneumophila antigenuria specificity only in children

^e total positive/total negative patient urines or CSF specimens tested using ImmuView assay

^f total positive/total negative patient urines or CSF specimens tested using BinaxNOW assays

^g total positive/total negative patient urines tested using ImmuView assay and BinaxNOW S. pneumoniae assay

^h total positive/total negative patient urines using BinaxNOW L. pneumophila assay

Table 2. Laboratory findings for patients with Legionnaires’ disease.

Site	Culture positive Lp1^a			Culture positive not Lp1	Culture not done			Total
	UAg+^b	UAg-	UAg Equiv+	UAg neg	UAg +	UAg Equiv +, Outbreak Suspect	UAg Equiv +
UPenn	44	0	0	7^c	54	3	3	111
SSI	19	28	1	0	0	0	0	55

Open in a new tab

^a Lp1, L. pneumophila serogroup 1

^b UAg, urine antigen test positive (+), negative (-), or borderline positive (Equiv+)

^c L. wadsworthii, L. bozemanae, L. longbeachae serogroup 1 (2 ea), L. pneumophila serogroup 2, L. pneumophila serogroup 4 (2 ea)

The BinaxNOW S. pneumoniae and L. pneumophila lateral flow assays, which are two separate tests, were performed according to the manufacturer’s instructions. The ImmuView S. pneumoniae and L. pneumophila lateral flow assay, which is a single test, was also performed according to the manufacturer’s instructions. Briefly, the BinaxNOW assay is performed by inserting a specimen-soaked kit-supplied swab onto a specimen pad on a card, followed by the addition of kit-supplied buffer to a 2^nd well. The card is closed and incubated for up to 15 minutes at room temperature. Colored lines on the card indicate both internal control and target positivity. A positive result can be read at any time up to the 15 minute maximum incubation time and a negative result is read at the 15 minute mark. Any visible line is considered positive, regardless of the color strength or line width, as long as the control line is also positive.

The ImmuView assay is performed by adding a specified amount of urine (120μL or three drops) to a kit-supplied test tube containing a specified amount of buffer (90μL or two drops). The ImmuView strip is immersed in the test tube, removed after 15 minute’s incubation at room temperature, and read while still wet. Three line positions can be read on the strip, the control line, a L. pneumophila line and a S. pneumoniae line, each of which are distinguished by a different color. Any line is considered positive regardless of color intensity as long as the line width is complete, with a requirement that the control line be positive for a valid assay. Partial width lines and colored spots are interpreted as invalid, requiring retesting of the sample in a new test.

Tests on the same specimen using different kits were not performed simultaneously or immediately sequentially to avoid reading bias. At least four hours separated the performance of each test type to help avoid reading bias, and if possible, a different technologist performed each test type on the specimen on a given day to further reduce reading bias. Results were recorded manually for each run, with a different recording sheet being used in the morning and afternoon readings for blinding purposes. Digital photographs of test strips and cards were made and stored after each run, to allow confirmation of the accuracy of the recorded results.

After both test types were performed, and the results recorded, the results were reviewed by a person not performing the testing to determine if the ImmuView and BinaxNOW results were discordant. If that were the case, then repeat testing of the specimen using the ImmuView and the target-specific BinaxNOW assay was performed, after boiling the urine for 10 minutes, centrifugation (~10,00 RCF) for one minute, and testing the supernate. The repeat testing was carried out one or more days after initial testing, with testing of the BinaxNOW and ImmuView assay separated by at least four hours, to reduce reading bias. Boiling and centrifugation was designed to reduce false-positive results due to heavy urine sediment and rheumatoid-like factors.

Calculations of test concordances were based solely on test results without regard to the known prior microbiology results. Because this trial was designed to determine the relative abilities of the two assays for L. pneumophila to detect Legionnaires' disease caused by confirmed L. pneumophila serogroup 1 infection, testing results of urines from patients with Legionella spp. infection other than L. pneumophila serogroup 1 or those whose initial urine antigen results were borderline positive were excluded from the primary analysis of test agreements and clinical sensitivity and specificity. A confirmed case of pneumococcal pneumonia required a positive blood culture for S. pneumoniae. Confirmed cases of Legionnaires' disease required either a positive respiratory tract culture for Legionella spp., a positive urine antigen assay for L. pneumophila, or both positive tests. Prior microbiology laboratory results for each patient were unblinded only after the trial was completed.

Each laboratory was instructed in the use of the ImmuView assay by SSI Diagnostica. Prior to testing the unknown specimens, each technologist performing the testing successfully completed extensive proficiency and reproducibility examinations using blinded specimens, and was also observed for proper technique by a SSI Diagnostica expert in use of the test. Instruction in the proper performance of the BinaxNOW tests was given by a different expert in the use of these tests, with technologists not allowed to perform testing on unknown testing until proper test performance was ascertained by observation and by completion of a panel of positive and negative specimens with complete accuracy. Thus, the participating technologists were highly trained, and accurately performed testing, prior to testing of the unknown clinical specimens.

Statistical testing of differences for the paired sample agreement assays were performed using a two-tailed McNemar test[6]. Analysis of clinical sensitivity and specificity uses standard definitions[7], for which 95% confidence intervals were calculated using the modified Wald method [8]. Comparisons of clinical sensitivity and specificity between sites used the two-tailed Fisher exact test (GraphPad InStat3.10).

The human studies protocols of all three study sites were approved by the Western Institutional Review Board (IRB) for human studies, protocol 20162534. In addition, the Western IRB reviewed and approved an additional protocol for the SSI site, protocol 1170324. The University of Pennsylvania Human Studies IRB accepted the Western IRB approval of the test protocol under protocol number 10060242. The University of Louisville Human Studies IRB approved the study under protocol number 16.0531. Patient consents were not obtained because these were not required by the IRBs.

The study sponsor, SSI Diagnostica, wrote the test protocols, paid for the study reagents and supplies, as well as for the performing technologist’s time at all sites except for the SSI site. This report was written independently of SSI Diagnostica using data spreadsheets supplied by the sponsor as well as separately kept spreadsheets for the UPenn site. SSI Diagnostica reviewed the manuscript for accuracy of the reported data but otherwise had no role in its writing. All authors vouch for the accuracy of this report.

Results

Urines from 166 and 59 patients with Legionnaires' disease and bacteremia pneumococcal pneumonia, respectively, were tested. Thirty actual or contrived S. pneumoniae-positive cerebrospinal fluids (CSF) and 56 urines from children were tested (Table 1).

When testing for bacteremic pneumococcal pneumonia the ImmuView and BinaxNOW assays had near perfect agreement (Table 3). Clinical sensitivities for these assays were 88.1 and 94.4% for the ImmuView and BinaxNOW assays, respectively, with overlapping 95% confidence intervals (Table 4). Test specificities were 99.6 and 97.0% for the ImmuView and BinaxNOW assay, respectively. Testing site-specific test sensitivity, but not specificity, was significantly lower for the UPenn than the SSI site for both assay types (P<0.01 by Fisher’s exact test) (S1 Table). These site specific differences in test sensitivity were not reflected in test agreements for the two different assays for each study site (S2 and S3 Tables), where no statistically significant differences were found by McNemar tests.

Table 3. S. pneumoniae specimens test agreements for urines from adults.

	BinaxNOW
ImmuView	positive	negative
positive	53	7
negative	8	323

Open in a new tab

p = 1.0, McNemar test

Table 4. Clinical sensitivity and specificity for urines from adults.

Assay	Target	Sensitivity^a (%)	Specificity^b (%)
ImmuView	S. pneumoniae	88.1 (77.1 to 94.3), 52/59	96.2 (93.5 to 97.8), 326/339
BinaxNOW	S. pneumoniae	94.4 (84.2 to 98.6), 51/54	97.0 (94.5 to 98.4), 327/337
ImmuView	*L pneumophila*	87.6 (81.3 to 91.9), 134/153	99.6 (97.3 to 100), 231/232
BinaxNOW	*L pneumophila*	84.2 (77.5 to 89.2), 128/152	99.1 (96.6 to 99.9), 225/227

Open in a new tab

a, mean (95% CI), total test positive patients with infection/total patients with infection specific to target

b, mean (95% CI), total test negative patients without infection/total patients without infection specific to target

The ImmuView L. pneumophila test detected nine positive urines that were BinaxNOW-negative versus five BinaxNOW-positives that were ImmuView-negative (p = 0.42) (Table 5). This imbalance in test sensitivity was solely due to a non-statistically significant difference (p = 0.11) in test concordance at the SSI site, but not the UPenn site (S4 and S5 Tables). These site-specific test agreement differences between the BinaxNOW and ImmuView assays had the same trend, albeit with overlapping confidence intervals when the data were analyzed for clinical sensitivity. No apparent difference was detected between sites for clinical specificity (S6 Table). The ImmuView and BinaxNOW assays had 92 and 94% sensitivity, respectively, at the UPenn site, but 80 and 67% sensitivity for the SSI site, for the ImmuView and BinaxNOW assays, respectively (UPenn vs. SSI p = 0.001 for comparison of sensitivity for each test type, Fisher’s exact test). Because test sensitivity has been reported to be higher for L. pneumophila serogroup 1 Pontiac monoclonal types, we analyzed the SSI site results (the only site having monoclonal typing results, for the 48 typed isolates) for BinaxNOW-ImmuView agreements by monoclonal antibody type (S7 Table). There was no association between BinaxNOW-ImmuView discordance and monoclonal antibody type (p = 0.2, Fisher’s exact test). We analyzed the effect of monoclonal antibody type on the seven specimens with available strain typing data that were falsely-negative for L. pneumophila by BinaxNOW assay, and truly-positive by ImmuView assay at the SSI site (S8 Table). Five of the seven urines were obtained from patients with Legionnaires' disease caused by the Pontiac monoclonal group, demonstrating no correlation between Pontiac monoclonal group and test sensitivity differences between the two assays. In addition, for all typed infection specimens regardless of discrepancies, ImmuView sensitivity for Pontiac and non-Pontiac infections was 87 and 68%, respectively, and BinaxNOW sensitivity was 65 and 60% for Pontiac and non-Pontiac infections, respectively (p>0.1 for comparisons within or between tests, Fisher’s exact tests).

Table 5. L. pneumophila test agreements for urines from adults.

	BinaxNOW
ImmuView	positive	negative
positive	125	9
negative	5	240

Open in a new tab

p = 0.42, McNemar test

Not included in the calculations of test performance for the L. pneumophila assays were urines from 13 UPenn patients, either because the patients had Legionnaires' disease not caused by L. pneumophila serogroup 1, or because they had only borderline-positive (“equivocal”) urine antigen test results, using either a commercial radioimmunoassay or ELISA, without positive cultures at the time of initial testing. Seven patients had culture-positive Legionnaires' disease caused by L. wadsworthii, L. bozemanae, L. longbeachae serogroup 1 (2 patients), L. pneumophila serogroup 2 and L. pneumophila serogroup 4 (2 patients). Six of the seven urines from these patients were not detected as positive by either assay, and one from a patient with L. pneumophila serogroup 4 disease was detected by both assay types. Three patients were suspected of epidemic Legionnaires' disease and were investigated as outbreak subjects during two different Legionnaires' disease outbreaks, both caused by L. pneumophila serogroup 1, Pontiac monoclonal group; none of these three patients had positive cultures and all three had borderline positive urine antigen assays when they were first tested decades previously. When retested during this trial two of the outbreak-related urines were positive, but only in the ImmuView assay, with the third being negative in both assays. Three additional urines were tested that were taken from patients with suspected sporadic Legionnaires' disease, with the only laboratory test indication of the disease being borderline-positive urine antigen tests at the time of original collection; one of these three was positive in both the BinaxNOW and ImmuView assays, with the other two being negative in both assay types. Inclusion of the data from these 13 urines into tests of agreement resulted in no significant difference for the UPenn site (p = 1.0, McNemar test) (S9 Table), and a change in the UPenn test sensitivity, but not specificity, with the assumption that all 13 of these patients truly had Legionnaires' disease. Test sensitivities for the UPenn site changed to 85.0% for both the ImmuView and BinaxNOW assays, respectively (S6 Table footnote). When these data are included in overall test performance for both the SSI and UPenn sites test agreements were not significantly different (p = 0.21 by McNemar test) with a change in test sensitivities to 83.1 and 79.4% for the ImmuView and BinaxNOW assays, respectively (S10 and S11 Tables).

Testing of CSF specimens from patients with pneumococcal meningitis, or contrived-positive CSF specimens, showed almost perfect agreement between BinaxNOW and ImmuView S. pneumoniae tests (p = 1, McNemar test) (S12 Table). Because the positive CSF specimens were comprised both of CSF specimens from patients with pneumococcal meningitis, and contrived positive CSF specimens, clinical sensitivity and specificity were not calculated, but rather positive and negative agreement. Positive agreement was high for both assays in comparison with the known result, with overlapping 95% confidence intervals, with values of 96.7 and 92.3% for the ImmuView and BinaxNOW assays, respectively (S13 Table). Both the BinaxNOW and ImmuView S. pneumoniae assays exhibited high negative agreement in comparison with the known result (98.9 and 97.7%, respectively) (S13 Table). The ImmunView L. pneumophila component of the dual target test strip, which is not indicated for testing of CSF, exhibited high negative agreement of 98.8% (95% CI 96.8 to 99.7%). No CSF testing was performed using the BinaxNOW L. pneumophila assay.

Testing of urines from children with lower respiratory tract infection, none of whom had pneumococcal pneumonia based on the absence of a positive culture result for S. pneumoniae, was performed to determine the specificity of urine antigen testing since nasopharyngeal pneumococcal colonization is known to lead to falsely-positive urine antigen assays. Thirteen of the 56 children tested for had pneumococcal colonization of the nasopharynx. Three of 56 urines were positive by both the BinaxNOW and ImmuView S. pneumoniae assays, and two urines were BinaxNOW-positive but ImmuView-negative (p = 0.5, McNemar test) (S14 Table). One of the three ImmuView-positive urines was from a child with pneumococcal colonization, as were two of the five BinaxNOW-positive tests. No BinaxNOW or ImmuView L. pneumophila test was positive for these urine samples (S15 Table). Clinical specificity for the S. pneumoniae target was 94.6 and 91.1% for the ImmuView and BinaxNOW assays, respectively, with overlapping 95% confidence intervals (S16 Table).

Since false-positive urine antigen tests have been reported as the result of rheumatoid-like factors in the urine that are inactivated by boiling, urine specimens that gave discrepant results between BinaxNOW and ImmuView assays were retested after boiling and centrifugation[9]. This was done only if there was sufficient specimen available to conduct a test, with priority given to the ImmView assay if there was insufficient urine to conduct both ImmuView and BinaxNOW assays. Priority was given to testing the ImmuView in the case of insufficient volume for both tests because less was known about the effect of specimen boiling on ImmuView results than is known about this effect on Binax assays. Sufficient volume was available at the UPenn site for retesting of all discrepant pairs, but for the SSI site only one of the 18 discrepant pairs were tested using both the ImmunView and BinaxNOW assays (S17 Table). The results after this treatment were not used to determine test agreements or clinical sensitivity and specificity. For the three BinaxNOW-ImmuView discordant UPenn L pneumophila results for urines from patients with Legionnaires' disease, only one discrepancy was resolved after boiling treatment. For all three UPenn urines from patients with Legionnaires' disease that tested positive with either the BinaxNOW or ImmuView S. pneumoniae assay, boiling resolved the discrepancies. One of two UPenn urines from patients with bacteremic pneumococcal pneumonia changed to a concordant BinaxNOW-ImmuView S. pneumoniae result.

Some of the results of boiling treatment were unexpected because three ImmuView S. pneumoniae truly-negative specimens became falsely-positive after treatment and some falsely-negative specimens became true-positives for both test types, as well as the expected false-positives becoming true-negative (S17 Table). For example, three of ten ImmuView-true positive but BinaxNOW-false negative L. pneumophila specimens reverted to negative after boiling; unfortunately, there was inadequate urine available to perform a test with the BinaxNOW assay on these boiled specimens. The three ImmunView true-positive to false-positive specimens were weakly positive prior to boiling based on the intensity of the positive band. Because many discordant pairs had post-boiling testing performed only using the ImmuView assay, comparisons of the frequencies of the various outcomes between BinaxNOW and ImmuView assays cannot be determined.

Discussion

This study showed that the ImmuView and BinaxNOW assays had excellent test agreements and clinical performances for both the S. pneumoniae and L. pneumophila targets for both urine and CSF specimens. The very large number of urines that were tested from patients with confirmed Legionnaires' disease provided a robust estimate of assay performance, as did the use of several hundred negative controls to determine assay specificities. Use of specimens from more than one site showed that relative assay performance differed between sites for the L. pneumophila target, suggesting the possibility of differing performance based on strain type or clinical severity.

There was very high test concordance between the two assays for the S. pneumoniae target for urine specimens from patients with pneumococcal bacteremia, with only two discordant results. Comparing these results with the two previously published studies of the ImmuView vs. BinaxNOW assays for the detection of S. pneumoniae antigenuria is impossible, because none of the prior studies revealed concordance data, only sensitivity and specificity rates[10, 11]. Jørgensen and colleagues reported that the ImmuView and BinaxNOW S. pneumoniae assays were 85 and 77% sensitive, respectively, for patients with pneumococcal bacteremia, and 98.7 and 100% specific[10]. Athlin and colleagues reported sensitivities of 60 and 62% for the BinaxNOW and Immunview tests, respectively, with 97% specificity for both tests[11]. These data contrast with the 88 and 94% sensitivities found by us, for the ImmuView and BinaxNOW assays, respectively, with respective specificities of 93 and 97%. The lower sensitivity of the ImmuView assay than the BinaxNOW assay at the UPenn site represents a difference of two results out of only seven total known positive urines tested, and is unlikely to represent a truly significant difference in assay performance. Based on our results, plus those of the two published studies, there are insignificant differences between the performances of these two tests for the detection of S. pneumoniae antigenuria.

Evaluation of the performances of the two different assays for the detection of L. pneumophila antigenuria was more complicated than for the S. pneumoniae target because of testing site differences in relative clinical sensitivity. Test sensitivities, but not specificities, were significantly greater at the UPenn site than at the SSI site. We were unable to explain this based on strain monoclonal antibody typing data from the SSI site. Another possibility for this difference is that 54% of the UPenn positive specimens tested were from antigenuria-positive only specimens, perhaps resulting in a selection bias. Whether other factors such as patient or unmeasured bacterial strain differences also accounted for these differences is unknown. Both test types had non-significantly different test result concordances within each site, and for the combined sites, good evidence for equivalent performance of these two test types for the detection of L. pneumophila serogroup 1 antigenuria.

In a previously published different study performed at the SSI, using different specimens, the ImmuView L. pneumophila assay detected 88 of 99 (89%) L. pneumophila serogroup 1 culture-positive or urine antigen-positive specimens, versus 71 of 99 (72%) specimens for the BinaxNOW assay[10]. For urines from solely L. pneumophila serogroup 1 culture-positive patients, these numbers were 48 of 55 (87%) and 43 of 55 (78%) positive tests for the ImmuView and BinaxNOW assays, respectively; without test concordance data it is impossible to perform appropriate statistical testing on these paired specimens, but the ImmuView assay appeared to be more sensitive in that study.

In the previously published SSI study both the BinaxNOW and ImmuView tests were better able to detect Legionnaires' disease caused by the L. pneumophila serogroup Pontiac monoclonal group than the non-Pontiac monoclonal groups, although these differences were not significantly different [10]. We observed the same differences in test sensitivity depending on monoclonal group, which were also not statistically significantly different between or within test types.

Neither the ImmuView nor the BinaxNOW L. pneumophila assays were able to detect urine antigen from six of the seven patients with Legionnaires' disease caused by non-L. pneumophila serogroup 1 bacteria, with both detecting one of the two L. pneumophila serogroup 4 infections. The inability to detect urine antigen from the four patients with Legionnaires' disease caused by Legionella spp. other than L. pneumophila was expected, has been reported previously[10], and is not claimed in the product insert of either test. A prior study showed that the ImmuView test detected 0/2, 0/5, 2/14, 10/28 and 1/1 of urines from patients with L. pneumophila serogroup 4, 5, 6, 3 and 15 infections, with the BinaxNOW assay only detecting 1 of 28 serogroup 3 urines[10]. We add to this the inability of either assay to detect urine antigen from a patient with L. pneumophila serogroup 2 infection.

Our test of CSF specimens showed excellent concordance between test types. Although the small number of positive specimens tested results in a broad estimate of test sensitivity, test performance for both tests parallels that found in large clinical studies for the BinaxNOW assay[4, 12], lending credence to our findings.

The use of urine antigen detection for the diagnosis of pediatric pneumococcal pneumonia is non-specific because 25 to 50% of children with pneumococcal nasopharyngeal colonization have been reported to have positive antigen test results[13, 14]. We found a much lower fraction of positive tests for either test type in colonized children, 5 to 9%. The reason for this difference is unknown, but regardless the specificities of each test were not significantly different for pediatric urine specimens.

Confirming a positive urine antigen assay by retesting the urine after boiling and centrifugation is commonly used to exclude false-positive tests due to rheumatoid-like factors, and for some tests such as the ImmuView test, this procedure is found in the kit instructions[15, 16]. Several studies have reported no change in test sensitivity using this procedure[9–11, 17]. It was therefore surprising that boiling and centrifugation of discrepant specimens led not only to falsely-positive tests becoming truly-negative as expected, but also truly-positive urines becoming falsely-negative. In addition, some urine specimens converted from falsely-negative to truly-positive. Since many of these urines were retested only with the ImmuView test, the greater frequency of these unexpected reactions with the ImmuView test does not necessarily point to this being more common with that test. It is possible that these aberrant results were due to prior freezing of the urines, to antigen levels very close to the detection limit or to some other factor. It would be wise to determine the frequency of such results in a prospective study for the both tests with both negative and positive freshly collected urines.

Several potential limitations of this study should be mentioned. First, positive urines for testing were unavoidably selected on the basis of a prior positive urine antigen test or a positive respiratory (Legionella spp.) or blood culture (S. pneumoniae). This results in a selection bias[18], because it was highly likely that these preselected urines would be found positive in the test assays. Thus our estimates of test sensitivity are likely an overestimate of the true sensitivity in patients without known prior positive tests or a high severity illness. Test specificity should not be affected in this case. There is also a likely spectrum bias, because urines from patients with pneumococcal pneumonia and bacteremia, and patients with positive respiratory cultures for Legionnaires' disease, were overrepresented in our test population. Therefore, comparative test agreement, rather than clinical sensitivity, is the best measure of the relative performances of these two different test kits. Another potential limitation is that all of the tested urines had been previously frozen, some several decades previously. While the antigen targets are relatively stable, there can be degradation of the antigen with prolonged storage[19]. This would have the opposite effect of the selection bias, in that urines from patients with Legionnaires' disease or pneumococcal pneumonia may have lost sufficient antigen to become negative and produce a falsely-negative test result.

In conclusion, both the ImmuView S. pneumoniae and L. pneumophila Urinary Antigen Test, and the BinaxNOW S. pneumoniae and L. pneumophila antigen cards for the same organisms performed equally well for the detection of Legionnaires' disease caused by L. pneumophila serogroup 1, bacteremic pneumococcal pneumonia and pneumococcal meningitis. Regional variations in L. pneumophila strains causing Legionnaires' disease may affect the relative performance of these tests but further delineation of this in prospective studies is required.

Supporting information

S1 Table. Clinical sensitivity and specificity for S. pneumoniae antigenuria by study site.

(PDF)

Click here for additional data file.^{(46.6KB, pdf)}

S2 Table. Agreement of S. pneumoniae antigenuria testing, UPenn.

(PDF)

Click here for additional data file.^{(41.9KB, pdf)}

S3 Table. Agreement of S. pneumoniae antigenuria Testing, SSI.

(PDF)

Click here for additional data file.^{(40.8KB, pdf)}

S4 Table. Agreement of L. pneumophila antigenuria testing Lp SG1 only, UPenn.

(PDF)

Click here for additional data file.^{(43.7KB, pdf)}

S5 Table. Agreement of L. pneumophila antigenuria testing, SSI.

(PDF)

Click here for additional data file.^{(41.2KB, pdf)}

S6 Table. Clinical sensitivity and specificity for L. pneumophila antigenuria by study site.

(PDF)

Click here for additional data file.^{(51.2KB, pdf)}

S7 Table. Correlation of BinaxNOW and ImmuView L. pneumophila results and monoclonal group for concordant and discordant results, SSI.

(PDF)

Click here for additional data file.^{(43.3KB, pdf)}

S8 Table. Correlation of BinaxNOW L. pneumophila falsely-negative tests with monoclonal group, SSI site.

(PDF)

Click here for additional data file.^{(45.5KB, pdf)}

S9 Table. Agreement for UPenn site only of BinaxNOW and ImmuView L. pneumophila assays including urines from patients with Legionella infections other than non-L. pneumophila serogroup 1, culture-negative outbreak suspects and culture-negative sporadic cases with initial borderline-positive urine antigen tests.

(PDF)

Click here for additional data file.^{(46.2KB, pdf)}

S10 Table. Agreement of BinaxNOW and ImmuView L. pneumophila assays including urines from UPenn patients with Legionella infections other than non-L. pneumophila serogroup 1, culture-negative outbreak suspects and culture-negative sporadic cases with initial borderline-positive urine antigen tests^a.

SSI and UPenn sites combined.

(PDF)

Click here for additional data file.^{(45.8KB, pdf)}

S11 Table. BinaxNOW and ImmuView L. pneumophila clinical sensitivity and specificity when including specimens from patients with Legionella infections other than non-L. pneumophila serogroup 1, culture-negative outbreak suspects and culture-negative sporadic cases with initial borderline-positive urine antigen tests.

SSI and UPenn combined data.

(PDF)

Click here for additional data file.^{(49.3KB, pdf)}

S12 Table. S. pneumoniae CSF agreements.

(PDF)

Click here for additional data file.^{(40.3KB, pdf)}

S13 Table. Positive and negative agreement for CSF specimens.

(PDF)

Click here for additional data file.^{(44.6KB, pdf)}

S14 Table. S. pneumoniae pediatric urines agreements.

(PDF)

Click here for additional data file.^{(41.9KB, pdf)}

S15 Table. L. pneumophila pediatric urines agreements.

(PDF)

Click here for additional data file.^{(40.3KB, pdf)}

S16 Table. Clinical specificity for pediatric urines.

(PDF)

Click here for additional data file.^{(44KB, pdf)}

S17 Table. Effects of retesting urine after boiling for ImmuView-BinaxNOW discordant result specimens for Upenn and SSI sites.

(PDF)

Click here for additional data file.^{(46.8KB, pdf)}

Acknowledgments

Pernille E. Landsbo and Sanne Otte of SSI Diagnostica wrote the clinical trial protocols, provided logistic support and trained site technologists in the performance of the assays. Ida T. Andersen of SSI Diagnostica collated data from the study sites. Christopher Bentsen, an independent consultant hired by SSI Diagnostica, assisted in clinical trial protocol development and human subject committee applications. Martha Edelstein and Karina Rios performed testing at the University of Pennsylvania. Subathra Marimuthu, Laura G. Schindler, S. Sabrena Garr and Jennifer L. Wick performed testing at the University of Louisville, and James Summersgill initially directed testing there. Anders Frische, Brian Hauschildt, Suheil Nasim, and Regina Hansen performed testing at the Serological Unit, Statens Serum Institut.

Data Availability

All relevant data are within the manuscript and its Supporting Information files.

Funding Statement

This study was supported by SSI Diagnostica (https://www.ssidiagnostica.com/) in the form of a salary provided to PHE. This salary support, which was required and administered by the University of Pennsylvania, amounted to 1% of his annual salary, and was used for study planning, study supervision, data analysis from all three study sites and writing the manuscript. The specific role of this author is articulated in the ‘author contributions’ section. SSI Diagnostica paid for the study reagents and supplies at each study site. Additionally, at the University of Pennsylvania and University of Louisville sites, SSI Diagnostica paid for the performing technologist’s time. SSI Diagnostica reviewed the manuscript for accuracy of the reported data. The funders had no further role in data analysis, decision to publish, or preparation of the manuscript.

References

1.Said MA, Johnson HL, Nonyane BA, Deloria-Knoll M, O'Brien KL, Team AAPBS, et al. Estimating the burden of pneumococcal pneumonia among adults: a systematic review and meta-analysis of diagnostic techniques. PLoS One. 2013;8(4):e60273 10.1371/journal.pone.0060273 [DOI] [PMC free article] [PubMed] [Google Scholar]
2.Centers for Disease Control. Legionnaires' disease surveillance summary report, United States 2014–20152019 2/28/2020. Available from: https://www.cdc.gov/legionella/health-depts/surv-reporting/2014-15-surv-report-508.pdf.
3.Helbig JH, Uldum SA, Bernander S, Luck PC, Wewalka G, Abraham B, et al. Clinical utility of urinary antigen detection for diagnosis of community-acquired, travel-associated, and nosocomial legionnaires' disease. J Clin Microbiol. 2003;41(2):838–40. 10.1128/jcm.41.2.838-840.2003 [DOI] [PMC free article] [PubMed] [Google Scholar]
4.Moïsi JC, Saha SK, Falade AG, Njanpop-Lafourcade BM, Oundo J, Zaidi AK, et al. Enhanced diagnosis of pneumococcal meningitis with use of the Binax NOW immunochromatographic test of Streptococcus pneumoniae antigen: a multisite study. Clin Infect Dis. 2009;48 Suppl 2:S49–56. 10.1086/596481 [DOI] [PMC free article] [PubMed] [Google Scholar]
5.Jorgensen CS, Uldum SA, Elverdal PL. Application of a lateral flow test as an additional serological tool for diagnosis of Legionella infections. J Microbiol Methods. 2014;96:12–5. 10.1016/j.mimet.2013.10.021 [DOI] [PubMed] [Google Scholar]
6.GraphPad QuickCalcs [cited 2020]. Available from: https://www.graphpad.com/quickcalcs/.
7.U.S. FDA Center for Devices and Radiologic Health. Statistical guidance on reporting results from studies evaluating diagnostic tests. Rockville, MD2007. p. 1–39.
8.Agresti A, Coull BA. Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions. Am Stat. 1998;52(2):119–26. 10.1080/00031305.1998.10480550 [DOI] [Google Scholar]
9.Sathapatayavongs B, Kohler RB, Wheat LJ, White A, Winn WC Jr., Girod JC, et al. Rapid diagnosis of Legionnaires' disease by urinary antigen detection. Comparison of ELISA and radioimmunoassay. Am J Med. 1982;72:576–82. 10.1016/0002-9343(82)90451-x . [DOI] [PubMed] [Google Scholar]
10.Jørgensen CS, Uldum SA, Sørensen JF, Skovsted IC, Otte S, Elverdal PL. Evaluation of a new lateral flow test for detection of Streptococcus pneumoniae and Legionella pneumophila urinary antigen. J Microbiol Methods. 2015;116:33–6. 10.1016/j.mimet.2015.06.014 . [DOI] [PubMed] [Google Scholar]
11.Athlin S, Iversen A, Özenci V. Comparison of the ImmuView and the BinaxNOW antigen tests in detection of Streptococcus pneumoniae and Legionella pneumophila in urine. Eur J Clin Microbiol Infect Dis. 2017;36(10):1933–8. 10.1007/s10096-017-3016-6 . [DOI] [PMC free article] [PubMed] [Google Scholar]
12.Samra Z, Shmuely H, Nahum E, Paghis D, Ben-Ari J. Use of the NOW Streptococcus pneumoniae urinary antigen test in cerebrospinal fluid for rapid diagnosis of pneumococcal meningitis. Diagn Microbiol Infect Dis. 2003;45(4):237–40. 10.1016/s0732-8893(02)00548-5 . [DOI] [PubMed] [Google Scholar]
13.Adegbola RA, Obaro SK, Biney E, Greenwood BM. Evaluation of Binax NOW Streptococcus pneumoniae urinary antigen test in children in a community with a high carriage rate of pneumococcus. Pediatr Infect Dis J. 2001;20(7):718–9. 10.1097/00006454-200107000-00018 . [DOI] [PubMed] [Google Scholar]
14.Hamer DH, Egas J, Estrella B, MacLeod WB, Griffiths JK, Sempertegui F. Assessment of the Binax NOW Streptococcus pneumoniae urinary antigen test in children with nasopharyngeal pneumococcal carriage. Clin Infect Dis. 2002;34(7):1025–8. 10.1086/339446 . [DOI] [PubMed] [Google Scholar]
15.Gutierrez F, Masia M, Rodriguez JC, Ayelo A, Soldan B, Cebrian L, et al. Evaluation of the immunochromatographic Binax NOW assay for detection of Streptococcus pneumoniae urinary antigen in a prospective study of community-acquired pneumonia in Spain. Clin Infect Dis. 2003;36(3):286–92. 10.1086/345852 . [DOI] [PubMed] [Google Scholar]
16.Edelstein PH. Urinary antigen detection for Legionella spp In: Plaeger SF, Denny TN, editors. Clinical Microbiology Procedures Handbook. 3 ed Washington, DC: ASM Press; 2010. p. 1–7. [Google Scholar]
17.Badoux P, Kracht-Kosten L, Herpers B, Euser S. Method comparison of the ImmuView L. pneumophila and L. longbeachae Urinary Antigen test with BinaxNOW Legionella Urinary Antigen Card for the detection of L. pneumophila serogroup 1 antigen in urine. J Clin Microbiol. 2019. 10.1128/JCM.01429-19 . [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Schmidt RL, Factor RE. Understanding sources of bias in diagnostic accuracy studies. Arch Pathol Lab Med. 2013;137(4):558–65. 10.5858/arpa.2012-0198-RA . [DOI] [PubMed] [Google Scholar]
19.Rigby EW, Plouffe JF, Hackman BA, Hill DS, Benson RF, Breiman RF. Stability of Legionella urinary antigens over time. Diagn Microbiol Infect Dis. 1997;28(1):1–3. 10.1016/s0732-8893(97)89153-5 . [DOI] [PubMed] [Google Scholar]

PLoS One. doi: 10.1371/journal.pone.0238479.r001

Decision Letter 0

Ray Borrow

24 Jun 2020

PONE-D-20-15595

Performance of the ImmuView and BinaxNOW assays for the detection of urinary and cerebrospinal fluid Streptococcus pneumoniae and Legionella pneumophila serogroup 1 antigen in Legionnaires’ disease or pneumococcal pneumonia and meningitis

PLOS ONE

Dear Dr. Edelstein,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised below during the review process.

Please submit your revised manuscript by Aug 08 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Ray Borrow, Ph.D., FRCPath

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please provide additional details regarding participant consent.

In the ethics statement in the Methods and online submission information, please ensure that you have specified (a) whether consent was informed;

(b) what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed). If the need for consent was waived by the ethics committee, please include this information;

3. Thank you for stating the following in the Financial Disclosure section:

'PHE received funding from SSI Diagnostica to conduct this study

CSJ received funding from SSI Diagnostica to conduct this study

LAW received funding from SSI Diagnostica to conduct this study

SSI Diagnostica: https://www.ssidiagnostica.com/

The study sponsor, SSI Diagnostica, wrote the test protocols, paid for the study reagents and supplies as well as for the performing technologist’s time. This report was written independently of SSI Diagnostica using data spreadsheets supplied by the sponsor as well as separately kept spreadsheets for the UPenn site. SSI Diagnostica reviewed the manuscript for accuracy of the reported data but otherwise had no role in its writing.'

We note that you received funding from a commercial source: SSI Diagnostica

a. Please provide an amended Competing Interests Statement that explicitly states this commercial funder, along with any other relevant declarations relating to employment, consultancy, patents, products in development, marketed products, etc.

Within this Competing Interests Statement, please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests). If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

b. Please include your amended Competing Interests Statement within your cover letter. We will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

4. We note that you have included the phrase “data not shown” in your manuscript. Unfortunately, this does not meet our data sharing requirements. PLOS does not permit references to inaccessible data. We require that authors provide all relevant data within the paper, Supporting Information files, or in an acceptable, public repository. Please add a citation to support this phrase or upload the data that corresponds with these findings to a stable repository (such as Figshare or Dryad) and provide and URLs, DOIs, or accession numbers that may be used to access these data. Or, if the data are not a core part of the research being presented in your study, we ask that you remove the phrase that refers to these data.

5. Please amend either the title on the online submission form (via Edit Submission) or the title in the manuscript so that they are identical.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Overall a good paper describing the results of a multisite study comparing the ImmunView and BinaxNOW urine antigen and CSF tests for Streptococcus pneumoniae and Legionella pneumophila. I would like to have seen more samples from patients with bacteraemic pneumococcal pneumonia tested at the UPenn site as I feel that the lack of these samples makes site comparisons for this particular test group difficult. I also have some minor points for you to consider.

1. Line 47- I don’t think it is necessary to describe the disease as “pneumonia caused by either pneumococcal pneumonia”. I would suggest something like “pneumococci” instead of “pneumococcal pneumonia” or alternatively remove the "caused by " and just list pneumococcal pneumonia.

2. Line 59- Please could you list the criteria that was used to define the children with lower respiratory infection not due to pneumococcal pneumonia. As cases of co-infections with pneumococci and other bacteria or viruses have been reported, was this considered when selecting these samples?

3. Line 172-174- The footnote for Table 1, specifically references e and f, are used to describe the total "urines" tested. However, the data in the first column of the table are supposed to represent CSF samples. Please can you clarify.

4. Line 265 - Please can you clarify as to what is meant by the ImmunView L. pneumophila test. I am assuming that it means that some of the samples reported a result for L. pneumophila when tested with the ImmunView test.

5. Line 283-286 -Please can you explain why priority was given to the ImmunView test over the BinaxNow when conducting repeat testing with insufficient sample volume.

6. Line 381-383 - Have you considered the sensitivity of the assays in regard to the different pneumococcal serotypes?

7. Table 4 - Could this table also include the figures for the number of samples reported positive and negative out of the total tested that was used to calculate the sensitivity and specificity?

Reviewer #2: In the present study, Edelstein et al compared the performances of ImmuView S. pneumoniae and L. pneumophila urinary antigen tests with that of the BinaxNOW S. pneumoniae and L. pneumophila assays using frozen samples and performing the testing at three laboratories. The study sponsor SSI Diagnostica wrote the test protocols and financed the study, but the report was written independently of them. The report adds information regarding the ImmuView test performance in comparison with the BinaxNOW assays on clinical samples.

My major concern is that samples are referred to as collected from pneumococcal pneumonia and Legionnaires’ disease patients but data are lacking on collection procedure and patient characteristics. When and where did the authors collect samples, and when were they analyzed? If the 55 Legionella serogroup 1 samples tested at SSI are the same 55 samples analyzed by Jorgensen et al (2015) previously this should be presented. Patient characteristics from those with suspected Legionnaires’ disease would help to interpret discordant results, especially since more than half of Legionnaires’ disease cases from the UPenn site are defined only by previous positive antigen tests (see below). It would be suitable to refrain from using the term “clinical” for defining test performance to avoid misunderstandings concerning the trial design.

Another concern is the lack of criteria for pneumonia diagnosis. Usually, a majority of patients with pneumococcal bacteremia have airway infections but, sometimes, may have meningitis, otitis or other foci only. If not described, maybe the authors should refer to patients with bacteremia and not bacteremic pneumonia. Furthermore, are these urine samples collected consecutively during certain time periods, or are they selected for some reason due to special features, e.g. if a positive pneumococcal antigen test was positive in the clinical routine along with positive blood culture? Could the selection of samples explain the high sensitivity yields for pneumococcal antigen testing for both tests in the study, and for site specific differences?

Line 115: How was serogroup 1 identified?

Line 118: How was pneumonia defined?

Line 155: Bacteremic pneumococcal pneumonia. Since the definition of Legionnaires’ disease and pneumococcal pneumonia is not clear, and based on positive microbiology results only, it should be referred to possible diagnoses.

Line 157: The sentence refers to Table 1 only.

Line 178: Supportive laboratory findings for patients with suspected Legionnaires disease?

Line 199: Not clinical performance, since the definitions of diagnoses have not yet been clarified.

Line 210: Not clinical specificity, since the definitions of diagnoses have not yet been clarified.

Line 261: Not clinical sensitivity, half the samples are spiked.

Line 269: Patient characteristics? Age? Previous urinary antigen test performed? Was urine sample and nasopharynx sample collected simultaneously, or colonization “at any time”?

Line 340: Is it 54% (60/111)? Please refer to Table 2 If appropriate.

Line 352: Due to absent test concordance…? Words missing.

Line 404: Little or no information of samples tested for specificity is presented in the study. Thus, if the samples are from healthy individuals, the results only reflects specificity on such a healthy population without respiratory symptoms. It would be of interest to study the test performances on patients with non-Legionella non-pneumococcal lower airway infections or bacteremia.

Line 408: Here, it is stated that the use of the term clinical sensitivity is not appropriate in the study, I agree.

Lines 388-391: If the selection of urine samples are based on positive Legionella urinary antigen tests only, with no clinical data on pneumonia, samples may be true-negative from the beginning, but false-positive before boiling. This is a major issue of the study, see above. Besides the discrepant specimens, could there be more positive tests that would have turned negative after boiling?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2020 Aug 31;15(8):e0238479. doi: 10.1371/journal.pone.0238479.r002

Author response to Decision Letter 0

31 Jul 2020

please see the rebuttal letter for these responses

Attachment

Submitted filename: response to reviewers.docx

Click here for additional data file.^{(25.2KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0238479.r003

Decision Letter 1

Ray Borrow

13 Aug 2020

PONE-D-20-15595R1

Performance of the ImmuView and BinaxNOW assays for the detection of urine and cerebrospinal fluid Streptococcus pneumoniae and Legionella pneumophila serogroup 1 antigen in Legionnaires’ disease or pneumococcal pneumonia and meningitis

PLOS ONE

Dear Dr. Edelstein,

Please submit your revised manuscript by Sep 27 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

We look forward to receiving your revised manuscript.

Kind regards,

Ray Borrow, Ph.D., FRCPath

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #1: Yes

**********

6. Review Comments to the Author

Reviewer #1: Thank you for submitting the revised manuscript. I am satisfied with the authors response to the reviewers’ comments and believe that the manuscript should be accepted. I would however like to suggest a couple of minor changes for the authors to consider.

1. In lines 304-305 you state “Testing of urines from children with lower respiratory tract infection, none of whom had pneumococcal pneumonia, was performed to determine the specificity of urine antigen testing...”

A number of publications have highlighted that the diagnosis of pneumonia caused by pneumococci can be insensitive when based solely on culture. In fact, this is a reason in favour of the use of urine antigen diagnostic kits such as the BinaxNOW and ImmuView. For this reason, I believe that it is incorrect to state that these children did not have pneumococcal pneumonia and instead suggest that you re-word this sentence to note that the children were deemed not to have pneumococcal pneumonia based on the absence of any culture.

2. Lines 327-328 – “For the three BinaxNOW-ImmuView discordant result UPenn L pneumophila results for urines from patients with Legionnaires' disease…” – perhaps consider removing the first "result" from this sentence for clarification.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

PLoS One. 2020 Aug 31;15(8):e0238479. doi: 10.1371/journal.pone.0238479.r004

Author response to Decision Letter 1

13 Aug 2020

please see the cover letter and response to reviewer file

Attachment

Submitted filename: response to reviewer.docx

Click here for additional data file.^{(18.2KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0238479.r005

Decision Letter 2

Ray Borrow

18 Aug 2020

PONE-D-20-15595R2

Dear Dr. Edelstein,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Ray Borrow, Ph.D., FRCPath

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

PLoS One. doi: 10.1371/journal.pone.0238479.r006

Acceptance letter

Ray Borrow

20 Aug 2020

PONE-D-20-15595R2

Performance of the ImmuView and BinaxNOW assays for the detection of urine and cerebrospinal fluid Streptococcus pneumoniae and Legionella pneumophila serogroup 1 antigen in patients with Legionnaires’ disease or pneumococcal pneumonia and meningitis

Dear Dr. Edelstein:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Prof. Ray Borrow

Academic Editor

PLOS ONE

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

S1 Table. Clinical sensitivity and specificity for S. pneumoniae antigenuria by study site.

(PDF)

Click here for additional data file.^{(46.6KB, pdf)}

S2 Table. Agreement of S. pneumoniae antigenuria testing, UPenn.

(PDF)

Click here for additional data file.^{(41.9KB, pdf)}

S3 Table. Agreement of S. pneumoniae antigenuria Testing, SSI.

(PDF)

Click here for additional data file.^{(40.8KB, pdf)}

S4 Table. Agreement of L. pneumophila antigenuria testing Lp SG1 only, UPenn.

(PDF)

Click here for additional data file.^{(43.7KB, pdf)}

S5 Table. Agreement of L. pneumophila antigenuria testing, SSI.

(PDF)

Click here for additional data file.^{(41.2KB, pdf)}

S6 Table. Clinical sensitivity and specificity for L. pneumophila antigenuria by study site.

(PDF)

Click here for additional data file.^{(51.2KB, pdf)}

S7 Table. Correlation of BinaxNOW and ImmuView L. pneumophila results and monoclonal group for concordant and discordant results, SSI.

(PDF)

Click here for additional data file.^{(43.3KB, pdf)}

S8 Table. Correlation of BinaxNOW L. pneumophila falsely-negative tests with monoclonal group, SSI site.

(PDF)

Click here for additional data file.^{(45.5KB, pdf)}

(PDF)

Click here for additional data file.^{(46.2KB, pdf)}

SSI and UPenn sites combined.

(PDF)

Click here for additional data file.^{(45.8KB, pdf)}

SSI and UPenn combined data.

(PDF)

Click here for additional data file.^{(49.3KB, pdf)}

S12 Table. S. pneumoniae CSF agreements.

(PDF)

Click here for additional data file.^{(40.3KB, pdf)}

S13 Table. Positive and negative agreement for CSF specimens.

(PDF)

Click here for additional data file.^{(44.6KB, pdf)}

S14 Table. S. pneumoniae pediatric urines agreements.

(PDF)

Click here for additional data file.^{(41.9KB, pdf)}

S15 Table. L. pneumophila pediatric urines agreements.

(PDF)

Click here for additional data file.^{(40.3KB, pdf)}

S16 Table. Clinical specificity for pediatric urines.

(PDF)

Click here for additional data file.^{(44KB, pdf)}

S17 Table. Effects of retesting urine after boiling for ImmuView-BinaxNOW discordant result specimens for Upenn and SSI sites.

(PDF)

Click here for additional data file.^{(46.8KB, pdf)}

Attachment

Submitted filename: response to reviewers.docx

Click here for additional data file.^{(25.2KB, docx)}

Attachment

Submitted filename: response to reviewer.docx

Click here for additional data file.^{(18.2KB, docx)}

Data Availability Statement

All relevant data are within the manuscript and its Supporting Information files.

[pone.0238479.ref001] 1.Said MA, Johnson HL, Nonyane BA, Deloria-Knoll M, O'Brien KL, Team AAPBS, et al. Estimating the burden of pneumococcal pneumonia among adults: a systematic review and meta-analysis of diagnostic techniques. PLoS One. 2013;8(4):e60273 10.1371/journal.pone.0060273 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0238479.ref002] 2.Centers for Disease Control. Legionnaires' disease surveillance summary report, United States 2014–20152019 2/28/2020. Available from: https://www.cdc.gov/legionella/health-depts/surv-reporting/2014-15-surv-report-508.pdf.

[pone.0238479.ref003] 3.Helbig JH, Uldum SA, Bernander S, Luck PC, Wewalka G, Abraham B, et al. Clinical utility of urinary antigen detection for diagnosis of community-acquired, travel-associated, and nosocomial legionnaires' disease. J Clin Microbiol. 2003;41(2):838–40. 10.1128/jcm.41.2.838-840.2003 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0238479.ref004] 4.Moïsi JC, Saha SK, Falade AG, Njanpop-Lafourcade BM, Oundo J, Zaidi AK, et al. Enhanced diagnosis of pneumococcal meningitis with use of the Binax NOW immunochromatographic test of Streptococcus pneumoniae antigen: a multisite study. Clin Infect Dis. 2009;48 Suppl 2:S49–56. 10.1086/596481 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0238479.ref005] 5.Jorgensen CS, Uldum SA, Elverdal PL. Application of a lateral flow test as an additional serological tool for diagnosis of Legionella infections. J Microbiol Methods. 2014;96:12–5. 10.1016/j.mimet.2013.10.021 [DOI] [PubMed] [Google Scholar]

[pone.0238479.ref006] 6.GraphPad QuickCalcs [cited 2020]. Available from: https://www.graphpad.com/quickcalcs/.

[pone.0238479.ref007] 7.U.S. FDA Center for Devices and Radiologic Health. Statistical guidance on reporting results from studies evaluating diagnostic tests. Rockville, MD2007. p. 1–39.

[pone.0238479.ref008] 8.Agresti A, Coull BA. Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions. Am Stat. 1998;52(2):119–26. 10.1080/00031305.1998.10480550 [DOI] [Google Scholar]

[pone.0238479.ref009] 9.Sathapatayavongs B, Kohler RB, Wheat LJ, White A, Winn WC Jr., Girod JC, et al. Rapid diagnosis of Legionnaires' disease by urinary antigen detection. Comparison of ELISA and radioimmunoassay. Am J Med. 1982;72:576–82. 10.1016/0002-9343(82)90451-x . [DOI] [PubMed] [Google Scholar]

[pone.0238479.ref010] 10.Jørgensen CS, Uldum SA, Sørensen JF, Skovsted IC, Otte S, Elverdal PL. Evaluation of a new lateral flow test for detection of Streptococcus pneumoniae and Legionella pneumophila urinary antigen. J Microbiol Methods. 2015;116:33–6. 10.1016/j.mimet.2015.06.014 . [DOI] [PubMed] [Google Scholar]

[pone.0238479.ref011] 11.Athlin S, Iversen A, Özenci V. Comparison of the ImmuView and the BinaxNOW antigen tests in detection of Streptococcus pneumoniae and Legionella pneumophila in urine. Eur J Clin Microbiol Infect Dis. 2017;36(10):1933–8. 10.1007/s10096-017-3016-6 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0238479.ref012] 12.Samra Z, Shmuely H, Nahum E, Paghis D, Ben-Ari J. Use of the NOW Streptococcus pneumoniae urinary antigen test in cerebrospinal fluid for rapid diagnosis of pneumococcal meningitis. Diagn Microbiol Infect Dis. 2003;45(4):237–40. 10.1016/s0732-8893(02)00548-5 . [DOI] [PubMed] [Google Scholar]

[pone.0238479.ref013] 13.Adegbola RA, Obaro SK, Biney E, Greenwood BM. Evaluation of Binax NOW Streptococcus pneumoniae urinary antigen test in children in a community with a high carriage rate of pneumococcus. Pediatr Infect Dis J. 2001;20(7):718–9. 10.1097/00006454-200107000-00018 . [DOI] [PubMed] [Google Scholar]

[pone.0238479.ref014] 14.Hamer DH, Egas J, Estrella B, MacLeod WB, Griffiths JK, Sempertegui F. Assessment of the Binax NOW Streptococcus pneumoniae urinary antigen test in children with nasopharyngeal pneumococcal carriage. Clin Infect Dis. 2002;34(7):1025–8. 10.1086/339446 . [DOI] [PubMed] [Google Scholar]

[pone.0238479.ref015] 15.Gutierrez F, Masia M, Rodriguez JC, Ayelo A, Soldan B, Cebrian L, et al. Evaluation of the immunochromatographic Binax NOW assay for detection of Streptococcus pneumoniae urinary antigen in a prospective study of community-acquired pneumonia in Spain. Clin Infect Dis. 2003;36(3):286–92. 10.1086/345852 . [DOI] [PubMed] [Google Scholar]

[pone.0238479.ref016] 16.Edelstein PH. Urinary antigen detection for Legionella spp In: Plaeger SF, Denny TN, editors. Clinical Microbiology Procedures Handbook. 3 ed Washington, DC: ASM Press; 2010. p. 1–7. [Google Scholar]

[pone.0238479.ref017] 17.Badoux P, Kracht-Kosten L, Herpers B, Euser S. Method comparison of the ImmuView L. pneumophila and L. longbeachae Urinary Antigen test with BinaxNOW Legionella Urinary Antigen Card for the detection of L. pneumophila serogroup 1 antigen in urine. J Clin Microbiol. 2019. 10.1128/JCM.01429-19 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0238479.ref018] 18.Schmidt RL, Factor RE. Understanding sources of bias in diagnostic accuracy studies. Arch Pathol Lab Med. 2013;137(4):558–65. 10.5858/arpa.2012-0198-RA . [DOI] [PubMed] [Google Scholar]

[pone.0238479.ref019] 19.Rigby EW, Plouffe JF, Hackman BA, Hill DS, Benson RF, Breiman RF. Stability of Legionella urinary antigens over time. Diagn Microbiol Infect Dis. 1997;28(1):1–3. 10.1016/s0732-8893(97)89153-5 . [DOI] [PubMed] [Google Scholar]

PERMALINK

Performance of the ImmuView and BinaxNOW assays for the detection of urine and cerebrospinal fluid Streptococcus pneumoniae and Legionella pneumophila serogroup 1 antigen in patients with Legionnaires’ disease or pneumococcal pneumonia and meningitis

Paul H Edelstein

Charlotte Sværke Jørgensen

Leslie A Wolf

Roles

Abstract

Introduction

Materials and methods

Table 1. Specimens tested at each testing site.

Table 2. Laboratory findings for patients with Legionnaires’ disease.

Results

Table 3. S. pneumoniae specimens test agreements for urines from adults.

Table 4. Clinical sensitivity and specificity for urines from adults.

Table 5. L. pneumophila test agreements for urines from adults.

Discussion

Supporting information

Acknowledgments

Data Availability

Funding Statement

References

Decision Letter 0

Ray Borrow

Roles

Author response to Decision Letter 0

Decision Letter 1

Ray Borrow

Roles

Author response to Decision Letter 1

Decision Letter 2

Ray Borrow

Roles

Acceptance letter

Ray Borrow

Roles

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases