Letter to the editor regarding “Clinical effectiveness and safety of powered exoskeleton-assisted walking in patients with spinal cord injury: systematic review with meta-analysis”

Dear editor

In the article “Clinical effectiveness and safety of powered exoskeleton-assisted walking in patients with spinal cord injury: systematic review with meta-analysis”, published in the March issue of Medical Devices: Evidence and Research, Miller et al1 present a meta-analysis of the clinical effectiveness and safety of powered exoskeletons for spinal cord injury (SCI) patients. A close examination of this article shows surprising coincidences, in that two primary studies (references 25 and 33 in the reference list) report the same proportions and 95% confidence intervals (CIs) of subjects able to ambulate with an exoskeleton without assistance (Figure 2 of the study), and two different primary studies (references 26 and 28) report the same mean and 95% CIs for the distance (in meters) walked in a 6-minute walk test (Figure 4 of the study).

A likely explanation is that a single group of authors described the same patients in different publications. In fact, nine of the 14 studies included in the meta-analysis by Miller et al1 can be assigned to three groups of studies that may contain duplicate patient information:

• New York City: references 23, 27, 32, and 34

• Philadelphia: references 25, 33, and 35

• Atlanta: references 26 and 27

Of course, these groups of researchers could have reported on a new case series in each publication, but the reported identical values for proportions and means would be surprising. Authors of literature on exoskeleton-assisted walking often list clinical and demographic information of individual subjects in a tabular format. Based on our estimates of the number of likely categories or values for gender, age, height, weight, years since injury, and level and completeness of SCI, the likelihood that Subject 1 of reference 32 is not the same person as Subject 2 of reference 27 is about 0.000015. Furthermore, sometimes, multiple individuals in two series match (eg, six of the seven subjects in reference 32 can be matched with all six subjects of reference 27). Therefore, it is clear that the chances that these studies contain information on unique individuals are infinitesimally small. Miller et al1 should have noted the overlap in author names, investigated the uniqueness of each case series, and followed up with the investigators.

The implications of including duplicate subjects in a meta-analysis are serious; the independence of observations is violated (contravening a key assumption of inferential statistics), the CIs for characteristics of the pooled subjects are too narrow, and the heterogeneity between studies is likely to be too small.

It is not uncommon for the same study to be published twice, with the same or different primary authors. Sometimes, for a case series that is reported more than once, new subjects are added to the series, but the outcomes reported are the same^2; in other instances, somewhat different outcomes are reported for the same or largely overlapping subjects, as is the case here with the New York City studies. If the primary authors do not explicitly report individual subject characteristics, as is true for most large case series and randomized controlled trials, it is difficult for casual readers, or even systematic reviewers, to determine the degree of subject overlap. The systematic reviewer should maintain a high level of suspicion, follow up on his or her leads, and report appropriately. According to the Cochrane Handbook, “It can be difficult to detect duplicate publication, and some ‘detective work’ by the review authors may be required.”3 The Handbook suggests that the most useful pieces of information for comparing reports are author names, specific details of the interventions, numbers of participants and baseline data, and the date and duration of the study, concluding that “Where uncertainties remain after considering these and other factors, it may be necessary to correspond with the authors of the reports.”3

We strongly recommend that Miller et al1 address our concerns and correct their report so as to remove erroneous information from the scientific literature.