Table 4.
Responses from trialists who had collected data on a prespecified outcome but not analysed them by the time of the primary publication (n=17)
| Trialist | Explanation from trialist | Category |
|---|---|---|
| 05 | “I think, you know, the issue with cost is it would have been relatively easy to get hospital charges, but actual costs are more complicated analysis, and given that there was no difference between the groups we didn’t go onto that.” | Secondary data not analysed because no difference in primary outcome |
| 08 | “Part of the problem is, as our economists keep on telling us, we have been collecting economic data, but we haven’t had enough of them. So for instance, for someone going into treatment, you know, we still don’t have a lot of participants who are receiving treatment. So, one thing that we now have is older participant rates, so the economists are just starting to work on that, saying: ‘OK, now that we have older rates, we can project out what potential benefits in terms of like earnings over a life course, we might expect.’” | Long term outcome data not obtained by the time primary outcome data published |
| 15 | “Unfortunately, the year we were funded the funders levied large across the board cuts to all approved projects. Our recruitment also took longer than expected. As a result, we had neither the time nor the resources to analyse for effects on the several less important secondary or exploratory outcomes we mentioned in our initial proposal.” | Insufficient time, resources, or both to analyse less important outcomes |
| 20 | “The problem was that they weren’t completed as well as all the other measures, it was only just over two thirds of people who consistently completed their diary, so it would have reduced our numbers considerably.” | Not analysed because of the amount of missing data |
| 23 | “It was missing, a lot of people were missing data, and because we were looking at pre- and post- if they were missing at either point then we had to throw out that person, so we just didn’t have enough, and then across three different conditions, so if you have only six or seven people with data in each condition it really wasn’t worth looking at.” “Oh I know what it was; I do know what it was. Well I know why it wouldn’t have been in the dissertation. I ended up getting pregnant, and so I actually defended the dissertation early, so I defended the dissertation before all of the six months data were collected, even though I said I would do all of it, and I did, but I wanted to get the defence done before I went on maternity leave, so they allowed me to defend just based on the pre-post data, and we didn’t collect the health services data until six months, so I didn’t have those data at the time that I defended. Now I did include the follow-up data in this published article but I probably yes, I didn’t go back and probably didn’t even code and clean all the data we collected on health services.” |
Not analysed because of the amount of missing data Long term outcome data not obtained by the time primary outcome data published |
| 27 | “Samples were taken and one could argue that this is unethical actually because those samples were taken, and stored, and frozen, and the results of analysis have never been published because the analysis was never done. Speaking as someone who is interested in trial ethics, that is probably unethical to take samples and not analyse them. Here is the answer to that question then, it’s really simple issue, which is the reality of doing a trial. This was an utter nightmare. For the analysis we did all the preparation abroad, it was an utter nightmare. I wouldn’t say I still wake up screaming. For a small aspect of a trial, you know, some of the more interesting outcomes and laboratory based things to support whatever the hypothesis is, it grew to assume gargantuan proportions because there was no electricity, it was 40 degrees, there were bugs crawling around everywhere, and this turned into a nightmare. Getting it back to . . . there were no facilities to do these in the country and getting it back to us was a nightmare. And, the reason for the reporting differences for this biochemical stuff is actually I suspect not to do with bias but to do with pragmatics.” | Not analysed because of practical difficulties |
| 31 | “The main reason was limited power, so we had fewer variables up front, because of small sample size. Most of it was driven by the small sample size, you know when you have 54 participants and you are trying to look at outcomes, so we made decisions about what seemed to be the most important variable. I think I tend to be conservative and part of this is just the sample was so small. I have tried to be very focussed in the papers and there is lots of data that is lying dormant essentially because of that.” | Limited analysis undertaken owing to poor recruitment |
| 39 | “No, we haven’t analysed it yet, you know we saw no effect on [primary outcome], which means that the drug is not ever going to get, well not in the foreseeable future, going to get on the market. So I think it’s less important, people aren’t going to have access to it, it’s less important to share, to make that data public given the low likelihood that it’s going to show anything. It’s not going to have any immediate clinical implications for anybody.” | Secondary data not analysed because no difference in primary outcome |
| 40 | “Well, we will be doing it, so what happens is you send someone down to a laboratory, they do the various procedures and then you have to have someone who scores the protocol, which requires various software and code writing, and then someone to spend the time to actually analyse the data. And, it just, it takes a while and so our primary interest was getting out the information, these other measures were more secondary outcome measures. It gets into issues around priorities, staff leaving, new staff having to be trained, it has just taken us a while to get this stuff analysed and as I said it’s, it has been analysed and we are hoping to write the manuscript for this, this fall.” | Delay in obtaining the data |
| 41 | “The cost effectiveness was really not conducted in this study. I am not an expert on cost effectiveness, there is a different team that works with us on cost effectiveness analysis, and this is their own survey now. And also the size, the effect sizes to be expected for the cost effectiveness analysis are much smaller and therefore the sample size did not really afford this type of analysis.” “There is also a limit of what number of statistical analyses you can actually reliably do with a small sample size, so considering the sample size of this particular study, we decided to report only on the really main primary selected outcomes and not run many analysis, because we understand that they lose the power, and they lose the power of the conclusion.” |
Not analysed because of poor recruitment |
| 42 | “Well that, it turns out that our funders [industry] kind of went through some internal reorganisation, and as a result there are delays in the analyses of this outcome, and so we are still waiting to receive the final data from that, to publish it. We have got partial data completed on that part of the project but not complete. The data, it’s really held by us. The funders have the samples, they just have that one set of samples, and they send us the data and we own the data and do the analysis, they just did the assay. We are interpreting it and analysing it and everything.” | Delay in obtaining the data |
| 43 | “The economic evaluation, it’s in processes. Well, we were supposed to have it analysed by the end of the month, I can’t give you any preliminary. We know about the utilisation already but I don’t know about the related costs. Our goal is to have it actually submitted this year.” | Long term outcome data not obtained by the time primary outcome data published |
| 44 | “So it was a negative trial, it was not a failed trial, we have statistical power, but it was negative basically, we showed that in the context of our environment the intervention did not increase the primary outcome so basically the outcomes were identical in the two groups. Given this there was absolutely no reason to expect the difference in anything we would measure in the blood. So, we had collected all that, it was sitting in the refrigerator, in the freezer, so we didn’t do the analysis because it was completely obvious that it would be negative.” | Samples not analysed because no difference in primary outcome |
| 45 | “That outcome is a paper still in the making. That data was kept completely aside, you know, as you can see the paper that we are talking about has got so much in it and we did hum and ah whether we were going to break it down or present it separately, or any of that sort of stuff, but when we did come down to it, this outcome was just going to be too much again for that paper and so we chose to leave it and it’s still a paper in the making, it’s still data which is sitting there and hopefully will be a standalone paper in itself. I haven’t got round to analysing it, I have gone out of a PhD and jumped straight into something else, I have still got data and all sorts of stuff that I haven’t even have time to enter. I get the impression from talking to others as well that it’s always common when you collect a lot of data to have one or two variables that never make it to the cutting board. So this outcome is not in my thesis. I would like to think that it will reach publication one day. I think at the moment I am very, I am very rushed with all the new things that I have got on my plate, but there is a number of variables that I breeze back on and think to myself ‘oh I will do something about that one day, I will do something about that one day.’ So yes I would like to think that it wasn’t, it wasn’t a wasted time in collecting it.” | Outcome not analysed because volume of data presented in primary publication |
| 53 | “Right, we did that and just because it takes absolutely for ever to score it, and although I am a trained scorer as a primary investigator I can’t do the scoring, so I had to send one of my team members to get trained and it just takes, it is taking forever to get through them all, so it’s still in the pipeline. I would have had to postpone the writing of the primary outcome paper if I had to wait for this data. Yes, we have almost just about done now and so in a separate publication we will report that.” | Delay in obtaining the data |
| 55 | “Doing the measurements is really hard and I wasn’t sort of specifically trained to do them. I mean the clinicians told me how to do them, but it can be quite difficult to do. Which is a shame because that is a nice sort of, a nice measurement to have. We did it the whole way through but kind of knowing that it was pointless. I think, because we ended up doing it for the trial participants rather than for us, because they expected it to be done. So we just did it and noted it down but never analysed it, because we didn’t believe in it, even if it had shown something brilliant, we wouldn’t have thought it was true.” | Not analysed because of practical difficulties and uncertainty about validity of data |
| 59 | “I think it had just fallen off our radar screen and we were focused on the primary outcome, because the story would have been ‘you reduce the outcome, and a higher proportion of people remain independent.’ In fact, we sort of then began telling another story. It didn’t seem important once we had already frightened ourselves by showing that we caused harm and I suppose the story could have been ‘you cause harm and you cause more people to have to go into care.’ I think it just fell off the radar screen because we then began to worry about why, why this didn’t work.” | Not analysed because harmful effect of intervention on primary outcome |