Table 2.
Areas of COS development where developers identified that guidance or research was needed
| Area of COS development | Illustrative quotes |
|---|---|
| Systematic review of outcomes | |
| Domain categorizationa | The domain categorisation I think is something which needs to be highlighted. I think you need to be really careful about how you do it. It is ultimately quite a subjective process, and I think you need to have methods in place that mean that, you have appropriate categorisation […] if you don't get that stage right then your Delphi just kind of doesn't matter. (023) |
| Delphi | |
| Response rates | The biggest challenge that we faced was trying to get enough responses to the online survey. Despite [name of charity organisation] help, we struggled to get as many responses as we would have liked and we went to, we looked at the variety of ways around that including advertisements in the press and other things. (O24) |
| Feedback | Within Delphi, I don't think anyone understands the best type of feedback to give people, the impact of what that feedback, you know of how that feedback affects responses. […] Do you combine different stakeholders, different results at different stages, do you merge it all at the end? Yeah these are all sort of mysteries to be honest with you. I could go on and on. (O23) |
| Balancing stakeholder groupsb | So there's a large imbalance in the two stakeholder groups at the moment, we've got about 110 patients for the patient group and we've got only 50 for the clinician group. (O19) |
| Retention of outcomes between rounds | The issues around whether you discard or retain outcomes or whether you keep the whole lot in until the end of the Delphi process, I don't think that's been resolved at all. (O23) |
| Sampling | How do you know, and how can you demonstrate if you do know, that the people you've chosen are the right people? (O28) |
| Sample size | I certainly think in terms of guidance, I think numbers of people for a Delphi, some information about that would be useful 'cause there really was nothing about that at all. (O29) |
| Bias | How do you motivate people to answer? If you're motivating people to answer them, is that not biased? (O32) |
| Consensus meeting | |
| Conduct and management | Over the, I guess, 20 years or so that I've been involved in chairing discussion meetings, I've learnt on the job, and I don't think that's necessarily the best way […] it's good to be able to benefit from other people's experience. But I didn't really come across guidance on how to run discussion meetings that I found useful. (P14) |
| Organization and efficiency | Guidance should include how best to organise a workshop like this to try and ensure that you use the expertise and time of the individuals actually face to face as efficiently as possible. (P14) |
| Objectives | The functionality of the consensus meeting is very, very vague. In the, in the literature, there is absolutely no consensus there about […] what are the objectives and the aims of having this consensus meeting, so that is something that needs to be looked at. (O26) |
| Reporting | |
| Reporting findings | How to report your findings, 'cause I think that it’s only very few papers that actually highlight, you know, best practice in reporting findings for the consensus. […] And especially when you compare it to previously done, previously published papers, you see that it sort of quite a few of the papers are missing very important points. (O26) |
| Patient participation and involvement | |
| Number of patients and meaningfulness of patient participation | I certainly think in terms of guidance […] number of patients as well, but […] for every clinical context that will probably differ and the amount of involvement that you have from patients will, you know, differ depending on the clinical context. (O29) |
| Resources | It's the same with PPI – patient involvement in research – a lot of groups won't have the resources to do that properly. So I think one has to be careful that um any kind of description of what gold standard is, is also tempered by a, a reality check about what's do-able within small resource envelopes. (O27) |
| Sampling and recruitment | How to identify stakeholders, it's hard to. Well for healthcare professionals […] you can more easily I guess identify people that are working on that field […] for parents it's much harder […] You might use a social media strategy […] but [if] you identify parents that have had [name of disease] at a national level, now does that add much in terms um the methods as compared to a parent workshop that is adequately sampled based on severity of the condition or a level of care? (O20) |
| Diversity | It worries me a lot that the people who are being involved in this way on a sort of patient research partner level, but probably as participants generally, um don't represent that diversity, and so we're missing potentially priorities, you know, a different set of priorities. (P17) |
a
Outcome domains are constructs used to classify similar outcomes or group broad aspects of the effects of interventions (e.g., functional status) into categories.
b
Delphis are used to elicit group opinion and reach consensus, using the rationale that “n heads are better than one” [26]. Opinions of individual participants are surveyed over several rounds and asked to review and rescore items based on feedback on aggregated scores from previous rounds. Different participant subgroups might have different sample sizes.