Abstract
Study Design
Overview of the methods used for a James Lind Alliance (JLA) Priority Setting Partnership (PSP).
Objectives
The objectives of this article are to (i) provide a brief overview of the JLA—facilitated PSP process; (ii) outline how research uncertainties were initially processed in the AO Spine RECODE-DCM PSP; and (iii) delineate the methods for interim prioritization and the priority setting workshop.
Methods
A steering group was created to define the scope for the PSP, organize its activities, and establish protocols for decision-making. A survey was created asking what questions on the diagnosis, treatment, and long-term management of DCM should be answered by future research. Results from the survey were sorted into summary questions. Several databases were searched to identify literature that already answered these summary questions. The final list of summary questions was distributed by survey for interim prioritization. Participants were asked to select the top ten most important summary questions. The questions that were ranked the highest were discussed at an in-person consensus workshop.
Results
The initial survey yielded a total of 3404 potential research questions. Of the in-scope submissions, 988 were related to diagnosis, 1324 to treatment, and 615 to long-term management of DCM. A total of 76 summary questions were developed to reflect the original submissions. Following a second survey, a list of the top 26 interim priorities was generated and discussed at the in-person priority setting workshop.
Conclusions
PSPs enable research priorities to be identified that consider the perspectives and interests of all relevant stakeholders.
Keywords: James Lind Alliance, priority setting partnership, degenerative cervical myelopathy, surveys and questionnaires, research uncertainties
Introduction
Priority setting partnerships (PSPs) are designed to develop research priorities for specific areas of health care in which there are considerable research uncertainties. 1 A research uncertainty is defined as any important question about a specific area of health care that cannot be convincingly answered by the existing body of evidence. 1 Specifically, a research uncertainty exists when there are no up-to-date, reliable systematic reviews, or clinical practice guidelines on a particular topic. PSPs consider the perspectives of clinicians, patients and caregivers. The James Lind Alliance (JLA) process consists of gathering research uncertainties from a wide range of key stakeholders, developing and refining summary questions that reflect these uncertainties and determining the top ten research priorities.
The JLA is a non-profit initiative that facilitates PSPs and ensures that the proposed research priorities reflect the interests of health care professionals, patients, and caregivers.2,3 To date, the JLA has published top ten research priorities for a wide range of medical and psychiatric conditions, including psoriasis, pancreatic cancer, depression, and asthma.4-6 According to a report by Staley et al (2020), the impact of a JLA PSP extends beyond identifying the top ten research priorities. 7 The authors suggest a JLA PSP can also encourage patients to become more involved in future research initiatives, influence clinical practice, and facilitate collaboration across funding organizations and health care departments.
As part of the AO Spine RECODE-DCM (Research Objectives and Common Data Elements for Degenerative Cervical Myelopathy) project, with the assistance of JLA, a PSP was initiated for degenerative cervical myelopathy (DCM) in order to reduce research inefficiencies and to set priorities for future investigation. 8 DCM is a progressive, degenerative spine disease and the most common cause of spinal cord dysfunction in adults worldwide. Based on a study by Smith et al (2020), the prevalence of DCM is approximately 2.3%. 9 Furthermore, as the global population continues to age, clinicians worldwide will be required to assess and manage an increasing number of patients with degenerative spine diseases. 10 Given the potential impact of DCM, it is essential to identify important knowledge gaps in the literature and to prioritize future research.
Although clinical research in DCM has steadily increased over the past 2 decades, numerous questions remain unanswered. 11 Given the challenges of conducting randomized controlled trials in this field, the level of evidence of published research is generally rated as insufficient, low, or moderate. Furthermore, current research largely reflects the academic and clinical interests of the treating physician, which may or may not be relevant to the patient or caregiver. As a result, research priorities must be established for DCM that consider the perspectives and interests of clinicians, patients, and caregivers.
The objectives of this article are to (i) provide a brief overview of the JLA-facilitated PSP process; (ii) outline how research uncertainties were initially processed in the AO Spine RECODE-DCM PSP; and (iii) delineate the methods for interim prioritization and the priority setting workshop.
Part I: An Overview of the James Lind Alliance Process
The steps of a JLA-facilitated PSP are summarized in Figure 1 and include 1 :
1. Create a steering group with equal representation of health care professionals and individuals with lived experience.
2. Gather research uncertainties by distributing a survey to key stakeholders asking what questions they believe should be answered by future research.
3. Summarize the responses gathered from the survey and sort them into overarching themes.
4. Develop clearly formatted summary questions that reflect the original research uncertainties.
5. Check existing evidence to determine whether the summary questions are true research uncertainties or whether they have already been answered.
6. Set interim research priorities by redistributing a survey. Health care professionals and individuals with lived experience are given a list of the summary questions and are asked to vote on the ones they believe are the most important.
7. Develop a list of the top ten research priorities that reflect the perspectives of health care professionals and individuals with lived experience by discussing the highest ranked 25–30 summary questions at a workshop.
8. Publish and promote the top ten research priorities.
Figure 1.
An overview of the priority setting partnership process as outlined by the James Lind Alliance.
The JLA also outlines the principles that PSPs should follow in order to ensure the end product is reliable. 1 These principles include:
• The process must be transparent and inclusive.
• The steering group must have equal representation from health care professionals and individuals with lived experience. Furthermore, the final list of research priorities must reflect the interests and perspectives of all key stakeholder groups.
• Non-clinician researchers must not participate in the voting process.
• Groups and organizations with significant conflicts of interest should not be included in the PSP.
• A detailed audit trail must be completed that outlines the process from submitted uncertainties to the final list of top ten research priorities.
• Priority setting should not be initiated until the summary questions have been formally verified as unanswered.
Part II: A Summary of the Methodology Used in the AO Spine RECODE-DCM Priority Setting Partnership
Table 1 provides an overview of important terminology that will be used throughout this section.
Table 1.
Important Terminology.
| Term | Definition |
|---|---|
| Research uncertainty | •Submitted by health care professionals and individuals with lived experience via survey |
| •Any important question about health care that cannot be answered by the existing body of evidence. Specifically, there are either no up-to-date, reliable systematic reviews or clinical practice guidelines that answer the question, or current evidence indicates that uncertainty exists | |
| •In this paper, the term “research uncertainty” is used interchangeably with survey response, survey submission, evidence uncertainty and in-scope question | |
| Summary question | •Developed by the information specialist with input from the steering committee |
| •An overarching question that summarizes similar submitted research uncertainties | |
| Research priority | •A summary question that is deemed to be one of the most important and should be answered by future research |
| Individuals with lived experience | •An individual either diagnosed with or treated for degenerative cervical myelopathy or a caregiver/supporter |
Create a Steering Group
The steering group of the AO Spine RECODE-DCM PSP consisted of 6 neurosurgeons, 1 orthopedic spine surgeon, 2 neurologists, 1 primary care physician, 3 physical medicine and rehabilitation specialists, and twelve individuals with lived experience. It was the responsibility of this group to define the scope of the PSP, organize its activities, and be accountable for the decisions made in the process. The steering group had full editorial independence from the sponsors of this initiative.
Gather Research Uncertainties
An initial survey was distributed to health care professionals and individuals with lived experience that consisted of the following four questions:
• Detecting DCM—What question(s) about the diagnosis of DCM would you like to see answered by research?
• Managing DCM—What question(s) about the treatment of DCM would you like to see answered by research?
• Living with DCM—What question(s) about the long-term care and follow-up of DCM would you like to see answered by research?
• What other question(s) about DCM that do not fit into the above categories would you like to see answered by research?
These questions were deliberately open-ended to encourage responses that reflect the experiences of a wide range of health care professionals and individuals with lived experience. There was also no limit on the number of research uncertainties that could be submitted by any one individual. The dissemination strategy and sampling results are detailed in a separate article within this special edition (Gathering Global Perspectives to Establish the Research Priorities and Minimum Data Sets for Degenerative Cervical Myelopathy: Sampling Strategy of the First Round Consensus Surveys of AO Spine RECODE-DCM)
Summarize the Survey Responses
Recruit an Information Specialist
An information specialist was specifically recruited to the AO Spine RECODE-DCM PSP to manage the data. The role of the information specialist was to review the initial responses of the survey, organize and categorize the submitted uncertainties into themes, generate summary questions for interim prioritization and check existing literature to verify each summary question is a true research uncertainty. The following sections will explain the steps used in the DCM PSP from analyzing the initial survey results to generating a list of the top ten research priorities.
Organize and Code the Responses from the Initial Survey
A key principle of a JLA-facilitated PSP is that the PSP must retain an audit trail from the original survey responses to the final list of research priorities. In order to do this, the information specialist downloaded an untouched copy of the original survey results. Survey responses, unique participant identifiers and demographic information were then organized into new documents based on the original survey question (i.e., diagnosis, treatment, long-term management and follow-up and other).
Blank responses were removed and survey submissions with multiple parts were divided into individual questions. For example, the following survey response was separated into 2 parts as it addressed different aspects of diagnosis:
“How can we manage general practitioners and other health care providers concerning early signs of DCM? How can we improve the imaging of the earliest signs of myelopathy?”
In some cases, however, submissions with multiple components were not divided if the information specialist believed the parts were related. For example, the following survey response was kept as one question:
“Why are people being misdiagnosed? What is it about the DCM condition that general practitioners are missing? What are the essential symptoms that should spike a general practitioner to consider DCM as a possibility? How can DCM be diagnosed at a much earlier stage?
After initial data processing, there was a total of 3404 research uncertainties that required further sorting. Figure 2 provides an overview of the process from the original 3404 research uncertainties to the top 10 research priorities.
Figure 2.
An overview of the process from initial survey results to the top ten research priorities.
Identify and Remove Out of Scope Questions
The information specialist identified and removed questions that were out of scope. These submissions were discussed with the members of the steering group to ensure that health care professionals as well as individuals with lived experience agreed with the decision to remove these questions. Examples of questions that were considered out of scope included:
1. Questions that did not fit within the defined scope of the project: “In patients who develop central cord syndrome after mild cervical trauma, in the presence of existing DCM, is acute decompression or delayed decompression the best option?”
2. Questions that asked for information or advice. These submissions typically came from individuals with DCM or their caregivers and were often already answered: “Any other treatment which can improve life?”
3. Questions that addressed the quality of or access to health care and social services: “Where can I get help from a doctor who is familiar with DCM as different stages develop?”
4. Questions that were too broad or unclear or comments that did not include a question: “Life after surgery, what is next? Weakness in both upper or lower limbs? How to prevent severe complications?”
A total of 414 original submissions were deemed out of scope and were removed from the data analysis process.
Sort Original Uncertainties Into Categories
A total of 2990 survey submissions were considered in-scope: 988 related to diagnosis, 1324 related to treatment and 615 related to long-term management and follow-up. An additional 63 research uncertainties were submitted that did not fit into one of the three original categories.
All in-scope questions were further sorted into themes based on content. These themes included defining DCM, investigations, timely diagnosis, disease natural history, nonoperative and surgical treatment, and long-term management.
Develop Clearly Formatted Summary Questions
Seventy-six clearly formatted questions were developed to reflect the original uncertainties. Similar submissions were grouped together into summary questions in order to significantly reduce the volume of data. The JLA offers important guidance on creating summary questions: (i) they should reflect the language of the original submissions and not introduce new concepts; (ii) they should not be written like research questions; and (iii) the language should be accessible to individuals with DCM and their caregivers but accurate enough to engage health care providers. Furthermore, it is important that the summary questions are not too specific or too broad as this could either dilute a key theme or make it difficult to interpret.
Supplementary Table 1 provides a list of 76 summary questions and examples of the original uncertainties. Some of these questions contain multiple parts in order to adequately capture the original submissions. The information specialist and the steering committee agreed that some of the original uncertainties were similar enough to be categorized under the same question, but different enough that the question should be multifaceted.
For example, the following summary question consists of 2 parts: “What are the main signs and symptoms that a patient with DCM presents with? What are the frequency, sensitivity, specificity and positive predictive value of symptoms and signs (clinical assessments) for DCM?” This multifaceted question better reflects the original uncertainties than either question does alone: “What are the main symptoms that patients present with? What is the prevalence of each commonly reported symptom; which symptom is most sensitive and specific?”
Present the summary questions to the steering group
Clinicians, patients and caregivers of the steering group must review the summary questions and confirm that they reflect the original submissions. Furthermore, members of the steering group must ensure that the summary questions are clear and unambiguous and that they do not overlap with one another.
The information specialist was responsible for presenting the summary questions to the steering group. Minor disagreements on the wording of the questions were resolved through discussion.
Check Existing Evidence
Existing evidence must be checked to verify whether the summary questions reflect true research uncertainties. According to the JLA, there are typically only a small number of questions that have been fully answered by existing research.
A broad search strategy was developed for each question in order to identify relevant and up-to-date systematic reviews and clinical practice guidelines. Several databases were searched, including MEDLINE, MEDLINE in Process, Cochrane Database of systematic reviews, NICE guidelines and clinicaltrials.gov. If a systematic review or clinical practice guideline were identified, the steering group and information specialist assessed its relevance, quality, and need for updating or extending. Systematic reviews and clinical practice guidelines published in the Cochrane database or NICE were assumed to be reliable and conducted according to methodological standards. If the supporting evidence was identified through MEDLINE or another database, its quality was assessed using standardized tools, including GRADE (Grading of Recommendations, Assessment Development and Evaluation), the Newcastle Ottawa Scale and AMSTAR (A MeasSurement Tool to Assess Systematic Reviews). Members of the steering group were also consulted and asked whether they were aware of any relevant reviews not identified in the search or if they knew of researchers working on the topic area. Supplementary Table 2 summarizes the relevant systematic and narrative reviews and clinical practice guidelines identified for each summary question.
Of the 76 summary questions, the information specialist determined that only two were already answered by existing or ongoing high-quality research. The steering committee agreed with this assessment.
Set Interim Research Priorities
The objective of interim prioritization was to condense a long list of summary questions to a shorter list that could be discussed at the final priority setting workshop. To accomplish this, a second online survey was distributed to a wide group of stakeholders that asked participants to choose the ten most important summary questions from a list of 74. This survey was disseminated through myelopathy.org (an international charity for DCM) and AO Spine (a non-profit organization focusing on spinal research, education, and community development) and via email to other health care professionals and organizations. There were 417 responses in total, of which 310 (74%) were from health care professionals and 107 (26%) were from individuals with lived experience. Each time a question was chosen, it was allocated one point. Separate tallies were maintained for each stakeholder group and were equally weighted when added together. From this process, a list of the top 26 summary questions was created to be presented at the final workshop. The 26 questions selected for the workshop are bolded in Supplementary Table 1.
Develop a List of the Top Ten Research Priorities
The final stage of a PSP is to establish a list of the top ten research priorities. This step is done at an in-person workshop, using a combination of small and whole group discussions. This workshop allows participants to exchange knowledge, clinical expertise and personal experiences in order to develop priorities that reflect the views of all key stakeholders. Furthermore, it facilitates discussions that enable individuals to broaden their minds when considering the management of various health care problems. Throughout the priority setting workshop, the JLA recommends using a Nominal Group Technique which supports the idea that no individual’s views or experiences are more valid than another’s. The process, although rigorous, is also flexible when disagreements arise and revisions need to be made.
The methods used at the priority setting workshop for DCM are summarized below:
1. A total of 25 individuals contributed to the priority setting workshop, including health care professionals who encounter patients with DCM, individuals with lived experience and caregivers. Three advisors from the JLA were also present and were responsible for facilitating the process and ensuring equal involvement by all stakeholder groups. This workshop was held on November 16th, 2019 in New York City.
2. The 26 questions from the interim prioritization were randomly sorted and designated a letter from A to Z. Before the workshop, each participant was required to review the shortlist of questions and rank them from 1 to 26. Each question was printed on an A4 sized card with interim priority setting data on the back. This data included how many survey respondents from each stakeholder group ranked the question in their top ten.
3. Participants were allocated to small groups for both a morning and afternoon session. Each small group had equal representation of health care professionals and individuals with lived experience as well as 1 adviser from the James Lind Alliance.
4. In the morning sessions, each participant was asked to present the summary questions he or she ranked as the top 3 and bottom 3. The adviser noted the rankings for each individual and laid out the cards in rough groups: questions thought to be the most important, those thought to be the least important and those either not mentioned or disagreed upon. Participants then discussed the ordering of the cards and agreed on a ranking from A to Z. In some cases, the back of the card was referenced in order to acknowledge the perspectives from the interim prioritization.
5. The 3 advisors entered each groups’ ranking into a spreadsheet in order to develop a combined rank list. This list was presented to the whole group.
6. Three new small groups were formed for the afternoon session in order to discuss and revise the combined rank list. The revised lists from all 3 groups were combined into a spreadsheet and an aggregate ranking was presented to the whole group.
7. The whole group discussed this aggregate ranking and agreed on the top ten research priorities by the end of the session.
Table 2 displays the top ten research priorities for DCM.
Table 2.
The Top Ten Research Priorities for Degenerative Cervical Myelopathy.
| 1. What strategies can increase awareness and understanding of DCM amongst health care professionals and the public? Can these strategies help improve timely diagnosis and management of DCM? |
| 2. What is the natural history of DCM? What is the relationship between DCM and asymptomatic spinal cord compression or canal stenosis? What factors influence the natural history of the disease? |
| 3. What are the diagnostic criteria of DCM? What is the role of imaging and when should imaging be used in the assessment of DCM? |
| 4. What assessment tools can be used to evaluate functional impairment, disability and quality of life in people with DCM? What instruments, tools or methods can be used or developed to monitor people with DCM for disease progression or improvement either before or after surgical treatment? |
| 5. What is the pathophysiology of DCM? What are the mechanisms of neurological injury and the molecular and anatomical consequences? |
| 6. What is the role of rehabilitation following surgery for DCM? Can structured postoperative rehabilitation improve outcome following surgery for DCM? What are the most effective strategies? |
| 7. Can novel therapies, including stem-cell, gene, pharmacological and neuroprotective therapies, improve the health and wellbeing of people living with DCM and slow down disease progression? |
| 8. What is the socio-economic impact of DCM? (The financial impact of living with DCM to the individual, their supporters and society as a whole) |
| 9. What is the role of dynamic or novel imaging techniques and neurophysiology in the assessment of DCM? |
| 10. Are there clinical and imaging factors that can help a surgeon select who should undergo surgical decompression in the setting of DCM? At what stage of the disease is surgery the preferred management strategy? |
Publish and Promote the Top Ten Research Priorities
The top ten research priorities are avaliable at aospine.org/recode and each contextualised in this Global Spine Journal Special Issue. A detailed knowledge translation strategy was developed to outline how these priorities will be promoted through publications, conference forums, and online platforms.
Conclusion
Priority setting partnerships enable research priorities to be developed that consider the perspectives and interests of all stakeholders, including health care professionals, patients, and caregivers. The rigorous methodology behind this process allows numerous initial research uncertainties to be condensed into a list of the top ten priorities.
Supplemental Material
Supplemental Material, sj-pdf-1-gsj-10.1177_21925682211062501 for James Lind Alliance Priority Setting Partnership for Degenerative Cervical Myelopathy [AO Spine RECODE-DCM]: An Overview of the Methodology Used to Process and Short-List Research Uncertainties by Lindsay Tetreault, Oliver Mowforth, Danyal Z. Khan, Toto Gronlund, Philip Garwood, Olesja Hazenbiller, James S. Harrop, Bizhan Aarabi, Vafa Rahimi-Movaghar, Shekar N. Kurpad, James D. Guest, Jefferson R. Wilson, Brian K. Kwon, Michael G. Fehlings, Benjamin M. Davies, Mark R. N. Kotter and On behalf of the AO Spine RECODE-DCM Steering Committee and AO Spine RECODE-DCM Consortium in Global Spine Journal
Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research priorities were organized and funded by AO Spine through the AO Spine Knowledge Forum Spinal Cord Injury, a focused group of international Spinal Cord Injury experts. AO Spine is a clinical division of the AO Foundation, which is an independent medically-guided not-for-profit organization. Study support was provided directly through the AO Spine Research Department. MRNK is supported by the National Institute for Health Research (NIHR) Brain Injury MedTech Co-operative based at Cambridge University Hospitals NHS Foundation Trust and University of Cambridge, and BMD is a NIHR Clinical Doctoral Research Fellows. The views expressed in this publication are those of the authors and not necessarily those of the NIHR.
Supplemental Material: Supplemental material for this article is available online.
ORCID iDs
Oliver Mowforth, MRCS https://orcid.org/0000-0001-6788-745X
Jefferson R. Wilson, MD, PhD FRCSC https://orcid.org/0000-0001-5965-0305
Michael G. Fehlings, MD, PhD, FRCSC, FACS https://orcid.org/0000-0002-5722-6364
Benjamin M. Davies, MRCS, BSc, MPhil https://orcid.org/0000-0003-0591-5069
References
- 1.The James Lind Alliance Guidebook, Version 9: National Institute for Health Research; 2020. Available from: https://www.jla.nihr.ac.uk/. [Google Scholar]
- 2.Cowan K. The James Lind alliance: tackling treatment uncertainties together. J Ambul Care Manag. 2010;33(3):241-248. [DOI] [PubMed] [Google Scholar]
- 3.Petit-Zeman S, Firkins L, Scadding JW. The James Lind Alliance: tackling research mismatches. Lancet. 2010;376(9742):667-669. [DOI] [PubMed] [Google Scholar]
- 4.Elwyn G, Crowe S, Fenton M, et al. Identifying and prioritizing uncertainties: Patient and clinician engagement in the identification of research questions. J Eval Clin Pract. 2010;16(3):627-631. [DOI] [PubMed] [Google Scholar]
- 5.Majeed-Ariss R, McPhee M, McAteer H, Griffiths CEM, Young H. The top 10 research priorities for psoriasis in the U.K.: Results of a James Lind Alliance psoriasis Priority Setting Partnership. Br J Dermatol. 2019;181(4):871-873. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Klotz R, Doerr-Harim C, Ahmed A, et al. Top ten research priorities for pancreatic cancer therapy. Lancet Oncol. 2020;21(6):e295-e6. [DOI] [PubMed] [Google Scholar]
- 7.Staley K, Crowe S, Crocker JC, Madden M, Greenhalgh T. What happens after James Lind Alliance Priority Setting Partnerships? A qualitative study of contexts, processes and impacts. Res Involv Engagem. 2020;6(41):41. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Davies BM, Khan DZ, Mowforth OD, et al. RE-CODE DCM (Research Objectives and Common Data Elements for Degenerative Cervical Myelopathy): A consensus process to improve research efficiency in dcm, through establishment of a standardized dataset for clinical research and the definition of the research priorities. Global Spine J. 2019;9(suppl l):65S-76S. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Smith SS, Stewart ME, Davies BM, Kotter MRN. The Prevalence of asymptomatic and symptomatic spinal cord compression on magnetic resonance imaging: A systematic review and meta-analysis. Global Spine J. 2020:2192568220934496. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Fehlings MG, Tetreault L, Nater A, et al. The Aging of the Global Population: The Changing Epidemiology of Disease and Spinal Disorders. Neurosurgery. 2015;77(suppl 4):S1-S5. [DOI] [PubMed] [Google Scholar]
- 11.Mowforth OD, Davies BM, Goh S, O'Neill CP, Kotter MRN. Research inefficiency in degenerative cervical myelopathy: findings of a systematic review on research activity over the past 20 years. Global Spine J. 2020;10(4):476-485. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplemental Material, sj-pdf-1-gsj-10.1177_21925682211062501 for James Lind Alliance Priority Setting Partnership for Degenerative Cervical Myelopathy [AO Spine RECODE-DCM]: An Overview of the Methodology Used to Process and Short-List Research Uncertainties by Lindsay Tetreault, Oliver Mowforth, Danyal Z. Khan, Toto Gronlund, Philip Garwood, Olesja Hazenbiller, James S. Harrop, Bizhan Aarabi, Vafa Rahimi-Movaghar, Shekar N. Kurpad, James D. Guest, Jefferson R. Wilson, Brian K. Kwon, Michael G. Fehlings, Benjamin M. Davies, Mark R. N. Kotter and On behalf of the AO Spine RECODE-DCM Steering Committee and AO Spine RECODE-DCM Consortium in Global Spine Journal