Abstract
Objective
The OMERACT Filter provides guidelines for the development and validation of outcome measures for use in clinical research. The ‘Truth’ section of the OMERACT Filter pre-supposes an explicit framework for identifying the relevant core outcomes that are universal to all studies of the effects of intervention effects. There is no published outline for instrument choice or development that is aimed at measuring outcome, was derived from broad consensus over its underlying philosophy, or includes a structured and documented critique. Therefore, a new proposal for defining core areas of measurement (“Filter 2.0 Core Areas of Measurement”) was presented at OMERACT 11 to explore areas of consensus and consider whether already endorsed core outcome sets fit in to this newly proposed framework.
Method
Discussion groups critically reviewed the extent to which case studies of current OMERACT Working Groups complied with or negated the proposed framework, whether these observations had a more general application, and what issues remained to be resolved.
Result
Although there was a broad acceptance of the framework in general, several important areas of construction, presentation and clarity of the framework were questioned. The discussion groups and subsequent feedback highlighted 20 such issues.
Conclusion
These issues will require resolution in order to reach consensus on accepting the proposed Filter 2.0 framework of Core Areas as the basis for the selection of Core Outcome Domains and hence appropriate Core Outcome Sets for clinical trials.
Introduction
OMERACT strives to develop core outcome sets for rheumatologic conditions. Such core sets specify, for each condition, the minimum set of outcomes (and associated measurement instruments) necessary to provide the best estimate of benefits of an intervention. After adoption of a core set, OMERACT recommends that all studies of a health intervention in that condition report the results of these outcomes, regardless of the primary study question and the intended primary outcome measure. The original OMERACT Filter (1) describes the procedure of consensus building regarding core outcome sets and the Filter's components are summarized in three words: truth, discrimination and feasibility. Truth captures the notion that a core set measures what is intended and is unbiased and relevant. However, as OMERACT members have applied the filter in a wider range of conditions and have become associated with a broader movement to identify core outcome sets in medicine as a whole (the Core Outcome Measures in Effectiveness Trials (COMET) group (2)), it has become clear that this definition pre-supposes an explicit way of identifying the relevant core outcomes that are universal to all studies. That this supposition may be inadequately elucidated has been highlighted since patients began to be included in the OMERACT process (3). For example, as presented by S Hewlett, in 2002 fatigue was identified by OMERACT patients as a relevant outcome not in the original core set (3), subsequently found to add to our understanding of RA (5-7) and finally recommended as an additional core set item in 2007 (8).
To address the question of ‘truth’, and in particular the basis upon which core areas of outcome can be identified, a discussion paper (9) and a literature review (10) were prepared for this OMERACT 11 session. These identified that several proposals to identify essential areas of outcome assessment (e.g. the International Classification of Functioning, Disability and Health (ICF)(11) and its predecessors (12, 13)). However, no proposal explicitly aimed at measuring outcome as a consequence of an intervention was found. In addition, the development process of previous proposals was insufficiently documented, Therefore, based on input from experts in the field and repeated consultations with and surveys of OMERACT and COMET attendees for more than 1 year, a new system for defining core areas of measurement was proposed specifically for discussion and possible adoption at OMERACT 11 (9).
The new proposal was laid out in detail in a pre-conference paper (9) and presented by M Boers: four Core Areas of outcome should be included in some manner in every clinical trial - Death, Life Impact; Resource Use; and Pathophysiological Manifestations. Under these headings, disease specific Core Domains would be specified by groups developing core sets. In addition the Contextual Factors should be specified that could influence the interpretation of outcomes in the setting in which they are applied. In this OMERACT session, small Discussion (‘breakout’) groups were presented with case studies drawn from Working Groups across the spectrum of OMERACT activity and invited to critically review the core area proposal in the light of the case study. Further formal and informal discussions during the OMERACT 11 meeting provided opportunities for clarifications and resolution of many areas of uncertainty before a final plenary vote at the last conference session.
Case studies and breakout discussions
Five illustrative case studies were reported, each to two breakout groups before a discussion among OMERACT 11 delegates. Case presenters addressed specific questions on their current work [Table 1]. Breakout discussion groups with a mixture of about 20 participants each including two patient partners, were first asked to consider the match between the presented case study and the proposed Core Area framework, its illustration of the importance of Contextual Factors, and to list any elements of the framework the case study had not addressed. Subsequently the groups considered in more general terms how outcome measurement sets can be developed and addressed the question, “In the light of these considerations, do you think the proposed concepts of Core Areas and Core Domains with Contextual Factors offers a useful model for Core Domain Set development?”
Table 1. Summary of case studies.
Case study | Full title | What are the outcome domains you are currently working with? | Why have you chosen them? | What contextual factors did you consider? | How did you make these decisions? |
---|---|---|---|---|---|
CTD-ILD | Connective tissue diseases - Interstitial lung disease | Dyspnea. Health Related Quality of Life. Lung Imaging. Lung physiology and function. Survival. Cough. |
These domains have been identified as the most relevant and measureable for clinical trials in CTD-ILD and interstitial pulmonary fibrosis by medical and patient experts. | Primary context is the randomized clinical trial, Also considered was the context of clinical relevance of the domains to patient outcome in usual clinical care. | Informed by expert Delphi involving Rheumatologists and pulmonologists A 3-round Delphi identified potential domains and measures. Patient perspective solicited via survey and patient focus groups. |
PMR | Polymyalgia rheumatica | Pain. Stiffness. Function. Systemic inflammation. |
Candidate outcome measures chosen for a postulated future interventional trial of an alternative to prednisolone for PMR. | Age. Gender. Cultural background. Time of day specified in the patient-reported outcome measures. |
Informal patient consultation. Systematic literature review. Work still in progress. |
Vasculitis | ANCA-associated vasculitis | Disease activity Disease damage Patient-reported outcome Mortality |
These domains have been measured in many trials and have been considered critical for both evaluating efficacy and guiding evaluation and treatment | Primary context was the randomized clinical trial, Trials have often been modeled on standard of clinical care. | Extrapolation from clinical trials Expert opinion Patient surveys |
Case study | Full title | What are the outcome domains you are currently working with? | Why did you decide not to use the existing Core Set? | What contextual factors did you consider? | How did you make these decisions? |
OA Hand | Osteoarthritis of the hand | Domains identified in OMERACT 3. Depending on the setting: pain, (physical) function and patient global assessment for symptom modifying trials and extra imaging for structure modifying trials. | In the selection process patients were not involved and there was limited attention for different settings, study populations, different hand OA subsets and the co-occurrence of OA at other joint sites. | Different hand OA subsets (thumb base OA, interphalangeal OA, erosive OA) and generalized OA. Additionally: aesthetic damage, inflammation and thumb base prosthesis were suggested. |
Discussions and Delphi exercises within a group of hand OA experts. Work still in progress. |
Flare in RA | Flares in rheumatoid arthritis | Pain Physical Function Tender Joints Swollen Joints Patient Global Assessor Global Laboratory Measures Fatigue Stiffness Participation Self-management |
A priori decision to use “bottom-up” approach to identify domains (literature review, qualitative studies (14) with patients, expert input) to ensure all potential domains considered. Domains ultimately recommended include existing RA Core Set plus domains initially identified by patients and then prioritized by both groups. | Area of application (RCT, LOS, clinical practice), disease duration, duration in current (low) disease activity state, co-morbid conditions, knowledge/experience with self-management strategies, individual context, environment | Domains identified through iterative 3 stage Delphi Process of > 200 patients and health care providers (15). Domains with more than 70% agreement as important or essential in measuring flare by either patients or HCPs were considered “Core”. |
Plenary report back and discussion
Each breakout group reported the main points from its discussion to a plenary session of all participants. While the case studies each brought to light specific points related to particular areas of work (helpful for the OMERACT group working in that area to consider further), 20 common issues emerged requiring clarification and resolution. These themes and the broad areas where existing work was entirely compatible with the new proposal were further explored during a highly participative plenary discussion session, and are summarised in Table 2.
Table 2.
Death | Death may not be an outcome of interest. |
Should states worse than death be mentioned? | |
| |
Life Impact | Should Life Impact be subdivided further? |
| |
Resource Use | What does this mean? |
Are there any surrogates? | |
What point of view is considered (patient, health system, society)? | |
Will measurement of resource use be impractical in many trials? | |
| |
Pathophysiological Manifestations | Can clinical signs (and sometimes symptoms) also indicate pathophysiological status? |
Need to be flexible about how this is defined. | |
| |
Contextual Factors | Can we better define what these factors are? |
Can we provide a list? | |
Can we better distinguish between factors? | |
Who decides what is required? | |
| |
Some general issues | Can we provide more concrete examples? |
Are adverse effects a core area in themselves? | |
Difference between domains and instruments unclear. | |
Will instruments crossing domains be a problem? | |
| |
Some process issues | Difference between core areas and primary and secondary outcomes. |
Does core set development come to a stop if one or more Core Domains does not have a validated instrument? | |
There should be provision for updating or revision of Core Outcome sets as further data accumulate. |
Many participants had difficulty using ‘Death’ as an outcome in all circumstances. In many case studies death was not a direct outcome of interest: it was not expected that any deaths would be related to the condition (e.g., hand osteoarthritis) or the treatment (e.g. physiotherapy), or occur in the time window under investigation (e.g. the short term response to intramuscular glucocorticoids). Participants did recognise that any death occurring during a clinical trial would need to be reported regardless of perceived causality. Other participants raised the possibility that ‘states worse than death’ may be experienced by patients and wondered how this would be dealt with within the framework.
The concept and importance of Life Impact as a Core Area (capturing function, quality of life, the ICF domains (11), patient perception of health, etc.) was widely accepted. Debate centred on whether, at a Core Area level, Life Impact should be subdivided further. Several different suggestions were made, such as work related problems, mobility and independence and social interactions, but these were often relevant in only one disease group or one particular context. No clear consensus about further subdivisions of Life Impact emerged.
Considering Resource Use as a Core Area produced the greatest discussion. Many participants saw this as an economic evaluation that was only worth undertaking in studies designed for that purpose. Some felt that ‘resources’ included family support, support at work to continue working, personal time and effort of the patient, opportunity costs to the health care system, etc. Other remarks addressed the costs/feasibility of adequately capturing resource use; and the early development phase of a therapy where true resource costs may not be relevant to the question (e.g. in 'proof of principle' studies), or not even calculable (e.g. the final cost of the therapy might depend on technical manufacturing issues and market forces). In sum this area, while very important and relevant in many circumstances, was felt to require additional discussion and delineation before it could be considered a definite Core Area.
Pathopysiological Manifestations also produced some debate as a Core Area. There was general recognition that some information concerning the underlying disease process and its activity was needed to measure the effects of any treatment, and that most of our existing outcome measures focused on evaluating this core area. For example most of the current RA core set instruments (joint counts, acute phase reactants and imaging) measure a pathophysiologic manifestation of the underlying disease process. There was some confusion as to the way this area might be assessed – could symptoms of pain and swelling in a joint be used as a measure of pathophysiology? There was a feeling that this needs to be defined in a flexible way.
There was wide recognition of a strong conceptual need to consider Contextual Factors. Confounding factors, co-morbidities, variation in health care systems and factors related to psychological status were all identified as potential contextual factors. However it was unclear to many participants how these factors would be identified and which would be labelled as ‘core’ to particular investigations. A number of general issues emerged from the breakout group reports and the plenary discussion. A recurrent theme was the request to provide concrete examples for the theoretical framework. Whether adverse effects should be a Core Area was also a topic of disagreement and uncertainty. All agreed that adverse effects should be reported in trials, and it was recognised that this often constitutes a specific section within a clinical trial report. However it was also agreed that any given adverse effect would occur under the umbrella of one of the proposed Core Areas.
There were many points raised in which issues related to choosing, testing and developing specific instruments became entangled with questions of whether a Core Area or a particular Core Domain would been adequately addressed and hence whether a Core Set of outcome measures would then be achievable within the proposed framework. There was also a recognition that many existing instruments, such as questionnaires, relate to more than one Core Area, and participants were unclear if this would be allow separate assessment of different Core Areas.
Some participants were unsure of the difference between Core Areas, primary outcomes and secondary outcomes and wondered if Core Areas were intended to be the primary or secondary outcome measures. They feared this might override the intention of trial designers in setting up the study protocol. There was also concern that the work of core outcome development might come to a halt if, in relation to a particular condition, a Core Domain was identified but no valid assessment instrument existed in that domain. This led to a fruitful exploration of the difference between Core Domains and Core Outcome Sets, and a clearer understanding that there is a two step process in defining first Core Domains within the Core Areas, and second identifying (or devising) instruments to include in the Core Outcome set. Finally, and perhaps unsurprisingly, there was a strong call for inclusion of a review process to ensure that as data accumulate the whole philosophy of the emerging Filter 2.0 framework would be regularly scrutinised and updated.
Summary and conclusions
This OMERACT session was deliberately constructed to test the proposed Filter 2.0 framework of Core Areas, Core Domains and Contextual Factors which had already been subject to discussion, debate and extensive development before the meeting. Using case studies from different working groups, participants were able to probe the theoretical and practical implications of the framework, and to look for areas of strength and weakness. There was a broad agreement with the need to formalise an overarching structure to justify the subsequent selection of Core Domains. Until challenged by the introduction of the patient perspective and the emergence of the COMET initiative, the OMERACT community has, in effect, been relying on clinicians' common understanding of the disease areas in which they are working. This workshop, which took place at the start of the OMERACT 11 conference, concluded from case studies and discussions that most of the current work of the OMERACT participants already fits into the principles of the new framework, but several important areas of uncertainty emerged, as described above. If sufficient consensus was to be achieved in time for the plenary session at the end of the conference [16], these areas would need to be clarified and addressed further by the Filter 2.0 development group.
Footnotes
Footline: Core Areas for outcomes
Contributor Information
John Richard Kirwan, University of Bristol Academic Rheumatology Unit, Bristol Royal Infirmary, Bristol, UK.
Maarten Boers, Departments of Epidemiology and Biostatistics, and Rheumatology, VU University Medical Center, PK 6Z 185, PO Box 7057, 1007 MB, Amsterdam, Netherlands.
Sarah Hewlett, University of the West of England, Academic Rheumatology Unit, Bristol Royal Infirmary, UK.
Dorcas Beaton, Department of Occupational Sciences and Occupational Therapy, Institute for Health Policy Management and Evaluation, University of Toronto, Toronto, Canada.
Clifton O. Bingham, III, Division of Rheumatology, Johns Hopkins University, Baltimore, MD, US.
Ernest Choy, Section of Rheumatology, Cardiff University School of Medicine, Cardiff, UK.
Philip G. Conaghan, Division of Musculoskeletal Disease, University of Leeds, & NIHR Leeds Musculoskeletal Biomedical Research Unit, UK.
Maria-Antonietta D'Agostino, Versailles-Saint Quentin En Yvelines University, Department of Rheumatology, Ambroise Paré Hospital, APHP, Boulogne-Billancourt, France.
Maxime Dougados, Paris-Descartes University, Medicine Faculty, APHP, Cochin Hospital, Rheumatology B Dpt, PARIS, France.
Daniel E. Furst, University of California, Geffen School of Medicine, Los Angeles, California, USA.
Francis Guillemin, Université de Lorraine, Université Paris Descartes, EA 4360 APEMAC, Nancy, France.
Laure Gossec, Université Pierre et Marie Curie (UPMC) - Paris 6, GRC-UMPC 08 (EEMOIS); AP-HP Pitié-Salpêtrière Hospital, Department of Rheumatology, Paris, France.
Désirée van der Heijde, Department of Rheumatology, Leiden University Medical Center, Leiden, Netherlands.
Margreet Kloppenburg, Departments of Rheumatology and Clinical Epidemiology, Leiden University Medical Center, Leiden, Netherlands.
Tore K. Kvien, Department of Rheumatology, Diakonhjemmet Hospital, Oslo, Norway.
Robert B. M. Landewé, Department of Clinical Immunology & Rheumatology, Academic Medical Center, University of Amsterdam & Atrium Medical Center Heerlen, the Netherlands.
Sarah Louise Mackie, NIHR-Leeds Musculoskeletal Biomedical Research Unit, Division of Rheumatic and Musculoskeletal Diseases, University of Leeds, Leeds, UK.
Eric L. Matteson, Division of Rheumatology, Mayo Clinic College of Medicine, 200 1st St SW, Rochester, MN 55905, USA. [Disclosures relevant to PMR: Grants: Novartis, American College of Rheumatology, European League Against Rheumatism. Consultant: Horizon Pharmaceuticals].
Philip J. Mease, Seattle Rheumatology Associates, Chief, Swedish Medical Center Rheumatology Research Division, Clinical Professor, University of Washington School of Medicine, Seattle, Washington, USA.
Peter A. Merkel, Division of Rheumatology, University of Pennsylvania, Philadelphia, PA, USA.
Mikkel Ostergaard, Department of Rheumatology, Copenhagen University Hospital at Glostrup, Copenhagen, Denmark.
Lesley Ann Saketkoo, Section of Rheumatology, Louisiana State University Health Sciences Center, New Orleans, Louisiana, USA.
Lee Simon, SDG LLC Cambridge, MA 02138, USA.
Jasvinder A. Singh, Birmingham Veterans Affairs Medical Center and University of Alabama at Birmingham, Birmingham, Alabama, USA [JAS has received research grants from Takeda and Savient and consultant fees from Savient, Takeda, Ardea, Regeneron, Allergan, URL pharmaceuticals and Novartis. JAS is a member of the executive of OMERACT, an organization that develops outcome measures in rheumatology and receives arms-length funding from 36 companies; a member of the American College of Rheumatology's Guidelines Subcommittee of the Quality of Care Committee; and a member of the Veterans Affairs Rheumatology Field Advisory Committee].
Vibeke Strand, Division of Immunology/Rheumatology, Stanford University School of Medicine, Palo Alto, CA, USA.
Peter Tugwell, Department of Medicine, University of Ottawa, Canada.
References
- 1.Boers M, Brooks P, Strand V, Tugwell P. The OMERACT Filter for outcome measures in rheumatology. J Rheumatol. 1998;25:198–9. [PubMed] [Google Scholar]
- 2.COMET initiative. [Accessed 24 July 2012]; at http://www.comet-initiative.org/
- 3.Kirwan J, Heiberg T, Hewlett S, Hughes R, Kvien T, Ahlmèn M, Boers M, Minnock P, Saag K, Shea B, Suarez Almazor M, Taal E. Outcomes from the Patient Perspective workshop at OMERACT 6. J Rheumatol. 2003;30:868–72. [PubMed] [Google Scholar]
- 4.Boers M, Tugwell P, Felson DT, et al. World health organization and international league of associations for rheumatology core endpoints for symptom modifying antirheumatic drugs in rheumatoid arthritis clinical trials. J Rheumatol. 1994;21(suppl 41):86–9. [PubMed] [Google Scholar]
- 5.Hewlett S, Cockshot Z, Byron M, Kitchen K, Tipler S, Pope D, Hehir M. Patients' perceptions of fatigue in rheumatoid arthritis: Overwhelming, uncontrollable, ignored. Arth Rheum (Arth Care & Research) 2005;53:697–702. doi: 10.1002/art.21450. [DOI] [PubMed] [Google Scholar]
- 6.Ahlmen M, Nordenskiold U, Archenholtz B, et al. Rheumatology outcomes: the patient's perspective. A multicentre focus group interview study of Swedish rheumatoid arthritis patients. Rheumatology. 2005;44:105–10. doi: 10.1093/rheumatology/keh412. [DOI] [PubMed] [Google Scholar]
- 7.Hewlett S, Hehir M, Kirwan J. Measuring fatigue in rheumatoid arthritis: a systematic review of scales in use. Arthritis and Rheumatism (Arthritis Care and Research) 2007;57:429–539. doi: 10.1002/art.22611. [DOI] [PubMed] [Google Scholar]
- 8.Kirwan JR, Minnock P, Adebajo A, et al. Patient perspective: fatigue as a recommended patient centered outcome measure in rheumatoid arthritis. J Rheumatol. 2007;34:1174–7. [PubMed] [Google Scholar]
- 9.Boers M, Kirwan JR, Wells G, et al. A Framework and Process to Develop Core Outcome Sets: OMERACT Filter 2.0. In preparation. [Google Scholar]
- 10.Idzerda L, Rader T, Tugwell P, Boers M. Can we decide which outcomes should be measured in every clinical trial? Systematic review of concepts and methodologies used to establish core sets for outcome measurement. In preparation. [Google Scholar]
- 11.International Classification of Functioning, Disability and Health: ICF..WHO. 2001 doi: 10.1080/0963828031000137090. Accessed at http://www.who.int/classifications/icf/en/ [DOI] [PubMed]
- 12.International Classification of Diseases (ICD-10) [Accessed 18-4-11];1990 at http://www.who.int/classifications/icd/en/
- 13.World Health Organization. International Classification of Impairments, Disabilities and Handicaps. Geneva: 1980. [Google Scholar]
- 14.Hewlett SE, Sanderson T, May JE, Alten R, Bingham CO, 3rd, March L, Pohl C, Woodworth T, Bartlett SJ. “I'm hurting, I want to kill myself”: Rheumatoid arthritis flare is more than a high joint count-an international patient perspective on flare where medical help is sought. Rheumatology (Oxford) 2012;51(1):69–76. doi: 10.1093/rheumatology/keq455. [DOI] [PubMed] [Google Scholar]
- 15.Bartlett SJ, Hewlett SE, Bingham CO, 3rd, Woodworth TG, Alten R, Pohl C, Choy E, Sanderson T, Boonen A, Bykerk V, Leong AL, Strand V, Furst DE, Christensen R the OMERACT Flare Working Group. Identifying Core Domains to Assess Flare in Rheumatoid Arthritis: an OMERACT International Patient and Provider Combined Delphi Consensus. Ann Rheum Dis. 2012 Jul 6; doi: 10.1136/annrheumdis-2011-201201. Epub ahead of print. [DOI] [PubMed] [Google Scholar]
- 16.Boers M, et al. How to choose core outcome measurement sets for clinical trials: OMERACT 11 approves Filter 2.0. J Rheumatol. doi: 10.3899/jrheum.131314. This issue. [DOI] [PubMed] [Google Scholar]