Abstract
Introduction
Clinical quality indicators are necessary to monitor the performance of healthcare services. The development of indicators should, wherever possible, be based on research evidence to minimise the risk of bias which may be introduced during their development, because of logistic, ethical or financial constraints alone. The development of automated methods to identify the evidence base for candidate indicators should improve the process of indicator development. The objective of this study is to explore the relationship between clinical quality indicators for asthma management in children with outcome and process measurements extracted from randomised controlled clinical trial reports.
Methods and analysis
National-level indicators for asthma management in children will be extracted from the National Quality Measures Clearinghouse (NQMC) database and the National Institute for Health and Care Excellence (NICE) quality standards. Outcome measures will be extracted from published English language randomised controlled trial (RCT) reports for asthma management in children aged below 12 years. The two sets of measures will be compared to assess any overlap. The study will provide insights into the relationship between clinical quality indicators and measurements in RCTs. This study will also yield a list of measurements used in RCTs for asthma management in children, and will find RCT evidence for indicators used in practice.
Ethics and dissemination
Ethical approval is not necessary because this study will not include patient data. Findings will be disseminated through peer-reviewed publications.
Keywords: HEALTH SERVICES ADMINISTRATION & MANAGEMENT
Strengths and limitations of this study.
Develop a novel method for comparing indicators with research evidence in clinical trial.
Provide a systematic and objective way to identify candidate indicators, identify research associated with current indicators, and identify potential inconsistencies between research evidence and clinical practice.
Generate a list of measurements used in randomised controlled trials of asthma management in children.
Introduction
The drive to improve the quality and safety of healthcare has resulted in the proliferation of clinical quality indicators. Indicators may be used to assess the quality of care, or to identify and prioritise areas for improvement.1 2 They describe “as much about a system as possible in as few points as possible”.3
A good indicator should be important, relevant, valid, reliable, meaningful and understandable.3 4 In addition, indicators should be easy to collect,5 and the costs of doing so should not outweigh the potential benefits.6 The development of indicators can be complex and resource intensive. Indicators can be developed using deductive approaches (moving from target concepts to supporting data), inductive approaches (using data to identify concepts) or both.7 Deductive approaches are more common.7 The steps to developing indicators deductively are: (1) selecting a topic, (2) forming a review team, (3) researching the evidence and practice behind candidate indicators, (4) selecting indicators, (5) designing their specifications, (6) pilot testing their collection and (7) implementing them.2 8
In practice, clinical indicators can be vulnerable to bias and lack rigour8 because of (sometimes necessary) logistic, ethical or financial constraints during their development. To minimise these risks, the development of indicators should, wherever possible, be based on research evidence.2 9 Different grades of evidence are available, and randomised controlled trials (RCT) are generally regarded as one of the highest levels of evidence.10 11
Asthma is a chronic respiratory disease that affects about 300 million people globally, with an estimated 250 000 deaths annually.12 It is the most common chronic disease among children. There is currently no cure for asthma, but it can generally be managed and controlled with appropriate care (care regarded as being in line with clinical guidelines13–15). Many quality indicators have been developed for asthma;16–20 most are based on clinical practice guidelines (CPGs). For example, the National Institute for Health and Care Excellence (NICE) quality standards for asthma are based on the British Thoracic Society/Scottish Intercollegiate Guidelines Network (BTS/SIGN) clinical guideline 101: British guidelines on the management of asthma;13 and the asthma clinical performance measurement tool promoted by the Physician Consortium for Performance Improvement is based on the National Asthma Education and Prevention Program Expert Panel Report 2.21
To better assess the level of evidence behind currently recommended indicators, this protocol describes a study of the relationship between outcome measurements used in RCTs and national level clinical quality indicators in the UK and the USA, using asthma management in children as a case study. The study will compare outcome terms extracted from RCT reports with clinical quality indicators by assigning concepts from the Unified Medical Language System (UMLS) using MetaMap22 to both sets of terms. This allows terms from different sources to be standardised for easy aggregation and comparison. MetaMap has been used for many tasks, for example, information retrieval23 and text mining,24 25 and has been shown to perform well in mapping biomedical concepts to text compared with human labelling.26
Methods and analysis
The study will examine asthma management in children aged less than 12 years. There are four basic steps: (1) identification and extraction of indicators, (2) identification and extraction of outcome measurements from RCTs, (3) mapping both sets of metrics to underlying concepts as defined in the UMLS using MetaMap and (4) matching and comparing clinical quality indicators and RCT measurements, with an evaluation of the basis for any differences.
Identification and extraction of indicators
National-level clinical quality indicators for asthma management in children from the USA and the UK will be collected. The National Quality Measures Clearinghouse (NQMC) is a database on healthcare quality measures sponsored by the US Department of Health and Human Services—Agency for Healthcare Research and Quality (AHRQ). The NQMC will be searched using the term ‘asthma’ within the respiratory tract diseases condition (50 measures in asthma of the 243). The NICE quality standard for asthma (11 asthma quality standards) will also be included. NICE plays a major role in commissioning and producing guidelines and developing quality standards for Britain's National Health Service (NHS).
The following inclusion criteria will be used to select indicators:
Any process or outcome indicator,
Any indicator of pharmacological or non-pharmacological management,
Applicable to children aged below 12 years,
Developed for national-level assessment.
Candidate indicators identified in the search will be independently assessed against inclusion criteria by two appraisers. Disagreements will be resolved by consensus.
The included indicators will then be normalised as follows:
Remove: Unit of analysis (eg, number of people/patients),
Remove: Time points/frames (eg, more than 24 h, within 2 days) for a process to occur,
Remove: The term “asthma” if it describes the characteristics of patients,
Extract: Noun terms describing the unit of analysis (examples in table 1).
Table 1.
Examples of classification of extracted terms
| Classification | Examples |
|---|---|
| Type of recommended care | Hospitalisation, asthma action plan |
| Medication | Steroids, inhaled corticosteroids, leukotriene inhibitors |
| Event | Exacerbation of asthma |
Table 2 shows examples of possible terms that may be extracted from the indicator sources.
Table 2.
Examples of terms extracted from clinical indicators
| Extracted terms | Classification | |
|---|---|---|
| Paediatric inpatients with asthma who received relievers during hospitalisation | Relievers; hospitalisation | Medication; type of recommended care |
| Paediatric inpatient discharge, age 2 years through 17 years, with an International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) Principal Diagnosis Code of asthma | Inpatient discharges | Type of recommended care |
| Number of patients with asthma who have an asthma discharge plan | Asthma discharge plan | Type of recommended care |
| Number of patients with asthma who were seen in an emergency department or hospitalised for asthma treatment | Emergency department; hospitalised | Type of recommended care; type of recommended care |
| The number of people in the denominator followed up by their own GP practice within 2 working days of treatment | GP practice | Type of recommended care |
| The number of people who received treatment in hospital or through out-of-hours services for an acute exacerbation of asthma | Hospital; out-of-hours services; acute exacerbation of asthma | Type of recommended care; type of recommended care; event |
Bold typeface indicates ‘extracted terms’.
GP, general practitioner.
Identification and extraction of outcomes from RCTs
Types of studies
Only RCTs will be included. Secondary/post hoc analysis/protocol of RCTs will be excluded.
Types of participants
Children with asthma aged below 12 years will be included.
Types of interventions
Studies that involved any type of medical or non-medical management of asthma will be included.
Search strategy
We will search PubMed for RCTs on asthma management for children aged below 12 years. Studies will be restricted to English language articles published in the past 10 years (2005–2014).
The following search terms will be used to retrieve PubMed articles:
Asthma
(randomised controlled trial[Publication Type] OR (randomised[Title/Abstract] AND controlled[Title/Abstract] AND trial[Title/Abstract]))
#1 AND #2
#3 AND “English”[Language]
The second search term above is the specific/narrow therapy filter for clinical queries. PubMed filters of Species (‘Humans’) and Ages (‘Infant: birth-23 months’, ‘Preschool Child: 2–5 years’, and ‘Child: 6–12 years’) will be applied.
Study selection
The titles and abstracts of the resulting references will be screened independently by two reviewers against the following inclusion criteria:
Articles describing RCT, excluding secondary/post hoc analysis/protocol of RCTs;
The participants of the trial include children with asthma aged below 12 years even if the trial also includes other participants;
The trial's focus is on medical or non-medical management of asthma.
Any control arm is acceptable as well as any outcome measure. Disagreements will be resolved by consensus. Inter-rater agreement will be measured periodically on screening decisions and only one screener may be used for the remaining abstracts if agreement is high (ie, Cohen's κ>0.8).27
The full text of the potentially eligible studies will be retrieved and reassessed if the abstract does not contain enough information to make a decision.
Outcome measures extraction from clinical trials and ambiguities resolution
RCTs generally report using the PICO criteria:28 Population, Intervention, Comparison, and Outcome. Two independent annotators will be asked to independently extract the Population (ie, number of participants in the study) and Outcome (outcome names) from the full text of every included RCT. Outcome terms will be any phrase that describes a quantifiable measurement.
Outcome names identified by this process will be normalised using the following rules:
Outcome deduplication: Similar outcome terms mentioned multiple times in an RCT will only be considered as a single occurrence.
Generalisation relationships: The most specific term mentioned in an RCT will be preferred. For example, spirometry and peak flow measurement both describe lung function tests, but the latter is more specific.
Ambiguities: Any ambiguities will be resolved by consultation with a paediatric respiratory physician, paediatric nurse or other relevant clinician.
Inter-rater agreement will be measured periodically on extraction decisions and only one extractor may be used for the remaining extractions if agreement is high (ie, Cohen's κ>0.8).
UMLS concept mapping using MetaMap
The list of terms extracted from indicator sources and the list of outcome measures taken from RCTs will next both be mapped into standardised terms (UMLS concepts using MetaMap) to facilitate aggregation and comparison. The UMLS Metathesaurus, from the National Library of Medicine, comprises over 1 million biomedical concepts from over 100 source vocabularies. MetaMap22 is a tool for mapping biomedical terms in free text to the UMLS Metathesaurus. MetaMap will be provided with extracted terms from clinical indicators and RCTs and will return a list of ranked mapping options. We will use all concepts identified by MetaMap in this study. If no UMLS concept is found, we will retain the extracted term. Table 3 shows examples of phrases mapped to UMLS terms. As shown in table 3, ‘exacerbation of asthma’ will be mapped to ‘asthma exacerbation’ and ‘PEF’ will be mapped to ‘peak expiratory flow’ even if they are expressed differently.
Table 3.
Examples of mapping to UMLS terms using MetaMap
| Phrases | UMLS mapping by MetaMap |
|---|---|
| Exacerbation of asthma | C0349790:Exacerbation of asthma [Finding] C0004096:–Asthma (Asthma) [Disease or Syndrome] C2984299:Asthma (Asthma Pathway) [Functional Concept] |
| Asthma exacerbation | C0349790:asthma exacerbation (Exacerbation of asthma) [Finding] C0004096:–Asthma (Asthma) [Disease or Syndrome] C2984299:Asthma (Asthma Pathway) [Functional Concept] |
| PEF | C0030771:PEF (Pefloxacin) [Antibiotic,Organic Chemical] C1518922:PEF (Peak Expiratory Flow) [Laboratory Procedure] C1542834:PEF (Peak expiratory flow rate) [Finding] |
| Peak expiratory flow | C1518922:Peak Expiratory Flow [Laboratory Procedure] C0857465:peak flow [Finding] C0231800:Expiratory (Expiration, function) [Organ or Tissue Function] C0444505:Peak (Peak level) [Quantitative Concept] C0806140:Flow (flow) [Natural Phenomenon or Process] |
UMLS, Unified Medical Language System.
Extracted term ranking
UMLS concepts will be ranked according to the total number of RCT participants that they were applied to. In most cases, this is the number of patients randomised or for whom there was an intention to treat.
Mapping and comparing clinical quality indicators and RCT measurements
The RCT outcomes and practice indicators will then be compared using UMLS concepts by two annotators. Terms in both lists will be mapped to each other if one or more UMLS concepts associated with them match. The mappings will be agreed by consensus and inter-rater reliability will be measured. Advice will be sought from appropriate clinicians if consensus cannot be reached.
The congruence between the mapped outcomes from RCTs and those from clinical indicators will be measured. Measured outcomes in RCT that are not used as clinical indicators may imply wastage either because metrics with research evidence behind them are not being used as indicators, or because clinical trials have measured outcomes that are not important in the assessment of routine care. Indicators that cannot be mapped to outcomes in RCTs will also be identified as they may have little or no research evidence to support their use.
Discussion
This protocol describes a novel method for comparing indicators with research evidence in clinical trials, and will help to determine if indicators associated with the management of asthma in children are supported by research evidence. This study will also generate a list of measurements used in RCTs of asthma management in children.
A limitation is that it will not examine other levels of evidence beyond level 1 (RCT) evidence. Some indicators may not be appropriate for use in RCTs and may appear in different study types, such as economic evaluations. Further, the protocol does not investigate if the RCTs support the use of indicators, and only looks for evidence of their use as a proxy indicator of utility. Some RCT outcomes may be important in early phases of the evaluation of an intervention, but will have limited application in routine practice. This study is also limited by searching only PubMed and assessing only English language RCTs.
However, it is important because it provides a systematic and objective way of finding candidate level 1 evidence for clinical indicators. It will assist in identifying candidate indicators, identifying research associated with current indicators, and identifying potential inconsistencies between research evidence and clinical practice.
Footnotes
Contributors: MKC, GT and EC designed the study and drafted the manuscript. All the authors were involved in devising the protocol and read and approved the final manuscript.
Funding: This work was supported by a National Health & Medical Research Council Program Grant APP1054146.
Competing interests: None declared.
Provenance and peer review: Not commissioned; externally peer reviewed.
Data sharing statement: This is a study protocol. Any data from the study can be requested from the corresponding author.
References
- 1.Crampton P, Perera R, Crengle S et al. What makes a good performance indicator? Devising primary care performance indicators for New Zealand. N Z Med J 2004;117:1–12. [PubMed] [Google Scholar]
- 2.Mainz J. Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care 2003;15:523–30. 10.1093/intqhc/mzg081 [DOI] [PubMed] [Google Scholar]
- 3.Pencheon D. The good indicators guide: understanding how to use and choose indicators. NHS Institute for Innovation and Improvement, 2008. [Google Scholar]
- 4.Institute of Medicine. Performance measurement: accelerating improvement (Pathways to Quality Health Care Series). Washington DC: The National Academies Press, 2006. [Google Scholar]
- 5.Wollersheim H, Hermens R, Hulscher M et al. Clinical indicators: development and applications. Neth J Med 2007;65:15–22. [PubMed] [Google Scholar]
- 6.Hibbert P, Hannaford N, Long J et al. Final report: performance indicators used internationally to report publicly on healthcare organisations and local health systems. Australian Institute of Health Innovation, University of New South Wales, 2013. [Google Scholar]
- 7.Stelfox HT, Straus SE. Measuring quality of care: considering conceptual approaches to quality indicator development and evaluation. J Clin Epidemiol 2013;66:1328–37. 10.1016/j.jclinepi.2013.05.017 [DOI] [PubMed] [Google Scholar]
- 8.Kötter T, Blozik E, Scherer M. Methods for the guideline-based development of quality indicators—a systematic review. Implement Sci 2012;7:21 10.1186/1748-5908-7-21 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Campbell S, Braspenning J, Hutchinson A et al. Improving the quality of health care: research methods used in developing and applying quality indicators in primary care. BMJ 2003;326:816 10.1136/bmj.326.7393.816 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Woolf SH, Battista RN, Anderson GM et al. Assessing the clinical effectiveness of preventive maneuvers: analytic principles and systematic methods in reviewing evidence and developing clinical practice recommendations. A report by the Canadian task force on the periodic health examination. J Clin Epidemiol 1990;43: 891–905. 10.1016/0895-4356(90)90073-X [DOI] [PubMed] [Google Scholar]
- 11.Sackett DL, Straus SE, Richardson WS et al. Evidence-based medicine: how to practice and teach EBM. 2nd edn Churchill Livingstone, 2000. [Google Scholar]
- 12.Cruz AA, Bousquet J, Khaltaev N. Global surveillance, prevention and control of chronic respiratory diseases: a comprehensive approach. World Health Organization, 2007. [Google Scholar]
- 13.British Thoracic Society Scottish Intercollegiate Guidelines Network. British guideline on the management of asthma. Thorax 2008;63:iv1 10.1136/thx.2008.097741 [DOI] [PubMed] [Google Scholar]
- 14.National Asthma Council Australia. Australian asthma handbook, Version 1.0. Melbourne: National Asthma Council Australia, 2014. [Google Scholar]
- 15.National Asthma Education Prevention Program. Third expert panel on the management of asthma. Expert panel report 3: guidelines for the diagnosis and management of asthma. US Department of Health and Human Services, National Institutes of Health, National Heart, Lung, and Blood Institute, 2007. [Google Scholar]
- 16.Australian Centre for Asthma Monitoring. Australian asthma indicators: five-year review of asthma monitoring in Australia. Canberra: Australian Institute of Health and Welfare, 2007. [Google Scholar]
- 17.Praena-Crespo M, Ruiz-Canela J, Aquino N et al. Evidence-based asthma indicators for primary care using RAND method. Eur Respir J 2013;42(Suppl 57):3834. [Google Scholar]
- 18.Tanne J. AMA develops measures of doctors’ performance. BMJ 2006;332:506–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.To T, Guttmann A, Lougheed MD et al. Evidence-based performance indicators of primary care for asthma: a modified RAND Appropriateness Method. Int J Qual Health Care 2010;22:476–85. 10.1093/intqhc/mzq061 [DOI] [PubMed] [Google Scholar]
- 20.Assurance. HEDIS 2014 volume 2: technical update. Washington DC: National Committee for Quality Assurance, 2014. [Google Scholar]
- 21.National Heart, Lung, and Blood Institute. National Asthma Education and Prevention Program Expert Panel report 2: guidelines for the diagnosis and management of asthma. NIH Report 1997(97–4051).
- 22.Aronson AR.Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. Proceedings of the AMIA Symposium American Medical Informatics Association, 2001. [PMC free article] [PubMed] [Google Scholar]
- 23.Choi S, Choi J, Yoo S et al. Semantic concept-enriched dependence model for medical information retrieval. J Biomed Inform 2014;47:18–27. 10.1016/j.jbi.2013.08.013 [DOI] [PubMed] [Google Scholar]
- 24.Osborne JD, Lin S, Zhu LJ et al. Mining biomedical data using MetaMap Transfer (MMtx) and the Unified Medical Language System (UMLS). Methods Mol Biol 2007;408:153–69. 10.1007/978-1-59745-547-3_9 [DOI] [PubMed] [Google Scholar]
- 25.Jimeno Yepes A, Berlanga R. Knowledge based word-concept model estimation and refinement for biomedical text mining. J Biomed Inform 2015;53:300–7. 10.1016/j.jbi.2014.11.015 [DOI] [PubMed] [Google Scholar]
- 26.Pratt W, Yetisgen-Yildiz M. A study of biomedical concept identification: MetaMap vs. people. AMIA Annu Symp Proc 2003;2003:529–33. [PMC free article] [PubMed] [Google Scholar]
- 27.Cohen J. A coefficient of agreement for nominal scales. Educ Psycholo Meas 1960;20:37–46. 10.1177/001316446002000104 [DOI] [Google Scholar]
- 28.Richardson WS, Wilson MC, Nishikawa J et al. The well-built clinical question: a key to evidence-based decisions. ACP J Club 1995;123:A12–13. [PubMed] [Google Scholar]