Abstract
The paradigm of evidence-based medicine (EBM) recommends that physicians formulate clinical questions in terms of the problem/population, intervention, comparison, and outcome. Together, these elements comprise a PICO frame. Although this framework was developed to facilitate the formulation of clinical queries, the ability of PICO structures to represent physicians’ information needs has not been empirically investigated. This paper evaluates the adequacy and suitability of PICO frames as a knowledge representation by analyzing 59 real-world primary-care clinical questions. We discovered that only two questions in our corpus contain all four PICO elements, and that 37% of questions contain both intervention and outcome. Our study reveals prevalent structural patterns for the four types of clinical questions: therapy, diagnosis, prognosis, and etiology. We found that the PICO framework is primarily centered on therapy questions, and is less suitable for representing other types of clinical information needs. Challenges in mapping natural language questions into PICO structures are also discussed. Although we point out limitations of the PICO framework, our work as a whole reaffirms its value as a tool to assist physicians practicing EBM.
Introduction
Clinicians have 0.7 to 18.5 questions for every 10 patients they care for.1,2 However, answers to two-thirds of the questions are either not pursued or pursued but not found.3,4 Subsequent analyses show that almost all unanswered questions could be answered through improved query formulation and better search.2 Therefore, helping physicians articulate their clinical information needs through well-built, focused questions has become one of the focal points of evidence-based medicine (EBM). 5 EBM provides an explicit framework for formulating a patient-specific clinical question.6 According to its guidelines, articulating a clinical question in terms of its four anatomic parts—Problem/Population, Intervention, Comparison, and Outcome (PICO)—facilitates searching for a precise answer.
This study investigates the suitability of the PICO frame as a knowledge representation for clinical questions posed in natural language by practicing physicians. To our knowledge, no researcher has studied the adequacy and flexibility of the PICO representation and whether it is complete in terms of being able to capture salient characteristics of clinical questions. We studied these issues by manually mapping real-world clinical questions into PICO frames and examining the results.
Background
One common approach to understanding the nature of clinical information needs is to collect and classify real clinical questions from physicians. Such analyses have yielded question taxonomies at varying levels of details.7,8 In addition, these studies have shown that not all question types occur with the same frequency: a large fraction of clinical questions can be “covered” by a smaller set of “question templates”. This distribution can be leveraged to guide system development and to better organize evidence resources. Nevertheless, previous studies characterize clinical questions broadly and do not explicitly take into account the principles of EBM.
The formulation of a focused clinical question containing well-articulated PICO elements is widely believed to be the key to efficiently finding high-quality evidence and also the key to evidence-based decisions.6,9 PICO frames originally developed for therapy questions were later extended to all types of clinical questions. 10 Empirical studies have shown that the use of PICO frames improves the specificity and conceptual clarity of clinical problems,11 elicits more information during pre-search reference interviews, leads to more complex search strategies, and yields more precise search results.12
There are few studies that examine the usability and acceptability of the PICO framework in general, and even less prior work on PICO applications in computerized information retrieval systems. A small questionnaire-based study reported that subjects considered a PICO interface for handhelds easy to use and useful in searching MEDLINE.13 However, the use of PICO-structured frames does not always translate into higher satisfaction.11,12
To better understand the adequacy and flexibility of the PICO framework as a knowledge representation, we coded a set of real-world questions asked by physicians into PICO frames. Through the mapping process and subsequent analysis, we addressed the following research questions:
How well are real-world clinical questions structured according to PICO standards?
How suitable is the PICO frame as a knowledge representation for clinical questions?
What concepts and relationships are not adequately captured by the PICO representation?
Is the PICO representation equally suitable for representing different types of clinical questions?
Methodology
Data Collection
We gathered 59 real-world clinical questions from two on-line sources: Family Practice Inquiries Network (FPIN)* and Parkhurst Exchange.† The question collection process was guided by typical instance sampling14 rather than random sampling, because the goal was not to obtain a fully representative, but rather a typical sample of clinical questions. According to the literature, approximately 33% of questions asked by clinicians are about treatment, 25% about diagnosis, and 15% about pharmacotherapy. Together, they account for over 70% of clinicians’ questions.7,11 Guided by this distribution, four types of clinical questions were gathered: therapy (25), diagnosis (15), prognosis (7), and etiology (12).
Coding Clinical Questions with PICO
The gathered questions were coded into PICO frames independently by the first and the third author (with backgrounds in library science and medicine, respectively). The process of comparing and reconciling the coded PICO frames was guided by the second author. This being an exploratory study and the first of its type that we are aware of, the primary purpose of independent coding was to preserve multiple perspectives, rather than to enforce uniformity for the sake of measuring inter-coder agreement. Therefore, no formal instructions or protocol beyond standard EBM guidelines were given to the coders.
Analysis of the Results
Our collection of 59 questions was first evaluated for structural completeness. Based on previous work, which found that clinical questions were less likely to go unanswered when the question identified the proposed intervention and desired outcome,15 we used the presence of these elements as an indication of the structural completeness of a question.
We then analyzed the prevalence of each PICO element. This analysis gave rise to structural frame patterns that represented prototypical therapy, diagnosis, etiology, and prognosis questions. In addition, semantic classes of concepts present in the 59 clinical questions were identified. This allowed us to construct the typical mapping relationships between semantic entities and PICO elements.
Finally, challenges encountered during the process of coding these clinical questions were gathered and organized into themes. This yielded a qualitative evaluation regarding the adequacy of PICO as a knowledge representation for clinical questions.
Results
Structural Completeness of Clinical Questions
In our collected corpus, only two out of 59 questions specify all four PICO elements and 37.3% of questions contain only intervention and outcome. Table 1 provides an overview of how often different PICO elements are found in each question type.
Table 1.
Structural Completeness of Clinical Questions, By Type
| A | B | C | |
|---|---|---|---|
| Therapy | 25 | 1 (4.0%) | 16 (64.0%) |
| Diagnosis | 15 | 1 (6.7%) | 5 (33.3%) |
| Etiology | 12 | 0 (0%) | 0 (0%) |
| Prognosis | 7 | 0 (0%) | 2 (3.4%) |
| Total | 59 | 2 (3.4%) | 22 (37.3%) |
A: # of questions in each type;
B: # of questions with all the elements;
C: # of questions with intervention & outcome
Independent of question type, the problem/population and intervention slots are the most frequently addressed PICO elements (50 and 49 out of 59 respectively), followed by population (29 out of 59), then by outcome (27 out of 59). In contrast, comparison is rarely mentioned (only 3 out of 59).
Prototypical PICO Representations
Manual mapping of clinical questions into PICO representations allows us to derive prototypical PICO structures.
All 25 therapy questions in our collection contain an identifiable intervention. All but two therapy questions describe the problem, the population, or both. Overall, 64% of the questions provide explicit statements of desired outcomes. Structural patterns of therapy questions and their frequencies are shown in Table 2. A question mark denotes the answer element, e.g., [O?] indicates that an outcome serves as the answer to the question.
Table 2.
Structural Patterns and Examples for 25 Therapy Questions
| Pattern | Example |
|---|---|
| [P][I][O?] (10) | Could stimulants be useful for chronic fatigue syndrome? |
| [P][I?] (8) | What is the best treatment for analgesic rebound headaches? |
| [I][O?] (2) | What protective effects do vitamins E, C, and beta carotene have on the cardiovascular system? |
| [P][I?][O] (2) | What regimens eradicate Helicobacter pylori? |
| [P][I][C][O?] (2) | Do acetaminophen and an NSAID combined relieve osteoarthritis pain better than either alone? |
| [I?] (1) | What is the most effective nicotine replacement therapy? |
For diagnosis questions, emphasis is placed on symptoms, coded in the population slot (they appear in 11 of 15 questions), hypothesized disease, coded in the problem slot (12 of 15), and diagnostic approach, coded in the intervention slot (10 of 15). In particular, one third of the diagnosis questions contain exactly the population and problem elements, e.g., “What is the differential diagnosis of chronic diarrhea in immunocompetent patients?” Although the PICO framework collapses the two “P’s” (population and problem), we discovered a need to explicitly distinguish between the two elements in diagnosis questions: “population” is used to describe the patient’s symptoms, while “problem” is used to describe the hypothesized disease.
The structure of the 12 etiology questions examined in this study is homogenous. All questions describe the problem and inquire about its etiology, following the pattern [P][I?], e.g., “What are the causes of hypomagnesemia?” Although counter-intuitive, causes are best captured in the intervention slot (see discussion section for more detail).
Prognosis questions focus on patient outcomes, given a diagnosed problem or a patient profile (population). Various structural patterns are shown in Table 3.
Table 3.
Structural Patterns and Examples for 7 Prognosis Questions (P1: problem P2: population)
| Pattern | Example |
|---|---|
| [P1][O?] (5) | What is the prognosis for acute low back pain? |
| [P1][P2][I][O?] (1) | A patient with stable creatinine and IgA Urology after a renal biopsy. His blood pressure and proteinuria are normal while he takes his enalapril. What is his prognosis? |
| [P2][O?] (1) | What is the prognosis for chronic active hepatitis, cirrhosis, and hepatoadenocarcinoma in an active asymptomatic 45- year-old man with no history of illness, strongly positive result for HBsAg and practically none for HBsAb ? |
Mapping of Semantic Classes
To investigate how specific semantic classes relate to PICO elements, we manually clustered concepts into semantically-related categories, which loosely correspond to the UMLS16 semantic types:
Problem
[Disease], e.g., “panic disorder”
[Behavior], e.g., “oppositional behaviors”
[Symptom], e.g., “leg cramps”
Population
[Age], e.g., “40-year- old”
[Gender], e.g., “male”
[Treatment Status], e.g., “delayed treatment”
[Physical Condition], e.g., “healthy”
[Medical History], e.g., “with prior attacks”
[Treatment & Drug], e.g., “taking hormone replacement therapy”
[Disease], e.g., “nonvalvular atrial fibrillation”
[Symptom], e.g., “chronic cough”
Intervention & Comparison
[Treatment & Drug], e.g., “warfarin”
[Procedure], e.g., “transvaginal ultrasound”
[Diagnostic Test], e.g., “Pap smear”
[Exposure], e.g., “maternal smoking”
[Disease], e.g., “a flare-up of the Crohn’s”
[Symptom], e.g., “a very low serum iron”
Outcome
[Treatment Outcome], e.g., “fibroid volume reduction”
[Patient Outcome], e.g., “decreased mortality”
We note that many semantic classes show strong, predictable associations with specific PICO elements. For example, [Age], [Gender], [Treatment Status], [Physical Condition], and [Medical History] are always mapped to the population slot. On the other hand, some semantic classes can be mapped to more than one PICO slot. Semantic classes such as [Treatment & Drug], [Disease], and [Symptom] take different roles for different question types and their mapping heavily depends on context. For example, [Treatment & Drug] is considered an intervention in the context of a therapy question, but may be part of the population in a prognosis question, i.e., a woman on hormone replacement therapy. The potential confusion in the associations between semantic classes and PICO elements presents a potential barrier to the formulation of clear clinical questions.
Discussion
As shown in Table 1, only 22 of 59 questions in our study contain both the intervention and outcome elements. This confirms the findings of Bergus et al.15 who discovered that few real-world clinical questions meet the minimum structural requirements for facilitating precise searches (i.e., contain identifiable intervention and desired outcome). In our corpus, therapy questions (64%) are most likely to be structured with both intervention and outcome, followed by diagnosis questions (33.3%). Prognosis and etiology questions are least structured (14.3% and 0%, respectively).
Challenges in Structuring Clinical Questions
Our study reveals a number of challenges in applying the PICO framework to analyzing clinical questions:
Inability to reconstruct the original question
Given a PICO frame, can we recover the original clinical question? Often, the answer is “no”. For example, does the representation [Problem: hypomagnesemia, Intervention: ?] correspond to “What is the most effective treatment for hypomagnesemia?” or “What are causes of hypomagnesemia?” This ambiguity, however, is easily resolved if the clinical task, e.g., therapy or etiology, is known. However, this suggests that the clinical task is an essential component of PICO representations, which would require a minor modification to the existing framework.
Inability to encode fine-grained relationship between frame elements
Consider the following question:
Is there any evidence to show that selective serotonin reuptake inhibitor (SSRI) use carries a risk of impulsive suicidal or homicidal behaviour, or is it just a case of association, in that those most likely to perform such acts are also most likely on mood-stabilizing medications for their underlying psychopathology?
It is difficult to represent the above question in a PICO frame without losing the fine-grained semantic relationships between concepts. The PICO representation mainly relies on inherent semantic relationships between concepts to connect different elements. For example, with etiology questions, the connection between interventions and problems is assumed to be causal. Thus, the PICO frame is ill-suited to questions that challenge these implicit relations.
No explicit temporal/state model
The PICO frame describes the state of affairs at a frozen point in time. However, temporal progress is a salient element of many clinical questions,17 and temporal concepts are often critical to retrieving precise results. For “medication states”, we can work around this problem by interpreting it as a part of patient profile, i.e., population, as in the following question:
What is the interval for monitoring warfarin therapy once therapeutic levels are achieved?
Population: therapeutic levels are achieved
Consider another common use of temporal concepts, as illustrated with the question: “Are there any advances in the treatment of motion sickness since 90s?” The PICO framework contains no provisions for capturing such temporal qualifications. At present, physicians must consider metadata requirements beyond the PICO frame in formulating their searches, e.g., restricting searches to specific publication dates.
Overloaded slots
Certain types of clinical questions cannot be intuitively captured by the existing PICO framework. For example, the standard PICO frame combines problem and population into a single “P” element. However, for diagnosis questions, as mentioned earlier, the most common structural pattern consists of a population and a hypothesized disease. To represent such questions, the “P” slot needs to be more finely articulated, explicitly separating problem from population. Otherwise, it would be problematic for questions like:
How would you manage a woman with brownish discharge from one of her breasts? She is pre-menopausal (less than 50 years old)
Another limitation of the PICO framework occurs with etiology questions, which, in our collection, all inquire about causes of diseases. Naturally, the disease fills the problem slot. But in what slot does the cause belong? Intervention is the closest match, but this placement is highly counter-intuitive. The intervention is generally thought of as “something done” to affect the problem, as in treating a disease with a drug. The encoding of etiology questions reverse the direction of causality normally associated with other question types. This is a cause for potential confusion in the formulation of well-defined clinical queries.
Inability to Capture Anatomical Relations
The PICO representation is unable to capture anatomical relations that may be relevant in a clinical question. Questions involving human anatomy are quite common, for example:
What protective effects do vitamins E, C, and beta carotene have on the cardiovascular system?
Quite simply, there isn’t a slot in the PICO framework capable of capturing “body parts”. Given the small size of our sampled questions, it is difficult to determine whether there are more concepts in real-world clinical questions that are not covered by the PICO frame.
Summary
Our study shows that the PICO framework is best suited for representing therapy questions, and considerably less-suited for diagnosis, etiology, and prognosis questions. In some cases, it is difficult to encode certain question classes without modifying the existing PICO structure or introducing counterintuitive elements. Given that the PICO framework is a well-established tool for formulating clinical queries, any limitations of the framework itself could potentially impact the quality of clinical evidence retrieved under its guidance. This study reveals a number of challenges associated with PICO analysis, which will serve as a basis for refining the principles of clinical query formulation.
Conclusions
This study investigated the adequacy and suitability of PICO as a knowledge representation for clinical questions. Our exploration focused on a manual analysis of 59 real-world clinical questions drawn from online sources. Overall, results reaffirm the value of the PICO framework as a method for structuring clinical questions, since natural language questions were found to lack the elements that comprise a well-formed query in most cases. Nevertheless, we encountered many challenges in employing PICO frames as a representation for clinical information needs. A better understanding of the advantages and limitations of this framework will translate into more effective strategies for retrieving relevant clinical evidence. We hope that these insights will ultimately translate into next-generation retrieval systems that leverage computational models of evidence-based medicine to automatically answer clinical questions.18
Footnotes
References
- 1.Ely JW, Osheroff JA, Chambliss ML, Ebell MH, Rosenbaum ME. Answering physicians’ clinical questions: Obstacles and potential solutions. J Am Med Inform Assoc. 2005 Mar–Apr;12(2):217–24. doi: 10.1197/jamia.M1608. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Gorman PN, Helfand M. Information seeking in primary care: How physicians choose which clinical questions to pursue and which to leave unanswered. Med Decis Making. 1995 Apr–Jun;15(2):113–9. doi: 10.1177/0272989X9501500203. [DOI] [PubMed] [Google Scholar]
- 3.Chambliss ML, Conley J. Answering clinical questions. J Fam Pract. 1996 Aug;43(2):140–144. [PubMed] [Google Scholar]
- 4.Currie LM, Graham M, Allen M, Bakken S, Patel V, Cimino JJ. Clinical information needs in context: an observational study of clinicians while using a clinical information system. AMIA Annu Symp Proc. 2003:190–4. [PMC free article] [PubMed] [Google Scholar]
- 5.Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996 Jan 13;312(7023):71–2. doi: 10.1136/bmj.312.7023.71. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP J Club. 1995 Nov–Dec;123(3):A12–3. [PubMed] [Google Scholar]
- 7.Ely JW, Osheroff JA, Gorman PN, Ebell MH, Chambliss ML, Pifer EA, Stavri PZ. A taxonomy of generic clinical questions: classification study. BMJ. 2000 Aug 12;321(7258):429–32. doi: 10.1136/bmj.321.7258.429. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Jerome RN, Giuse NB, Gish KW, Sathe NA, Dietrich MS. Information needs of clinical teams: analysis of questions received by the Clinical Informatics Consult Service. Bull Med Libr Assoc. 2001 Apr;89(2):177–84. [PMC free article] [PubMed] [Google Scholar]
- 9.Ebell MH. Information at the point of care: answering clinical questions. J Am Board Fam Pract. 1999 May–Jun;12(3):225–35. doi: 10.3122/jabfm.12.3.225. [DOI] [PubMed] [Google Scholar]
- 10.Armstrong EC. The well-built clinical question: The key to finding the best evidence efficiently. WMJ. 1999 Mar–Apr;98(2):25–8. [PubMed] [Google Scholar]
- 11.Cheng GY. A study of clinical questions posed by hospital clinicians. J Med Libr Assoc. 2004 Oct;92(4):445–58. [PMC free article] [PubMed] [Google Scholar]
- 12.Booth A, O’Rourke AJ, Ford NJ. Structuring the pre-search reference interview: a useful technique for handling clinical questions. Bull Med Libr Assoc. 2000 Jul;88(3):239–46. [PMC free article] [PubMed] [Google Scholar]
- 13.Fontelo P, Nahin A, Liu F, Kim G, Ackerman MJ. Accessing MEDLINE/PubMed with Handheld Devices: Developments and New Search Portals. Proc. of the 38th Hawaii International Conference on System Sciences. 2005:1–5. [Google Scholar]
- 14.Lindlof TR, Taylor BC. Qualitative Communication Research Methods. 2. Thousand Oaks, CA: Sage Publications; 2002. [Google Scholar]
- 15.Bergus GR, Randall CS, Sinift SD, Rosenthal DM. Does the structure of clinical questions affect the outcome of curbside consultations with specialty colleagues? Arch Fam Med. 2000 Jun;9(6):541–7. doi: 10.1001/archfami.9.6.541. [DOI] [PubMed] [Google Scholar]
- 16.Lindberg DAB, Humphreys BL, McCray AT. The Unified Medical Language System. Meth. Inform. Med. 1993;32:281–291. doi: 10.1055/s-0038-1634945. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Green ML, Ruff TR. Why do residents fail to answer their clinical questions? A qualitative study of barriers to practicing evidence-based medicine. Acad Med. 2005 Feb;80(2):176–82. doi: 10.1097/00001888-200502000-00016. [DOI] [PubMed] [Google Scholar]
- 18.Demner-Fushman D, Lin J. Computational Linguistics 2006. Answering Clinical Questions with Knowledge-Based and Statistical Techniques. in press. [Google Scholar]