Skip to main content
NCBI home page
Annals of Family Medicine logoLink to Annals of Family Medicine
. 2013 Nov;11(6):559–567. doi: 10.1370/afm.1565

Number Needed to Benefit From Information (NNBI): Proposal From a Mixed Methods Research Study With Practicing Family Physicians

Pierre Pluye 1,✉, Roland M Grad 1, Janique Johnson-Lafleur 2, Vera Granikov 2, Michael Shulha 2, Bernard Marlow 3, Ivan Luiz Marques Ricarte 4
PMCID: PMC3823727  PMID: 24218380

Abstract

PURPOSE

We wanted to describe family physicians’ use of information from an electronic knowledge resource for answering clinical questions, and their perception of subsequent patient health outcomes; and to estimate the number needed to benefit from information (NNBI), defined as the number of patients for whom clinical information was retrieved for 1 to benefit.

METHODS

We undertook a mixed methods research study, combining quantitative longitudinal and qualitative research studies. Participants were 41 family physicians from primary care clinics across Canada. Physicians were given access to 1 electronic knowledge resource on handheld computer in 2008–2009. For the outcome assessment, participants rated their searches using a validated method. Rated searches were examined during interviews guided by log reports that included ratings. Cases were defined as clearly described searches where clinical information was used for a specific patient. For each case, interviewees described information-related patient health outcomes. For the mixed methods data analysis, quantitative and qualitative data were merged into clinical vignettes (each vignette describing a case). We then estimated the NNBI.

RESULTS

In 715 of 1,193 searches for information conducted during an average of 86 days, the search objective was directly linked to a patient. Of those searches, 188 were considered to be cases. In 53 cases, participants associated the use of information with at least 1 patient health benefit. This finding suggested an NNBI of 14 (715/53).

CONCLUSION

The NNBI may be used in further experimental research to compare electronic knowledge resources. A low NNBI can encourage clinicians to search for information more frequently. If all searches had benefits, the NNBI would be 1. In addition to patient benefits, learning and knowledge reinforcement outcomes are frequently reported.

Keywords: electronic knowledge resources, family physicians, information assessment method, information management, information storage and retrieval, mixed methods research, number needed to benefit from information, patient-centered care, patient health outcomes

INTRODUCTION

Electronic knowledge resources help physicians find answers to clinical questions, thus enabling the use of clinical information in routine professional practice. Hereinafter, clinical information refers to research-based articles or summaries or synopses, and knowledge syntheses for clinical practice, eg, guidelines—not patient-specific data.1–4 In observational studies, information retrieved from electronic resources has been associated with a positive cognitive impact on physicians.5 No study, however, has systematically described patient health outcomes associated with the use of information directly retrieved by family physicians.6 Our research question therefore was for how many patients does a physician need to retrieve clinical information from electronic resources to report a health benefit for 1?

We critically reviewed 29 qualitative and quantitative studies on information use and subsequent patient health outcomes.6 No study was grounded in a conceptual framework or a theoretical model. No study systematically tracked information objects retrieved by physicians, such as Web pages, and comprehensively assessed their use and subsequent benefits. Moreover, these 29 studies reported qualitative and quantitative data separately. None systematically linked quantitative data by tracking information use in log files with qualitative data to produce trustworthy clinical stories.

In contrast, we conducted a pilot mixed methods study where searches conducted by 17 family medicine residents were systematically tracked for 2 months.7 Among 84 searches for patients, 12 were associated with patient health benefits (reported by residents). This finding suggested a new concept we refer to as the number needed to benefit from information (NNBI). We define the NNBI as the number of patients for whom clinical information (eg, the answers to physicians’ clinical questions) was retrieved for 1 patient to benefit. In our pilot study, the NNBI was 7.

The purpose of the present mixed methods study is twofold. We first describe family physicians’ use of information from an electronic knowledge resource for answering clinical questions and their perception of subsequent patient health outcomes, and we then estimate the NNBI. Our study is based on a theoretical model and merges complementary quantitative and qualitative data on searches done by practicing physicians for clinical information. Given that one-half of all clinical questions in practice are not pursued by family physicians,8,9 the wealth of information available to guide primary health care practice remains unused to its full potential. Thus, knowing that their patient can benefit may stimulate physicians to search more often when they feel the need.

METHODS

We undertook a mixed methods study10 in which we combined a quantitative longitudinal study with a qualitative multiple case study,11 the design of which is described in Figure 1. Cases were searches for information by practicing physicians where information was used for specific patients. To obtain a systematic and comprehensive description of these cases, quantitative and qualitative data were integrated at both collection and analysis stages.12 Both types of data were merged into clinical stories (hereinafter called vignettes). Concepts and terms from mixed methods research and information studies are defined in Supplemental Appendix 1, available at www.annfammed.org/content/11/6/559/suppl/DC1, and a detailed methods section is available in Supplemental Appendix 2.

Figure 1.

Figure 1

Open in a new tab

Mixed methods convergence design.

IAM=Information Assessment Method.

This study is based on a theoretical model called the acquisition-cognition-application/levels-of-outcomes model (hereinafter the ACA-LO model),13 which can be summarized as follows. Clinicians may find one or more information objects to fulfill a search objective, eg, a Web page for addressing a clinical question (acquisition/level 1). Next, they integrate with their previous knowledge the information objects deemed relevant for achieving this objective (cognition/level 2). Only a fraction of this information may be used for 1 specific patient (application/level 3). On occasion, 1 information object is applied to the management of this patient and associated with health benefits (patient outcomes/level 4).

In the quantitative longitudinal study, 41 Canadian family physicians used Essential Evidence Plus (Wiley-Blackwell) on a handheld computer to search for clinical information, and rated their searches using the Information Assessment Method (IAM), which operationalizes the ACA-LO model (www.mcgill.ca/iam).14,15 For all searches, physicians were prompted to complete the 2007 version of the IAM questionnaire (Figure 2).

Figure 2.

Figure 2

Open in a new tab

The 2007 Information Assessment Method (IAM) questionnaire.

In the multiple case study, qualitative data consisted of observations, log reports, archives, and interviews with respect to a subsample of most recently rated searches conducted over an average of 86 days. Physicians were then interviewed on 2 occasions regarding their most recent searches (midstudy and end of study). Before each interview, log reports containing IAM ratings were analyzed by the interviewer to identify searches rated as “I used (or will use) this information for a specific patient”; then, to stimulate recall of past events, log reports and IAM ratings were reviewed by telephone with each physician. Cases were critical searches that were identified using the critical incident technique.16–18 In line with this technique, a search was critical when the clinical situation and the use of information were clearly described.

We then analyzed data on critical searches (cases) to describe physicians’ use of clinical information and subsequent effects on patient health. We integrated qualitative and quantitative data using specialized software (NVivo 7, QSR International). For each case, data were comprised of content from the information objects, ie, pages from Essential Evidence Plus, the corresponding log file containing IAM ratings, and interview transcripts. We combined a deductive with an inductive thematic analysis.19,20 We assigned data to predefined themes derived from IAM items, such as types of information use and health benefits. These items were based on prior literature reviews, qualitative studies, and mixed methods studies.14 Data were also analyzed to explore new themes (emerging types of information use and benefits). A vignette contains both qualitative and quantitative data on the same case. Finally, to estimate the NNBI, we divided the total number of searches for a specific patient (reported via IAM ratings) by the number of critical searches with at least 1 patient health outcome (reported in vignettes).

Ethics approval (A09-E28-07B) was obtained from the McGill University Institutional Review Board.

RESULTS

We interviewed 39 physicians from 9 of 10 Canadian provinces about their searches. Twenty-six (66.7%) reported that they had a faculty appointment and were involved in teaching or research. There were 23 men and 16 women, all in active practice, ranging in age from 28 to 70 years (median = 44 years). Over an average of 320 days, 2,131 searches for clinical information were documented, and 1,767 (82.9%) searches were rated; of those, 1,193 (56.0%) were covered by interviews (Figure 3). In 715 (59.9%) of 1,193 searches, the search objective was directly linked to a patient (to address a clinical question, or to share information with a patient, or to plan/manage aspects of patient care with other health professionals). Of 365 (30.6%) searches rated “I used (or will use) this information for a specific patient,” 188 (15.8%) were critical searches. The 188 vignettes corresponding to these cases are available online (iamclinicalvignettes.mcgill.ca). A sample vignette is shown in Table 1. Results are presented in accordance with the 188 cases.

Figure 3.

Figure 3

Open in a new tab

Mixed methods data collection: flow diagram.

IAM=Information Assessment Method.

Table 1.

Avoidance of a Medical Test Associated With Found Information: A Clinical Vignette

MD08S03: Celiac disease
Bottom line: Information on celiac disease was used to justify the management of the patient (no testing), and it contributed to avoid an unnecessary diagnostic procedure.
Acquisition: On May 30, 2008 [quantitative data], MD08 did a search at work, with a patient, and during the encounter [qualitative data]. They retrieved one information hit about celiac disease: The reported search objectives were: to address a clinical question and to share the information with the patient [quantitative data]. “[I was] looking up the utility of doing the test in someone who did not have symptoms; […] a first degree relative [of the patient] had been diagnosed with celiac disease; […] [the patient] was asking some questions for herself, as to whether some of her symptoms from her gastrointestinal tract could have been caused by this [disease] […] and she was asking me about doing the testing. […] She was pretty much asymptomatic, but she had this family member, and she wanted to know whether she could have the same problem.[…]It wasn’t very clear whether people who had the specific test […], transaminase antibody, whether just having the positive antibody, without symptoms, whether those people would be considered a false positive, whether it really meant anything. […] She [the patient] had a question about it. I mean she was sharing information with me and I think was asking for my feedback” [qualitative data]. According to MD08, Essential Evidence Plus (EE+) was the only source of information, and the found information was relevant [qualitative data].
Cognition: One hit was associated with a report of highly positive cognitive impact (practice improvement) [quantitative data]. “I learned something about the sort of nuances interpreting these antibody results, […] knowing when to use it [the test] and also how to interpret the result. I think that was the [practice] improvement part of it” [qualitative data].
Retrieved information hit(s):
EBMG clinical topics: Celiac Disease (MD08S03H01) [quantitative data]
Application: Information on celiac disease was used for a patient [quantitative data] to justify their management (information used as presented in EE+) [qualitative data]. “It was basically to discuss the significance of her having a positive result, but not having symptoms; […] Just because you have a positive test doesn’t mean that you have the disease. That’s the message that I gave her.”
Patient outcomes: Regarding patient health, the information contributed to avoid an unnecessary diagnostic procedure. “[The information had an impact] in terms of not having to go into further testing” [qualitative data].
Open in a new tab

Searches were conducted before, during, and/or after the patient-physician encounter in 21 (11.2%), 71 (37.8%), and 98 (52.1%) cases, respectively. The sum is superior to 100% because in 2 cases the same information was consulted twice: once with the patient and once without, ie, during and after seeing the patient. Before the encounter referred to situations where physicians prepared for the patient visit. After the encounter referred, for instance, to situations where physicians reviewed laboratory results after the visit. Searches were conducted in a clinic or office setting in 146 (77.7%) cases, or outside a professional setting, eg, at the physicians’ home, in 42 (22.3%) cases. Physicians searched most often by themselves, less commonly in the presence of the patient, or with a colleague in 127 (67.5%), 34 (18.1%), and 27 (14.4%) cases, respectively. In 133 (70.7%) cases, Essential Evidence Plus was the only resource consulted by the physician. When another source was consulted, interviewees consulted 1 or more than 1 type of source, such as another electronic resource, textbooks or journals, and other professionals in 41 (21.8%), 11 (5.9%), and 4 (2.1%) cases, respectively.

In all cases, information from Essential Evidence Plus was relevant and met the physician’s objective. The most frequently reported search objective was to address a clinical question about a specific patient (181; 96.3%). The most frequently reported cognitive impact was confirming that the physician did the right thing (132; 70.2%). The most frequent use of the information was confirming that the patient was being managed correctly (109; 58.0%). The most frequently anticipated health benefit was to avoid an inappropriate diagnostic procedure or treatment (28; 14.9%).

These 28 perceived benefits resulted from 4 types of patient advice: (1) the patient was not asked to do something that the physician had planned to do before the search (20; 10.6%), eg, consult another specialist; (2) the patient was asked to avoid something (4; 2.1%), eg, planned use of a natural product that could interact with prescribed medication; (3) the patient was asked to do something different, compared with what was planned before the search (2; 1.1%), eg, to reduce a drug dose; and (4) the patient was asked to do something more appropriate, which the physician had not planned to do before the search (2; 1.1%), eg, further testing.

Paths and NNBI

The 188 vignettes are represented in the path diagram (Figure 4), where paths are characterized by types of patient outcome (patient behavior vs patient knowledge or satisfaction vs no outcome). The 53 vignettes with outcomes are listed in Supplemental Appendix 3 (www.annfammed.org/content/11/6/559/suppl/DC1). Of those vignettes, 42 describe types of patient behavior outcome: in 28 cases (14.9%), interviewees associated the use of retrieved information with avoidance of an unnecessary or inappropriate diagnostic procedure, treatment, or preventive intervention; in 18 cases (9.6%), interviewees associated information use with prevention of disease or health deterioration; and in 13 cases (6.9%), interviewees associated the use of retrieved information with improvement of patient health or functioning or resilience. In summary, interviewees conducted 715 searches for a specific patient and associated the retrieved clinical information with health benefits in 53 (7.4%), ie, they retrieved information on average for 13.5 patients to report health benefits for 1. This outcome suggests an NNBI of 14.

Figure 4.

Figure 4

Open in a new tab

Physician perceptions of patient outcomes.

Note: This diagram and corresponding vignettes are available at iamclinicalvignettes.mcgill.ca.

DISCUSSION

Based on a new concept (NNBI) and a mixed methods study, our results suggest how frequently patients may benefit when their physicians search for information in electronic knowledge resources. The NNBI will vary depending on the type of electronic knowledge resource and the clinicians’ characteristics, such as information-seeking skills (acquisition), knowledge (cognition), and behavior in specific clinical situations (application). In a concomitant study with similar mixed methods, we documented searches for treatment recommendations (summaries) conducted by 30 primary health care professionals (10 nurse practitioners, 10 family health team pharmacists, and 10 family medicine residents), using another resource (e-Therapeutics+) on a desktop computer: the NNBI was lower (NNBI = 7), ie, expected benefits were more frequently reported.13 In our pilot study involving family medicine residents, the NNBI was also 7.

In a qualitative research study, Australian researchers interviewed 29 clinicians (13 hospital-based physician specialists and 16 clinical nurse consultants) about their use of an electronic knowledge resource called the Clinical Information Access Program.21 Interviewees described 85 searches. Of those, 19 (22.4%) were associated with patient health benefits, such as health improvement (eg, quality of life improved) and avoidance of an unnecessary procedure or medication. Their findings suggests an NNBI of 4. The main reason for the difference between this lower (better) NNBI compared with our results might be that in the Australian study, interviewees were asked to recall examples of searches that were helpful or unhelpful in their clinical work. Thus, participants selectively reported on searches with potential health benefits. In contrast, in our study all searches were systematically tracked whatever their helpfulness, and interviews of clinicians were informed by a log report of rated searches.

In the present study, our estimation of the NNBI may be too high for 3 reasons. First, to be conservative, we considered that forgotten searches never led to patient benefits (n = 130) (Figure 3). Forgotten searches that were beneficial for patients would have led to a lower NNBI. Second, we interviewed participants only once with respect to only 1 patient, but follow-up times needed to observe a health benefit vary as outcomes are time dependent.22 Had participants observed benefits for patients after the interview, the NNBI would have been lower. In addition, longer follow up on all searches could yield additional evidence of benefit.23 Longer follow-up may also allow the detection of other patients for which the same information could be used; observed benefits for other patients would have also led to lowering the NNBI. Third, it is likely the 2008 version of Essential Evidence Plus was somewhat difficult to use for searches on a small hand-held computer. Our results indicate that participants rated about only 1 search per week (Figure 3), while some participants reported technical problems, making it likely that searches were occasionally done at a PC workstation (and therefore not tracked). Just after our data collection ended, Essential Evidence Topics was added to Essential Evidence Plus. This resource was specifically created to make it easier for physicians to find relevant information.

One implication of this study is that further improvements are needed to facilitate information seeking at the point of care. Improvements can be done by optimizing search engines and by making access to electronic knowledge resources easier. Such improvements may increase the frequency of searches, the relevance of information for patient management, and patient health benefits. For example, we propose developing a system to make it easier for family physicians to search for answers to their clinical questions within the electronic health record and to document the outcome of these searches using the IAM.24 Such a system will facilitate searches by embedding links to favorite knowledge resources directly within the patient record and help physicians to automatically earn continuing education credit for their searches. Such improvements can make it worth the time and energy to search for answers to clinical questions.

What is encouraging in our work is that searches for information are often associated with educational outcomes for physicians themselves (published elsewhere).15 Considering all participants’ searches (n = 2,131), our results indicate that information was relevant in 76% of rated searches. Our results also indicate that information often had positive cognitive impact. Participants rated 3,300 information objects associated with these searches. Regarding 52% of them (n = 1,708), they reported an actual use or an intention to use for a specific patient. Regarding 46% (n = 1,516), they reported, “This information confirmed I did (will do) the right thing.” Regarding 38% (n = 1,246), they reported, “I learned something new.”15 Although reinforcement of knowledge and learning outcomes do not bring direct benefit to patients, they help physicians to build confidence for decision making.

We propose the NNBI concept by analogy with the number needed to treat (NNT), a widely used concept to communicate the magnitude of treatment effect, defined as the number of patients that need to be treated for 1 to benefit compared with a control in a clinical trial context. In line with this definition, further experimental research can measure the NNBI in a variety of contexts and consider potential adverse effects to arrive at a net NNBI (benefit after deduction of harm). For example, experiments can use the NNBI metric to compare electronic knowledge resources using objective outcome measures, such as data from the electronic medical record.25,26 These resources are expensive, and such a comparison, eg, Essential Evidence Plus vs UpToDate, can help academic institutions and professional organizations select the most efficient resource in their context (resources with an optimal ratio of benefit divided by cost). Information retrieval and use are complex, thus further research may measure composite outcomes (ideally patient-oriented), eg, a combination of different kinds of health benefit associated with answers to therapeutic questions.

The lower the NNBI, the more value physicians will see in looking for information (if all searches had benefits, the NNBI would be 1). One-half of all clinical questions in practice are not pursued by family physicians; thus, a low NNBI can encourage clinicians to search for information systematically when they perceive an information need and may justify further continuing education or financial incentive for searching. Applying what is already known will have an important impact on health care and health. Moreover, a low NNBI can help to stimulate clinicians to take information retrieval training. If used by patients, the NNBI concept may enhance their confidence in asking clinicians or librarians to seek more information in high-stakes situations, eg, a treatment associated with serious adverse effects.

Our research faces at least 2 limitations. First, our results may not generalize to community physicians because of selection biases: (1) two-thirds of study physicians had a faculty appointment and were involved in teaching or research, (2) participants rated relatively few searches considering the broad scope of clinical questions that come up in primary health care, and (3) we described patient health benefits arising from the use of information directly retrieved by practicing family physicians within 1 type of electronic knowledge resource on 1 device (hand-held computer). Second, the NNBI is difficult to estimate because patient health benefits were based on physician perception rather than objective measures, such as blood pressure. Adverse effects that might result from searches for information were not reported (eg, by spending time with 1 patient searching for information, a physician may have less time for other patients with potentially adverse consequences). Our clinical vignettes document physicians’ reported behavior (information use) and their observations or expectations in terms of patient behavior, knowledge, and satisfaction (benefits). We did not prospectively follow patients to determine, for example, whether recommendations made by their physician were implemented. In addition, the association between preventive or therapeutic interventions and observed benefits are at best probabilistic because there was no control group. As mentioned by an interviewee, it is hard to determine for individual patients in routine clinical settings whether a specific intervention contributes to health outcomes.

The strength of our research was the ability to establish a chain of evidence between information retrieved in routine practice and reported benefits for a specific patient. Documenting the relative influence of all sources of information on the process of care and patient outcomes is challenging, and mixed methods allowed us to address this challenge. The combination of quantitative and qualitative data strengthened our work for at least 2 reasons. First, the computerized IAM questionnaire reduced recall bias because IAM-based log reports guided interviews and stimulated participants’ recall.27 Second, the critical incident technique is known to be valid, and the triangulation of multiple sources of qualitative evidence, namely, archives, log reports, interviews, and observations, permitted us to critically examine physician interviews.

In conclusion, we report the first systematic and comprehensive examination of patient health benefits associated with the use of information directly retrieved by practicing family physicians from an electronic knowledge resource. Our mixed methods study supports a new concept, the number needed to benefit from information (NNBI), which can be used in experimental research with objective measures of patient health outcome to compare resources, to identify kinds of clinical questions that are difficult to answer, and to measure benefits of specific searches for information.

Acknowledgments

Authors gratefully acknowledge assistance from Yuejing Meng for the data collection, and from the members of the Information Technology Primary Care Research Group for the discussion.

Footnotes

Conflicts of interest: authors report none.

Funding support: Pierre Pluye holds a New Investigator Award from the Canadian Institutes of Health Research (CIHR) (200609MSH-166957-HAS-CFAA-109294). The present study was funded by the CIHR: Operating grant # 200610KAL-167781-KAL-CFAF5366; and # 200703MOP-173703-KTE-CFAF53066.

Previous presentation: Preliminary findings from this study were presented as a ‘work-in-progress’ poster at the North American Primary Care Research Group Meeting, Seattle, Washington, November 13–17, 2010.

Additional information: The Information Assessment Method (IAM) is protected by Registered Copyrights (2008): # CA 1057518 “A scale to assess the cognitive impact, relevance and use of information hits derived from electronic resources,” and # CA 1057519 “Une échelle pour évaluer l’impact cognitif, la pertinence et l’utilisation des informations issues de ressources électroniques.”

References

  • 1.Magrabi F, Westbrook JI, Coiera EW. What factors are associated with the integration of evidence retrieval technology into routine general practice settings? Int J Med Inform. 2007;76(10):701–709 [DOI] [PubMed] [Google Scholar]
  • 2.McGowan JL, Grad R, Pluye P, et al. Electronic retrieval of health information by healthcare providers to improve practice and patient care. Cochrane Database Syst Rev. 2009;3(3):CD004749 10.1002/14651858.CD004749.pub2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Gagnon MP, Légaré F, Labrecque M, et al. Interventions for promoting information and communication technologies adoption in healthcare professionals. Cochrane Database Syst Rev. 2009;CD006093. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.College of Family Physicians of Canada, Canadian Medical Association, Royal College of Physicians and Surgeons of Canada National Physician Survey 2010. http://www.nationalphysiciansurvey.ca/nps/ Accessed Oct 26, 2011
  • 5.Pluye P, Grad RM, Dunikowski LG, Stephenson R. Impact of clinical information-retrieval technology on physicians: a literature review of quantitative, qualitative and mixed methods studies. Int J Med Inform. 2005;74(9):745–768 [DOI] [PubMed] [Google Scholar]
  • 6.Pluye P, Grad RM, Shulha M, Granikov V, Leung K. Using electronic knowledge resources for person centered medicine - I: an evaluation model. Int J Pers Cent Med. 2011;1(2):385–394 [Google Scholar]
  • 7.Pluye P, Grad RM, Mysore N, Shulha M, Johnson-Lafleur J. Using electronic knowledge resources for person centered medicine - II: the number needed to benefit from information (NNBI). Int J Pers Cent Med. 2011;1(2):395–404 [Google Scholar]
  • 8.Ely JW, Osheroff JA, Chambliss ML, Ebell MH, Rosenbaum ME. Answering physicians’ clinical questions: obstacles and potential solutions. J Am Med Inform Assoc. 2005;12(2):217–224 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Ely JW, Osheroff JA, Maviglia SM, Rosenbaum ME. Patient-care questions that physicians are unable to answer. J Am Med Inform Assoc. 2007;14(4):407–414 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage Publications; 2010 [Google Scholar]
  • 11.Yin RK. Case Study Research: Design and Methods. Thousand Oak, CA: Sage Publications; 2009 [Google Scholar]
  • 12.O’Cathain A, Murphy E, Nicholl J. Three techniques for integrating data in mixed methods studies. BMJ. 2010;341:c4587. [DOI] [PubMed] [Google Scholar]
  • 13.Pluye P, Grad RM, Repchinsky C, et al. Four levels of outcomes of information-seeking: a mixed methods study in primary health care. J Am Soc Inf Sci Technol. 2013;64(1):108–125 [Google Scholar]
  • 14.Pluye P, Grad RM, Repchinsky C, et al. IAM: A comprehensive and systematic information assessment method for electronic knowledge resources. In: Dwivedi A, ed. Handbook of Research on IT Management and Clinical Data Administration in Healthcare. Hershey, PA: IGI Publishing; 2009:521–548 [Google Scholar]
  • 15.Grad R, Pluye P, Granikov V, et al. Physicians’ assessment of the value of clinical information: operationalization of a theoretical model. J Am Soc Inf Sci Technol. 2011;62(10):1884–1891 [Google Scholar]
  • 16.Flanagan JC. The critical incident technique. Psychol Bull. 1954;51(4):327–358 [DOI] [PubMed] [Google Scholar]
  • 17.Urquhart C, Light A, Thomas R, et al. Critical incident technique and explicitation interviewing in studies of information behavior. Libr Inf Sci Res. 2003;25(1):63–88 [Google Scholar]
  • 18.Anderson BE, Nilsson SG. Studies in the reliability and validity of the critical incident technique. J Appl Psychol. 1964;48:398–403 [Google Scholar]
  • 19.Boyatzis RE. Transforming Qualitative Information: Thematic Analysis and Code Development. Thousand Oaks, CA: Sage Publications; 1998 [Google Scholar]
  • 20.Pluye P, Grad RM, Levine A, Nicolau B. Understanding divergence of quantitative and qualitative data (or results) in mixed methods studies. Int J Mult Res Approaches. 2009;3:58–72 [Google Scholar]
  • 21.Westbrook JI, Coiera EW, Sophie Gosling A, Braithwaite J. Critical incidents and journey mapping as techniques to evaluate the impact of online evidence retrieval systems on health care delivery and patient outcomes. Int J Med Inform. 2007;76(2–3):234–245 [DOI] [PubMed] [Google Scholar]
  • 22.Suissa D, Brassard P, Smiechowski B, Suissa S. Number needed to treat is incorrect without proper time-related considerations. J Clin Epidemiol. 2012;65(1):42–46 [DOI] [PubMed] [Google Scholar]
  • 23.Shershneva MB, Wang M-F, Lindeman GC, Savoy JN, Olson CA. Commitment to practice change: an evaluator’s perspective. Eval Health Prof. 2010;33(3):256–275 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Grad RM, Pluye P, Shulha M, et al. EBM CME and the EMR. Evidence Based Medicine Online First. http://ebm.bmj.com/content/early/2013/03/05/eb-2013-101231.full?sid=48c5357f-47dd-4742-8e07-e35f2d8ddffc [DOI] [PubMed]
  • 25.Banzi R, Liberati A, Moschetti I, Tagliabue L, Moja L. A review of online evidence-based practice point-of-care information summary providers. J Med Internet Res. 2010;12(3):e26. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Alper BS, White DS, Ge B. Physicians answer more clinical questions and change clinical decisions more often with synthesized evidence: a randomized trial in primary care. Ann Fam Med. 2005;3(6):507–513 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Shiffman S. Real-time self-report of momentary states in the natural environment: Computerized ecological momentary assessment. In: Stone AA, Turkkan JS, Bachrach CA, Jobe JB, Kurtzman HS, Cain VS, eds. The Science of Self-report: Implications for Research and Practice.Mahwah, NJ: Lawrence Erlbaum Associates, 2000:277–296 [Google Scholar]

Articles from Annals of Family Medicine are provided here courtesy of Annals of Family Medicine, Inc.

RESOURCES