Table 2.

Summary of the existing empirical studies (presented in reverse chronological order)

Citation and PubMed identifier	Study description	Measures or themes identified	Theory and validation	Inclusion or exclusion in this study
van der Sijs et al,77 20171929	An experimental study observing how participants responded to CDS alerts, followed by structured interviews.	Better training, improved concise alert texts, and increased specificity were identified as facilitating factors.	Loosely based on Reason's model of accident causation; validation does not apply.	All critical facilitating factors were included.
Hor et al,78 20067624	A survey among GPs in Ireland regarding perceived benefits of and barriers to adopting CDS in ePrescribing.	27 questions related to value of CDS and barriers to adoption, such as high sensitivity of alerting.	Self-developed survey instrument; underlying theory not indicated; validation not reported.	All value- or barrier-related questions were included, except those at the practice level (eg, those related to standardized product software).
Vashitz et al,79 19000935	Development and validation of a conceptual model of clinicians' responses to CDS reminders related to cholesterol management.	Conceptualized four principal types of user responses: compliance, reliance, spillover, and reactance.	Response types derived from cognitive engineering concepts on end user responses to warning systems.	The spillover effect is difficult to assess via self-reported surveys; a related perceptual measure, the incidental learning effect, was added instead.
Weingart et al,80 19786683	A survey among ambulatory care clinicians regarding their experiences in using drug–drug and drug–allergy alerts provided in an ePrescribing system.	42 items assessing perceived value, satisfaction, barriers, behavioral effects, and impact on safety, efficiency, and cost of care.	Survey developed based on focus groups with practitioners; validation conducted but results not reported; underlying theory not indicated.	Questions about the frequency of events related to behavioral alteration and impact were revised to a leveled scale.
Weingart et al,81 19395307	A focus group study leading to the survey instrument used in the paper above.	Relevant themes included an excessive number of alerts of uncertain value, high sensitivity, trivial alerts interrupting workflow, and appropriate polypharmacy not acknowledged by CDS.	The semi-structured facilitator guide was pilot-tested with an unknown number of physicians and nurses; underlying theory not indicated.	All relevant themes were incorporated.
Mollon et al,82 19210782	A systematic review of prescribing decision-support systems to identify which features predict implementation success and changes in user behavior and patient outcomes.	41 papers independently assessed by two reviewers to study the association between outcomes and 28 predefined system features.	Does not apply	All features were assessed to varying degrees.
Ko et al,83 17068346	A survey among VA prescribers and pharmacists regarding their opinions about and suggestions for DDI alerts.	Prescriber survey (33 items) covered measures such as alert burden and outcomes; and pharmacist survey (39 items) covered additional measures such as their interactions with prescribers regarding alerts.	Self-developed survey instrument; underlying theory not indicated; pilot-tested but detail not reported.	Questions specific to the VA setting or only applicable to pharmacists were not included.
Mayo-Smith and Agrawal,84 16935025	An alert log review investigating the relationship between reminder response rates and practice (primary care facilities at a VA site), and provider and reminder characteristics, followed by a user survey.	Various facilitating and impeding conditions at the practice, provider, and reminder levels; the user survey contained 13 questions assessing providers' perceived value of CDS reminders and adequacy of facilitating conditions.	Self-defined characteristics measures and self-developed survey instrument; underlying theory not indicated; pilot-tested but no formal validation reported.	Very specific characteristics, for example, minimization of keystrokes, were not included.
Grizzle et al,33 17927462	An alert log review investigating prescribers' rationales for overriding DDI alerts at six VA facilities.	14 categories of common prescriber-provided reasons for overriding, such as lack of relevance and availability of alternative management plans.	Does not apply.	All relevant categories were incorporated.
Graham et al,85 17617908	A survey among physicians from multiple specialties soliciting their perceptions of computerized decision aids and intention to use.	43 items on value of CDS for patients and clinicians, content/format, quality of implementation, and intention to use.	Based on the Ottawa Model of Research Use, technology diffusion theories, and prior work by the research team; validation results reported.	Patient-oriented questions and use intention questions were not included.
van der Sijs et al,32 16357358	A systematic review paper summarizing extant literature on alert overrides.	Various facilitating or impeding conditions at the environment, task, team, and individual levels.	A foundational paper of this study, proposing an adapted accident causation model to account for unexpected use behaviors by prescribers.	All measures were incorporated.
Sittig et al,86 16451720	A survey delineating factors affecting primary care providers' acceptance of CDS reminders.	Factors related to patient and provider characteristics, type and volumes of alerts, and configuration of use environments.	Self-developed survey instrument based on a prior observational study conducted by the research team (Saleem et al, 2005).90	Questions specific to the primary care setting (eg, examination room layout) were not included.
Glassman et al,87 16501396	A survey at a VA facility regarding clinicians' knowledge about DDI (as a result of alert use) and their perception of and experiences with DDI alerts.	Research methods based on Glassman et al, which included a 21-item survey soliciting perceived benefits of and barriers to using CDS alerts.96	Self-developed survey instrument; underlying theory not indicated; validation not reported.	All questions were incorporated; increased knowledge about DDI was added as an additional measure of benefits (incidental learning).
Abarca et al,88 16602224	A national survey assessing community pharmacy managers' perception of DDI alerts.	34 questions on perceived value of alerts, meaningfulness, and facilitating conditions such as provision of additional information.	Self-developed instrument; validation not reported; underlying theory not indicated.	All questions were incorporated except for a few that specifically addressed pharmacists' work (eg, coordination with providers).
Niès et al,89 17238410	A systematic review characterizing common success factors of CDS functionality provided through CPOE systems.	Included four success characteristics: system-initiated interventions, assistance without user control over output, automated data retrieval, and provision of corollary actions.	Does not apply	Most success factors were incorporated.
Saleem et al,90 15802482	An observational study conducted at four VA facilities to assess barriers and facilitators related to use of preventive care and chronic disease management reminders.	Five impeding conditions (eg, workload) and four facilitating conditions (eg, workflow integration).	Ethnographically based observations.	Most barriers and facilitators were incorporated.
Kawamoto et al,91 15767266	A meta-analysis investigating success factors of CDS systems.	Four key success factors identified: (1) automatic provision of decision support as part of clinician workflow, (2) provision of recommendation rather than just assessments, (3) provision of decision-support at the time and location of decision-making, and (4) computer-based decision-support.	Does not apply	Success factors (2) and (4) were not included because they do not usually apply in the research context that the survey instrument of this study is designed for.
Taylor and Tamblyn,92 15360983	A chart audit study assessing Canadian GPs' overrides of medication alerts and common reasons for overriding.	Seven common reasons for physician non-adherence, such as alerts not clinically important and interaction already known.	Does not apply	All seven reasons were assessed.
Patterson et al,93 14527974	Observations followed by semi-structured interviews at six VA sites to study human factors barriers to effective use of computerized reminders related to HIV screening, intervention, and progression monitoring.	Six common human factors barriers such as workload, inapplicability of reminders, and limited training.	Self-developed observation and interview protocols; underlying theory not indicated; validation not reported.	All human factors barriers identified were incorporated to varying degrees.
Venkatesh et al65	A theory development study consolidating existing models related to technology adoption and acceptance.	16 relevant questionnaire items assessing the four conceptual constructs in addition to three questions assessing perceived adoption intention.	A foundational paper of this study proposing the unified theory of acceptance and use of technology.	Several questions specific to general business applications were excluded (eg, enabling me to accomplish tasks more quickly). Social influence measures were substantially revised based on relevant research in healthcare.72–75
Weingart et al,29 14638563	A chart review study examining primary care physicians' overrides of medication safety alerts.	Eight categories of common reasons for overriding.	Does not apply.	General categories, such as ‘alerted interaction not clinical significant,’ were included, while context-specific ones such as ‘medication list out of date’ were not.
Ahearn and Kerr,94 12831382	A focus group study among GPs in Australia regarding their options regarding pharmaceutical decision-support systems.	Seven semantic themes ranged from GPs' reaction to computerized alerts to suggested improvements and attitudes to evidence-based guidelines.	Self-developed focus group protocol; detail not revealed.	All themes were incorporated to varying degrees.
Magnus et al,95 12383140	A survey among GPs in the UK assessing their views about computerized alerts and perceived rates of override.	Nine questions on perceived usefulness, applicability, relevance, and quality of information presentation; and six questions on main reasons for overriding.	Self-developed survey instrument; underlying theory not indicated; validation not reported.	All relevant categories were incorporated.
Glassman et al,96 12458299	A survey study conducted at a VA facility soliciting clinicians' knowledge about DDI alerts (as a result of alert use) as well as perceptions of and experiences with computerized alerting.	A survey instrument consisting of 19 questions and 67 items; an adapted version was used in Glassman et al.87	Self-developed survey instrument; underlying theory not indicated; validation not reported.	Most questions were incorporated.
Krall and Sittig,97 11825206	A survey among Kaiser Permanente primary care clinicians regarding the usability and usefulness of different approaches to presenting reminders and alerts, in addition to the desirability of six alert types.	Six characteristics contributing to user acceptance of computerized clinical alerts: number, priority, accuracy, subject domain, relevance, presentation mode, and usefulness.	Self-developed survey instrument; underlying theory not indicated; validation not reported.	All characteristics were incorporated to varying degrees.

CDS, clinical decision-support; CPOE, computerized prescriber order entry; DDI, drug–drug interaction; GP, general practitioner; VA, Veterans Affairs.