Appendix III.

Survey items loaded to each factor

Items that cluster on the same factor	Survey items
Factor 1 Perceived Alert Noise or Overload	Noise (Overload): Too Many (Q: I receive too many alert notifications per day.) Noise (Overload): Too Many FYI (Q: I get too many FYI (For Your Information) only alert notifications that require my signature, even though no action on my part is required.) Noise (Overload): Too Many Unnecessary (Q: I often receive alert notifications where I am unsure as to why they were sent to me.) Alert Fatigue: More Than Manageable (Q: The number of alert notifications I receive on any given day exceeds what I can effectively manage.) LOG Transformation of Signal to Noise (% of alerts necessary for quality care) Noise (Overload): Too Many Duplicative (Q: I receive too many lab alert notifications that are duplicative, i.e. pertain to the same panel.)
Factor 2 Perceived Effectiveness of Alert System	Performance Expectancy: Alerts Increase My Productivity (Q: Using alert notifications in CPRS increases my productivity.) Q: My colleagues believe the alert notifications in CPRS help them get their job done effectively.) Performance Expectancy: Alerts Enhance My Effectiveness (Q: Using alert notifications in CPRS enhances my effectiveness on the job.) Ease of Use (Q: I find the alert notification system in CPRS easy to use.)
Factor 3 Patient Notification Support”	Patient Notification Support: Have Help (Q: I have the help I need for notifying patients of test results.) Patient Notification Support: Have Staff (Q: I have support staff to assist with management of test result alert notifications (i.e. acting upon and following-up) Patient Notification Support: CPRS Has Features (Q: CPRS has convenient features for notifying patients of test results.)
Factor 4 Represents “Experience”	Years as PCP at VA Age Ever Used EMR Other Than CPRS (Q: Have you ever used an electronic medical record system other than CPRS?)
Factor 5 Outcomes of Patient Safety	Outcomes of Patient Safety: I Missed Alerts Leading to Poor Patient Care (Q: In the past year, I missed abnormal lab or imaging test results that led to delayed patient care.) Outcomes of Patient Safety: Colleagues Missed Alerts Leading to Poor Patient Care (Q: In the past year, colleagues I work with missed abnormal lab or imaging test result alert notifications that led to delayed patient care.
Factor 6 Volume of Alerts	LOG Transformation of Time spent on alerts (Hours per week spent managing alerts) LOG Transformation of volume (Number of alerts received per day) % of abnormal imaging results lacked follow-up within 14 days.