End-user support for primary care electronic medical records: a qualitative case study of users’ needs, expectations and realities

Abstract

Support is considered an important factor for realizing the benefits of health information technology (HIT) but there is a dearth of research on the topic of support, especially in primary care. We conducted a qualitative multiple case study of 4 family health teams (FHTs) and one family health organization (FHO) in Ontario, Canada in an attempt to gain insight into users’ expectations and needs, and the realities of end-user support for primary care electronic medical records (EMRs). Data were collected by semi-structured interviews, documents review, and observation of training sessions. The analysis highlights the important role of on-site information technology (IT) staff and super-users in liaising with various stakeholders to solve technical problems and providing hardware and functional (‘how to’) support; the local development of data support practices to ensure consistent documentation; and the gaps that exist in users’ and support personnel’s understanding of each other’s work processes.

INTRODUCTION

Health Information Technology (HIT) is considered essential for any modern, high performing health care system. At the core of HIT infrastructure is the electronic health record (EHR). Ideally, EHR is an integrative system that includes an individual’s health risks, health promotion, and clinical and diagnostic reports from the various providers involved in one’s care (Hersh, 1995; The National Alliance for Health Information Technology (NAHIT), 2008). However, many systems today are local applications which do not have the full integrative capacity of an EHR. These systems are usually referred to as electronic medical records (EMR) (The National Alliance for Health Information Technology (NAHIT), 2008). The potential and actual benefits of EMRs, EHRs, and other HIT applications, include improved quality of care, patient safety and efficiency. In particular, improvements in preventive care, adherence to guidelines, surveillance and monitoring, medication error prevention, and decreased utilization of care have been reported (Chaudhry et al., 2006).

Although Canada and the US lag behind other developed countries in EMR adoption, especially in primary care, there has been a steady growth in EMR uptake (Schoen et al., 2009; Schoen et al., 2012), which resulted in a shift of focus from adoption to “meaningful use” and “benefits realization” (Blumenthal & Tavenner, 2010; Lau et al., 2011; Schoen et al., 2012). Organizational factors, including support, are widely acknowledged as important for successful implementation and promoting effective use of HIT (Lluch, 2011; Cresswell & Sheikh, 2012). However, there is a dearth of research on support for HIT in general, and EMRs in particular. Most previous studies focused on technical support for HIT by information technology (IT) units in hospitals (Baum et al., 2004; Fernando, 2010; Petersen, 2010).

The purpose of this study was to gain in-depth insight into users’ expectations and needs, and the day to day realities of end-user support for primary care EMRs. It differs from previous research in that:

1. The context of this study is primary rather than acute care.

2. We took a holistic view of end-user support, defined as “any information or activity that is intended to help users solve problems with, and better utilize, the software” (Shachak et al., 2011). The holistic conceptual framework we employed is described in detail below.

3. This study looked at on-going support for existing users as opposed to initial implementation.

CONCEPTUAL FRAMEWORK

The conceptual framework that informed our holistic view is based on a scoping review of the Medical Informatics and Information Systems (IS) literature. It has been described in detail in Shachak et al. (2011). The framwork consists of the following 4 facets; support source, location of support, support activities, and (perceived) characteristics of support and support personnel. Support sources may be formal or informal i.e., provided by a person or agency whose job it is to do so or by colleagues, champions or super-users, respectively (Govindarajulu & Reithel, 1998; Munkvold, 2003), as well as from impersonal resources such as user manuals (Munkvold, 2003). End-user support may be provided locally or remotely (e.g., help desk) and it involves a wide variety of activities including infrastructure, software, data and functional (‘how to’) support as well as training and counseling (Leitheiser & Wetherbe, 1986; Speier & Brown, 1997; Govindarajulu & Reithel, 1998; Bygholm, 2001). Finally, users’ perceptions of end-user support characteristics such as the timeliness of support provision, problem solving capability, and communication skills of training and technical support personnel are also included.

Another framework that informed this study is the DeLone and McLean (D&M) Model of Information Systems Success (DeLone & McLean, 1992, 2003). We originally included the D&M model in our conceptual framework because one of the goals for the broader research project was to study the relationships between end-user support and EMR success. Although in the present article we focus on a different question (i.e., EMR users’ expectations, needs, and realities around end-user support), we found that information collected in an attempt to understand EMR success attributes was useful to understanding support issues as well, and vice versa.

Although it originated from the field of management information systems (MIS) the D&M model has become widely accepted in health informatics. Several models of HIT success are partly based on it including Canada Health Infoway Benefits Evaluation Framework (Lau et al., 2007), the Clinical Adoption Model (Lau et al., 2011), and the Human, Organization and Technology-fit factors (HOT-fit) model (Yusof et al., 2008). The D&M model and its derivatives have been successfully employed as a framework for literature reviews as well as in original studies of HIT success (van der Meijden et al., 2003, Yusof et al., 2008, Lau et al., 2013).

The original D&M model, published in 1992, suggested the following attributes of information systems success: system quality, information quality, system use, user satisfaction, and individual and organizational impacts. In 2003, the model has been revised. The construct of use has been divided to intended and actual use, individual and organizational impacts were merged into a single construct of net benefits, and a new construct—service quality—was added. However, for this study, we decided to use the original D&M model (DeLone & McLean, 1992) because of the higher level of granularity it offers for classifying the impact of technology (individual and organizational levels as well as impact on patient care which we added).

In addition, the revised D&M model includes the new construct of service quality. There has been much debate in the IS literature on the measurement of service quality (Pitt et al., 1997, Jiang et al., 2002, Kettinger & Lee, 2005, Landrum et al., 2007). SERVQUAL, a commonly used measure of service quality developed by Parasuraman et al. (1985), includes for example the following constructs: “1. Tangibles: The appearance of physical facilities, equipment, and personnel; 2. Reliability:: The ability to perform the promised service dependently and accurately; 3. Responsiveness: The willingness to help customers and provide prompt service; 4. Assurance: The knowledge and courtesy of employees and their ability to inspire trust and confidence and; 5. Empathy: Providing caring and individualized attention to customers” (Jiang et al., 2002). These constructs are closely related to some of the items in our end-user support framework—especially the support characteristics of knowledge, problem solving capability, communication skills and timeliness. However, in contrast to the D&M model that includes service quality as part of the broad dependent variable of IS success, end-user support in our view is an independent variable that affects it. For this reason, and because our end-user support framework (Shachak et al., 2011) is more specific to the context of HIT and primary care, we decided to use the framework described above.

METHODS

We conducted a qualitative case study using-semi structured interviews, document analysis, and non-participant observation of training sessions. End-user support is a complex multi-factorial phenomenon that may vary greatly, depending on contextual factors. Therefore, we selected a case study approach which is especially suitable for in-depth holistic investigations of phenomena in context [Yin, 2009]. Ethics approval was obtained from the Research Ethics Boards of the University of Toronto and participating sites (where applicable).

Case selection and participants’ recruitment

Four family health teams (FHTs) in Ontario, Canada that use the same commercial EMR system were recruited to participate in the study. Family health teams (FHTs) were first introduced as an inter-professional model for family medicine in Ontario, Canada in 2004. They are currently serving approximately 2 million people (Rosser et al., 2011). In addition to family physicians, FHTs usually involve registered nurses or nurse practitioners as well as other health professionals (e.g., social and mental health workers, dietitians, etc.) Physicians in the FHT are paid by the Ontario Ministry of Health and Long Term Care using blended funding models which combine fixed payments (fixed salary or capitation-based) with payment for specific services and bonuses for meeting preventive care targets (Ontario Ministry of Health and Long Term Care, 2009). With the help of OntarioMD, a provincial agency that supports the implementation of EMRs in Ontario, we identified FHTs that use the leading commercial EMR system in the province. Letters were sent to the lead physicians of these FHTs and of those who agreed to participate we selected four that capture the diversity of FHTs across several dimensions including: 1. size; 2. distribution (multiple sites vs. co-located); 3. geographic locations (urban, suburban, or rural area); and 4. affiliation with a hospital (community vs. academic within a hospital).

As a critical case—i.e. “a single case… [that] can confirm, challenge, or extend the theory” (Yin, 2009, p. 47) —we selected a fifth site: a family health organization (FHO) that used a different, open source, EMR system. FHOs are similar to FHTs in that funding is also based on blended models; however, by definition, they are not necessarily inter-professional and often include only physicians and administrative staff, as in our case (Rosser et al., 2011).

Similarities between this case and the other FHTs would support the generalizability of the findings.

When a FHT or FHO was recruited, an initial visit to the site was made to describe the study in detail. Consent forms were left with a representative of the FHT/FHO who distributed them to all eligible staff i.e., all physicians, other health care professionals, and administrative staff members who used the EMR in their daily work. Those who agreed to be interviewed returned their signed consents by mail or fax directly to the study team and were all contacted to arrange an interview time. In addition, the EMR software vendor was contacted and trainers, technical support personnel, and managers were recruited to participate in the study using the same procedure.

Data Collection

Interviews

We conducted a total of 58 semi-structured interviews. The interview questions focused on the sources of support, support activities and characteristics, and the impacts of EMR use on the individual user, the organization, and the patients. Questions about the system’s quality, its use, and the quality of information were also included as well as probes to follow-up on participants’ answers. Interviews were conducted in person and on–site and took between 30 to 60 minutes to complete. 20 % of the interviews were conducted by two researchers together in order to reflect on and revise the interview protocol, provide feedback on the interviewing technique, and ensure consistency in the approach. The remaining interviews were completed by either one of these two researchers. Interviews were audio recorded and transcribed verbatim by a professional transcriptionist.

Document Review

Formal and informal documents from participating sites and the vendor were collected and reviewed. During interviews, users and support personnel were asked if they have any documents related to the EMR that they could share with the research team. Copies of these documents (either in electronic or print format) were handed or sent to the research team. Formal documents included service agreements, user manual, and training workbook from the vendor. Informal documents created by the users included planning documents related to the implementation process as well as user-generated tutorials and manuals. These documents were reviewed to provide insight into the needs of the various users, identify the planned support sources and activities, and compare these plans for support with reality. We triangulated this information with the themes identified by interviews and observations. Finally, tutorials and manuals collected from the vendor and participating FHTs were more comprehensively analyzed in a different study (Shachak et al., in press).

Observation of training sessions

Two training sessions for a new client—a small solo specialist practice with 3 EMR users in a small town—were observed. Both training sessions were offered by the commercial EMR software vendor and were delivered on site by the same trainer with a one week interval between sessions. During observations, researchers took notes to describe the various training activities, their delivery methods (e.g., lecture, hands-on exercise, demonstration, etc.) scope (e.g., system features and use, workflow, etc.), as well as the time dedicated to each training activity. Characteristics of the trainer (e.g., rapport with users, patience, knowledge of the software, understanding of workflow, and ability to answer questions) and trainees (e.g., mastery of computer usage, comfort with the software, signs of frustration, apparent understanding of content, type of questions, and feedback to the trainer) were also noted with illustrative examples. Each of the training sessions was observed by a different researcher (AS, CM) and the two researchers then met to compare their notes and discuss their interpretations.

Data Analysis

Interview transcriptions, documents, and observation notes were first read by members of the research team to familiarize themselves with their contents and annotate them. We employed the framework approach to qualitative data analysis (Ritchie & Spencer, 1994; Lacey & Luff, 2007) using the conceptual framework of end-user support and the D&M model of information systems succes described above. Based on this framework and the initial interpretation of the data, we developed a coding and categorization scheme (Appendix A) that we employed to identify and match specific data elements with corresponding theoretical concepts.

Each interview was coded by at least one researcher (case E: RD; all other cases: CM) using NVivo 8 qualitative data analysis software. 60% of the interviews (purposely selected to reflect interviewee roles) were also coded independently by at least one other member of the study team (AS, CM, or RD). Coding disagreements greater than 5 % (as calculated by the software) were discussed among team members to reach consensus and revise the coding scheme. Two researchers (AS, CM) independently coded all observation notes and then reviewed and discussed coding discrepancies greater than 5% until consensus was reached.

Next, three researchers (AS, CM, RD) independently reviewed and interpreted the data, looking for patterns and themes and identifying emergent concepts. Initial agreement on themes was high and the researchers discussed their interpretations to further refine concepts and reach consensus. Identification of themes was initially done on a case by case basis and then extended to all cases so that overarching and unique themes, as well as contextual factors, could be identified. Finally, we reviewed and annotated documents and triangulated this information with the themes identified by interviews and observations.

FINDINGS

Characteristics of settings and participants

Although the selection of the 4 FHTs and the FHO was based on convenience, these sites partially represent the diversity of primary care settings in Ontario in terms of size, affiliation with hospitals, and geographic location (Table 1). All 4 FHTs used the same proprietary EMR software. The FHO implemented a different, open source, EMR and employed a third party authorized service provider.

Table 1.

Salient characteristics of study settings

Site	A	B	C	D	E
Attribute
Size	Large (>30 physicians)	Small (<10 physicians)	Small (<10 physicians)	Medium (10–30 physicians)	Small (<10 physicians)
Area	Small Town/Rural	Urban	Suburban	Urban	Small town/semi-rural
# of Clinic Sites	>10	1	3	1	1
Affiliation with a hospital	No	Some affiliation with an adjacent hospital.	Some affiliation with an adjacent hospital.	Located within a teaching hospital.	No
Time using the EMR (months)	>24	>24.	12–24	<12	12–24
In-house IT support	None	Some assistance from the hospital IT unit	Some assistance from the hospital IT unit	Support from the Hospital IT unit	Local contractor on an on-call basis.

Of the 58 interviews, 49 were with EMR users, including physicians, nurses, allied health professionals (e.g., dieticians, social workers, etc.). The remaining 9 interviewees were employees of the vendor of the EMR software which all four FHTs were using including help desk staff, trainers, and managers of training and support. Table 2 provides an overview of the interviewees’ occupations, age group, gender and experience with EMRs.

Table 2.

Descriptive statistics of interviewee characteristics

Site	A	B	C	D	E	Vendor	Total
Profession
Physicians	9	4	2	4	3	0	22
RN	2	1	1	3	0	0	7
NP	0	1	0	0	0	0	1
Allied Health	2	0	1	0	0	0	3
Tech Support	0	0	0	2	0	3	5
Training	0	0	0	0	0	2	2
Administrative	4	1	1	4	4	4	18
Gender
Male	8	1	2	2	2	4	19
Female	9	6	3	11	5	5	39
Age (years)
<30	1	1	1	3	0	0	6
30–39	3	4	1	2	0	3	13
40–49	8	0	1	4	0	4	17
50–59	5	1	1	3	3	2	15
>60	0	1	0	0	3	0	4
Unknown	0	0	1	1	1	0	3
EMR Experience*/time with vendor (years)**
<1	0	0	0	2	1	1	4
1–2	9	3	4	6	6	4	32
3–5	4	0	0	1	0	2	7
6–8	1	3	0	0	0	2	6
9–12	3	0	0	2	0	0	5
>12	0	1	0	1	0	0	2
Unknown	0	0	1	1	0	0	2
Total	17	7	5	13	7	9	58

RN= Registered nurse; NP= Nurse practitioner

^*includes experience with any EMR/EHR system; not only the ones studied here.

^**Interviewees from the vendor only

Themes

Our data analysis revealed several themes, which are organized into the following categories: location and sources of support, support activities, and perceived characteristics of support and support personnel.

Location and sources of support

Local, on-site, support was perceived as a critical factor for EMR success in all of the cases. All FHTs agreed that for effective and efficient use of the EMR someone with technical expertise needs to be available, at least on an ‘on-call’ basis. This role was important not just to deal with problems but also to ensure that security measures and procedures for data back-ups are in place and regular maintenance procedures are completed. However, the type and degree of formality of in-house support varied greatly between cases. In contrast, functional support (i.e., support for how to use the system) was often provided by peers or ‘super-users’. The main characteristics of in-house support available at each of the sites are summarized below:

• Users in FHT A relied mostly on informal support from peers—especially the lead physician. Since the FHT is distributed over large geographic area, this support was not always available in remote, isolated, clinics. There was some dissatisfaction that the lead physician had to deal with infrastructure support: “He [lead physician] shouldn’t have to look at the server and figure out, ‘Oh, somebody put it on plug 7 when it should be on plug 9’.”(Interviewee A.1.5)

• FHT B had strong internal support from the medical director (lead physician) and administrative manager, who provided most functional support, trained new staff and residents, and liaised with the vendor and hospital when technical problems arose: “The technical support probably comes more from our medical director so if I am stuck with a certain function….or I’m having difficulty ….he can figure out a more effective way” (B.3.1)

• FHT C hired a part-time technical support person who provided functional support, liaised with the hospital and vendor when infrastructure issues arose, and developed training documents for new staff and residents: “We have an IT person and she deals with these problems; so when the system is down she calls the hospital or vendor right away [and] she is also providing local support…”(C.1.1)

• FHT D was located in a teaching hospital which provided infrastructure support. The FHT trained three super-users who provided most functional support, liaised with the vendor in cases of technical problems, trained new staff and residents, and developed a comprehensive user manual contextualized to the setting: “She [physician manager] received extra training and she’s been training us a lot. Our nurse manager, also did and then we hired one of our support staff—one of our office coordinators—so the three of them are our main people that we go to.”(D.1.2); “The other thing they helped us was developing our own training because the manual you get from the vendor is very generic. It’s how to run their system—it’s not how to run their system in your environment.”(D.6.1)

• FHO E contracted a local IT person to provide infrastructure support on an on-call basis. Software support was provided by an authorized service provider for the open source EMR used by the FHO. Some functional support was provided by one of the physicians and one of the administrative assistants (who also liaised with the service provider when technical problems occurred). Additional support, especially for creating custom forms and standardizing data entry, was obtained from family members: “My daughter...she actually took all the templates from the hospital lab, all the requisitions, and she actually put those into the system for us.”(E.1.1)

Although all cases had identified some in-house sources of support, they also required support from the vendor or from an authorized service provider (FHO E) to resolve some problems with the software. Vendor provided support was available to users in a variety of methods as described in Table 3.

Table 3.

support provided by the commercial EMR vendor and open source EMR service provider

Support activity	Details
System installation	The vendor (or service provider) advised clients on hardware, network and ancillary devices requirements. They set up and configured the system for users.
Initial training	The training provided to the users of the OS EMR was very basic and included one-day training session on-site. The training by the commercial system’s vendor changed over time. The current model includes three sessions: preliminary phone call to set up expectations, a session for administrative staff (which covers administrative functions such as appointments and billings), and a session for clinicians (on problem list, progress notes, labs, prescription, etc.)
On-going training	Provided through user conferences (for the OS EMR this is organized by the society of users). Training for new staff is available for fee upon request.
Technical support	Telephone help desk. The vendor’s help desk has 3 tiers: Tier 1 receives calls from users and solves most (~70%) of the problems, Tier 2 handles more difficult problems which usually take longer to solve, and transfers unresolved problems (usually system bugs) to Tier 3, which is the development team.

Support activities

Infrastructure support

Users in the study reported that many of their support needs were not related to the EMR software but to the infrastructure (hardware, network, and ancillary devices) which supports it. These issues were often technically complex and beyond the expertise of the in-house super users. They often required coordination among several organizations including network providers, software and hardware vendors and, where applicable, the IT unit at the hospital which hosted the EMR server. Sometimes, there was no agreement on who was responsible for solving the problem and users were shunted back and forth between various organizations. On-site support people often played an important role in liaising with these bodies and coordinating the resolution of technical problems.

To further complicate matters, users often viewed the infrastructure and software as a single package. Representatives of the commercial EMR vendor confirmed that a portion of the calls they received were for assistance with items for which they were not responsible and did not support (e.g., printers, internet connection, MS-Office software). While they could offer some troubleshooting advice, they sometimes had to inform the caller that he or she needed to contact someone else. This left some users with the impression that no one was taking responsibility for problems.

Support for data quality

Users from all four FHTs (cases A–D), and some users from FHO E, recognized the value of the EMR for managing information. They reported that with the EMR, information was legible and easy to retrieve, and that it could be easily shared with other providers. Information was searchable electronically, which meant they could identify patients who required specific care and be proactive in providing it (e.g., send reminders for colonoscopies; call in patients for immunizations; or invite diabetic patients for foot examinations). The ability to perform practice wide searches also allowed them to assess their performance in meeting established clinical targets set by the Ontario Ministry of Health and Long Term Care.

Users gradually understood that to obtain these benefits, activities needed to be undertaken to ensure the quality of data in the EMR. This was particularly important for the specific commercial EMR system used, which allows free text data entry and multiple places to document them. While users liked this freedom and associated it with user friendliness of the EMR, it also meant that they had to agree on conventions to ensure data are entered consistently, updated regularly, and that all relevant information is documented.

The vendor identified this need and emphasized the importance of consistency in data entry. First, the user manual contained a section on the importance of consistency which explained the problem and provided some advice (e.g., inclusion of multiple terms in searches and creating standard templates and forms). Second, during the training sessions we observed, the trainer made several comments about the need for consistency in data entry and demonstrated some system tools to facilitate it e.g., templates and creating links to standard classifications or terminologies. However we are not clear as to whether trainees fully understood the implications of lack of data consistency at this early stage.

All cases made some effort to develop rules around consistent data entry; however, there was variation in the extent and types of activities undertaken and the success in implementing them. Table 4 describes some of these activities.

Table 4.

Activities taken by study participants to support data consistency

Activity	Sample quotation/examples
Forming a committee to decide on conventions and share them with all users (FHT A)	“We had a committee started about a year and a half ago. I was a member of that. You know, we met a few times and set some nomenclature around that we suggested people use.” (A.1.3)
Providing practice wide tools such as templates or custom forms to facilitate consistent data entry. In some cases one person was in charge of creating these templates and sharing them with all users (All cases)	“…one of the other things that I have done recently is I’ve taken a Custom Forms Course and I am now entering in custom forms into the database, that will hopefully help all the doctors speed up that process.” (A.5.4) “…then we also picked one of the office coordinators…we pulled her out to be our sort of – learn the system really, really well and so she got involved in – there were about 5 of us that took the course to make custom forms, but she really ended up doing most of them.” (D.5.5) “We have templates for letters, we have templates for anything we can standardize that we use repetitively, we have a template for, as well as Custom forms.” (B.5.1)
Adoption of a standard terminology (FHT B)	“So we’ve actually implemented a data entry terminology called ENCODE. And so people could be coding on the fly, much less – but without that it’s really a chore.” (B.1.1)
Connecting data entry to reminders so the reminder would only disappear after the data were entered (FHT B)	“They all have to agree when they’re doing the foot exam that they use a particular template called the DB Foot Stamp. And so there’s a tech stream that goes in there that the reminder is able to read and say: Okay, that’s been done. I’ll bring up the reminder again in six months.” (B.1.1)
Undertaking practice wide activities to review charts and ensure data were complete, correct, and entered consistently; (FHTs B,D)	“We do spend a fair amount of time cleaning the data up, trying to streamline it.” (B.2.1)

Training

All FHTs received initial training from the EMR vendor when they implemented the system. Some of the FHTs were trained several years ago and some within the year prior to the interviews. In that period of time the vendor reported that they changed their training curriculum. Therefore, not all users received the same training and their degree of satisfaction with it varied. FHTs A and C, which received training early on were the least satisfied. Users in FHT D, which was the last to implement the system, were generally satisfied with the vendor’s training. In all of the FHTs, there was also on-going training for new staff and—in teaching sites—for residents or students. On-going training was done internally by a super-user or the on-site support person, who—in three of the FHTs—developed their own tutorials or user manuals for this purpose. FHO E received initial training from an authorized service provider for their open source EMR. Users in the FHO reported that the initial training was short, and that after using the system for some time they needed additional training on more advanced features.

Clinicians in all sites reported that use of the EMR affected their communication with patients during consultations. They noted that there was some initial difficulty in learning how to look at the monitor, use the keyboard, and at the same time maintain eye contact with the patient. The formal training did little to address these issues and some users expressed a desire for advanced training in this area. To some extent, this was done internally in Case D where the teaching environment enabled it (e.g., by reviewing video recordings of the consultation and providing feedback to trainees).

(Perceived) Characteristics of support and support personnel

A number of overarching themes related to the characteristics of support and support personnel were identified. Users expected support to be timely and provided by knowledgeable people; that those providing support would be patient, able to adapt to varying degrees of users’ knowledge and computer skills, and have the ability to articulate solutions in a way users could understand. However, interviewees indicated that, in reality, there was variation in the degree of support and training personnel’s knowledge and communication skills; that support was not always timely; and that often support and training staff did not understand clinical practice and its complexity. These issues are presented in detail in Table 5.

Table 5.

Themes related to support characteristics

Theme	Main findings	Sample quotations/observations
Timely Support	Since the FHTs are dependent on the EMR, support needs to be provided at the time the problem is encountered.	“They’ve got to be available right when you need them. If you’re using only a computerized system and ALL of your information is on it and the system goes down, you’re paralyzed. You can’t run your clinic and you can’t see your patients.” (D.1.2)
	For users, timely support means “now”	“…but it has to be fixed NOW, not tomorrow – NOW!” (E.5.4)
	In a busy clinical setting, clinicians were often only available to receive support for a limited period of time	“I don’t find calling a 1–800 number and then waiting, --, for them to call me back helpful, ---You have to move on to the next patient and so it really can stall your timing and your day and so I find that really frustrating. (B.1.5)
	All too often users had to leave a message and the call would not be returned for some time, or not be returned at all.	“We would call and would leave a message with the receptionist as to what our problem was and then they would forward that onto a support person who would call you back,---so sometimes it would be 2 or 3 days before you even heard from them. All right, not heard from them, but until you speak to them live.” (C.5.1)
	A growing number of clients and staff shortage contributed to delays in providing support. The vendor was in the process of hiring new staff.	“Um....in general right now it’s more call volumes, because we’re a fastly growing company and we don’t have the staff up to par yet, which they’re working on obviously, to hire more people.” (V.1.1)
	The vendor emphasized that some technical problems cannot be fixed immediately	“We wish that when we identified a problem we would have it fixed the next day. It doesn’t work that way. No, because we don’t want to give them a fix that we didn’t test and it breaks something else. That’s – in the end that’s not a good thing.” (V.1.3)
	Users in FHO E felt that their problems were not solved in a timely manner because the service provider had bigger clients which were more important to them.	“… our software provider works with [names] now as clients. So it may be that --we were so small that maybe we weren’t a priority”. (E.1.1)
Support from a knowledgeable person	There was variance in support personnel’s knowledge	“I find them [support personnel], many of them knowledgeable and many of them not knowledgeable.” (B.1.1)
Support people’s communication and counseling skills	Users want support staff to be patient, communicate in a way they can understand and provide reassurance.	“I think, maybe he didn’t know how to – because he knows how the computer runs so well, he didn’t know how to come down to my level. Maybe that was the problem, you know, because I’m not the technical kind”. (A.1.10)
		“…to recognize that people have different learning styles and also I think that, you know, to be prepared that everybody with computers is very different. You can’t just assume that everybody’s going to pick it up overnight. (D.2.1)
		“Do you know what? You’re doing exactly right! It’s perfect. You’re doing it right.’ And they go, ‘Oh, okay good. I just wasn’t sure.’ And it’s like, so that’s part of our day as well is just encouraging and cheerleading. “ (V 1.3)
Support providers’ understanding of clinical work	Physician users (especially in FHTs A and C) expressed their dissatisfaction with the initial training from the vendor and attributed it to the trainers’ lack of understanding of clinical processes.	“The people who provide the training are not clinical professionals, they are computer guys so they really don’t have a sense of what we need to know, how we use the [Name] program so I would say that the training sessions we had which were quite expensive were pretty bad.” (A.1.2)
	Experienced trainers we interviewed argued that it was not necessary for a trainer to be a physician, but that understanding the client’s workflow was an essential skill for them.	“Well, it’s really workflow. It’s not,, you don’t have to have a medical background, because I certainly don’t and I’m training medical software and doctors. So you know, a lot of people are like, ‘Oh, you don’t have any medical background and yet you’re training medical EMR software in a doctor’s office. And really what it’s narrowed down to is just more workflow rather than medical knowledge. So you need to know what sort of items they need to be able to work with. (V.3.2)
		In our observations, the trainer started the session on using the clinical module of the system by reviewing the workflow with the client. She then discussed with them how to best adapt the use of the system to it and customized the training accordingly.
	The vendor tried to hire people who worked in physician offices for their support team.	“Well, some of them have a background in medical office experience. I have one trainer who is actually an RN as well, and then a few of the trainers have a more technical background with more of an interest in being able to provide training.” (V.2.3)

DISCUSSION

We explored the needs, expectations and day to day realities of end-user support for EMRs. Many of our findings are similar to those of previous research on HIT implementation. Thus, the contribution of this study is mostly in that it provides additional evidence for the importance of special people, IT support and the gaps in users and vendors’ understanding of each other’s work, to name a few. It is part of a growing body of research (e.g., Ash, et al., 2003; Terry et al., 2008; Yusof, et al., 2008; Gagnon et al., 2010; Novak, et al., 2012; Vedel et al., 2012) that, taken together, show that these issues are universal and cut across countries, healthcare systems, and organizations.

Another important contribution of this study is the provision of rich detailed descriptions on several end-user support issues such as the various forms of on-site support, the informal activities taken to ensure the consistency of data entry, and the perceived characteristics of support and support personnel. This level of detail is sometimes missing from the medical informatics literature and may be useful for identifying issues that are important for realizing the potential benefits of EMRs and help formulate recommendations for users, vendors, and policy makers. Table 6 summarizes the main issues we identified and corresponding recommendations for EMR users and vendors. We further discuss some of these issues and how our findings relate to previous research below.

Table 6.

Main themes and corresponding recommendations for EMR users and vendors

Main theme	Recommendations for users	Recommendations for vendors
On-site support was important in all participating sites; Its form and degree of formality varied depending on contextual factors (e.g., practice size, geographic distribution, and affiliation with a hospital). Users often sought support from the vendor for items that were not covered by the service agreement (hardware, network, ancillary devices and other software).	If your clinic is not affiliated with a hospital, consider hiring a local person to support the IT infrastructure (hardware, network, and ancillary devices), at least on an on-call basis.	Recommend users to hire local IT people where applicable, or consider expanding your service package to cover infrastructure-related issues.
	Appoint super-users to provide functional (“how to”) support locally and liaise with the various service providers. Consider the training, responsibilities, and time allocation requirements for these super-users.	Endorse the appointment of super-users and provide them with more extensive training.
All cases recognized the importance of data quality and developed practices to ensure data are entered and documented in a consistent manner.	Ensure data are entered accurately and consistently by all users: appoint a person to develop and share templates and forms; agree on data entry conventions or adopt a standard terminology; undertake data quality assurance actions.	Provide means for ensuring data consistency (e.g., built-in templates and forms), and emphasize the importance of consistency through training and documentation.
There were gaps in users and support personnel’s understanding of each other’s work, and between users’ need for immediate assistance and reality.	Adapt the user manual and training materials to your local context and workflows.	Support contextualization of user manuals and tutorials by providing workflow information, allowing modular design, or participatory documentation (similar to participatory system design).
	Familiarize yourself with basic IT concepts; distinguish critical issues that need to be resolved now (e.g., the system is not available) from less critical problems; understand that some problems may be hard to solve, or require changes to the software, and therefore may take time to fix.	Hire support and training personnel with good interpersonal communication skills who understand the users’ workflow. Make sure they understand users’ need for timely support.

Our study is related to previous research in several ways. First, specific studies on support focused mostly on technical support for HIT in hospital settings (Fernando, 2010; Petersen, 2010; Lluch, 2011). One difference between hospital and many primary care settings is the availability of formal on-site support. Our findings indicate that on-site support, including infrastructure, data, and functional support as well as on-going training, was perceived as critical in primary care settings, and is required not only for initial implementation but also in later stages when FHTs (or other primary care groups) have already used the EMR for some time and the benefits of the system need to be realized. All cases developed mechanisms to provide end-user support locally, although their forms and degree of formality varied. For some, the help of a clinical leader or super-users was sufficient, while others needed to hire someone to provide on-site technical support.

The important role of a champion and local “go to” people is well documented in the literature on HIT implementation (Ash, 1997; Ash et al., 2003; Terry et al., 2008; Cresswell & Sheikh, 2012; Vedel et al., 2012) and our study provides additional support for and an in-depth analysis of the “go to” person function. Some of the roles and characteristics of on-site support people we found are similar to those of ‘bridgers’ described by Ash et al. (2003) e.g., connecting with both users and vendors. However, while bridgers in Ash’s study communicated required system changes to developers, in our study they often liaised with multiple service providers to negotiate technical problem ownership. A possible explanation for this difference is that the EMR in our study is an off-the-shelf system that is purchased by networked groups of physicians who are not always affiliated with hospitals or large health care organizations. Thus, the system only allows for limited customization such as creating templates and custom forms. Second, the scope of our study was support for on-going use of the system rather than early stages of implementation. It is possible that the liaison role of bridgers or on-site support people changes over time; however, longitudinal research is required to verify this explanation.

Another new role of on-site support people, which has not been described before, was the development of tutorials and user manuals. It has been reported that developers of open source software often do not invest sufficient resources in documentation (Berglund and Priestley 2001; Lanier, 2011). However, in our study, formal user manuals were provided to all sites by the commercial EMR vendor or the authorized service provider for the open source EMR in both print and electronic (PDF file) formats. Nevertheless, users in all sites developed tutorials and manuals of their own, which triggered our curiosity regarding the differences between formal and informal tutorials and manuals. We reported in a separate article (Shachak et al., in press) that the most important difference was that user-developed tutorials and manuals were customized to the specific workflow, user roles, and patient characteristics of the clinics whereas the formal manual from the vendor was generic in nature. Thus, an important role of on-site support people is the provision of customized impersonal support for EMRs.

Second, previous studies suggest that the use of EMRs improve information quality, and especially its completeness, legibility, availability, and timeliness (van der Meijden et al., 2003; Yusof et al., 2008). However, concerns have been raised that the quality of data in health information systems is not optimal (Arts et al., 2002). Consistent with the literature, users in this study found that the proprietary EMR—which allows for free text entry—easily captured the patient’s narrative and was fast and easy to use (Patel et al., 2002; Johnson et al., 2008; Rosenbloom et al., 2011). However, they also realized that the lack of consistency in data entry hindered their ability to effectively retrieve and use information for preventive care, disease management, and quality improvement purposes.

In a setting similar to some of our cases, there was no significant difference in the provision of preventive care services between physicians who implemented EMRs and others who used paper records (Greiver et al., 2011). In contrast to this study, a recent report from Canada Health Infoway (2013), suggests that EMR use can improve preventive care and chronic disease management but this positive impact depends on maturity of use. Infoway’s maturity of use definition includes self-reported use of EMR functionalities such as reminders for patient follow-up and decision support features. This idea seems to be consistent with our finding that most users in the FHTs but only some in the FHO realized the importance of the EMR for information management that is enabled by consistent documentation. Overall, users in the FHO seemed to be still struggling with basic functionalities of the EMR as opposed to the FHTs which exhibited various levels of more advanced use of the system. This may be attributed to the older age of users in the FHO, their lack of computer and EMR experience, or the shorter formal training they had received.

In our study, support activities to improve information quality evolved locally as the FHTs and FHO moved from basic to more advanced use of the EMR (except for FHT B where the lead physician was well aware of this issue from the beginning) and varied from one case to another. Therefore, we propose that the definition of maturity of use be expanded to include consistency of documentation, which may partially explain why in some studies (e.g. Greiver et al., 2011) there was no improvement in preventive care following EMR implementation. The extent to which the activities taken to improve information quality actually affect the provision of preventive care and chronic disease management needs further exploration. Nevertheless, the detailed description in the results section of this article may help establish best practices or assist other users, vendors, and policy makers in determining what forms of data support may fit their needs.

Our study also sheds some light into the impact of procurement model (proprietary or open source) on end-user support. Proponents of open source software argue that access to the source code promotes innovation by enabling more creative input and garnering additional corrective feedback (McDonald et al, 2003). Furthermore, since open source development costs are often lowered, developers and distributers can shift resources towards end-user training and support, customizing software to specific user settings, and implementation (Kantor et al., 2003, Wynn et al., 2012). Under open source license agreements, “EMR vendors can become professional service providers (the economic model of medicine itself), competing on service quality rather than on the basis of software secrets” (Kantor et al. 2003). The FHO selected for this study provides a good critical case for testing these arguments because of its small size, location in a semi-rural area approximately two hours’ drive from the nearest big city, and relatively older users (all of whom were over 50 years old) with limited computer experience and little or no previous experience with EMRs. If support for this case was indeed proven to be superior to that of the closed source counterpart EMR, this could support the open source advocates’ claim of better service. However, our findings suggest that there are many similarities between end-user support for the open source and proprietary systems; for example: reliance on formal sources of support (i.e., their EMR vendor or service provider, respectively) for solving problems related to the quality of EMRs and dissatisfaction with the timeliness of support. These findings challenge the promise of open source for better service and are more in line with Morgan and Finnegan (2007), whose field study of information systems managers suggests that poor service quality resulting from a lack of accountability is a concern with open source software.

Finally, in a previous paper, we reported on an in-depth analysis of the interviews with the vendor support staff, how they perceive their roles, the challenges they face, and how they deal with them (Shachak et al., 2012) Here we expand this work and describe the gaps in support personnel and users’ understanding of each other’s work. Our findings support those of previous works on end-user support (Fernando, 2010; Petersen, 2010; Walkinshaw, 2011), which revealed similar issues. Thus, it provides additional evidence that these are universal challenges faced by HIT users and vendors in different countries, healthcare systems, and organizational settings. It identifies additional gaps—especially the tension between users’ need for immediate assistance and the difficulty of the vendor in providing timely support for complex technical problems—and further highlights the importance of human and not just technical skills in bridging them.

Limitations and directions for future research

This study has a number of limitations. First, although theoretical saturation (Corbin & Strauss, 2008) was reached, generalizability remains a limitation as only two EMR systems were included and only two training sessions were observed. Second, our data provide only limited insight into some of the support activities. In particular, interviewees did not provide much information on functional support, which may be especially important for facilitating formal support personnel’s understanding of users’ needs. This issue may be studied in future research using more naturalistic approaches (e.g., ethnography). In the same vein, users at the FHO reported that the training they received was shorter than the current training model for the proprietary EMR; however, we were unable to observe a training session for the open source EMR to gain an in-depth insight into the differences in training for using the two systems and their potential impact on EMR success.

Finally, cases varied in several attributes, which makes it difficult to isolate the impact of specific contextual factors. More research is required to better understand these contextual differences. Of particular interest are the differences between closed and open source EMR systems. Open source EMRs have been promoted as a viable alternative to proprietary systems for their potential to reduce the initial cost of implementation, result in more standardized and interoperable products, and be customized to the users’ needs (Vest and Stephens, 2012). Although as discussed above, our findings do not support the idea that open source procurement improves end-user support we are in the process of conducting a more comprehensive analysis of the differences between the critical and all other cases. In the future, additional sites that use open source primary care EMRs should be studied to verify the generalizability of the findings. Future research may also explore the relationships between end-user support and EMR success attributes from the DeLone and McLean model (DeLone & McLean, 1992, 2003) that informed our analytical framework e.g., satisfaction, use and impact on individual users, the organization and by extension, patient care.

APPENDIX. Code Definitions and Guidelines*

General Comments

• When coding an interview one quotation may deal with several issues and may require more than one code.

• Two codes within the same attribute can be used; e.g. support may be formal/personal.

• It is not necessary to look for connections between the codes at the initial coding; relationships will be looked for at a later point.

• Unless it is necessary for understanding the quotation, do not code the interviewer question.

• When occasional comments are made that are unusual and do not easily fit into any code, make an annotation; (1)

• Comments related to activities required to support the transition to EHR do not need to be coded; transition activities are beyond the scope of this project.

• When coding support characteristics, if possible also code the support source and support activity; e.g. counseling skills/formal/training and education could indicate a comment was made about the counseling skills of someone from the vendor company who was provide a training program;

Attribute	Definition
Source	These codes deal with characteristics of support for use of the EMR
Formal	Support provided by an official source such as manuals created by the vendor, a help desk, or personnel from the vendor. Personnel from the FHT whose job includes at least some component of IT related duties are considered to be formal sources of support. Use this code to code negative comments (ie “support from outside agencies like the vendor and OntarioMD is lacking”) as well as positive comments.
Informal	Support provided from peers whose job is not IT related; can include a local champion or local expert user (e.g. one of the other physicians in our group is very familiar with the EMR so he is able to help us with things like searches”). Manuals created internally by the FHT are considered informal sources. Use the informal code to code comments about “super users” that originate from the vendor;
Personal	Support provided directly by a person either on site or by telephone. A help desk is an example of a formal/personal source of support.
Impersonal	Support provided by documents or websites. No direct contact with a person is involved.
Location	The location from where support is provided
On site	Refers to support provided on site regardless of who is providing it.
Off Site	Refers to support provided from an offsite location.
Activities	Actions provided to help those using the EMR
Infrastructure Support	Assistance or lack of assistance with the acquisition, maintenance, or use of EMR infrastructure including items usually thought of as hardware ie computers, printers, and also of items such as network connections, log in mechanisms and connections to external providers such as a hospital based system.
Data Support	Activities undertaken to ensure data is entered consistently and completely. This code is used to refer to activities related to data, not the quality of the data itself.
Functional Support	Assistance provided (or not provided) to use or solve problems related to the PS program itself. Assistance with learning the various functions; i.e. how to do searches, how to make templates, short cuts etc. (E.g. “They come to me if they are having problems with the scheduler or problems with messaging.”
Training and Education	Refers to teaching users how to use the program initially when the organization is converting to an EHR and also the training that is required on an on-going basis when new staff are hired.
Project Management Support	Refers to the overall activities and efforts the organization must take in order to ensure the successful operation of the EHR. Use this code to code comments that interviewees from the vendor use to describe things that need to happen to ensure successful training and adoption of the EHR;
Characteristics	Describe the attributes of the support provided to users of the EMR
Counseling Skills	The ability of the person providing support to listen, to reinforce training or usage, to communicate patiently and in an empathetic manner, and with a willingness to try various alternatives. “ (E.g. patience and the ability to multi task and drop what I am doing to help someone who needs help are the most important factors)”
Knowledge	Includes technical knowledge and the ability of those providing the support to understand the problem being described and provide an appropriate answer.
Homophily/Heterophily	Refers to comments that indicate there is, or is not, a gap between the technical knowledge the support person has and their understanding of the day to day work of the user. Must be used in the context of the support provided.(Eg The problem is the person that did it, I don’t think they ever did clinical work so they were OK at showing us the system but I don’t think they really understood how that applied to a patient.”)
Service Quality	Comments related to the overall quality of the support provided including timeliness, responsiveness and accessibility (Eg “the service we get is very prompt” or “They usually have the problem fixed in an hour but an hour is a long time in a doctor’s office.)Add continuity to the definition ie the desire of the client for someone who will deal with the entire problem and/or the desire of the client to deal with someone they have dealt with previously regarding a problem.
Business/Operational Model	Refers to issues around the need to pay, and the amount of payment, required to obtain support from the vendor. Also refers to comments from the vendor about the way they organize support.
System Quality	This section refers to the characteristics of the EMR itself. Note if there are quality issues identified that do not fit into one of the attributes below, code as system quality (sys qual)
Availability	Is the program and the computers that run it generally available to the user when needed; use this term to capture comments related to down time. (e.g. “,It’s improving all the time; when I first started it would go down a lot but it is better over time.”)
Response Time	Does the program and the computers respond quickly to commands or do the users find it slow; e.g.,“ For the most part it works well but sometimes it’s very slow and that’s very frustrating”
Interoperability	Comments related to the ability of the users of the EHR to communicate with other providers outside the FHT such as labs; pharmacies or hospitals who use different information systems
Information Quality	Comments related to the completeness and accuracy of the information available from the EMR.
Completeness	Is the information that is in the EMR complete and accurate; is all patient data entered?
Consistency	Is data entered in the same way, using the same terms and acronyms? Use this code for comments that include a garbage in/garbage out component.
Legibility	Refers to the ease of reading information on the EHR.
Usage	Describes the use of the EMR in the FHT or among the staff.
Functions	What functions or features of the EMR are used or are particularly liked or disliked or that are most useful to particular categories of users; (e.g. we chart all information about the visit, the past medical history; all the labs”). Do not use this code for desired functions; an additional node has been established under desired functions which is now added to the usage node. Also use for comments about the extent to which or how much of the system capability is used or the efficiency of use (e.g. “people probably are using about 20 to 30 % of what this thing is capable of”) Use for positive and negative comments about functions built into the system (eg “the system wants you to put in drug orders in a certain way such as 1 tablet for 30 days but then after 30 days the prescription will disappear and so you have to kind of look to see was the drug prescribed” Code functions/workflow when the function of the EMR causes a change to the way work is done; If a function causes a problem to the user and you want to note that, then write an annotation(1)
Desired Functions	Use when the interviewee comments about attributes or functions that she/he wishes were available in the system
Satisfaction	Comments related to how satisfied the interviewee is with the system and how they perceive the satisfaction of others.
Overall satisfaction	Use to code comments about the degree to which the interviewee is satisfied or not satisfied with the use of the EMR. (“I don’t have a lot of complaints about it-I really like it”) and also how the interviewee describes how other users like the system. Use also to code comments they make about their satisfaction with specific features of the system; (Eg Its user friendly, I don’t have to be afraid of making mistakes”)
Individual Impact
Workload	Use to describe any comments about the effect of the EMR on the amount of work required of the person being interviewed (e.g., “I don’t find it adds to the workload but it takes just as much time.”)
Workflow	Changes in work processes that occur as a result of the EMR; Use this code for comments about changes to the way a provider communicates with other providers and for comments that relate to the ability to access information quickly and easily, from remote sites, or by other providers. (e.g.” it’s a lot easier to find things-- to keep track of things)”
Patient provider Communication	Code comments related to communication between providers and their patients. (Eg you find you are not paying as much attention to them in terms of eye contact initially although I am getting better at that.”)
Organization Impact	Refers to changes at the organizational level as a result of use of the EHR.
Technical dependency	Use to code comments related to the inability of the organization to do any work if the system is down.
Coordinated Care	Refers to quotes that indicate how EMR use affects the process of communication among providers and facilitates coordination of care; eg I think it puts everybody involved in the health care in a position to recognize that the information goes into a central place, to be coordinated, if the chips are down, by just one person whose job it is to pull all your health, all that information, and I think that’s a tremendous benefit to health care.
Patient Care Impact	Use for comments on the impact of the EMR in patient care.
Preventative Health Care	Used for comments related to use of EMR for preventative health care such as immunizations and periodic tests (mammography, pap smears, colonoscopies, etc.). May also refer to adhering to preventive care guidelines
Monitoring and Surveillance	Use to describe the use of the EHR for follow up on patients with chronic conditions, effect of treatments, or test results (HbA1c, Vit. D levels, blood pressures…)
Patient Education	Used for comments related to use of EMR as an education tool for patients’ (Eg “you can trend data to show patients, provide patients access to guidelines”.
Patient Safety	Used to note the effect on patient safety that results from use of the EMR ie drug interaction alerts;
Emerging Themes	Themes that emerge from the data and do not fit with any of the other categories
Digital Island	Use to code comments related to the fact that users are operating in an electronic arena but working in a community where their colleagues are using paper systems.
Learning by trial and error	Use to code comments related to the interviewees need or ability to learn the functions of the program on their own, on a trial and error basis.
User Characteristics	This is a new node added to code characteristics of the users of vendor provided support and training.
Age	Comments related to the age of those seeking training support or training.
Practice Characteristics	Code comments relating to the number of physicians in the practice or the geographic location i.e. rural vs. urban practices
User Role	Use to identify the role of those who participate in training programs or who call for support; could include doctors, admin staff or super users.
Computer Experience	Use to code comments related to the expertise, or lack of computer expertise, of those participating in training or requesting support;
Resistance	Use to code comments related to computer fear or other comments that might indicate resistance to working with computers.

^*Code definitions were first published in Shachak et al., 2012, Informatics in Primary Care, 20(3): 185–195 and are reprinted here with permission of the publisher.