Abstract
Background
Health information exchange (HIE) is a potentially powerful technology that can improve the quality of care delivered in emergency departments, but little is known about emergency physicians’ current perceptions of HIE.
Objectives
This study sought to assess emergency physicians’ perceived needs and knowledge of HIE.
Methods
A questionnaire was developed based on heuristics from the literature and implemented in a Web-based tool. The survey was sent as a hyperlink via e-mail to 371 attending emergency physicians at 12 hospitals in New York City.
Results
The response rate was 58% (n = 216). Although 63% said more than one quarter of their patients would benefit from external health information, the barriers to obtain it without HIE are too high—85% said it was difficult or very difficult to obtain external data, taking an average of 66 minutes, 72% said that their attempts fail half of the time, and 56% currently attempt to obtain external data less than 10% of the time. When asked to create a rank-order list, electrocardiograms (ECGs) were ranked the highest, followed by discharge summaries. Respondents also chose images over written reports for ECGs and X-rays, but preferred written reports for advanced imaging and cardiac studies.
Conclusion
There is a strong perceived need for HIE, most respondents were not aware of HIE prior to this study, and there are certain types of data and presentations of data that are preferred by emergency physicians in the New York City region.
Introduction
Emergency departments (EDs) in the United States are in crisis. Between 1992 and 2003, the number of ED visits in the United States increased more than 26% from 90.3 million to 113.9 million visits annually, whereas during the same period the number of hospital EDs decreased by 425 or 12.3% and the number of hospital beds decreased by 198,000. 1 The recent Institute of Medicine report Hospital-Based Emergency Care: At the Breaking Point specifically calls for increased use of information technology and, more specifically, for regional health information organizations (RHIOs) and a nationwide health information network (NHIN) that would provide health information exchange (HIE), bringing vital information to the bedside to help improve efficiency in the health care system, combat the problem of overcrowding, and improve safety and quality. 2
Many of the problems with safety and quality in EDs are due to a lack of provider access to vital patient data at the point of care due to information gaps. 3,4 These information gaps are likely caused to a large extent by patient migration among providers, institutions, and health plans and decentralization and fragmentation of the overall healthcare system. 5–8 The promise of HIE to fill these information gaps by providing essential data at the point of care is substantial; for this reason more than 165 RHIOs in 49 states nationwide are currently developing HIE in one form or another according to a survey conducted in 2006, and 26 of these are fully operational. 9 This is an increase from the 109 RHIOs found in the same survey just one year prior. 10
The ED is an information-intensive environment, yet it functions at a baseline information deficit. Emergency physicians have become accustomed to making decisions without sufficient access to relevant information, and this information deficit is an inherent characteristic of the current practice patterns and standards by which care is delivered. 11,12 With the advent of HIE networks and a large increase in access to extrainstitutional, outside clinical information, the standards of care in the ED may change. 13
To maximize the benefits of this improved access to outside clinical information, preliminary work needs to be done to determine the data needs of the clinicians and the proper way in which to implement HIE systems to be well integrated into ED workflow. Previous studies have shown that ED workflow falls into one of three types depending on current health information technology (HIT) adoption and that this may affect how HIE data might be presented, 14 that when longitudinal (HIE-like) data from within an organization is made more accessible to emergency physicians usage increases and admissions may decrease (A. Wilcox, unpublished data, 2007), that emergency physicians will use different types of longitudinal data to varying degrees, 15 that the ED has specific information needs for clinical practice and research that overlap to some degree, 16,17 and that these information needs vary with diagnoses. 18 Overhage et al. 19 conducted a previous survey of emergency physicians prior to conducting a randomized controlled trial of an HIE intervention looking specifically at the problems they experienced trying to get outside information prior to the intervention, and which of the data elements provided by the HIE system were most useful. As part of a project to implement HIE in EDs in the New York City region, this study was conducted to address the following four questions: (1) What is the perceived need for external health information, and what are the barriers to its access with current practices? (2) What is the current state of awareness among emergency physicians of HIE? (3) What is their perception of the costs and benefits of HIE? (4) Prior to the implementation of HIE, what are their preferred data elements a priori, and how would emergency physicians like to have the information presented?
Methods
Context
NYCLIX (New York Clinical Information Exchange), Inc., is a not-for-profit organization with a mission to implement HIE in the New York metropolitan region to improve the quality, efficiency, and safety of patient care. The current participants of NYCLIX are 14 stakeholder organizations including hospitals, ambulatory care organizations, and a regional home care agency. 20 In July 2005 NYCLIX convened its Clinical Advisory Subcommittee. The mission of the Clinical Advisory Subcommittee is to involve future end-users of the system in NYCLIX planning and implementation phases to gain valuable input and better ensure end-user acceptance and a successful implementation. This subcommittee is composed of clinical faculty and administrators, including ED clinicians from 12 hospitals across the New York metropolitan area who were designated by their chairperson or director. One of the group’s first orders of business was to conduct a survey of all participating departments, which is outlined in this article, to help inform the future development of the NYCLIX project.
Questionnaire Development
The survey methodology literature was reviewed with a focus on Web-based surveys. 21–29 Heuristics derived from this literature search were followed in the instrument design (e.g., no forced responses; number of remaining questions was displayed at bottom of page; “N/A”, “Do not know”, or free-text entry answer choices were made available where appropriate). A preliminary instrument was created to answer the four principal questions of the study that are stated above and then implemented in SurveyMonkey, a Web-based survey tool. 30 The survey was administered to members of the Clinical Advisory Subcommittee to test for clarity and face validity. Feedback was solicited, and the survey instrument was refined through an iterative process by the investigators. A exemption from the institutional review board at Mount Sinai Medical Center was granted because the survey was voluntary, required no interaction with patients, contained no protected health information, and kept individual responses anonymous.
Subjects
All attending emergency physicians who were currently employed at the 12 sites were considered subjects. The Clinical Advisory Subcommittee representative from all but one site provided a list of e-mail addresses for their department’s entire clinical faculty. One site representative was required to ask for their clinical faculty to provide their e-mail addresses for the study ahead of time on a voluntary basis, and therefore only a subset of this department’s faculty received the questionnaire.
Survey Implementation
Once the survey instrument was finalized, it was deployed through the Subcommittee members to their respective sites via an e-mail hyperlink so that the request came from a known clinical colleague. The hyperlink took the respondent to a survey that collected data separately for each of the 12 sites. Recipients of the e-mail could opt out by following a second hyperlink at the bottom of the e-mail message, which permanently blocked them from further correspondence related to the survey. Nonresponders were sent weekly e-mail reminders for a total of six weeks until they responded or opted out. After six weeks the survey was closed.
Analysis
Data were downloaded directly into Microsoft Excel, and identifiers for individual respondents were immediately removed. Simple frequencies were calculated and, where appropriate, data were transferred into SPSS version 12 (SPSS Inc., Chicago, IL) for additional statistical analyses including calculation of confidence intervals.
Results
A total of 371 surveys were sent to attending emergency physicians at 12 EDs in New York City. Department locations are broken down as follows: Manhattan 8, Queens 1, Bronx 1, and Brooklyn 2. All hospitals in the study are considered teaching hospitals with some degree of resident training, but three of the departments in Manhattan did not have emergency medicine residency programs and were among the smaller departments. The mean size of departmental faculty at each site was 31 (range 13 to 60).
The final overall response rate was 58% (n = 216). Across institutions, the mean response rate was 56% (range 21% to 93%). Results can be broken up into four sections, corresponding with the original four study questions that are listed above.
Current State and Need for HIE
As shown in ▶, when asked if access to clinical information from providers outside of their hospital’s network would be helpful, 39% said more than half of the patients under their care would benefit (95% confidence interval [CI] 32.4% to 45.6%), and 63% said more than one quarter of the patients under their care would benefit (95% CI 56.8% to 69.9%). That notwithstanding, 86% said it is currently difficult or very difficult to obtain clinical information from outside providers with current phone, fax, and paper consent requirements (95% CI 80.7% to 90.2%); 66% said that when they do try to obtain data from external sources their efforts fail more than half of the time (95% CI 63.1% to 69.6%). When asked “How long do you estimate it currently takes to get clinical information from providers outside of your hospital’s network?”, they reported an average of 66 minutes (95% CI 48.2–83.7 min); 56% reported that they currently attempt to obtain clinical information from providers outside their hospital’s network for only 0 to 10% of their cases (95% CI 49.4% to 62.7%).
Table 1.
Table 1 Current Access to External Health Information, Perceived Costs and Benefits, and Current State of Awareness of Health Information Exchange
| Response (N = 216 respondents) | Rate | 95% Confidence Interval |
|---|---|---|
| >1/2 of emergency department patients would benefit if access to outside clinical information were immediate | 39% | 32.4%–45.6% |
| >1/4 of emergency department patients would benefit if access to outside clinical information were immediate | 63% | 56.8%–69.9% |
| It is currently difficult or very difficult to obtain relevant outside clinical information at the point of care in the emergency department | 86% | 80.7%–90.2% |
| Current attempts to obtain clinical information from other area providers fail more than 1/2 of the time | 66% | 63.1%–69.6% |
| Estimated number of minutes it currently takes to get clinical information from outside providers | 66 min | 48.2–83.7 min |
| Attempt to get outside clinical information ≤10% of the time | 56% | 49%–62.7% |
| HIE would benefit the delivery of clinical care | 97% | 94.2%–99.1% |
| HIE would benefit the emergency department | 92% | 88.7%–95.9% |
| HIE would benefit the hospital | 91% | 87.5%–95.2% |
| HIE would benefit the health care system overall | 98% | 95.5%–99.7% |
| HIE would increase or greatly increase the efficiency of clinical care | 88% | 83.6%–92.4% |
| HIE would decrease or greatly decrease the number of tests ordered | 78% | 72.1%–83.4% |
| HIE would decrease or greatly decrease the number of medical errors | 64% | 57.4%–70.5% |
| HIE would increase or greatly increase the time to disposition decision on individual patients | 54% | 47.1%–60.6% |
| HIE would decrease or greatly decrease health care costs | 64% | 57.1%–70.2% |
| The benefits of HIE outweigh or greatly outweigh the risks | 85% | 80.3%–89.9% |
| Never heard of HIE prior to this survey | 76% | 70.4%–82.0% |
| HIE will eventually exist locally | 71% | 64.7%–77.1% |
| HIE will eventually exist regionally | 64% | 57.7%–70.8% |
| HIE will eventually exist nationally | 50% | 42.9%–56.6% |
| HIE will eventually exist internationally | 18% | 12.8%–23.3% |
HIE = health information exchange.
Perceived Costs and Benefits of HIE
Respondents were asked what benefits might be realized if outside clinical information were made available: 97% said the way care is delivered would benefit (95% CI 94.2% to 99.1%), 92% said their department would benefit (95% CI 88.7% to 95.9%), 91% said their hospital would benefit (95% CI 87.5% to 95.2%), and 98% said the overall health care system would benefit (95% CI 95.5% to 99.7%). With HIE, 88% thought the efficiency with which care is delivered would increase or greatly increase (95% CI 83.6% to 92.4%), 78% thought the number of tests ordered would decrease or greatly decrease (95% CI 72.1% to 83.4%), and 64% thought the number of medical errors would decrease or greatly decrease (95% CI 57.4% to 70.5%). When asked to take into account what they thought would be the expense of designing and implementing HIE in New York, 64% thought the effort would decrease or greatly decrease the health care costs in the region (95% CI 57.1% to 70.2%). Even though 85% thought the overall benefits of rapid and easy access to outside clinical information outweighed or greatly outweighed the risks (95% CI 80.3% to 89.9%), 54% still responded that they thought the time to disposition decision on individual patients would increase or greatly increase with the additional information (95% CI 47.1% to 60.6%).
Current State of Awareness of HIE
Respondents were given the following definition: “Health information exchange (HIE) would allow relevant clinical information to be easily and rapidly delivered through a computer interface at the point of care in your department.” In addition, they were told that HIE is already in place in several other U.S. cities. Seventy-six percent responded that they had never heard of HIE before this survey (95% CI 70.4% to 82.0%). When asked if they thought it would succeed locally, 71% said yes (95% CI 64.7% to 77.1%), regionally 64% said yes (95% CI 57.7% to 70.8%), nationally 50% said yes (95% CI 42.9% to 56.6%), and internationally only 18% said yes (95% CI 12.8% to 23.3%).
Preferred Data Elements and Presentation
Respondents were asked “Do you have thoughts on what data elements you would like to have available in an HIE system” and if they answered yes, they were taken to an optional fourth page of the survey. Eighty-seven respondents elected to fill out this optional page, for a response rate of 40% of all respondents.
Respondents were asked to choose, in order of preference, their top five data elements out of a list of 11 possible data elements (▶). They included electrocardiograms 80% of the time (95% CI 71.5% to 88.5%), and these were chosen first by 34% of respondents (95% CI 24.0% to 44.2%). Discharge summaries were chosen in the top five by 66% (95% CI 55.8% to 76.0%), and were the next most frequently chosen first by 15% of respondents (95% CI 7.6% to 22.9%). Medication lists were chosen in the top five by 65% (95% CI 54.5% to 74.9%); laboratory results, radiology reports, and problem lists each by 59% (95% CI 48.4% to 69.3%); provider information by 44% (95% CI 33.0% to 54.1%); cardiology reports by 38% (95% CI 27.4% to 48.0%); allergy information by 21% (95% CI 12.5% to 29.9%); endoscopy reports by 3.5% (95% CI 0 to 7.5%); and patient demographics by 2.4% (95% CI 0 to 5.6%).
Figure 1.
Percentage of respondents who chose each data element in their top five from a pick list.
When asked whether they would prefer images or written reports for certain data types (▶), written reports were preferred for cardiac catheterization by 100% of respondents, for echocardiograms and nuclear medicine by 98% (95% CI 94.5% to 100%), for ultrasound by 98% (95% CI 94.6% to 100%), and for endoscopy by 100%. Reports were preferred for computed tomography scans by 63% (95% CI 53.1% to 73.4%). For plain X-rays, 74% preferred the actual image (95% CI 64.3% to 82.8%), and 98% preferred the images for electrocardiograms (95% CI 94.4% to 100%). When asked how important it was to have both images and written interpretations, the percent of the time respondents answered absolutely necessary or very important for each study was as follows: plain X-rays 60% (95% CI 49.5% to 70.1%), computed tomography scans 48% (95% CI 37.1% to 58.2%), electrocardiograms 63% (95% CI 53.0% to 73.4%), echocardiograms 13% (95% CI 5.8% to 20.1%), ultrasound 17% (95% CI 8.6% to 24.4%), nuclear medicine 11% (95% CI 4.0 to 17.1%), and endoscopy 12% (95% CI 4.9% to 18.6%).
Figure 2.
Respondents were asked to choose either images or written reports for the following types of data with a mutually exclusive response.
Discussion
Emergency physicians work on the front lines of the health care system, where a patient’s full data set often is not available. There is an obvious need for information to flow more easily among providers, and also a need for access to this information when new patients present themselves to an ED, especially when they are very sick or unable to provide a complete or accurate history because of their condition.
The emergency physicians in this study perceived a strong need for easy and rapid access to information from providers outside of their own health care institution, but currently are unable to access this data in a reasonable manner, and often have to make decisions without a complete picture. A majority believe that access to this information will help them improve efficiency while reducing errors and costs, and thought that their patients would benefit.
This study contrasts with the one mentioned in the introduction by Overhage et al. 19 because this study was conducted in a larger number of practice environments (12 hospital EDs versus two in the Overhage et al. 19 study), included six times the number of respondents (216 vs. 36), and had a slightly higher response rate (58% vs. 50%). Some of the results from the previous study are similar and lend themselves to comparison with the data presented here. For example, physicians in the study by Overhage et al. 19 estimated that they attempted to obtain medical record data by facsimile from other institutions for 5% of patients, and in this study more than half of respondents said they attempt to get data from providers outside their hospital’s network 10% of the time or less. Other results that lend themselves to comparison are quite different. For example, physicians in the study by Overhage et al. 19 estimated that these attempts to get data without HIE took 15 minutes, whereas respondents in this study estimated that time to be 66 minutes per attempt. The reason for such a discrepancy could be due to differences in the way each survey’s questions were phrased, or in the study populations or their practice environments, none of which can be determined with available information. Although the previous study does ask about preferences regarding types of data presented, this was determined based on data that were already being presented to them on abstracts printed through use of their HIE network. The data presented in the current study explore the perceived data needs of emergency physicians a priori, without having already implemented HIE.
Interestingly, 54% thought the time to disposition decision on individual patients would increase or greatly increase with the additional information. There is probably some concern regarding the amount of time that would be necessary to acquire and synthesize a new, possibly much larger data set. For this reason it is of the utmost importance that the user interface be well designed and that clinicians be involved in the planning and implementation process. Being able to sort by time, location, test, or abnormal findings and drill down to exactly the information that is needed would be tremendously useful in helping the clinicians filter through a potentially massive data set for individual patients.
There are several important limitations to this study. First, the results from the Preferred Data Elements and Presentation section are from a smaller number of respondents than the overall survey, and therefore might not be representative of the entire study population. This section was made optional because it was thought that the length it added to the survey and the more technical nature of the questions in this section would lead respondents to stop completion of the survey and cause an overall loss of data.
Second, these results are limited to the clinical domain of emergency medicine. RHIOs are developing HIE networks across the country testing a variety of use cases, and NYCLIX itself is planning to expand into additional clinical domains. It is unlikely that radiologists, for example, when given access to an HIE network, are going to be satisfied with only a written report for advanced studies. They will likely want to see the actual images to draw comparisons and formulate their own interpretation. Likewise, cardiologists will probably want to see the actual video or images from advanced cardiac studies to draw conclusions and make decisions for their patients. To know what the clinical information needs are for these other specialties, similar studies with new study populations will need to be conducted.
Another limitation is that these results are limited in scope geographically to the New York metropolitan area. There may be idiosyncrasies in the way that emergency medicine is practiced in this region, and the practice patterns and data needs in other geographic areas could vary from those presented in this study. Additionally, some EDs in the study spanned more than one physical hospital site, but shared a common faculty or shared a common academic departmental structure and little information was gathered on the distribution of individual characteristics of their faculty (e.g., age, gender, years in practice). Furthermore, due to the voluntary nature of the responses, there is a potential bias introduced because of differences in the types of subjects who actually completed the survey versus those who did not. Although monetary or other tangible incentives may have helped reduce this bias, other measures that fit within the study budget were used (described above in Methods). Finally, this survey elicited the perceptions and opinions of these individuals, and their estimates may not be accurate if compared with actual measured impacts. Also, the respondents answered questions on topics in which they may not be expert, such as health care economics and human–computer interaction.
Conclusion
There is a tremendous need for health information exchange in the EDs, and emergency physicians believe that having access to currently siloed data at outside institutions will benefit patient care and the efficiency with which health care is delivered. This study begins to tell the story of what clinicians in the ED want and need from an HIE network. These results may prove very useful in creating a requirements document to build an HIE network. As RHIOs and HIE progress and expand into other clinical domains, both inpatient and outpatient, it will be important to seek similar input from clinicians to ensure that their information needs are being met.
Acknowledgments
The authors thank the NYCLIX Clinical Advisory Subcommittee members for helping to design the survey instrument and coordinating the survey deployment in their respective departments. Subcommittee members include: Buz Abrams, MD, Montefiore Medical Center; Gregg Almond, MD, Metropolitan Hospital; Carol Barsky, MD, Saint Vincent’s Catholic Medical Center; Phillip Fairweather, MD, Elmhurst Hospital Center; Peter Fromm, RN, New York Downtown Hospital; William Goldberg, MD, New York University Medical Center; Joseph Kannry, MD, Mount Sinai Medical Center; Past Chair, Clinical Advisory Subcommittee; Stuart Kessler, MD, Elmhurst Hospital Center; Paris Lovett, MD, New York Presbyterian Hospital; Pravene Nath, MD, New York University Medical Center; Peter Peacock, MD, Kings County Medical Center; Kenneth Sable, MD, Maimonides Medical Center; Jason S. Shapiro, MD, Mount Sinai Medical Center; Chair, Clinical Advisory Subcommittee; Benjamin Stein, MD, Lennox Hill Hospital; Gabriel Wilson, MD, St. Luke’s - Roosevelt Hospital.
Footnotes
NYCLIX Clinical Advisory Subcommittee membership listed in the Acknowledgments section
Supported by training grants from the National Library of Medicine (T15 NLM 007079 13–15 2005-2006) and Pfizer (2006-2007) (Dr. Shapiro). Preliminary results for these data have been presented as poster abstracts at the Spring American Medical Informatics Association 2006 meeting and the American College of Emergency Physicians 2006 meeting.
References
- 1.McCaig LF, Burt CW. National Hospital Ambulatory Medical Care Survey: 2005 emergency department summary Adv Data 2004;340:1-34. [PubMed] [Google Scholar]
- 2.Institute of Medicine Hospital-Based Emergency Care: At the Breaking PointWashington, DC: National Academy Press; 2006.
- 3.Stiell A, Forster AJ, Stiell IG, van WC. Prevalence of information gaps in the emergency department and the effect on patient outcomes CMAJ 2003;169:1023-1028. [PMC free article] [PubMed] [Google Scholar]
- 4.Smith PC, Raya-Guerra R, Bublitz C, Parnes B, Dickinson LM, Van VR, et al. Missing clinical information during primary care visits JAMA 2005;293:565-571. [DOI] [PubMed] [Google Scholar]
- 5.Institute of MedicineCommittee on Quality of Health Care in America To Err Is Human: Building a Safer Health SystemWashington, DC: National Academy Press; 2000.
- 6.Institute of MedicineCommittee on Quality of Health Care in America Crossing the Quality Chasm: A New Health System for the 21st CenturyWashington, DC: National Academy Press; 2001. [PubMed]
- 7.Finnell JT, Overhage JM, Dexter PR, Perkins SM, Lane KA, McDonald CJ. Community clinical data exchange for emergency medicine patients AMIA Annu Symp Proc 2003:235-238. [PMC free article] [PubMed]
- 8.Greater New York Hospital Association. [newsletter on the Internet] New York: GNYHA Members Initiate Clinical Data Exchange Project. GNYHA 2005 July. Available at: http://www.gnyha.org/pubinfo/skyline/skyid20050725.pdf. Accessed November 9, 2005.
- 9.Bordenick JC, Marchibroda J, Welebob E. Improving the Quality of Healthcare through Health Information Exchange: Selected Findings From eHealth Initiative’s Third Annual Survey of Health Information Exchange Activities at the State, Regional and Local Levels 2006. eHealth Initiative September 25. Accessed March 10, 2007. Available at: http://toolkits.ehealthinitiative.org/assets/Documents/eHI2006HIESurveyReportFinal09.25.06.pdf.
- 10.Marchibroda J, Bordenick JC. Emerging Trends and Issues in Health Information Exchange: Second Annual Survey of State, Regional and Community-Based Health Information Exchange Initiatives and Organizations 2005. eHealth Initiative. Accessed October 5, 2005. Available at: http://ccbh.ehealthinitiative.org/communities/register_download.mspx.
- 11.Cordell WH, Overhage JM, Waeckerle JF. Strategies for improving information management in emergency medicine to meet clinical, research, and administrative needsInformation Management Work Group. Acad Emerg Med 1998;5:162-167. [DOI] [PubMed] [Google Scholar]
- 12.Cordell WH, Overhage JM, Waeckerle JF. Strategies for improving information management in emergency medicine to meet clinical, research, and administrative needsThe Information Management Work Group. Ann Emerg Med 1998;31:172-178. [PubMed] [Google Scholar]
- 13.Shapiro JS, Kannry J, Lipton M, et al. Approaches to patient health information exchange and their impact on emergency medicine Ann Emerg Med 2006;48:426-432. [DOI] [PubMed] [Google Scholar]
- 14.Friedmann BE, Shapiro JS, Kannry J, Kuperman G. Analyzing workflow in emergency departments to prepare for health information exchange AMIA Annu Symp Proc 2006:926. [PMC free article] [PubMed]
- 15.Hripcsak G, Sengupta S, Wilcox A, Green RA. Emergency department access to a longitudinal medical record J Am Med Inform Assoc 2007;14:235-238. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Teich JM. Information systems support for emergency medicine Acad Emerg Med 1998;5:271-274. [DOI] [PubMed] [Google Scholar]
- 17.Teich JM. Information systems support for emergency medicine Ann Emerg Med 1998;31:304-307. [DOI] [PubMed] [Google Scholar]
- 18.Shablinsky I, Starren J, Friedman C. What do ER physicians really want?A method for elucidating ER information needs. Proc AMIA Symp 1999:390-394. [PMC free article] [PubMed]
- 19.Overhage JM, Dexter PR, Perkins SM, et al. A randomized, controlled trial of clinical information shared from another institution Ann Emerg Med 2002;39:14-23. [DOI] [PubMed] [Google Scholar]
- 20.The New York Clinical Information Exchange (NYCLIX). [homepage on the Internet] Available at: http://www nyclix org 2007 Accessed March 13,2007.
- 21.Wyatt JC. When to use web-based surveys J Am Med Inform Assoc 2000;7:426-429. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Eysenbach G, Wyatt J. Using the Internet for surveys and health research J Med Internet Res 2002;4:E13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) J Med Internet Res 2004;6:e34. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Ritter P, Lorig K, Laurent D, Matthews K. Internet versus mailed questionnaires: a randomized comparison J Med Internet Res 2004;6:e29. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Leece P, Bhandari M, Sprague S, et al. Internet versus mailed questionnaires: a randomized comparison (2) J Med Internet Res 2004;6:e30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Braithwaite D, Emery J, De LS, Sutton S. Using the Internet to conduct surveys of health professionals: a valid alternative? Fam Pract 2003;20:545-551. [DOI] [PubMed] [Google Scholar]
- 27.Kellerman SE, Herold J. Physician response to surveysA review of the literature. Am J Prev Med 2001;20:61-67. [DOI] [PubMed] [Google Scholar]
- 28.Schleyer TK, Forrest JL. Methods for the design and administration of web-based surveys J Am Med Inform Assoc 2000;7:416-425. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Dilman D, Tortora RL, Boweker D. Principles for Constructing Web Surveys 1998. Presented at the Joint Meetings of the American Statistical Association. Dallas, Texas: August. Available at: http://www.sesrc.wsu.edu/dillman/papers/websurveyppr.pdf. Accessed March 24, 2006.
- 30.SurveyMonkey. Available at: http://www surveymonkey.com/2006. Accessed March 24, 2006.