AMIA members’ “vital signs”: what the HIT implementation listserv says about goals for AMIA and for medical informatics

Abstract

The health information technology (HIT) implementation listserv was conceived as a way to combine a substantial portion of American Medical Informatics Association (AMIA) members who belonged to four working groups (WGs): CIS, Evaluation, ELSI, and POI. Other AMIA members joined in significant numbers. It immediately became a major forum for discussing medical informatics, informatics policies, and discussion of the purpose of AMIA itself. The listserv membership approximates 25% of AMIA’s members and has generated over 6,000 posts. We report on a survey of the listserv’s members: what members think about the listserv; what participants want for medical informatics; how they think those goals should be achieved, and what AMIA’s role should be in this process. The listserv provides vital signs about AMIA and hopes for informatics. We combine qualitative analysis of members’ comments and responses about the listserv using ATLAS.ti qualitative text analysis tool and a word cloud generator.

Introduction

The lifeblood of a membership organization like the American Medical Informatics Association (AMIA) is, not surprisingly, its members. Among the reasons members join is because they value interacting with other members through conferences, journals, committees, working groups, and listservs. Each AMIA Working Group (WG) has its own listserv, through which members exchange information, carry on discussion, post announcements, organize conference sessions, and keep in touch with each other. In addition, one cross-group listserv serves these functions for the large section of the membership interested in implementation issues.

Twelve hundred members from AMIA (~25% of the organization’s members) subscribe to the AMIA “Implementation and Optimization Forum” listserv (hereafter, the “listserv”). This listserv was created in 2012 when multiple cross-postings to the four separate AMIA WGs’ listservs frustrated participants with overlapping and redundant conversations. The postings’ high volume and engaging content motivated the four groups to form a new combined listserv to which all members of the four WGs – the Clinical Information Systems WG, the Evaluation WG, the Ethical, Legal and Social Issues WG, and the People and Organizational Issues WG – were automatically subscribed. Other AMIA members also were invited to join. By October of 2014 the participants had posted 5392 messages in 824 separate discussion threads. As expected, there is a Pareto-like distribution of members’ postings, i.e., some members post far more frequently than many others.

The lively discussions and frequent interchanges reflect the pulse of AMIA membership. Informaticians from different disciplines, countries, and traditions discuss a range of topics, including implementation experiences, ethical conundrums, change management, organizational issues and politics, regulatory environment, workflow, open source, influential publications, design, and cutting-edge content that has not yet reached medical informatics textbooks or journals. Moreover, the listserv was intended to create synergy and energy across working groups. This has apparently been accomplished. Many from outside of the previous four groups have joined the new listserv.

While listservs are not new, they still create community, enhance communication, enable sharing information and resources, and provide a venue for engaging discussion. [1–9] However, there is little research on how they can be used to support knowledge dissemination and how individual scientists use a listserv for various purposes, including education, conversation, and knowledge dissemination. In December 2014, we surveyed members of the listserv to gain insight into their use of and perceptions about the listserv itself. The survey solicited members’ views on what they most appreciated about this listserv, their suggestions to improve it, and topics they would like to have discussed. We report responses from over 200 AMIA listserv subscribers who responded to fixed-choice and open-ended questions.

We previously reported on some of the listserv content and discussions [10–11] and also have reported some preliminary survey findings from early respondents that focused on the fixed-choice survey responses [12]. This paper complements that work in that we focus on the free text comments. They indicate the reasons AMIA members find the listserv of considerable value and ways they think it might be improved. Our survey results indicate how a significant proportion of AMIA members interact, and would like to interact, suggesting ways AMIA can augment on-line member benefits and services. They also point to topics members consider important for improving implementation of health information technologies (HIT), and, ultimately, health care, that AMIA members, and AMIA as an organization, can address. The survey results indicate what members consider important, showing the vitality of both the listserv and of members’ interest in expanding both its functionality and discussion. The survey, therefore, as well as the listserv, measures some of AMIA’s “vital signs.”

Methods

Three methods were used. First, a survey was conceived to gain insight into AMIA members’ perceptions about the listserv. The authors developed the survey instrument between July and September 2014. The survey was pre-tested starting in mid-September 2014 with a group of 50 volunteer AMIA members who were asked to answer the questions and provide suggestions for revising them. Based on the feedback from the pretest, the authors then formulated and iteratively refined the questions via pretesting with 17 of the volunteers. The final version of the survey had 8 questions that ranged from how often people read and make listserv postings to what they like and dislike about the listserv, as well as what they would like to see improved. With the exception of Question 8, all questions were fixed-choice, and Questions 4, 5, 6, and 8 also asked people for free text comments. The fixed choice questions were not limiting in choices, so a respondent could select as many choices as they wish. The survey was IRB approved and respondents were offered anonymity. The survey was officially launched on December 2, 2014 via SurveyMonkey®. [13] All 1200 Forum members were invited to participate and, after several reminders, by February 1, 2015, 223 people responded to the survey.

Second, the numerical data was analyzed by SurveyMonkey® using descriptive statistics. Third, the free text comments were exported into ATLAS.ti [14] and analyzed to identify themes related to the numerical data. Three of the authors (BK, KR, and CK) used ATLAS.ti to independently code the free text data for Questions 4, 5, and 8, focusing on what people most appreciated about the listserv and their suggestions for improvement. After completing their independent coding the three authors then integrated their analyses and used an iterative process to develop thematic consensus. Free text quotations are used throughout the findings to emphasize or further illustrate certain themes. The other authors (MA, SA, and RK) also analyzed the free text data from Questions 4, 5 and 8 questions to identify respondents’ suggestions for AMIA. The four comments to Question 6, which asked participants if they referred others to the listserv, were not analyzed and are not germane to this paper.

For further clarity, a word cloud provided a visual display of each of the Questions 4, 5, and 8 free text comments. For this analysis, the word cloud only gives size as a visualization of word frequency, not necessarily the importance of the words, a task that falls to the authors for the interpretation. [15]

Results

The response rate for the survey was 18.5% and the final number of participants was 223. The findings are structured according to the data from questions 4, 5, and 8. For each question we briefly summarize the fixed choice responses, rounded to the nearest integer, then describe the themes from the free text responses, and conclude with a Word Cloud that illustrates the free text responses. When we describe the themes from the free text data we use quotations from the data to emphasize or clarify certain themes.

The first question about the listserv asked “What aspects of the listserv do you most appreciate?” Two hundred and twenty three fixed choice responses were received and were overwhelmingly positive: Approximately 60% of the respondents said the value of the listserv is in “interesting comments from colleagues (64%), “different perspectives” (64%), and “learning about important issues” (59%). Others commented on “the relative absence of political discourse” and the benefit of access to an international perspective. Moreover, in answer to Question 6, 52% said they have recommended the listserv to others.

While many respondents favored leaving the listserv as is (27% said “don’t change a thing”), the survey responses also indicated that some see room to improve. The answers to a subsequent open-ended question regarding how the listserv could be improved revealed three broad categories of recommendations: content of discussions, social rules, and technological improvements. We later discuss each of these in turn.

The free text (additional) comments expanded on the themes noted above. Twenty-one free text comments were received for the question of what respondents most appreciated. Seven of the 21 comments from this question highlighted how much people appreciated getting different perspectives from reading listserv postings. They liked reading different points of view and “the chance to discuss possible different way of doing things….” Perhaps most important is that the interdisciplinary perspective of the listserv was something that both clinical and non-clinical participants appreciated.

(Example response) As a clinician, it shows me how non-clinicians and academic personalities view issues…and helps me understand what physician informatics leaders are thinking.

Participants also reported reading real life experiences from actual HIT implementations. Some noted “real world experiences…show how the grass is not always greener across the pond.”

The second most appreciated value of the listserv was keeping readers informed of current developments. Four comments: new publications, “what’s hot,” upcoming webinars, AMIA activities, and instant dispersion of information.

Fig. 1’s word cloud reflects the sentiments expressed in both the fixed choices and free text comments about the aspects most appreciated in the listserv. Predominant terms are generally positive and welcoming: “learning,” “new developments,” “funny,” “humor,” “informs me,” “new developments,” “experiences” (explained, related), “value,” “discussions of problems,” “helps me understand,” “shows me,” (discusses) “reasons for actions,” “studies,” “publications,” “rapid information,” and “collaboration.”

Figure 1:

Word cloud reflecting responses (both fixed choice and free text) for Question 4: “What aspects of the listserv do you most appreciate?”

In tabular form, the most common responses were:

What aspects of the listserv do you most appreciate?
Interesting comments from colleagues	64%
Differing perspectives	63
Learning about important issues	58
Wide range of experiences reflected in the posts	56
Helps me stay abreast of developments	53
I appreciated the controversies reflected in the posts	52
Help me stay connected to the discipline	47
Informs me about new studies and publications	41
Some are well written	36
Some are funny	34
Help me learn about other participants	31
Some are written by people I follow/read	27
Help me stay connected to colleagues	22
All of the above	26

Note: Respondents could choose multiple items.

The next question asked for “Suggestions for improving the listserv” (N=209). The most popular fixed choice response requested that we “Ask people (politely) to make the headers reflect the actual topic,” which was chosen by 57% of respondents. As noted above, however, 27% of respondents selected “don’t change a thing” as a suggestion for improvement.

Of the 19 free text responses, some addressed technical issues, such as links not loading from postings, but most primarily referred to social aspects of the listserv: certain people dominating the discussions, some “rants” and “rock throwing,” and too many interpersonal discussions.

People’s suggestions to improve the social issues were varied. Many loved the freedom and spontaneity of the listserv, but some felt this led to tangents of limited interest. Respondents were equally divided on whether the list should be moderated. For some, the volume, repetition, and negativity made the listserv less attractive, while others thought it is a “must read.” While some wished for more intervention and listserv moderating (12 of the comments mentioned this) to improve netiquette and reduce the “bashing” and “beating the same drum over and over again,” they were equally split (11 of the comments) because others liked the “spontaneity.” Many welcomed new perspectives and the “frank exchange” of problems being exposed via the “savvy” postings.

(Example response) A few suggested the use of “providing polite training for those with knowledge gaps of how to respond to single individual vs all.”

The word cloud for this question (Fig. 2) reflects both the joy and frustration of the readers. On the joy side – the predominant side – we see terms like: “useful,” “original,” “great sprit(ed),” “constructive,” “discussions,” “conversations,” and “communications.” On the “frustrated” side, we find terms such as: “header” (as in fails to keep headers up to date), “attacks,” “personal,” “soapbox,” and “personal” (as in responses to personal comments or the rounds of congratulatory messages that some experience as clogging the list).

Figure 2:

Word cloud reflecting comments (free text) in response to question 5, about improving the listserv.

Finally we inquired: “What else would you like to tell us? For example, some have suggested we encourage more discussions of: cyber security of EHRs and PHRs; the role of the CMIO; specific concerns about the cost of healthcare IT (both initial software and the cost of implementation); inclusion of clinicians at key decision points in HIT design and implementation.” This question had only free text responses and 97 were received. Suggested topics included: improving EHRs and their implementation, usability or lack of it, HIT errors, standards (e.g., interoperability standards), the continuum of change (from talk to change), liability for lost data, HIT risks, change management, measurement and improvement of quality and safety, the barriers between IT and clinical and fiscal or monetary influences, gaps between technical people and healthcare professionals, and the topics listed in the question itself. Some suggested a topic of the month, while others liked that topics evolved, responded to the news, and reflected member interests and issues. A few commented that they liked the political discussion and ways to effect change, while others wanted less or no political discussion. Some also believe more presence of vendors as well as healthcare students who are currently trained using EMR systems in the listserv could be helpful. Overall, the freedom and spontaneity of the listserv was appreciated by some users but disliked by others who believe it leads to less meaningful dialogue.

(Example response) The beauty of the listserv is the spontaneous presentation of topics represented based on what’s relevant to all of us in the field and front lines. Attempting to influence the content detracts from this. You always have the choice to ignore a post or present a topic.

However, others would prefer to have set topics to focus the discussion.

(Example response) I wish it could somehow be focused, as the volume is far (!) too great. Really limit it to implementation and optimization issues, not every topic under the Sun. Some people seem to spend virtually all their time posting to the list…a more circumspect approach would be better.

More specifically, the diversity of participants meant that there were different ideas for selected topics. One idea was topics with a greater academic focus such as social or economic theories or foundations in applied informatics research. Another suggested topic was more perspectives for CMIOs. Other requests were for either more moderator involvement (help focus topics, encourage polite behavior) or for less moderator involvement.

Several commented on the volume of postings and the chaotic or vestigial headers (they don’t change when topics change). Some asked for summaries, search and visualization tools, and more digest options. Some wanted previous messages to be repeated, finding it easier to make sense of the thread, while others “hated” it and wanted them either eliminated or “snipped.” Others argued for mobile apps and greater access via the AMIA website.

In Fig. 3 we see a reflection of the themes noted above: on the one hand, an appreciation of the freedom and evolving nature of the listserv, on the other, a desire to focus and control the discussion.

Figure 3:

Word cloud for Question 8 (entirely free text) asked for any comments and/or specific topics for discussion and the word cloud reflects the themes noted above.

Discussion

The listserv evolved from the intersection of several AMIA WGs. The informality continues in this combined service and is clearly appreciated by members who enjoy the natural development and evolution of topic threads. Nonetheless, some members indicated that they missed specific (types of) HIT implementation topics or that some topics were not discussed adequately and suggested that the list moderator could ensure that these topics were raised, for example by posting a “topic of the month.” Recurring themes in the suggested topics were gaps in applied informatics research, theories and frameworks, standards, interoperability, EHRs/PHRs, and the role of the CMIO. Other members suggested not trying to influence content because one of the listserv’s benefits is its spontaneity of posting topics and the evolution that threads take.

Several respondents felt that just exchanging ideas on the listserv is not enough. In the free text, respondents indicated that the list members should try to move from merely discussing barriers or problems to finding solutions (“moving from talk to change”). Suggestions for doing so included AMIA members collaborating to develop a medical record system, picking up a Quality Improvement initiative or study, and trying to resolve the seemingly ever-present gap between professionals (e.g., technicians versus physicians).

Social rules

Some respondents further indicated that the added value of the listserv can become clouded by the amount or nature of posts from a few users with personal agendas. They suggested that a moderator or others should say what types of posts are appropriate for a listserv and what types of posts might be more appropriate via, for example, personal email, blog, or tweet. Moreover, the respondents suggested a level of mutual respect – recognizing that others may have a different opinion from one’s own and that some members might be more critical than others in their comments. That said, there were far fewer negative comments than positive, and respondents split on the desirability of a moderator.

Additional social rules included awareness of the tone of messages, indicated in statements such as “I really don’t appreciate the sarcastic humor,” and the need to discourage “rants” or “beating the same drum.” Some respondents were careful to say that they do not explicitly favor censorship except in extreme cases where the postings are detrimental to what readers can get out of the list or involve personal attacks.

To address these concerns, some members suggested solutions such as establishing rules of engagement for newcomers to the list or distributing a regular operational message to the list with a policy statement. An additional suggestion was to offer training (to new and existing members) regarding proper listserv etiquette. There were also suggestions to let the moderator play a stronger role, both in the discussions and behind the scenes by explicitly discussing problematic behavior with the person posting to the listserv.

Technology

Some responses also indicated that the flurry of discussion that naturally evolves should be balanced against email overload. These respondents therefore asked AMIA administrators to examine technical solutions to help reduce the volume of messages in people’s inboxes. Some members favor a periodic summary of trending topics.

One of the most prominent comments involved threading and individual posting behavior vis-a-vis thread/header hygiene. Few pay attention to the headers even when the topic shifts. Fifty-seven percent indicated that as discussion evolves headers should be adjusted to reflect the actual topic. However, respondents also indicated that this is sometimes difficult within the current listserv software and ease of use with headers. They ask for personal accountability and/or improvement to the current email interface.

Conclusions

The implementation listserv, within less than a year, became a major forum for discussing medical informatics, informatics policies, and of the purpose of AMIA itself. At the time of the survey, the listserv was subscribed by 25% of AMIA’s members and, by March 12, 2015 had generated over 6,000 posts. In summary, the survey results analysis serves to advance research evidence about knowledge dissemination, particularly how members interact and what they consider priorities for the topics under discussion and for the listserv itself. This report has relevance both to AMIA and to a more general audience interested in sociotechnical aspects of online knowledge transfer, engagement among community-of-practice members, organizational use of IT, and implementation and optimization issues in deploying and using IT.

For the respondent sample size, 18.2%, we make no claims of representativeness – in part because we turned off the email tracing function as part of our promise of confidentiality and anonymity, and thus cannot report on the identities of individual respondents. We also did not collect demographic data out of the same concern, as we personally know respondents we might have been able to identify. We do know that AMIA membership includes clinicians, researchers, HIT practitioners, and vendors, no doubt reflected in the pool of subscribers to this listserv. In addition, we may assume that those more engaged in the listserv’s discussions were more likely to respond to the survey. We note, however, that some respondents said they seldom if ever contributed to the listserv, which may indicate at least some inclusion of “lurkers.”

Of course, the usual limitations of question wording apply. We also know that some for whom English is not a mother tongue may be especially reluctant to contribute to the listserv. On the other hand, we doubt that many members of the listserv are illiterate, making this concern less relevant.

Next steps will be more in depth studies of the listserv and its data such as social network analysis and studies of specific topics (e.g., data standards, workflow, HIT evaluation, international contexts). Our plan is to engage the listerv participants to form groups corresponding to topics of interest.