Abstract
National initiatives on Electronic Health Records (EHRs) recognize the vital role of public health and recommend reporting to Immunization Information Systems (IIS) and access of its clinical decision support for immunizations (CDSi). The objective of this study was to collect stakeholder feedback on access and utilization of CDSi from the Minnesota Immunization Information Connection (MIIC), Minnesota’s IIS. Input was solicited using a semi-structured questionnaire developed by experts, and from a sample of 17 key informants from February 2015 through May 2016. Analysis highlighted the appreciation of MIIC services, comprehensive vaccination history across providers and CDSi functionality, with public health users relying on MIIC. It also identified issues such as data entry due to read-only view, data quality and communications for improvement. These findings underscore the critical role of IIS, need to engage stakeholders, ensure CDSi updates, maintain good data quality, and promote bi-directional data exchanges across EHRs-IIS.
Introduction
Immunization Information Systems (IIS) are population-based, secure computerized systems present in most US states and territories1. IIS serve as a hub for immunization data given across providers over time. As delivery of certain preventive services, including immunizations, spreads beyond the confines of traditional healthcare organizations, IIS play a critical role in collating immunizations obtained through a spectrum of stakeholders. It aims to present a comprehensive vaccination history and addresses the potential record scatter across providers. IIS contains clinical decision support for immunizations (CDSi), incorporating recommendations from the Advisory Committee on Immunization Practices (ACIP)2. These guidelines are complex and include a range of factors like age, number of doses, dose intervals, medical conditions and vaccine contraindications. IIS are recognized as one of the strategies to utilize to meet Healthy People 2020 goals of decreasing vaccine-preventable infectious diseases by improving vaccination rates.
National initiatives that promoted adoption of electronic health records (EHRs)3,4 have also advocated for data exchange across settings using nationally recommended standards5. These regulations have recognized the vital role of public health and recommended reporting to Immunization Information Systems (IIS) as one of the priority areas. Recent regulations have built upon this to include bi-directional communications across EHRs-IIS6. The Advancing Care Information (ACI) measures for Merit-Based Incentive Payment System (MIPS) includes submission of immunization data and receipt of immunization forecasts and histories from the public health immunization registry/immunization information system7. Given these regulatory drivers, the Preventive Taskforce recommendations8, studies on IIS effectiveness9,10 and the increasing use of CDS with embedded immunization guidelines in EHRs, there is a need to understand the stakeholder access and utilization of IIS CDSi.
EHR-IIS research to-date comprises of single clinical setting reports11,12 and assessing automated reporting from EHR to IIS13,14. The overlap of functionalities between EHRs and IIS have been outlined with recommendations for guiding their integration15. Bi-directionality across IIS and EHRs has been advocated wherein benefits of integration supersede the challenges posed by variability in EHR technologies and various IIS16. Given the increasing importance of CDSi, there have been efforts to examine EHRs for functionality17 and creation of computable CDSi logic18. Study on access to IIS decision support within EHR12 concluded that visual integration of external registries into an EHR was feasible with improvement in provider satisfaction and registry reporting. Recent studies have examined EHR-IIS data exchange and access of IIS CDSi, but are focused in single system/setting19,20. A collaborative strategy emphasizing shareable solutions has been proposed with goals of making CDSi sustainable for private and public sectors21. Given this landscape, there is a growing need for research to understand the access and use of CDSi at point of care, specifically through EHRs.
The Minnesota Immunization Information Connection (MIIC)22, the IIS in Minnesota, has been operational since 2002 and currently has 84 million immunizations and 7.8 million individuals across the life span. Overall, there are approximately 5,300 active organizations in MIIC ranging from primary care clinics, specialty providers, hospitals, pharmacy, schools and public health. MIIC currently offers an option to access MIIC from the provider EHR and is branded as ‘Alternate Access’23. This solution generates a query to MIIC for vaccination history and forecasting and is based on demographics of the record in the EHR. It does not require separate MIIC log-in and addresses the issue of data entry for the query. With this functionality, the history and forecast are presented to user within their EHR. This data can either be displayed as a ‘read-only’ view or can be ‘integrated’. Many have implemented a ‘read-only’ option, but moving towards ‘integrated’ with recent EHR system upgrades.
Prior research by authors has examined the technological and organizational context around immunization reporting to MIIC to portray increasing electronic reporting from EHRs using standards and real-time methods24. Recent MIIC CDSi studies have focused on understanding the queries and its variability across providers and EHR implementations25 and analyzing the CDSi presentation through direct access by IIS interface and by access through electronic health records (EHRs) to outline similarities and differences26. Building upon prior work and on national recommendations which promote IIS CDSi use, the goal of this research is to understand various aspects of MIIC CDSi from end-user/stakeholder perspectives. The main study objective was to collect stakeholder feedback on access and use of MIIC CDSi with the intent of utilizing results for prioritizing system enhancements and for program improvement. The overarching goal is to enhance the utility of IIS and its CDSi functionality. The study will have potential implications for using IIS CDSi as a strategy to increase immunization rates and improve population health.
Methods
Subject and Site Selection
Data was collected from a purposive sample of subject matter experts selected from healthcare systems and public health clinics. A total of 17 professionals were interviewed from 12 organizations representing 324 individual care sites. Their expertise spanned across public health and clinical care settings, and across clinical and administrative roles. These experts were selected based on their prior collaborations with MIIC and for their familiarity with the tool. The interviewee selection was also based on their ability to share information at a health system level vs. a site level, as approximately nine healthcare delivery systems hold 90% of the health services market in Minnesota due to the high presence of integrated delivery networks. Likewise, the interviewees of local public health were chosen from relatively high volume clinics to get granular information on MIIC CDSi access and some had an EHR in place for the last few years. Table 1 presents their various roles which range from vaccine coordinator, clinical analyst, nurse administrator, quality improvement manager and EHR project manager.
Table 1:
Case Demand Index Assessment Scale
| Roles | Settings | |
|---|---|---|
| Clinical Care Setting Health System / clinic | Public Health Setting Local health department / public health clinic | |
| Clinical | Clinical Analyst (1) Vaccine Coordinator (2) Licensed Practical Nurse (1) Vaccine and Infection Prevention Specialist (1) Health Outreach Coordinator (1) | Clinical Director (1) Vaccine/Immunization Coordinator (2) Immunization Manager (1) Nurse Epidemiologist (1) Public Health Nurse (2) |
| Administrative | Nurse Administrator (1) Quality Improvement Manager (1) | EHR Project Manager (1) Support Staff (1) |
Development of the Interview Questionnaire
The study utilized qualitative methodology to collect information from main stakeholders and users of MIIC CDSi. These are subject matter experts from healthcare systems and local health departments / public health clinics. A semi-structured interview questionnaire was developed by study authors (SR, AB) and refined through input from MIIC leadership and experts in CDSi and IIS. The tool aimed to collect data across four domains and consisted of both structured and open-ended questions to facilitate the gathering of information (Refer to Table 2 for the Interview Questionnaire). The tool was pilot tested with a clinic first, minor edits incorporated and was then utilized as the main data collection instrument.
Table 2:
Interview Questionnaire
| A. BACKGROUND |
| 1) What is your role at your organization? |
| 2) We understand that MIIC CDSi data is sent to your EHR. What is the number of sites that have access to this data and what number of these sites use the CDSi data in their practice? |
| 3) When was this feature installed in your system? Why? What led to this decision? |
| 4) What was the process for assessing client immunization history and vaccine recommendations (decision support) prior to that? |
| 5) Is this functionality included as part of EHR training or other user education efforts? |
| B. AWARENESS AND USE OF MIIC CDSi |
| 6) How many users are there in your clinic/health system that use the CDSi data from MIIC? |
| 7) Who typically accesses this functionality? Nurse, provider, clinic manager? |
| 8) How many client visits does your system have annually? |
| 9) Is MIIC accessed during all visits or only for select visits / client group? |
| 10) How is this integrated with workflow? Is it similar across various clinic types (pediatric vs. other settings)? When is the CDSi data reviewed? |
| 11) When viewing CDSi information from MIIC, does data not present in EHR get entered into your system? If not, why not? |
| 12) If there is data in EHR and not in MIIC, do they get reported back to MIIC? |
| 13) If errors noticed in the MIIC data (CDSi or immunization data), are they noted and reported back to MIIC? |
| 14) Are users aware that the source of the immunization data (history and forecasting) is MIIC? |
| C. VALUE OF MIIC CDSi |
| 15) What do users typically look for/what are users most interested in? New immunization data or validate current ones and/or look for vaccine recommendations? |
| 16) Please speak to the value of the MIIC CDSi information. Does your organization find this information valuable? If not, why not? |
| 17) Does your system use CDSi offered by any sources other than MIIC? How is this integrated into workflow? |
| 18) If you didn’t have access to MIIC for CDSi what would your organization use? Has your organization looked into alternate CDSi solutions? |
| 19) Do you track immunization rates by clinic and try to find “missed opportunities”? If so does the MIIC CDSi data help you determine a missed opportunity, or do you use something else to calculate that? |
| 20) What would like to see happen to improve use and value of this feature? (e.g. software improvement, ideas to promote use) |
| D. CLOSING |
| 21) What are the initiatives under which this functionality is promoted/monitored? (E.g. quality improvement, public health, clinical decision support); if not, why not? |
| 22) Who else in your organization/clinic would be involved with this and could be subject matter expert (SME) for this purpose? |
| 23) Any additional information you would like to share or relevant topic that needs to be addressed? |
The background section aimed to understand the number of sites in the organization that had access to MIIC CDSi functionality (both embedded EHR access and also access by MIIC interface); time and reason for installation of EHR access feature and if it was included in EHR training or other user education efforts. This was followed by a section on the awareness and use of MIIC CDSi collecting data on number of users, type of users (e.g. provider, nurse, clinic manager) and if it was accessed for all visits/select visits. This section also focused on immunization data across EHRs and MIIC and if data missing across either systems was shared/reported, including highlighting of errors. It also aimed to solicit awareness of MIIC CDSi. The third section addressed the value of MIIC CDSi by gathering data on perceived value of the tool, information that is sought (e.g. vaccination history, immunization forecast) and integration with workflow. Tracking of missed opportunities by the system/clinic (if any) and ideas for MIIC CDSi improvement were some essential elements of this interview process. The closing section aimed to collect information on organizational initiatives under which MIIC CDSi is promoted (if any) and other organizational subject matter experts.
Data Collection
Subject matter experts were contacted over email and interview questions were shared a couple of weeks in advance to help them prepare and gather additional information as needed. An opportunity was given to extend the interview to their colleagues as needed. These interviews were conducted over the time period of February 2015 through May 2016. WebEx was used for set-up and recording of the meetings. Three study authors (SR, AB and SS) were present during all the interviews and two authors (SR and SS) took detailed notes during the interview process which was utilized for the study analysis. Verbal consent was obtained prior to the interview and each lasted for approximately 45 minutes. Notes were exchanged immediately after the interview and cross referenced so that any discrepancy can be resolved.
Data Analysis
Preliminary analysis and exchange of notes (SR, SS) occurred between the interviews. Analysis of qualitative open-ended answers was supported by NVivo 11 software (QSR International)27. One of the study authors (SR) participated in all the interviews, was one the main note takers, has worked in public health and in immunization program for many years and was the main coder. These codes were synthesized through discussion (SR, AB and MM) which lead to consolidating categories that overlapped and identifying higher level themes. The responses to structured questions were analyzed using descriptive statistics and tabulation was utilized to present results. The responses considered important to highlight were selected through group consensus (SR, AB, MM).
Results
Table 3 presents the responses by the 17 subject matter experts from 12 organizations, and highlight the key areas with MIIC access. Themes and categories identified in the analysis along with sample quotes are in Table 4.
Table 3:
Access to MIIC across Clinical Care and Public Health Settings
| Health Care Settings | Access to MIIC | |||||
|---|---|---|---|---|---|---|
| Part of EHR Training | Type of MIIC Access | Integration with Workflow | MIIC Data / CDSi in Care Process | Main User Base | ||
| Clinical Care (Health System / Clinic) | Organization A | ✓ | From EHR; read-only view | Part of patient preparation & before visit | MIIC lookup only when vaccine due/overdue in EHR | RN; Pediatric providers |
| Organization B | ✓ | From EHR; read-only view | Previsit access for routine & in visit for same day appts | First look at EHR & then access MIIC as needed | Nurses | |
| Organization C | ✓ | From EHR; read-only view | Pre-visit access for routine / physicals | All pediatric visits include MIIC lookup | Nurses | |
| Organization D | ✓ | From EHR; bidirectional data exchange | Currently check in visit; moving to pre-visit planning | Depends on clinic & visit type | Nurses; occasionally providers | |
| Organization E | ? | From EHR; read-only view | No set protocol; varied access | Only if associated with infectious disease | Medical Assistant | |
| Organization F | × | No integrated EHR access; separate MIIC log-in | No set protocol; varied access | Only for well child visits & in cases of uncertain history in EHR | Front desk staff | |
| Organization G | × | From EHR; read-only view | Chart scrub 3 days advance; same day for sick visits | For family practice, general medical & sick visits; not for specialty visits | Mainly clinic assistants | |
| Organization H | ✓ | From EHR; bidirectional data exchange | Pre-visit, but some variation in practice | Most visits except for urgent care | Nurses; medical records team | |
| Public Health (Local Health Department) | Organization I | × | No integrated EHR access; separate MIIC log-in | Part of patient preparation & before visit | All pediatric & prenatal diabetic visits; inconsistent for other visits | Nurses; providers; billing |
| Organization J | ✓ | From EHR; bidirectional data exchange | MIIC look-up during vaccine appointment scheduling | All visits to public health clinic | Public health nurses | |
| Organization K | ✓ | From EHR; bidirectional data exchange | MIIC look-up during vaccine appointment scheduling | All visits to public health clinic | Public health nurses; registration staff | |
| Organization L | ✓ | From EHR; bidirectional data exchange | Part of patient visit | Look up during immunization walk- in clinics & home visits | Public health nurses | |
| ✓ - included; × - not included;? - not sure; EHR - Electronic Health Record; RN - Registered Nurse; appts - appointments | ||||||
Table 4:
Themes, Categories and Quotes regarding MIIC
| Themes | Categories | Sample Quotes | Setting and Role of Interviewees |
|---|---|---|---|
| Recognition of the Value of MIIC | Vaccination history | ….“Valuable in public health – see clients which are mobile population – so using MIIC which is shared system is important (as clients drop on and off insurance)” | Public health; Public health nurse |
| Immunization forecast | …“MIIC forecasting is nice as it gives earliest date – if kids come in early, it is easy to make a judgement call” …“no, we don’t use any other – rely on MIIC” …..”like the forecasting functionality” | Clinical and public health setting; Clinical analyst, Vaccine coordinator | |
| Appreciation of MIIC services | …“MIIC is valuable to end-users; to have all that information handy” …“very easy to access and use….print reports” …“honestly, MIIC is amazing ….have used for 16 years….seen it only get better” … “Absolutely of value – I am in MIIC constantly – at the click of the button – rely on MIIC to make sure immunizations are given correctly” | Clinical care and public health setting; Licensed practical nurse, Vaccine / immunization coordinator | |
| Benefits of System and Workflow Integration | EHR Integration | …“what’s helpful now – within Epic ambulatory module – with click of a button, can get into MIIC” | Clinical care setting; Nurse administrator |
| Fit with Workflow | …“I do the day before (check MIIC) to see if there are any catch-up opportunities” …“part of our routine pre-visit planning” | Public health; Public health nurse | |
| Functionality and System Requirements to Address | Vaccine Details | …“more details on vaccine received, specifically the brand. Who is the manufacturer? Who gave the vaccine?” | Clinical care setting; Nurse administrator |
| Forecaster Improvement | ….“foreign born don’t fit into usual schedule routine” …“love for the forecaster to be improved” | Public health; Nurse epidemiologist | |
| Data Quality | …“MMR rates in MIIC is really low – not due to missed opportunities, but due to missing shots in MIIC” | Public health; Immunization manager | |
| Data entry | …“need to manually enter MIIC data” …“need for two-way interface” | Clinical care setting; EHR proj. manager | |
| System Capacity | …“Sometimes MIIC is extremely slow – doesn’t query quick enough as I have hundreds of people (but slows during the day) – sometimes 4-5 min” | Clinical care setting; Outreach coordinator | |
| Need for Better C ommunications | Information Sharing | …“communications from MIIC to sites for error corrections will be good” | Public health; Immunization coordinator |
| Outreach | …“when MIIC is down, we are not informed – so, an issue when kids are in the clinic” | Clinical care setting; Clinical analyst |
Most of the organizations interviewed (10 of 12, 80%) had MIIC access within their EHRs. Of the 10 organizations with embedded EHR access to MIIC, almost all had included this functionality as part of their EHR training (8 answered yes and 2 were uncertain). All the public health clinics interviewed (with the exception of 1) had bi-directional data exchange across MIIC and EHRs which indicated that data was being transferred from MIIC to EHRs and integrated into the EHR record. Of the organizations which had pre-visit planning in place (10 of 12), almost all had access to MIIC as part of patient preparation/arrival. Scenarios which prompted access to MIIC varied across the organizations, and predominantly part of pre-visit and MIIC look-up during a clinical encounter was done only for urgent care visits. Two organizations (A and B) which are the dominant integrated health systems in the state utilized the CDSi present in their EHR and looked up MIIC only for vaccination history. MIIC was accessed by nurses, clinic assistants and care support professionals (e.g. front desk, scheduling, billing) in all the organizations (12 of 12, 100%) and had limited access by providers (e.g. physicians).
Four main themes were identified which included feedback on MIIC CDSi tool and its utlity and items which needed action: Recognition of the Value of MIIC; Benefits of System and Workflow Integration; Functionality and System Requirements to Address and Need for Better Communications. MIIC value proposition was due to the comprehensive vaccination history provided, the immunization forecaster (CDSi) and other services/functionality offered (e.g. Reminder/Recall, Improbable shots report, Vaccine Management). The access which was integrated within the EHR eliminating the need for a separate user interface and data entry was highly appreciated by the users. Overall, MIIC access seemed well integrated within the workflow of various organizations. Five functional/system requirements were identified for improvement ranging from need for additional details on the vaccine given; improvement of the forecaster by inclusion of guidelines for those who don’t fit into regular schedule; improving the quality of data in MIIC; need for bi-directional data exchange/dynamic data flow to eliminate dual data entry and finally enhancing the capacity of the system to facilitate quick response time to queries. Need for information sharing and outreach was mapped into the final theme on the need for better communications (from MIIC to its users/stakeholders) which underscored the need to inform users ahead of time on planned system upgrades/down time, to facilitate planning of clinic vaccinations.
Discussion
Results point to strong support of MIIC and recognition of its value and the services/tools offered. Almost all participating organizations (12 of 12), which include 324 individual sites had access to MIIC in varying formats embedded within their care delivery. These represent approximately 25% of the sites/clinics in the state and so a reasonable indicator of the overall pattern. Majority of the organizations interviewed (10 of 12, 80%) had MIIC access within their EHRs, but some of those did not have dynamic data exchange which supported data entry from MIIC into EHRs. But, this issue should be resolved in the near future, as current version of the prominent EHR product used in Minnesota (Epic) supports this functionality and organizations are upgrading to this version.
One of the findings was the systematic difference between private and public health clinics in their access and use of MIIC and its perceived utility, with public health clinics/local health departments much more appreciative of a public health information system such as MIIC. This is likely due to the patient population they serve, many of whom have sporadic encounters with the healthcare system and have providers who are spread across locations and systems resulting in an immunization record scatter. IIS serves as a hub to collate this vaccination history across providers and locations and presents immense value to these settings. The appreciation of MIIC is also due to the fact that low resource settings likely have EHRs with fewer capabilities such as clinical decision support and hence rely more on MIIC services such as CDSi and various tools/reports (client assessment, vaccine management).
As initiatives are being explored to increase the utility of MIIC CDSi, this study yielded important findings on the main user base, who are primarily nurses in both clinical and public health settings along with support professionals (e.g. front desk, scheduling, billing), with limited access by providers (e.g. physicians). The literature on clinical decision support and national recommendations advocate for access during clinical encounters/point of care. But the majority of settings in this study pointed to access of MIIC CDSi as part of previsit planning. As per study participants, this timing of access seems to integrate better with their workflow as vaccinations are typically administered during scheduled well-child visits for pediatric population. As the recommendations are to access CDSi during the point of care, additional research needs to be done to understand the obstacles (if any) to access the CDS tools during a clinical encounter. This is an important issue which will affect the design and implementation of EHRs and their integration with IIS.
This study pointed to some system/functionality needs which required actions by MIIC program management. The need for better communications was acted upon by MIIC leadership with regular email updates sent to stakeholders on scheduled MIIC downtimes and also immediate information shared on unexpected data exchange malfunctions so that an organization can re-submit their data. Though MIIC had communication mechanisms on error corrections, the interviews pointed to the need for selecting the right professional at the provider end to receive the messages and to address those errors.
The study emphasized the need for good quality of data in the IIS (MIIC in this study) as the forecasting is dependent on IIS data and becomes irrelevant with incomplete data in the IIS. The issue of missing shots impacting completeness of data identified by stakeholders is due to non-submission of data by some organizations and efforts are being made to increase provider participation in MIIC. Likewise, the need for better data quality in MIIC was addressed by creation of a data quality coordinator position whose responsibility is to monitor quality of data and identify systematic errors with incoming data and also implement protocols for regular data quality assessment. The increasing bidirectional movement of data across MIIC and EHRs due to capacity of EHRs to support this data interchange highlights the increasing need to monitor quality of data to ensure data is correctly being attributed to organizations which have the shot and those that incorporate and report them to IIS. These findings also supported the need to use a data quality monitoring tool on a regular basis for data validity checks.
One of the limitations of this study are that it did not cover all the sites that accessed MIIC CDSi. The participating organizations represent approximately one fourths of user base and so findings can be applicable to other sites that are similar in size and patient population. Participants were chosen based on their expertise and their position within the organization to share practice information from a system level, but it’s possible that there is variation across sites. The other shortcoming is that, after completion of this research, three of the main healthcare systems with hundreds of clinic sites have either upgraded their EHR product or switched to a new EHR platform and so the functionalities (specifically the capacity for dynamic data interchange across MIIC and EHRs) may have since improved.
Another limitation of this study is that it focuses on a single state IIS. MIIC is part of WIR consortium28 which is a collaboration of group of states on the same vendor platform as MIIC and is likely that some of these findings are applicable to them, but need additional studies for validation. Additional aspects which are unique to this study setting and can potentially limit generalizability is the predominance of integrated health care delivery network structure in the state and the market dominance by a single EHR vendor. Yet another limitation is that this research did not examine the role of third-party immunization clinical decision support29 and its role. This study focused on understanding the overall access and utilization of MIIC CDSi and additional studies are needed to understand the variability in CDSi across MIIC and third-party solution. The American Immunization Registry Association (AIRA)30, a membership and advocacy organization for IIS across the various states should share findings from this and other IIS studies to share lessons learned.
Conclusion
With the recognition of the importance of public health reporting and the recommendation to look up and use CDSi offered by IIS, there is a need to understand the current status of utilization. This study collected stakeholder feedback on access and use of vaccination history and MIIC CDSi with the intent of utilizing results for prioritizing system enhancements and program improvement. Research insights can be applied to other scenarios of public health decision support (e.g. case reporting) and other situations of HIT implementation and evaluation. The overarching goal is to enhance utility of IIS and its CDSi functionality and study will have potential implications for using IIS CDSi as a strategy to increase immunization rates and improve population health.
Acknowledgements
The authors would like to thank the various stakeholders from private health systems and from public health clinics who participated in this study for their time and valuable input.
References
- 1.Centers for Disease Control and Prevention. Immunization Information Systems (IIS) 2014. [cited 2017 March 2]. Available from: http://www.cdc.gov/vaccines/programs/iis/index.html.
- 2.Advisory Committee on Immunization Practices (ACIP) Immunization Schedules. 2012. [cited 2017 February 23]. Available from: http://www.cdc.gov/vaccines/schedules/hcp/index.html.
- 3.Centers for Medicare and Medicaid Services. EHR Incentive Programs. 2010. [cited 2017 March 2]. Available from: http://www.cms.gov/ehrincentiveprograms.
- 4.Centers for Medicare & Medicaid Services. Electronic Health Record Incentive Program—Stage 3 and Modifications to Meaningful Use in 2015 Through 2017; Final Rule. 2015. [cited 2017 February 23]. Available from: https://www.federalregister.gov/documents/2015/10/16/2015-25595/medicare-and-medicaid-programs-electronic-health-record-incentive-program-stage-3-and-modifications.
- 5.Office of the National Coordinator for Health Information Technology. 2015 Edition Health Information Technology (Health IT) Certification Criteria, 2015 Edition Base Electronic Health Record (EHR) Definition, and ONC Health IT Certification Program Modifications. 2015. [cited 2017 February 23]. Available from: https://www.federalregister.gov/articles/2015/03/30/2015-06612/2015-edition-health-information-technology-health-it-certification-criteria-2015-edition-base. [PubMed]
- 6.Centers for Medicare and Medicaid Services. Medicare Access & CHIP Reauthorization Act of 2015 (MACRA) 2015. [cited 2017 March 2]. Available from: https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Value-Based-Programs/MACRA-MIPS-and-APMs/MACRA-MIPS-and-APMs.html.
- 7.Centers for Medicare and Medicaid Services. Advancing Care Information, Quality Payment Program. 2016. [cited 2017 February 1]. Available from: https://qpp.cms.gov/measures/aci.
- 8.Community Preventive Services Task Force. Recommendation for use of immunization information systems to increase vaccination rates. J Public Health Manag Pract. 2015;21(3):249–52. doi: 10.1097/PHH.0000000000000092. Epub 2014/06/10. [DOI] [PubMed] [Google Scholar]
- 9.Groom H, Hopkins DP, Pabst LJ, Murphy Morgan J, Patel M, Calonge N, et al. Immunization information systems to increase vaccination rates: a community guide systematic review. J Public Health Manag Pract. 2015;21(3):227–48. doi: 10.1097/PHH.0000000000000069. Epub 2014/06/10. [DOI] [PubMed] [Google Scholar]
- 10.Patel M, Pabst L, Chattopadhyay S, Hopkins D, Groom H, Myerburg S, et al. Economic review of immunization information systems to increase vaccination rates: a community guide systematic review. J Public Health Manag Pract. 2015;21(3):253–62. doi: 10.1097/PHH.0000000000000100. Epub 2014/06/10. [DOI] [PubMed] [Google Scholar]
- 11.Agency for Healthcare Research and Quality (AHRQ) DEVISE: Data Exchange of Vaccine Information between an Immunization Information System and Electronic Health Record (New York) 2013. [cited 2017 February 23]. Available from: http://healthit.ahrq.gov/ahrq-funded-projects/devise-data-exchange-vaccine-information-between-immunization-information.
- 12.Stevens LA, Palma JP, Pandher KK, Longhurst CA. Immunization registries in the EMR Era. Online journal of public health informatics. 2013;5(2):211. doi: 10.5210/ojphi.v5i2.4696. Epub 2013/08/08. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Merrill J, Phillips A, Keeling J, Kaushal R, Senathirajah Y. Effects of automated immunization registry reporting via an electronic health record deployed in community practice settings. Applied clinical informatics. 2013;4(2):267–75. doi: 10.4338/ACI-2013-02-CR-0009. Epub 2013/07/23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Schauer SL, Maerz TR, Verdon MJ, Hopfensperger DJ, Davis JP. The Wisconsin immunization registry experience: comparing real-time and batched file submissions from health care providers. WMJ : official publication of the State Medical Society of Wisconsin. 2014;113(3):102–6. Epub 2014/08/15. [PubMed] [Google Scholar]
- 15.HLN Consulting. IIS and EHR Feature Overlap. 2014. Available from: https://www.hln.com/assets/pdf/HLN-IIS-EHR-Overlap-White-Paper.pdf.
- 16.Dombkowski KJC, S. J. Redefining meaningful use: achieving interoperability with immunization registries. Am J Prev Med. 2012;42(4):e33–5. doi: 10.1016/j.amepre.2012.01.009. Epub 2012/03/20. [DOI] [PubMed] [Google Scholar]
- 17.CNI Advantage LC. Immunization-Related Capabilities for Clinical Software. 2015. [cited 2017 February 23]. Available from: http://www.immunizationsandhealthit.org/
- 18.National Center for Immunization and Respiratory Diseases - Centers for Disease Control and Prevention. Clinical Decision Support for Immunization (CDSi) 2013. [cited 2017 March 2]. Available from: http://www.cdc.gov/vaccines/programs/iis/cdsi.html.
- 19.Stockwell MS, Catallozzi M, Camargo S, Ramakrishnan R, Holleran S, Findley SE, et al. Registry-linked electronic influenza vaccine provider reminders: a cluster-crossover trial. Pediatrics. 2015;135(1):e75–82. doi: 10.1542/peds.2014-2616. Epub 2014/12/31. [DOI] [PubMed] [Google Scholar]
- 20.Stockwell MS, Natarajan K, Ramakrishnan R, Holleran S, Forney K, Aponte A, et al. Immunization Data Exchange With Electronic Health Records. Pediatrics. 2016;137(6) doi: 10.1542/peds.2015-4335. Epub 2016/06/01. [DOI] [PubMed] [Google Scholar]
- 21.Arzt NH. Clinical Decision Support for Immunizations (CDSi): A Comprehensive, Collaborative Strategy. Biomedical informatics insights. 2016;8(Suppl 2):1–13. doi: 10.4137/BII.S40204. Epub 2016/10/30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.MIIC. Minnesota Immunization Information Connection 2016. [cited 2017 March 7]. Available from: http://www.health.state.mn.us/divs/idepc/immunize/registry/index.html.
- 23.Minnesota Immunization Information Connection. MIIC Resources. 2009. [cited 2017 March 7]. Available from: http://www.health.state.mn.us/divs/idepc/immunize/registry/index.html.
- 24.Rajamani S, Roche E, Soderberg K, Bieringer A. Technological and Organizational Context around Immunization Reporting and Interoperability in Minnesota. Online journal of public health informatics. 2014;6(3) doi: 10.5210/ojphi.v6i3.5587. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Rajamani S, Bieringer A, Muscoplat M. Characterizing the Access of Clinical Decision Support Offered by Immunization Information System in Minnesota. Online journal of public health informatics. 2015;7(3) doi: 10.5210/ojphi.v7i3.6282. Epub 12/31/2015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Rajamani S, Bieringer A, Wallerius S, Jensen D, Winden T, Muscoplat MH. Direct and Electronic Health Record Access to the Clinical Decision Support for Immunizations in the Minnesota Immunization Information System. Biomedical informatics insights. 2016;8(Suppl 2):23–9. doi: 10.4137/BII.S40208. Epub 2017/01/05. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.QSR International. NVivo Qualitative Data Analysis Software (QSR International, Doncaster, Australia) 2012.
- 28.Kurilo MB, Flynn M. Leveraging a Multi-state Collaborative for Developing Immunization Information System Enhancements. 2012. [cited 2017 July 4]. Available from: http://www.immregistries.org/resources/iis-meetings/michael flynn aira 2012 conf deip.pdf.
- 29.HLN Consulting. Open-Source Immunization Forecasting Software. 2013. [cited 2017 July 4]. Available from: https://www.hln.com/services/open-source/ice/
- 30.American Immunization Registry Association (AIRA) AIRA Vision and Mission. 2000. [cited 2017 February 23]. Available from: http://www.immregistries.org/about-aira/vision-mission.