Critical Evaluation of Pharmacy Automation and Robotic Systems: A Call to Action

Health care is an industry primed for innovation. Health care spending in the United States is approximately 50% more per person than in many other developed nations, with this spending accounting for 17.8% of the gross domestic product (GDP).¹ It has been suggested that the increased use of technology and automation within health care could provide better patient care while reducing total costs.² Pharmacy leaders continue to be pressured to minimize costs while providing pharmaceutical care in a safe and effective manner.

Human error is inevitable. Estimates for the number of medical errors that occur in hospitals in the United States have increased significantly from 98 000 to 400 000 since To Err is Human: Building a Safer Health System was published in 1999.^3,4 To reduce these errors, potential problems must first be identified and then measured and reported before systems can be designed to mitigate the problems.

One subset of medical errors is medication errors (MEs). Medication errors have previously defined as “any error occurring in the medication-use process”.⁵ Due to differences in how MEs are determined, there is a wide range in the number of MEs that are reported in both the inpatient and outpatient settings.⁶ Flynn et al found a 17.9% ME rate in hospitals and skilled nursing homes when multiple detection methods were used.⁷ This high rate is important because MEs can lead to drug complications, the most common cause of adverse events in hospitalized patients.⁸

Complexities within the medication-use process contribute to the overall number of MEs. Figure 1 identifies the major steps in the medication-use process and each can involve various technologies. Complexities arise when workflows are not standardized. This is often due to the lack of interoperability between systems and the unpredictability of human behavior that occurs when health care staff use these technologies. When taken in aggregate, these systems, behaviors, and interactions complicate our understanding of the potential failure points in the medication use process and their subsequent resolution. Preparation and dispensing of medications resides mostly within pharmacies and frequently involves both pharmacy automation and robotics systems blended with human work actions. Pharmacy robotics are “mechanical devices that perform programmed, complex, and repetitive manipulations which mimic human behavior without continuous input from an operator” p. 1601.⁹

Figure 1.

Informatics within the medication-use system.

The advent of the randomized controlled trials (RCTs) began in the 1940s to eliminate bias observed in study methodologies. By the 1980s, the RCT was considered to be the “gold-standard” of medical evidence. Although RCTs have become the standard in pharmaceutical research, the application of RCTs in other areas affecting medication use, including the incorporation of technology, has been challenging.¹⁰ The cautious, systematic approach to drug approval contrasts sharply with the fast-paced nature of technology development and innovation. Despite challenges with the applications of RCTs, they remain the “gold-standard” due to their revolution of medical research and improved quality of health care.¹⁰ The question remains: Should pharmacy automation and robotics systems be tested in a similar, robust manner to prove their efficacy and safety?

One of the largest drivers of increased use of automation and technology within the medication-use process is the intuitive belief that it leads to an increase in patient safety and reduction in pharmacy errors.^11,12 However, the presence of new technology can create new problems in the medication-use process because not all factors are understood at the time of implementation. A patient safety situation occurred at a hospital in Bend, Oregon, despite the hospital having an intravenous (IV) automated IV workflow system on-site. The IV workflow system was not used due to an incompatibility that occurred with their newly implemented electronic health record (EHR). The compounded medication preparation was not validated by an IV workflow system prior to preparation and several mistakes were made in the manual preparation and verification process, which resulted in the patient being erroneously administered rocuronium instead of the prescribed fosphenyotin. The investigation revealed the error could been prevented if the IV workflow system was used as intended.¹³ To challenge our intuitive beliefs that errors are “always” reduced and safety is “always” enhanced in the presence of automation and technology, we ask the following questions: Does the literature represent selection bias toward publishing favorable outcomes studies? Is there uniformity among health systems in selection of metrics to measure impact of medication-related technologies?

The goal of this article is to review evidence related to the dispensing and preparation of medications. This article is not a comprehensive review of the entire body of literature related to the topic; rather, it is a review of the articles with the most robust study designs for each of the listed pharmacy technologies. These were selected from the current literature by reviewing the abstracts, methods, and results sections of the available articles. The study designs and outcomes were critiqued using a hierarchy starting with the highest scientific rigor (eg, randomized control trial), progressing to sequentially less rigorous studies when such methodology was lacking. The goal of the article is to inform pharmacy leaders with details about the availability of data when justifying the decision of whether or not to use specific technologies. Our observation has been that pharmacy automation and robotics system implementation decisions are frequently based on personal experiences conveyed between health care professionals, consultant recommendations, and/or case reports published and/or marketed by vendors. These types of interactions can result in overly optimistic expectations (and/or confirmation bias) with little regard for the potential negative consequences. It is imperative to understand the quality of evidence that does exist related to these technologies.

Pharmacy Bar Coding

Pharmacists and pharmacy technicians are responsible for ensuring the accuracy of dispensed medications. Bar code technology is often deployed to support that responsibility. There is limited, high quality evidence regarding the benefits of implementing bar code technology within the pharmacy.^14,15

Poon and colleagues designed a before-and-after observational study to compare target dispensing errors and potential adverse drug events (ADE) following implementation of a multipronged workflow change including bar code technology and a medication carousel for retrieval and filling. Poon et al demonstrated a reduction in target dispensing errors by 85% and target potential ADEs by 74% following implementation of bar coding. Despite the benefits of qualitative assessment in the study design, statistical analysis of the primary outcomes was not performed due to differences in the various dispensing processes. The a priori secondary analyses demonstrated significant improvements in target dispensing errors within each dispensing area when the pre/post bar code periods were stratified according to similar dispensing processes.¹⁴ Two of the dispensing processes found significant improvements in rates of potential ADEs. One dispensing process (“alternate zone filling”) saw an increase in potential ADEs and potential life-threatening ADEs in the post bar code period following errors in filling high-risk IV medications.¹⁴ Although a quantitative improvement was demonstrated in medication dispensing errors and potential ADEs, a statistical analysis was not completed on the primary outcome. The secondary analyses indicated improvement in medication dispensing errors and potential ADEs when each dose was scanned or doses were scanned upon restocking and retrieval from a medication carousel.

There is limited data demonstrating the value of bar code technology within pharmacy dispensing. Poon et al indicated that a new workflow incorporating bar code technology improved dispensing from a pharmacy in certain scenarios.¹⁴ The inclusion of the carousel, in addition to bar coding, added complexity to the analysis and our understanding of the study as several workflow steps were introduced simultaneously. There is a lack of confirmatory studies related to the use of barcodes in the stocking and order fulfillment process. However, with the absence of contradictory data in the literature, these data suggest that bar code technology should be used in medication dispensing to minimize potential MEs and ADEs related to the manual selection of medications, with bar code scanning conducted for every dose.

Automated Dispensing Cabinets (ADCs)

This technology differs from the other evaluated systems, as most of the routine use of ADCs is located in direct patient care areas outside of the physical walls of the pharmacy. Although pharmacy departments often ensure that the appropriate medications are stocked within the ADC units, medication selection and preparation from ADC units often resides with nursing or other providers. Beginning in the 1990s, many before-and-after observational implementation studies involving the use of ADC technology were published.^16-22 The initial ADC studies lacked robust statistical analyses and consistency in measured outcomes.^16-20 Recent ADC studies placed an increased emphasis on assessing the reduction in medical errors and their potential severity.^21-23

The most robust ADC studies were published by Chapuis and colleagues in France.^24,25 Their first study analyzed the effect of the implementation of an ADC on the rate of MEs within a medical intensive care unit (MICU) between a study unit (ADC) and control unit (medication cabinet). This study used a randomized design to compare usual care with an ADC technology intervention and the primary outcome was MEs. There was a significant improvement in overall error rate in the medication dispensing processes (selection, preparation, administration) following implementation of an ADC. The reduction in ME rate was driven by an improvement in preparation errors with no improvement in selection and administration errors. Although overall error rate decreased, the errors rated with the potential to cause harm increased. Further analysis of the results demonstrated that selection errors, which the system is designed to improve upon, were not impacted as expected.²⁴ Despite the randomized design, MICU nurses rotated between units which introduced possible crossover bias. Furthermore, the study units did not have prospective pharmacist review of medications, limiting its applicability to many hospitals across the United States.²⁴

Initially, there were limited data regarding the financial impact of implementing ADCs within hospitals.^16,19 Chapuis et al demonstrated a positive net present value (NPV) and global cash flow associated with implementation of ADCs within three surgical intensive care units. The analysis took both operation (variable) and investment (fixed) costs into account and was able to demonstrate a positive return on investment.²⁵ Although Chapuis demonstrated a positive financial impact when implementing ADCs, much of the savings were based on nurses’ time gained following a reduction in their role in medication-use process. There is no guarantee that the nurses’ time saved resulted in increased productivity from nurses. Given many hospitals have removed nurses from preparing the majority of medication doses on the patient unit, there may be limited external applicability to hospitals throughout the United States. A more complete economic analysis would incorporate patient outcomes in the economic analysis, such as a robust cost-benefit analysis.

ADC technology ensures medication dispensing occurs closer to the patient bedside; however, the success of the technology is reliant upon nursing workflow and behaviors. Although there have been many studies published involving the implementation of ADC systems with varying outcomes analyzed, the two most robust studies did show favorable medication safety and financial outcomes. Further analyses of medication safety outcomes in the Chapuis study demonstrated that selection errors did not improve with implementation of the technology despite what anecdotal experience would suggest.²⁴ Both of these studies had limitations and there is a lack of confirmatory studies. Despite the widespread adoption of ADC technology, there is insufficient evidence available demonstrating improved medication dispensing accuracy to patient care areas. Therefore, nurses and providers must place continued emphasis on appropriate selection of medications from the ADCs.

Automated Pharmacy Carousel Systems

Other than the Poon bar code study, only two studies were performed which employed before-and-after observational study designs to analyze the impact of automated pharmacy carousel systems implementation. These studies failed to define their primary outcome or complete meaningful statistical analyses, which limits their overall utility. Both studies demonstrated an improvement in pharmacy technician filling errors across certain dispensing processes.^26,27 However, the rate of medication dispensing errors following the pharmacists’ final verification did not improve across all dispensing processes following implementation of the system.²⁷

Based on the lack of literature and the limitations of these study designs, it is challenging to make robust conclusions about the technology. Automated Pharmacy Carousel Systems are intended to improve medication storage within pharmacies. Although anecdotally this may have occurred with implementation of these systems, the lack of improvement in medication dispensing errors indicates that there was insufficient evaluation of the technology prior to their widespread deployment and use.

This concept is similar in nature to the widespread adoption of ADC technology without definitive conclusions that the systems reduce medication dispensing errors. Part of the reason for the growth of these systems is the organic evolution that often occurs with technology. These systems were designed as improved medication storage devices, which allowed secure and compact medication storage in the pharmacy (carousel) or patient care areas (ADC). Their use has evolved over time to include dispensing for the provision of direct patient care, such as guiding pharmacy technicians or nurses in selection of first dose medications. This transition is akin to the off-label use of approved medications. Medications may be approved for one indication, but other uses become apparent over time where research may be lacking. Ideally, these medications are approved for additional indications following more studies. In a similar fashion, additional safeguards (eg, electronic data interchange, bar code scanning of each medication upon receipt and dispensing) would be deployed and their impact studied to ensure the patient is provided the correct medication.

Intravenous Automated Workflow Systems

Intravenous automated workflow systems implemented multiple technologies into the high-risk area of IV compounding (eg, bar code verification, image capture, and/or gravimetric analysis). The technologies are designed to ensure proper product selection of medications and standardization of the IV preparation process. The initial studies of automated IV workflow systems were observational and identified errors captured with this new technology.^28-30

Wright and colleagues compared error rates between a manual IV preparation process and an automated IV workflow system with bar code verification and image capture. Baseline error data were collected over 4 days and compared with a 48-week post-implementation period. The primary outcome was number and types of errors before and after implementation.³¹ The study was unique given its before-and-after design with comparing a manual compounding process with a newly implemented IV workflow system. There was no significant difference in error rates between baseline and post-implementation period.³¹ Wright and colleagues found 8.5% of total errors were introduced by the automated IV workflow system which is less that reported by Moniz et al.^30,31 The differences in reported error rates are likely due to variations in study design and definition of errors. Given the lack of standardized study design and measured outcomes, the impact of these systems cannot be assessed across multiple studies. The post hoc analysis by Wright et al found a statistically significant difference in error detection rate in the first 3 of 4 months following implementation compared with baseline.³¹ This result could indicate that the number of errors are reduced as staff become more familiar with the system over time. Limitations of the study include a short pre-implementation data collection period and lack of clinically significant metrics.

Reece and colleagues studied the implementation of a gravimetric IV workflow system. The study reported a 74-fold increase in errors reported with the gravimetric IV workflow system over self-reporting during manual compounding.³² However, the study design was not well described and the IV gravimetric workflow system comparator was the manual compounding of medications that were unable to be utilized in the new system. Therefore, the study results did not provide a reliable comparison between automated and manual processes for IV gravimetric compounding processes.

Similar to the other technologies previously discussed, the lack of well-defined outcomes with the IV automated workflow technologies limits the conclusions that can be made.^28-32 Observational studies, without sufficient pre-implementation data, limit the ability to understand the true impact of these systems in real-world practice. Before-and-after implementation studies must have reliable comparators. Although logic leads us to believe the technologies added to the IV compounding process would improve potential MEs, the published literature has yet to provide that proof. While there is a lack of detrimental effects described with implementation of automated IV workflow solutions, further confirmatory studies are required to ensure these systems have a clinically meaningful impact on patients receiving IV compounds.

Intravenous Compounding Robotics

Intravenous Compounding robotic systems were developed to improve the safety of high-risk IV compounding for both patient and pharmacy personnel. The goal of these systems is to improve compounding accuracy of robotic systems over manual preparation and create self-contained compounding areas to reduce potential hazardous medication exposure for pharmacy staff. Several studies involving the use of IV compounding robotics have been performed with conflicting findings.^33-35

Seger and colleagues completed a before-and-after, direct observation study to evaluate robotic antineoplastic IV compounding technology to manual compounding processes. The primary outcomes were MEs with the potential for patient harm and errors with the potential for staff harm. The study found significant improvements in the number of drugs exceeding accuracy measurements (±5%) in the post-intervention period.³⁶ For compounded products, the US Pharmacopeia (USP) has established an accuracy standard threshold of ±10%, while many IV compounding robotics studies chose a more stringent threshold of 5% to evaluate the systems.^33,34,36-38 Seger and colleagues found no significant differences in the severe MEs with the potential for patient harm between the pre-implementation and post-implementation periods. The study found a significant decrease in the number of errors with propensity of causing staff harm.³⁶ The ability to reduce staff harm when compounding hazardous medications directly aligns with the principles set forth within USP Chapter <800>.³⁹

Masini and colleagues utilized a crossover design, over the month-long study period, to evaluate robotic antineoplastic IV compounding technology in comparison with manual processes (including gravimetric analysis). The accuracy of both processes was compared based on the difference in relative errors between the measured versus the expected amount of compounded medication. Masini et al demonstrated a worsening in doses exceeding the 5% threshold when comparing IV compounding robotics to manual process. Of note, the worsening of this metric was driven by an error in preparation of one medication that was later corrected during the study. Although considered a manual process in the study, the use of gravimetric analysis is a component of some IV automated workflow solutions, which complicates the comparator in this study design. A cost analysis was also performed that analyzed variable unit costs and fixed costs of an IV robotics solution.³⁷ The cost analysis was distinct as it took the fixed cost of the technology into account.^36,37 The study found a break-even point of 34 000 compounded products annually.³⁷

Many of the initial studies related to the IV compounding robotics technology are descriptive reports.^33-35 Although important to describe individual institution’s experiences with implementation of these technologies, descriptive reports contain no control group and lack statistical comparators to allow robust conclusions regarding the technologies. The robotics studies with manual comparators demonstrated conflicting results with compounding accuracy.^36,37 The lack of sustained improvement in MEs related to patient harm found by Seger et al demonstrated the limited favorable impact on quality.³⁶ Additional improvements in these technologies with confirmatory studies are warranted. The noted improvement in staff safety using the systems could provide justification for use if the quality is not worsened and if staff efficiency can be improved with the use of these systems.³⁶ Based on the financial data available, an institution would have to maintain a high volume of infusions with IV chemotherapy robotics to justify a return-on-investment (ROI) to offset the high cost of the technology.^36,37

Health information technology (HIT) systems were designed to improve the care of patients. Unintended consequences arise from the unanticipated and/or undesired effects when these tools are used in practice. The informatics literature has previously described the unintended consequences seen with the implementation of informatics systems such as computer physician order entry (CPOE).⁴⁰ However, there has been limited mention with regards to pharmacy automation and robotics systems. Pharmacy automation and robotics systems are designed with the intent to improve the safety and efficacy of the medication-use process. However, there are potential unintended consequences that are associated with these technologies, which could ultimately result in the potential for patient or staff harm. Seger et al specifically discussed unintended consequences attributed to the use of IV chemotherapy preparation robotics system. In this study, there were 45 unintended events associated the implementation of the technology, though none were considered serious medication-related errors.³⁶ Another IV chemotherapy preparation robotics system described considerable downtime and decreased productivity as a result of technical issues related to the implementation of the technology.³⁴ Overall, there is limited discussion of unintended consequences across these pharmacy technologies despite a large amount of automation/robotics and human interaction.

Discussion

To put proper perspective on unintended consequences related to pharmacy automation and robotics systems, consider cases that have arisen within the traditional US Food and Drug Administration (FDA) drug development process. Medications such as Vioxx (rofecoxib) and thalidomide were removed from the market due to post-market adverse events that have been associated with significant morbidity and/or mortality.^41,42 These adverse events were not discovered until the medications were released to the general population. The FDA expects drug manufacturers to monitor for adverse events post-market in phase IV studies. Should there be similar vendor expectations when implementing technologies in health systems to ensure the systems are working as intended? Based on our review of the evidence, post-marketing research studies should be designed to detect these to the greatest extent possible. Who will design the studies, who will fund them, and who will ensure the scientific integrity of the outcomes measured?

As pharmacy leaders, it is important to learn from experiences with unintended consequences in the informatics industry to ensure systems meet their original intent. A more conscious effort must be made to measure the impact of implementation and the ongoing use of pharmacy automation and robotics systems to more fully describe their impact on the medication-use process. Given the large variability in study designs, it is challenging to interpret the available data and apply this to make decisions related to which systems to implement.

Donabedian described three categories that must be considered when assessing the quality of health services: structure, process, and outcomes.⁴³ Health care providers are well versed in the structural aspects of health care delivery, by ensuring the appropriate facilities and equipment are available for use. Process describes how systems and technologies are used, the interactions between people and systems, and the interactions between providers and patients. Technologies often help by standardizing and systemizing the possible actions that interacting humans can and should take. However, technologies can also hurt if the interacting workflow is not well designed or if humans develop work-around actions in the interest of improving their self-assessed efficiency. The best way to measure the ultimate effect of these structures and processes is through outcome assessment, which measures the impact of these systems and technologies on the health of patients. The pharmacy automation and robotics systems literature is ripe with description of structure and process, but outcome literature is lacking.⁴³ The proper selection of appropriate outcomes is critical to demonstrating value. There is a lack of consistency within medication-related technologies as it relates specifically to medication safety.¹² Accuracy rates often serve as a surrogate outcome, but accuracy rates alone do not provide insight into the consequences of potential errors. Methods such as rating of potential MEs on their propensity of causing patient harm is a better representation of the potential impact of the automation and robotic systems on patient safety.

One technique to analyze large amounts of data related to specific health-related topics includes the use of systematic reviews and meta-analyses. Jadad et al demonstrated that Cochrane Reviews were more robust than other systematic reviews published in peer-reviewed journals.⁴⁴ Cochrane reviews were designed with explicit inclusion/exclusion criteria and formal assessment of trial quality which reduced the potential for bias compared with other systemic reviews and meta-analyses.⁴⁴ Tsao and colleagues performed a systemic review of 8 ADC studies which evaluated ME rates, efficiency, and/or costs. Similar to our assessment, the study was impeded by the lack of availability of rigorous studies, which limited the conclusions that could be made.²³

In reviewing the available data regarding pharmacy automation and robotics systems, the majority of the literature are descriptive reports of implementation at individual institutions. Actual studies are typically single-centered and completed within academic medical centers. Given the complexity of unique pharmacy workflows, technology implementations are often individualized to each institution. Although it is important to learn about experiences at individual institutions, pharmacy leaders must understand that the implementations are not one-size-fits-all and each leader must understand the unique nuances of their institution. It is important to consider that the many published studies have been completed in academic medical centers which often have resources available for implementing and assessing medication-related technologies that may not be available other institutions. Although the majority of studies come from the academic medical centers, they also represent the minority of hospitals. In the current state, the data available may not directly translate to the majority of institutions across the country.

Another issue is that there is a noticeable lack of confirmation studies in the pharmacy automation and robotics systems literature. Why is this a problem? Because one study is just that—a snapshot in time of the experience of one institution based on their implementation. There is an adage often used among technology personnel that states, “once you have seen one implementation, you have seen one implementation”. An informatics research study is subject to the same design characteristic limitations inherent in any single site study, including institution-specific factors (such as size, staffing models, unique workflows, and concomitant use of other technologies), differences in outcome assessment methods or definition, and probabilities associated with type 1 or type 2 errors. There is also a noticeable lack of available multicentered studies assessing the implementation of technologies across multiple institutions. Consider the most robust studies within the medical literature include experiences from multiple institutions to capture uniqueness in patient populations, provider experiences, and institution-specific characteristics. Why should the approach be any different when analyzing pharmacy automation and robotics systems technologies?

Confirmation studies for technology should follow similar principles to the latter stages of the drug approval process. Phase III studies are completed in a controlled environment and demonstrate efficacy for a specific medication. Phase IV studies are completed to demonstrate efficacy and safety in a “real-world” environment following drug approval. Following initial studies, other researchers complete studies using similar methodologies to confirm or refute the findings of the initial researchers. The completion of confirmatory studies would provide pharmacy leaders with an additional layer of evidence with regard to medication safety and justification to hospital leadership when determining if these systems should be purchased.

Finally, there is a demonstrated bias toward the publication of positive results within the medical literature.^45,46 Pharmacy automation companies often lead their discussions regarding outcomes of their particular system(s) with a marketing focus on “perceived benefits” rather than evidence-based outcomes. Pharmacy leaders must hold these companies accountable for real-world, statistically valid results. Health systems must publish both positive and negative results to accurately represent the products on the market. There is a danger in creating a false sense of security if only positive studies are published.

Given the rising cost of health care within the United States and internal competition for finite capital dollars, it is important to identify solutions that will improve quality and safety while being fiscally responsible. Pharmacy automation and technology solutions are capital investments with major budget implications costing millions of dollars annually. Based on a recently published commentary, “capital expenditures are an important driver of health care performance and should garner more attention as a potential value lever” p. 1543. Capital investments have a large impact on health care operating budgets and the decisions last for years following their selection; therefore, value must be engrained in the decision-making process.⁴⁷

Health care is constantly changing as a result of new technologies and financial pressures and incentives. As a result, pharmacy faces pressures to conduct rapid cycle evaluation and implementation of solutions to meet increasing demands to improve efficiency, reduce the overall cost of care, and maintain or improve the quality of patient care. Often what suffers is the ability of pharmacy leaders to critically evaluate pharmacy automation and robotics systems prior to implementation. Given the currently available literature, pharmacy leaders are often swayed by marketing promises and anecdotal information from colleagues when selecting systems to optimize medication-use within their facilities rather than demanding evidence that these claims are substantiated. Similar to the drug approval process, we are advocating for the availability and use of evidence as part of the decision-making process for pharmacy automation and robotics systems implementation. To achieve this, we are providing the following recommendations:

1. National/international professional organizations should facilitate the creation of expert panels to standardize the process related to the evaluation of pharmacy automation and robotics systems

2. Expert panels should describe the core metrics that should be used in these evaluations, including measures related to patient safety and financial stewardship.

   1. Metrics should include appropriate efficacy outcomes, consistent measures of ME, and potential adverse events including severity ratings, and meaningful financial metrics.

3. Health-system pharmacy leaders should advocate for adequate funding to support high-quality research studies that measure outcomes associated with the implementation of pharmacy automation and robotics systems.

4. Principles utilized in clinical research (randomization, pre/post implementation, control groups) should be applied to evaluation of pharmacy automation and robotics systems

5. Research should be conducted at a diversity of institutions (eg, varying size and levels of care) to better reflect real-world challenges; consideration should be given to multisite studies.

6. Validation studies should be completed to confirm results from single-center studies.

7. Standard-setting organizations (eg, the Joint Commission, State Boards of Pharmacy) in conjunction with national organizations (eg, the American Society of Health-System Pharmacists) should review evidence regarding pharmacy automation and robotics systems using a scientific approach, promote the use of these technologies where sufficient evidence exists, assist pharmacy leaders in seeking local capital funding through advocacy, standards, and tool kits, and help determine future directions for evaluation of these systems.

8. Corporations who provide pharmacy automation and robotics systems should work with standard-setting organizations to ensure meaningful outcomes are measured and to compare like-products on the market

Pharmacy automation and robotics systems will continue to be developed and marketed at a fast pace. Health care technology has become a great business but the question remains: Where is the science? The current literature regarding pharmacy automation and robotics systems does not support critical evaluation of these systems. Standardization of relevant outcomes related to implementation of these systems would benefit pharmacy leaders internationally to ensure they are equipped to make more-informed decisions about these large capital investments. To accomplish this goal, there will need to be collaboration between professional organizations, hospitals and health systems, pharmacy leaders, health-system leaders, standard-setting organizations, grant funding organizations, and manufacturers of these systems. The ultimate goal for all of these groups should be consistent: the implementation of pharmacy automation and robotics systems that deliver pharmaceutical care in a safe, effective, and cost-efficient manner.