Systems Engineering and Point of Care Testing: Report from the NIBIB POCT/Systems Engineering Workshop

Abstract

The first part of this manuscript is an introduction to systems engineering and how it may be applied to health care and point of care testing (POCT). Systems engineering is an interdisciplinary field that seeks to better understand and manage changes in complex systems and projects as whole. Systems are sets of interconnected elements which interact with each other, are dynamic, change over time and are subject to complex behaviors.

The second part of this paper reports on the results of the National Institute of Biomedical Imaging and Bioengineering (NIBIB) workshop exploring the future of point of care testing and technologies and the recognition that these new technologies do not exist in isolation. That they exist within ecosystems of other technologies and systems; and these systems influence their likelihood of success or failure and their effectiveness.

In this workshop, a diverse group of individuals from around the country, from disciplines ranging from clinical care, engineering, regulatory affairs and many others to members of the three major National Institutes of Health (NIH) funded efforts in the areas the Centers for POCT for sexually transmitted disease, POCT for the future of Cancer Care, POCT primary care research network, gathered together for a modified deep dive workshop exploring the current state of the art, mapping probable future directions and developing longer term goals.

The invitees were broken up into 4 thematic groups: Home, Outpatient, Public/shared space and Rural/global. Each group proceeded to explore the problem and solution space for point of care tests and technology within their theme. While each thematic area had specific challenges, many commonalities also emerged. This effort thus helped create a conceptual framework for POCT as well as identifying many of the challenges for POCT going forward.

Four main dimensions were identified as defining the functional space for both point of care testing and treatment, these are: Time, Location, Interpretation and Tempo. A framework is presented in this paper.

There were several current and future challenges identified through the workshop. These broadly fall into the categories of technology development and implementation. More specifically these are in the areas of: 1) Design, 2) Patient driven demand and technology, 3) Information Characteristics and Presentation, 4) Health Information Systems, 5) Connectivity, 6) Workflow and implementation, 7) Maintenance/Cost, and 8) Quality Control. Definitions of these challenge areas and recommendations to address them are provided.

Part 1: Systems Engineering and it role in Technology Evaluation and Implementation

Introduction

The United States has the highest annual per-capita health expenditures of developed countries, yet its population still faces significant health challenges. Many changes to the system are being proposed to address current inadequacies. To respond to the needs of an increasingly unhealthy population comprised of individuals with multiple chronic conditions, one goal of these changes is higher quality care at reduced cost, with a shift in focus from utilization of specialized care for the treatment of late-stage disease to an emphasis on patient-centered approaches and coordinated care teams that promote wellness and effective disease management. The evolving healthcare system includes new delivery models in which primary care physicians and nurses are assuming more significant roles, with the patient more involved in decision-making and self-care. These changes require the development of inexpensive and easy-to-use medical devices and information sharing tools that provide timely health status information at the point of care.

To address these challenges, the National Institute of Biomedical Imaging and Bioengineering (NIBIB) created the Point-of-Care Technologies Research Network (POCTRN) in 2007(1). POCTRN’s purpose is to drive the development of appropriate point-of-care diagnostic technologies through collaborative efforts that merge scientific and technological capabilities with clinical need.

The POCTRN is currently comprised of three centers, each with different focus. These are:

• Center for Point-of-Care Technologies Research for Sexually Transmitted Diseases(2)

  • Principle Investigator: Charlotte Gaydos DrPH

  • Johns Hopkins University

• Center for Innovation in Point of Care Technologies for the Future of Cancer Care(3)

  • Principle Investigator: Catherine Klapperich, PhD

  • Boston University (Charles River Campus)

• Point of Care Technology Research Center in Primary Care(4)

  • John Parrish, MD

  • Massachusetts General Hospital

Each POCTRN Center performs the following functions in their domain area:

• Assessment of clinical and user needs to inform device design and further define publicly available clinical needs information

• Evaluation of promising point-of-care prototype devices from the perspective of device performance and potential for clinical impact

• Completion of clinical testing appropriate for the stage of development of the chosen prototype and the target clinical application, to facilitate translation and commercialization

• Training and education of relevant stakeholders in the development and utilization of POC technologies, including technology developers, industry partners, practitioners and the lay community (as potential users)

• Development of partnerships with industry and other stakeholders to facilitate commercialization

• Utilization of Network collaboration as needed to achieve goals and increase visibility of point-of-care testing

• Provision of grants to industry and academic investigators to develop POC technologies

Systems Engineering and POCT meeting

The purpose of this meeting evolved out of a recognition that new technologies do not exist in isolation. They exist in the context and within ecosystems of other technologies and systems; and these systems influence the success or failure, effectiveness or not of any new technology. The Institute of Medicine (IOM) and National Academy of Engineers explicitly recognized this in their report "Building a Better Delivery System"(5), and the need to apply industrial engineering and system engineering tools to the healthcare system.

Technologies succeed or fail for many reasons. Broadly the reasons can be classified as functional or system-related. At the functional level, a technology can fail when the technology is not capable or mature enough to fulfill designed for function. For example, a test with too high a false positive or false negative rate or a device that requires too much investment in maintenance to keep working. Functional failures or immature technologies tend to remove themselves from the market. A systemic failure occurs when the technology is not designed for the end-users needs, the technology is incompatible or cannot work in the system it is inserted into or the technology creates systemic breakdowns, excess demands or shortages or overstresses the environments into which it is inserted. It is not news to report that our healthcare delivery system is in crisis. We remain in an environment where demands for new technologies outstrip supply.

New technology is fundamentally changing the delivery landscape. These changes can be disruptive or sustaining(6).

A disruptive innovation is an innovation that helps create a new market and value network, and eventually disrupts an existing market and value network (over a few years or decades), displacing an earlier technology. In healthcare one might consider new immunologic modulating agents as disruptive, displacing many other classes of therapeutics.

A sustaining innovation does not create new markets or value networks but rather only evolves existing ones with better value. In health care examples are new MRI or CT technologies which incrementally improve sensitivity and specificity but don't generally change how care is delivered.

Point-of-care technology (POCT) is potentially disruptive. As this technology rapidly evolves we are seeing profound changes in accuracy, expanded capability to test and treat and robustness. Medical tests that once required a large infrastructure, several days to produce results, and a robust logistical base are now being conducted in outpatient offices, people's homes and in resource constrained areas such as developing countries. This increase in test capability is changing where and when this technology is deployed.

The purpose of the May 21 2014 NIBIB meeting was to lay the groundwork for:

• Anticipating the near, mid- and long-term shape and needs of our healthcare delivery system

• Helping to decide what development to support, encourage, avoid or allow to proceed on its own

• Helping to guide new entrants, who are developing POCT, as to where their efforts may be best spent and how to succeed in a complex unfamiliar environment

• Helping to understand the complex healthcare delivery systems needs from a systems perspective.

What is systems engineering?

The role of system engineering is to better understand and manage changes in the system.

The big picture as noted above is that our healthcare delivery system is in crisis. We are experiencing increases in demand and costs and our material and human resources and our time are often constrained or poorly allocated.

Diagnosis

The Institute of Medicine (IOM) in 2000 in their report "To Err is Human"(7), gives us the (in)famous estimate of > 98,000 deaths per year due to poorly designed or broken health care processes. One year later in 2001 the IOM report "Crossing the Quality Chasm(8)", documented the potential quality of care people receive versus the care they actually receive and the comparison reflected poorly the on the state of the system. In 2005, Bodenheimer in his 4 part series(9–12) pointed out that one of the major causes of high and rising health costs and poor quality is a lack of systems perspective and thinking, as well as poor management of technological innovation.

Therapeusis

In response the National Academy of Engineering (NAE) and IOM published their report (2005) "Building a better delivery system"(5). They noted that system engineering (ISyE) tools have been used in a wide array of manufacturing and service industries, contributing to major improvements in quality, efficiency, safety, and customer-centeredness, and that the healthcare delivery system would benefit by bringing in these tools to address it problems.

What is a system?

At its simplest, a system is a set of connected persons, parts or things forming a more complex whole. Central to this definition is the concept of connectedness. Connectedness is not the same as association. Association implies co-incidence or co-eventuality of events without their being a causal relationship. Connectedness also goes beyond simple causality because connectedness implies feedback and some form of communication between the elements of the system. One of the consequences of this is that systems are dynamic, changing over time, and subject to complex behaviors.

Feedback may be positive or negative, promoting or suppressing the activity of another element in the system respectively. With these complex interactions over time system behavior is often more or different than the sum of its parts and can result in unexpected events. For example, a school of fish is comprised of elements, specifically, fish that often follow simple rules such as: keep one length ahead of the fish behind you, keep one length away from the fish to your right and flee when you see a predator. The resulting schooling behavior is an example of system behavior called emergent behavior. The school emerges as a meta-entity with its own behavior yet comprised of smaller units. Other examples include, traffic, the brain and our healthcare delivery system.

Systems are complex and complexity is hard, therefore, what tools do we have available to understand or manage it? In the 1940s Bell Labs(13) was the first to explicitly develop industrial engineering and system engineering (ISyE) tools.

Today system engineering is defined as an Interdisciplinary field focusing on how to design and manage complex systems and projects as a whole. Conventionally, it deals with:

• Work-processes

• Optimization methods

• Evaluation methods

• Coordination/integration of different stakeholder teams

• Reliability

• Logistics

• Risk management

• Industrial engineering

• Industrial design

• Organizational and operations research.

In Healthcare the goal of systems engineering is to achieve the 3Rs, getting the right therapy, test or resource to the right person, in the right place at the right time. This can only be achieved consistently by understanding how the system behaves.

Connecting ISyE and Healthcare

Despite ISyE’s obvious benefits to many industries such as manufacturing, aerospace, service industries and other industries, healthcare has been slow to adopt it. There have been many barriers to adopting this technology, and many others, in healthcare, both for positive and negative reasons. Specific barriers include the primary objective function of health care, “First, do no harm”. This has led healthcare as whole to be cautious in adopting change. Another barrier is years of negative experiences with process improvement initiatives over time. The tools of Lean and Six sigma, while powerful, have often been applied in a partial fashion or for specific functional outcomes such as throughput without consistently paying attention to the global and system needs of the key elements in the system the end-users, i.e., the clinicians and staff. As a result, much trust has been eroded. Another barrier is related to the incremental approach to change in health care, without an abrupt crisis change is less likely to happen. Incremental change allows systems to adapt and adapt until the limits of the system are reached and the system breaks, a "boiling frog" syndrome. Another key barrier to change in the healthcare system relates to its strong system of stakeholder feedback characterized as the 4Cs (Community, Control, Cure and Care) by Mintzenberg and Glouberman (14, 15) (see Figure 1). Finally, there is the issue of external and internal skill development. At present, healthcare must import these skills from the outside. Any skill set from the outside gives the stakeholders an opportunity to ignore it. It is common to hear in healthcare that outsiders do not understand our problems because they don't live it. Hence, there is the need to have "home grown" experts in ISyE.

Figure 1.

Systems view of Healthcare: Four C

Adapted from S Glouberman, H Mintzberg Managing the Care of Health and Cure of Disease 1998 INSEAD

Overcoming Barriers

In order to overcome these barriers the health care delivery system community in general and the POCT community in specific will have to do several things. First is developing facts on the ground. To do this we need to generate evidence that decision-makers and end-users deem reliable, objective and, useful, i.e., linked to clinical outcomes. Specifically we need to be able to get the right information to the right place at the right time. Second, we need to cultivate a systems perspective in the healthcare delivery community. This will require sustained effort from both the healthcare community and the ISyE community. Specifically, ISyE practitioners need to pay concerted attention to Healthcare’s unique system of stakeholders and the Healthcare community needs to be trained look at how our individual actions affect the system. In part this will also require a joint effort to develop the skills and resources in health care. Finally, it also means cultivating trust; trust in the tools, trust in the practioners, trust in the end-users and trust in the decision makers(16–25) Figure 2.

Figure 2.

A Systems engineering/solution model for creating change in health care

An iterative model has several steps:

1. Interdisciplinary envisioning/designing system solutions together(26).

   This is the co-development of an understanding of what the problem space is and its solution space. Intrinsic to this process is developing trust in each other’s view of the problem and the feasibility and desirability of potential solutions

2. Agreed upon system, measures, and outcomes

   This means all stakeholders’ agree on what is important and what we are observing. This is particularly critical in systems where there is complex and robust feedback between the stakeholders

3. Understanding and predicting the effect on all end-users

   This means developing tools to anticipate not just the immediate outcomes but also the longer term effects on all system stakeholders and end-users. The stakeholders and end-users must trust there will be benefit or minimized harm by the proposed change.

4. Implement and verify

   Trust is necessary but it is also necessary verify that the project achieved its goals against objective reality.

5. Iterate

The above model creates a path from problem specification to system trust.

Systems Thinking

Systems thinking is a holistic approach to analysis that focuses on the way that a system's constituent parts interrelate and how systems work over time and within the context of larger systems. Systems thinking is difficult because it is hard to think “outside the box” that we make for ourselves and “to see the forest for the trees”. It is difficult because of the prevalence of the Cartesian reductionist approach in decision making and analysis. Much of the culture of science is based on the dualism of Rene Descartes. Cartesian reductionism has been incredibly useful as far as it goes but has tended to result in a scientific culture that believes only one tool is valid or useful. However, as Thomas Kuhn noted, culture change even in science is difficult at the best of times. Reductionism, however, only gets you so far. Eventually, every individual item examined must be placed in context and as noted above systems are intrinsically more complex than the sum of its parts. An alternative philosophic approach may be found in the Philosophy of Systems as developed by Lazlo and Von Bertalaffny(27, 28). Another challenge to systems thinking is the limit our cognitive abilities. There are a limited number of variables we can juggle or solve for. Lack of control or certainty makes many people uncomfortable and shy away from these types of problems.

To practice systems thinking it requires:

• Understanding systems context

• Understanding relationships

  1. within the system

  2. between the system and its context

  3. systems at different scales

• Understanding complexity

  1. Relationships between elements of systems and time yield uncertainty, and dynamic nonlinear states and situations

• Communicating across disciplines

  1. Any single perspective is inherently limited. The broader the set of disciplines examining a problem the more perspectives and experience can be brought to bear to create a solution, taking advantage of a diverse principles, models, methods and tools.

  2. However, one must first create an environment where people from widely divergent origins can successfully share ideas.

• Practice

  1. Like any skill one must practice systems thinking if one is to internalize it and use it successfully.

Operating Room of the Future (ORF)

The demand for surgical services is rising faster than our ability to supply them(29). In an increasingly resource constrained environment efficiency is critical for maintaining access to quality care. Surgeons, nurses and OR administrators often turn to new technologies to help solve this problem. However, in the OR setting, new technologies are usually introduced in an ad hoc manner without rigorous analysis of their potential costs and benefits, interactions with other systems or their impact on the end-user. In the ORF project we used computer and physical simulation methods(30) to design and build a new OR suite at the Massachusetts General Hospital. This project was a collaboration between CIMIT and the Massachusetts General Hospital and a consortium of scientists and technologists. This OR suite was to serve as a living laboratory where we could measure the effect of new staffing regimes, peri-operative systems and equipment on the delivery of surgical care. The specific objectives were:

• Prospectively analyze cost and effectiveness of new peri-operative processes, integrated surgical systems and technologies

• Prospective evaluation of new measurement technologies

• Prospective evaluation of acceptability of new technology

• Observe and characterize learning of new technologies

The ORF design features a central operating room with attached induction and recovery bays and a control room to provide a workspace for surgeons between cases (Figure 3). The redesigned peri-operative process allows staff to care for patients in parallel (Figure 4) or in an interleaved fashion in contrast to the traditional sequential process - where all patient care from induction of anesthesia through surgery and extubation occur in the same single space. Parallel processing allows two (theoretically up to three) patients to be cared for simultaneously in the induction bay, the OR, and recovery bay. However, this process change requires more staff (an additional 0.5 anesthesia FTE [full time equivalent] and 1 nurse FTE) than in the standard OR process.

Figure 3.

Architectural mock up of OR of the Future prior to implementation

Figure 4.

Comparison of workflow in ORF and Standard OR

We hypothesized that redesigning the operating room and the peri-operative process (Figure 4) would improve throughput and that this improved efficiency would reduce the cost/patient treated when compared with standard ORs. The evaluation of the ORF was conducted in three steps: 1) prospective case-controlled evaluation of the operational characteristics of the room, 2) retrospective case-controlled exploration of the variables affecting outcomes and costs, and 3) cost-effectiveness analysis combining cost and effectiveness data. This process broadly followed the systems engineering and design cycle described above. See Figure 5.

Figure 5.

Systems engineering and design problem solving cycle for ORF

Team construction

The project team was comprised of as wide a range of disciplines as was possible. The group included:

• Surgery

• OR administration

• Anesthesia

• Nursing

• Industry

• Health outcomes research

• Operations research

A key to the success of this project was an emphasis on cross-disciplinary communication. Each stakeholder group has different approaches to problem solving, and uses different language and jargon to describe problems and solutions. These barriers were overcome by developing trust through shared experience and the creative use of visualization and analysis tools which allowed all the stakeholders to share a vision of the projects goals and how to proceed.

Envisioning Tools: Simulation

Simulation is a tool that allows the investigator to create a functioning model of the real system that captures the underlying cause, effect and interactions in the real system. These simulations allow the investigator to develop an understanding of the ‘nature’ of the ‘real’ system through experiments on the in silico system. Simulation is a particularly useful when evidence and understanding are needed but direct experimentation is not feasible due to: Cost, Time, Ethics. This is often the case in healthcare. Simulation in general is a tool that allows the investigator to go DEEPP(31). Specifically Describe, Evaluate, Explore, Predict, system behavior and Project for stakeholders their options. It is also a tool that allows stakeholders to explore options and make mistakes prior to implementation (Figure 6)

Figure 6.

The relative cost of errors in the implementation process by phase after Ghosn et al

In the case of the ORF Stakeholders needed convincing to proceed. They needed to be convinced that the propose system would work and they needed to be reassured we were measuring the correct outcomes. In this case this involved multiple scenario analysis on permutations of the physical space and staffing levels. It also required specific projections of cost-effectiveness and cost-benefit as well as return on Investment.

Common Outcome Measures in systems analyses

• Health outcomes

  1. Morbidity, Mortality, Life-expectancy

• Quality

• Economic outcomes

  1. Costs

• System performance outcomes

  1. Process measures

  2. Error rates

  3. Reliability

• System outcomes

  1. When comparing systems you want to choose measures that are a product of system behavior

  2. Every technology affects users ability to perform the task

  3. Operates within the context of a system, specifically, a new technologies should effect the operational characteristics of system in which they are introduced

  4. Common measures

     • Throughput

     • Flow time

     • Wait time

     • Utilization

Inter-disciplinary communication of POCT Working Group

In order to maximize our ability to envision the future of POCT, it was felt that it was critical to have as many different perspectives on the problem as possible. The more perspectives the less risk of missing an important trend or phenomena that will shape the future.

Potential Dangers of within domain thinking

The potential dangers of within domain thinking are well known. An often cited example is that of the Polaroid camera company. Polaroid is a good example for the POCT world because its major breakthrough innovation was to come up with a technology that shortened the time from gathering information with an instrument to generating end-user desired outcome, specifically, shortening the interval from taking a picture with a camera to having printed photos in hand from days to seconds. The parallel to POCT is clear.

Polaroid's failure occurred when they mistook a technologic object, the photographic print, for its function to the end-user, a preserver of memory. This resulted in large part from looking at its industry only from a within-print media perspective(32). As a result, they did not see early enough or take seriously enough the changing environment or take advantage quickly enough of the new tools, such as electronic media.

Multi-disciplinarity vs. Inter-disciplinarity

The terms multi-disciplinarity and interdisciplinarity are often used interchangeably(33, 34). We would like to draw some distinctions which will help clarify when each is most useful. The distinction to is to some extent a matter of degree. Multi-disciplinarity involves the interaction and hopefully collaboration of individuals from separate but related fields. In clinical care, this might involve having care providers from different disciplines (for example, physicians, nurses, occupational therapists, and social workers) working together to develop appropriate care plans for a given patient, whose needs may be complex or to create a working collaborative such as a "heart center" or a “medical home”. In these settings knowledge is shared within a commonly understood framework such as healthcare. In a sense the combined knowledge is additive and follows easy to understand rules, i.e. 2 + 2 = 4.

Inter-disciplinarity on the other hand is a process through which parties who see different aspects of a problem and have different conceptual frameworks, can constructively explore their differences and search for solutions that go beyond their own intrinsically limited vision of what is possible. For example, bringing together clinicians, engineers, artists and patients with their own distinct conceptual models and problem solving approaches would be inter-disciplinary. There is a growing recognition that complex problems like those often found in healthcare require bringing in the outside perspectives of many disciplines working together to achieve workable solutions. Interdisciplinarity combines diversity in ways that are often unexpected often producing new forms of knowledge.

Approaches to Problems

Interdisciplinary groups not only can differ in where they are coming from and their sources of knowledge and experience, they can also differ on their approaches to problems. This may be a greater hurdle to interdisciplinary communication than differing knowledge bases.

Two common approaches to problems may be characterized as problem solving and problematizing. A problem solving approach tends to ask first "What are my constraints?" It then proceeds to approach a problem as practical process that frequently requires compromises to reach solutions that can be implemented in a short time frame. A problematizing approach tends to ask first, “Why are these constraints?”. Problematizing is a thoughtful process that aims to identify and explore root causes of problems, regardless of whether or not those root causes can be practically addressed within a given time frame. Both are valuable with one seeking to solve problems with what is known and the other challenging our assumptions on what is known.

Barriers and facilitators to interdisciplinary envisioning

With participants coming from many backgrounds there are many potential barriers to success(26). For example, it may be easier to work within a shared approach to problems than within a shared discipline. The specifics of a knowledge base may be less contentious than reconciling individuals world-views. However, one shouldn't underestimate the ability of disciplinary gaps in language, ambiguous use of terms or terms that are used to point to different mental objects in different disciplines, e.g. the word “model”, to sow confusion. Other key potential barriers to collaboration are Ego, Ownership of ideas and hierarchy. Each can inhibit the exchange of ideas or allow them to improved upon by others. Finally, poor communication and disrespect can scuttle the best efforts to share and grow ideas.

Facilitators for interdisciplinary collaboration are superficially simple but often hard. These include: Listening, systems and lateral thinking, compassion/humility and a willingness to trust.

Deep dives Workshop

The Deep dive is technique derived from the world of industrial design. It is a method whereby a team of involved end-users and designers get together to explore in depth the nature of the end-users’ dilemma. These deep dives often are held within the end users’ natural environment rather than through interviews or focus groups. They are oriented towards abstractive reasoning where problem solvers look beyond the current circumstance and dilemma and articulate and speculate on potential long- and near-term solutions. Deep dives grew out of the concept of brainstorming but explicitly use facilitators to mitigate evaluation apprehension, free riding, production blocking, negative feedback and other potential stumbling blocks.

The modified Deep dive method used in the POCT workshop was derived from the work of Witteman and Stahl (Radcliffe Foundation)(35) which in addition the traditional facilitated format also includes the use of improvisation and rapid visualization to speed up the communication and teamwork process and allow end-users through visualization tools to literally be on the same page when crafting ideas.

The goal of the workshop was to help develop a preliminary conceptual framework for POCT and prioritization of action items for the point-of-care community. The invitees were broken out into 4 facilitated groups organized around the themes of: Home, Outpatient, Community/Public/Shared space, Global/Rural (Table 1). The types of stakeholders represented, the range of test touched upon and modalities for POCT are listed in tables 2–4.

Table 1.

Work shop groups

Home	Outpatient	Public/Shared	Rural/Global
Facilitators
Heather McGowan, facilitator	Stahl, James, MD, CM, MPH, facilitator, Massachusetts General Hospital, Senior Scientist, Institute of Technology Assessment; POCTRC in Primary Care, Systems Engineering Core Leader.	Ellen Di Resta, facilitator	Gaydos, Charlotte, DrPH, MPH, MS, facilitator, Johns Hopkins University, Center for Point-of-Care Tests for Sexually Transmitted Diseases, Principal Investigator.
Participants:
Abello, Paola MBA, Harvard Center for Primary Care, Innovation Center Program Manager.	Carleton, Penny, RN, MS, MPA, MSc., CIMIT, Point-of-Care Technology Center in Primary Care, Program Manager; Clinical Innovation Director.	Daaboul, George, Ph.D., Boston University, Post- Doctoral Researcher, Electrical and Computer Engineering.	Cunningham, Brian, Ph.D., University of Illinois, Professor of Bioengineering; Director, Center for Innovative Instrumentation Technology; Founder and CEO, Exalt Diagnostics.
Edgman-Levitan, Susan, PA, Massachusetts General Hospital, Executive Director, MGH Stoeckle Center for Primary Care Innovation; POCTRC in Primary Care, Needs Assessment Core Leader.	Clarke, William, Ph.D. , Johns Hopkins University, Director of Clinical Toxicology and Point-of-Care Testing.	Dempsey, Michael, CIMIT, Entrepreneur-in- Residence; Director, Accelerator Program.	Klapperich, Catherine, Ph.D, Boston University, Center of Future Technologies in Cancer Care, Principal Investigator.
Fawcett, Helen, Ph.D. , Boston University, Center of Future Technologies in Cancer Care, Program Manager.	Crocker, J. Benjamin, MD, Massachusetts General Hospital, Medical Director, Ambulatory Practice of the Future.	Dixon, Ronald, MD, Massachusetts General Hospital, Director of the Virtual Practice Project at MGH; Associate Medical Director of MGH Beacon Hill Practice.	Korte, Brenda, Ph.D., NIBIB, Program Director, Point-of- Care Technology Research Network; Program Director in the Division of Discovery Science and Technology.
Filerman, Mark, CIMIT, Accelerator Team	Jackman, Joanie, Ph.D., Johns Hopkins University, Center for Point-of-Care Tests for Sexually Transmitted Diseases, Co-Investigator.	Goldberg, Bennett, Ph.D., Boston University, Center of Future Technologies in Cancer Care, Educational Core Leader.	Lewandrowski, Kent, MD, Massachusetts General Hospital, Director, of Clincial Laboratories Point-of-Care Testing; Associate Director, Clinical Laboratories; Associate Chief of Pathology; POCTRC in Primary Care, Validation Core Leader.
Kane, Barbara MD, MGH, Primary Care Physician, Bulfinch Medical Group.	Parrish, John, MD, CIMIT, Founder and Executive Director; Point-of-Care Technology Research Center in Primary Care, Principal Investigator.	Merchak, Todd, B.S., NIBIB, Biomedical Engineer in the Extramural Science Program; POCTRN Science Officer.	Otenti, Desiree, NP, Boston Health Care for the Homeless Program, Nurse Practitioner.
Kelley, Chris, Ph.D., NIBIB, Director of the Division of Discovery Science and Technology; POCTRN Science Officer.	Sims, Nat, MD, Massachusetts General Hospital, Serial inventor; Director, Sims Lab.	Klepser, Michael, Pharm D., FCCP, Ferris University, Professor, Pharmacy Practice.	Sauer-Budge, Alexis, Ph.D., Boston University- Fraunhofer Alliance for Medical Devices, Senior Research Scientist, Instrumentation and Diagnostics.
Mascoli, Nicholas, MD, Newton- Wellesley Hospital, Primary Care Internist.	Wang, Catharine, Ph.D, Boston University, Community Health Services.	Goldman, Julian MD, Partners Healthcare, Medical Director of Biomedical Engineering, a practicing anesthesiologist at the Massachusetts General Hospital, and Director of the Program on Medical Device Interoperability at MGH and CIMIT.	Schachter, Steve, MD, CIMIT, Chief Academic Officer; POCTRC in Primary Care, Scientific Core Leader.
	Remote Participants:	Whitehead, Susan, CIMIT, program manager of the Medical Device “Plug-and- Play” (MD PnP) Interoperability Program.	Spiliotis, Diane, CIMIT, Chief Financial Officer and Chief Information officer
	*Krosnick, Steve, NIBIB, Program Director; Extramural Science Programs; Science Officer POCTRN.	Lee, Debbie, CIMIT, Grants Manager
	*Sabourin, Stephanie , NIBIB, Biomedical Engineer, Division Of Discovery Science & Technology; Science Officer, POCTRN.

Table 2.

POCT Stakeholders

Patients

Providers

Family

Patient Groups

Community

Culture

Insurers

Regulators

Developers/Industry

Investors

Table 4.

Examples of POCT Modalities

Fluid
	Blood
	Urine
	Saliva
	Stool
Vapor
	Breath
Ultrasound
Electromagnetic
	IR
	X-ray
	Conductance
Mass
	Weight
	Accelerometry

The workshop followed the following format:

• Loosening Exercises

• Focus and framing for each group

• Problem space mapping

• Solution space mapping

• Identifying Barriers and facilitators

• Summarizing results

Part 2: Findings from the POCTRN workshop

LESSONS LEARNED AND RECOMMNDATIONS

The invitees were broken up into 4 thematic groups: Home, Outpatient, Public/shared space and Rural/global. Each group “deep dive”(36) was led by a facilitator and with aid of rapid visualizers proceeded to explore the problem space and of Point of Care testing for their theme. While each thematic area had specific challenges, many commonalities also emerged. This effort thus helped create a conceptual framework for POCT as well as identifying many of the challenges for POCT going forward.

THE DIMENSIONS OF POCT

There are 4 main dimensions which define the functional space for both point of care testing and treatment, these are: Time, Location, Interpretation and Tempo. See Figure 7.

Figure 7.

Framework for evaluating and comparing Point of care Tests

TIME

Time describes the interval between the time the test is performed and the time the resulting information is fed back to the end-user. Broadly, the meaningful distinctions described by the working group were: 1) Continuous, 2) Within encounter and 3) Circa Visit turnaround. Continuous means information is fed back to the end-user at a short enough interval that the information is perceived as always available to the end-user when they ask for it. An example might be a continuous pulse oximeter or a pedometer. Within encounter means that the interval between test and resulting information is short enough to allow the end-user to use it within the patient- clinician encounter. Patients may receive the test prior to the encounter but within the same day of the visit. the working group defined this interval as less than 15–20 minutes. This is the time budgeted for most patient encounters in scheduling templates. This might allow the clinician and patient to work with the resulting data simultaneously as it becomes available and make decisions accordingly. Circa visit implies an interval of 1–2 working days between an onsite test given at the point of care, specifically, where care will be delivered, such as, a clinic, and the resulting information.

LOCATION

Location describes primarily where the test is performed but can also describe where the information is received. The distinctions made by the working group were: home, outpatient, public/shared, remote, mobile and virtual. Home was described as the primary dwelling of an individual or group. Outpatient described a outpatient medical facility as distinguished from in hospital. Public or shared space described and space where multiple individuals can go and interact, exclusive of home and clinic, where members of various communities may interact, where physical privacy may not be readily available. Remote was characterized as either distant from population centers and/or resource constrained in a way as to limit access to care and technical infrastructure. Mobile refers to mobile facilities or labs that can travel to or near the patient. Virtual refers to tests performed away from a clinical facility but connected to a clinical facility, for example, telemedicine functions such as tele-ophthalmology. This also could refer to “quantified-life” type applications linked to the internet.

INTERPRETATION

Interpretation is the dimension describing the requirements needed to translate the information generated by the POCT into a usable form. The primary driver of this dimension is who is able to translate this data into a usable form. Specifically, can the POCT be used by an untrained individual or is a trained person distinct from the end-user, such as a clinician or expert patient, needed to translate the POCT information before the end-user can use it. An example of POCT that is usable by the untrained end-user is a pedometer. Here it is assumed the average untrained individual can collect, translate and act on the information generated without outside help. POCT that meets the untrained standard may lend themselves to self-management of problems. An example of a POCT requiring a trained individual is an ECG. Even the most sophisticated algorithms used in some of these devices have their true positive and false negative rates and provide information in a way that can potentially be misinterpreted by the untrained and even the moderately trained user.

This dimension has many different subdomains as what it means to be “trained” and “untrained” have many components. Endogenous factors that contribute include issues of literacy and numeracy. Exogenous factors include issues of industrial design, the dimensionality of the information, integration with and access to EHRs, decision support tools, accuracy, ambiguity, reliability, and implied risk.

Ultimately a test is as good as its ability to inform a decision or clarify which strategy to pursue. Thus health Interpretation also includes how actionable is the information provided and in practice how likely is it to help provide the impetus or support behavior change.

Finally, interpretation contains an ethical component as the ability of an end-user to interpret the POCT information on their own or with help reflects on the ethical domains of autonomy, beneficence, non-maleficence and justice.

HEALTH TEMPO

The tempo of a disease or illness refers to the pace and urgency of the individuals health issue. In the context of POCT we consider a spectrum ranging from acute to chronic to well. Acute refers to a problem that requires attention from within 1 day to 1 week. Chronic refers to an issue that has a steady tempo potentially punctuated with acute events. Well refers to the state where there are either: no active problems, where only screening is appropriate, or where the focus is on maintenance of wellness rather than treatment of disease.

CHALLENGES FOR POCT

The challenges facing POCT broadly fall into the categories of technology development and implementation. These issues surrounding technology development are familiar to most technologists. Specifically, developing tools which do what they are intended to do and the challenge of developing tools end-users are willing to use. Each is necessary but not sufficient for success. In this paper we assume most technologists will be familiar with the technical challenges of their own domain such as microfluidics and optics and will focus on the systems and implementation issues. It should be noted however, that many of the items identified below are interconnected, e.g., health information systems, data presentation, and work flow, and will require systems type thinking to solve them.

DESIGN

Industrial design is the use of applied art and science to improve the functionality and usability of a product and facilitate its uptake by end-users. POCT exists at the interface between the patient and practitioner. Experience is one of the domains of industrial design particularly relevant to POCT. This domain describes the environment, context, usability, ergonomics, cultural and behavioral effect of technologies and the study and practice of making them work best in these dimensions. This issue was raised as a challenge by all of the thematic working groups. Ultimately a satisfactory patient and practitioner experience will be one of the necessary conditions for the successful implementation and commercialization of POCT. Another related domain is personalization and customization. This is of particular import in the home and wellness settings and the growing area of the "quantified life" movement.

Finally, with the increased gathering of personalized information in a wider range of environments come risks to privacy and confidentiality. Solutions to this last item will require the education of patients and providers, new information systems and designing the POCT tools with the best design affordances to minimize the risk of privacy loss.

Recommendation: As part of the development of POCT is critical to include early on the resources to develop the technologies’ usability, ergonomics, and to pay attention to the technologies behavioral and cultural effects on the individual and society. All of these will be essential in any implementation process.

Recommendation: With the increasing capability of POCT, increased focus must be brought to privacy and confidentiality concerns that these technologies raise in parallel to their development.

PATIENT DRIVEN TECHNOLOGY

A challenge identified by the groups was that as off the shelf technology becomes more competent and the "maker movement" becomes more competent, more POCT type activity will be driven by the patients themselves. The advent of using "Dr Google" by patients to take more control of their health is an example. This raises significant concerns over quality control and risk assumption.

Recommendation: The POCT community should 1) regularly survey the landscape of end-users and their needs. 2) Provide a mechanism for engaging community driven technology.

INFORMATION CHARACTERISTICS AND PRESENTATION

A challenge related to both design and information system is how the information generated by POCT is characterized and presented. The goal is to provide information in such a way that it can advance personal wellness or clinical diagnostic, prognostic or therapeutic goals. Specifically, the data should drive action either at the moment of the encounter or downstream. This will depend in part on whether or not the end-user is untrained, e.g., the public, or trained, e.g., clinicians, in how to use and derive meaning from the data. We know even in the trained group there is a large range in domain literacy and general numeracy.

The density and dimensionality of the data also impacts its usability. The denser the data or the more different variables presented will directly impact how fast the information can be processed by the end-users and how much meaning they can derive from the data.

The POCT must also be careful not to generate too many false positives and alarm fatigue. This happens regularly in the hospital dulling clinicians response time and may make them less able to distinguish signal from noise. It can only be assumed that this effect will be multiplied as POCT reaches out into the broader community. The effect on the “worried well” is likely to generate more healthcare transactional costs as false positives drive more follow on activity. The effect on the “unworried ill” is also problematic. In this group it is not the false positives that are of concern but rather elevating the signal generated from the test to the point that it can help support behavior change.

The data also must be accurate and reliable and must minimize ambiguity.

Recommendations: It is recommended that POCT begin the process of setting interoperability standards and standards of reliability and accuracy. It is recommended that the POCT community devote resources towards developing communication and educational standards for clinicians and patient end-users.

HEALTH INFORMATION SYSTEMS

One of the challenges raised by the groups was inter-connectivity. In this context we are referring to the ability of POCT to connect with the various Electronic Health and Medical Records. Without adequate integration, the information generated by POCT runs the risk of creating more information noise in the clinical environment, disrupting workflow and essentially becoming unused and unusable. This is a significant risk as demonstrated by the work of Dr J Goldman’s Medical Device Plug-and-Play project (www.cimit.org/programs-mdplugandplay.html) which has tried to address the lack of communication between vendors and EHRs in the ICU and ORs by endeavoring to create information standards.

Recommendations: It is recommended that the POCT begin the process of setting interoperability standards and standards of reliability and accuracy.

CONNECTIVITY

Connectivity is a challenge related to the HIS challenge. Here however the emphasis is on the challenges surrounding integrating a synchronous telemedicine type application that requires a relatively high bandwidth relative the information being transferred and asynchronous systems where there is a delay between test and result. The latter by separating test and result in time may negatively affect the ability to act and instigate behavior changes at teaching and clinically actionable moments.

Recommendations: It is recommended that the POCT community develop standards around connectivity and requirements for information transfer for different clinical applications.

WORKFLOW, IMPLEMENTATION AND ADOPTION

One of the greatest challenges to POCT will be in the area of workflow. Where and when are clinicians going to fit yet another process into their clinical workflow. Unless this problem is solved the chances of large scale adoption are small. Most clinics are tightly coupled systems, meaning most activities are tightly scheduled with very little spare capacity. What happens as a result is that disruption in any one spot in the system can create a multitude of negative downstream effects. Therefore adoption of a new technology or process in a clinical setting involves trade-offs and competing priorities. Therefore POCT will need to either increase capacity to do work without simultaneously increasing demand or the increased activity has to have a sufficiently valued pay-off to make the extra work worthwhile. In addition to not creating additional work the new technology will need to provide feedback in a timely fashion so we can get the right resources to the right person at the right time.

From the patients perspective, POCT will have to provide feedback and engagement in the clinical setting in conjunction with the clinician and/or in the flow of their daily activities. Finally with the advent of patient driven activity the POCT will need to be able to provide feedback to informal and more formal networks in the community such as peer support groups.

Recommendations: It is recommended that the POCT community set up resources to evaluate new systems in the context in which they are implemented - before, during and after implementation

MAINTENANCE/REPAIR/OVERHEAD

As many of these new POCT assets will be distributed in resource constrained environments it is not only the per unit cost that is important. The further from the supply chain the POCT is used the higher maintenance, repair and overhead costs will rise. The reverse calculation is likely to come into play in the use of POCT in hospital settings, i.e., unit cost, may dominate purchasing decisions.

Recommendations: Along with evaluating workflow and design, it recommended that as part of any implementation that the functions of maintenance and repair be reported and integral to the plan.

QUALITY CONTROL

Quality control is a major concern for the POCT community. Currently, tests used in clinical environments in the United States are regulated by the Clinical Laboratory Improvement Amendment (CLIA). The Centers for Medicare & Medicaid Services (CMS) regulates all laboratory testing (except research) performed on humans in the U.S. through CLIA. In total, CLIA covers more than 200,000 laboratory entities. The Division of Laboratory Services, within the Survey and Certification Group, under the Center for Clinical Standards and Quality (CCSQ) has the responsibility for implementing the CLIA Program. POCT falls under CLIA.

As POCT is implemented further away from immediate oversight the risks of poorly made tests increases, i.e., poor quality control, collection errors, test variability, etc. However, some of the main uses of POCT are on the frontlines in primary care, rural care and in the home. This is compounded by off label use and patient driven demand and technologies. Unfortunately, our current regulatory mechanism must struggle to keep up with the avalanche of new technologies. A solution will require both increased regulatory resourcing and regulatory reform to adapt to the changing landscape.

Recommendations: It is recommended that the regulatory bodies develop multiple tracks to examine and regulate new POCT technologies. These tracks should be based on a risk assessment and stratification of the technologies.

Recommendation: It is recommended that the regulatory bodies develop surveillance and outreach programs to help facilitate and guide quality control for new and developing technologies.

Summary

In summary, as part of the development of POCT it is critical to include early on the resources to develop the technologies’ usability, ergonomics, and to pay attention to the technologies behavioral and cultural effects on the individual and society. All of which will be essential in any implementation process. With the increasing capability of POCT, increased focus also must be brought to privacy and confidentiality concerns that these technologies raise in parallel to their development.

The POCT community should regularly survey the landscape of end-users and their needs and provide a mechanism for engaging community driven technology. The POCT community should also begin the process of setting interoperability standards and standards of reliability and accuracy and connectivity and information transfer for different clinical applications. The community should also devote resources towards developing communication and educational standards for clinicians and patient end-users.

In addition it is critical that stakeholders, including government, insurance entities, healthcare systems, technologists and investors set up resources to evaluate new systems in the context in which they are implemented - before, during and after implementation. Along with evaluating workflow and design, the functions of maintenance and repair need to be reported and integral to the plan. Finally it is recommended that the regulatory bodies develop multiple tracks to examine and regulate new and emerging POCT technologies. These tracks should be based on a risk assessment and stratification of the technologies. Along with this, regulatory bodies should develop surveillance and outreach programs to help facilitate and guide quality control for new and developing technologies.

Table 3.

Examples of POC Tests

Pregnancy test	Heart monitoring
Blood pressure monitoring	Flu test (Influenza A&B)
Weighing yourself	Mononucleosis (Mono)
Urine testing	Cholesterol Screen (Lipid Panel)
Glucose testing	Quick Strep
HIV testing	STD testing
Drug testing	Stress testing
DNA testing	Vitamin level testing
Sleep apnea	Food/water testing