Abstract
Most entrepreneurial ventures fail long before the core technology can be brought to the marketplace because of disconnects in performance and usability measures such as accuracy, cost, complexity, assay stability, and time requirements between technology developers’ specifications and needs of the end-users. By going through a clinical needs assessment (CNA) process, developers will gain vital information and a clear focus that will help minimize the risks associated with the development of new technologies available for use within the health care system.
This article summarizes best practices of the principal investigators of the National Institute of Biomedical Imaging and Bioengineering point-of-care (POC) centers within the National Institute of Biomedical Imaging and Bioengineering POC Technologies Research Network. Clinical needs assessments are particularly important for product development areas that do not sufficiently benefit from traditional market research, such as grant-funded research and development, new product lines using cutting-edge technologies developed in start-up companies, and products developed through product development partnerships for low-resource settings. The objectives of this article were to (1) highlight the importance of CNAs for development of POC devices, (2) discuss methods applied by POC Technologies Research Network for assessing clinical needs, and (3) provide a road map for future CNAs.
Keywords: user needs assessment, brain trauma, sexually transmitted infections, disaster medicine, global health
Most diagnostic product development builds on existing platforms or creates new, usually improved versions of existing products. It also usually aims to add analytical targets and improve assay performance and utility. This approach is appropriate for applications in which the desired assay specifications and use cases are fairly well established. The development of product specifications for such products is usually relatively straightforward and can be achieved through conventional market research activities.
The situation is frequently different in academic, nonprofit, or commercial start-up diagnostics research and development (R&D) settings. Here, the R&D goals tend to be more audacious—the development of entirely new platforms, assays for new analytes, or the transfer of an assay from laboratory settings to point-of-care (POC) or home-care settings. It is frequently necessary to develop and clinically validate a new device at the same time as identifying potential end-users, formulating reimbursement approaches, educating the medical and diagnostic establishment about new and better testing modalities, and developing use algorithms. At the same time, the latter R&D settings usually lack the expertise, the resources, and, to some extent, the tradition to perform proper market research, for which also little guidance or formal requirements exist from grant funders, professional associations, and regulators. Therefore, one of the overlooked ways to improve both performance and acceptance in clinical practice is to create better market and product requirement documentation at the outset of diagnostic product development through a process called clinical or user needs assessment. This article represents the consensus opinion on the subject of POC diagnostics clinical needs assessments (CNAs) of a group of diagnostics development centers, the Point-of-Care Technologies Research Network (POCTRN).
The POCTRN is funded by the National Institute of Biomedical Imaging and Bioengineering (NIBIB), which is part of the National Institutes of Health. It is composed of 4 centers: The Point-of-Care Center for Emerging Neurotechnologies at the University of Cincinnati focuses on diagnosing neurological emergencies such as stroke and head injury at the point of need; the Center for Point-of-Care Technologies Research for Sexually Transmitted Diseases at Johns Hopkins University develops POC tests for sexually transmitted diseases; the Center for Point-of-Care Technologies for Disaster Readiness at the University of California, Davis develops novel POC diagnostics for critical emergency-disaster preparedness; and the Center to Advance POC Diagnostics for Global Health, led by the Program for Appropriate Technology in Health in Seattle, Wash, focuses on diagnostics for low-resource settings, specifically developing countries.
The POCTRN is focused on all aspects of the development of POC diagnostic assays. Individual centers develop novel diagnostic devices that will help eliminate health disparities by making predictive, preemptive, preventative, and personalized care accessible to all communities worldwide. The network advances diagnostic development for high-risk diseases to be used in clinical settings with undefined or risky market potential. The POCTRN is a translational group that helps commercial companies during the risky product development stages finalize and market diagnostic devices for unmet clinical needs.
This article was developed specifically to provide guidance to technology developers and other stakeholders who currently pursue their R&D activities with limited input from formalized CNA methods. Most entrepreneurial ventures fail long before the core technology can be brought to the marketplace because of a disconnect between technology specifications and clinical needs. By performing a CNA, developers of POC tests gain vital information and a clear focus that minimizes the risks associated with the development of new technologies for health care. The objectives of this article were (a) to highlight the importance of CNAs for development of POC devices, (b) to discuss methods applied by POCTRN for assessing clinical needs, and (c) to provide a road map for future CNAs. Figure 1 illustrates the ultimate goal of diagnostics product development, which is to arrive at an optimal intersection between technical feasibility and the needs of users within a sustainable business context.
FIGURE 1.
The sweet spot: the goal of CNA-guided diagnostics development is to identify a set of product specifications and features that result in a device that fulfills an important user need, is technically feasible, and can be realized in a format that is commercially sustainable in the target market.
Clinical Needs Assessment
Clinical needs assessment is a process by which information is gathered regarding the scope and potential impact of gaps or deficiencies in the current delivery and practice of health care. The processes and techniques used to gather and analyze information regarding clinical needs encompass a diverse set of qualitative and quantitative techniques.
Clinical Needs and POC Diagnostics
Point-of-care diagnostics refers to testing that is performed at or near the site of the patient care with the result leading to possible improvement in the care of the patient.1,2 Most diagnostic product development in academic and start-up settings focuses on technical improvements. This allocation of resources is appropriate for applications where the desired product requirements are fairly well established. However, in POC diagnostics development, it is frequently necessary to develop and clinically validate a new device at the same time as identifying potential end-users; determining reimbursement approaches; educating the medical and diagnostic establishment about new, faster, and better testing modalities; and developing use algorithms. Therefore, one way to improve both performance and acceptance in clinical practice is to create better market and product requirements documentation at the initiation of diagnostic product development through a CNA.
Figure 2 shows an appropriate development process for R&D-heavy products such as POC diagnostics. Needs assessment and concept development are followed by feasibility demonstration, prototype R&D, transfer to production, then introduction, and activities designed to sustain the success of the product in the marketplace. Each step is iterative, and product managers need to ensure that the findings of the initial CNA are revisited every time the product concept changes or is refined.
FIGURE 2.
Clinical needs assessment–guided product development: CNAs should be performed before commencement of R&D activities and periodically reiterated.
MATERIALS AND METHODS
Clinical needs assessment activities may include any of the following:
The process of gathering and critically evaluating information that identifies and characterizes an existing gap in health care.
The process of identifying methods or technologies that have the potential to bridge identified gaps in health care.
The process of identifying and characterizing the requirements and desires of stakeholders (providers, patients, payers, regulators, public health agencies, etc) in the delivery of health care.
The process of identifying barriers to current uptake to POC technology using transdisciplinary approaches including all stakeholders in the process.3,4
The purpose of needs assessments is 2-fold:
To provide background information on a problem and its context so that technology developers can make decisions on product design and development of POC diagnostic tools.
To guide development of appropriate commercialization and marketing strategies needed for eventual product introduction.
Researchers in the POCTRN used needs assessments to guide diagnostic product development for a variety of applications (see Table 1 for a comparative overview). Clinical needs assessments for products for these diverse types of assays are similar in concept and definition but vary in scope and approach. Assessment of needs focuses resources for better utilization, organization, and planning.5,6 Similarly, the POCTRN identified and resolved technological gaps in clinical diagnostics by assessing stakeholders’ needs for high-impact critical missions. Clinical needs assessments strive to provide guidance in the areas of market requirements, product requirements, product cost, user desires, and product specifications. In commercial product development, these areas are commonly represented by specific forms called the Market Requirement Document, Product Requirement Document, and Product Specifications that have to be completed, updated, and agreed on by various stakeholders at various stages of the development process.
TABLE 1.
Comparative Overview of CNAs Performed by the POCTRN Principal Investigators
| POC Center for Emerging Neurotechnologies |
Center for POC Tests for STDs |
UC Davis POC Technologies Center |
GHDx Center | |
|---|---|---|---|---|
| Purpose/research question | To inform and prioritize the current and future technology development activities for POC devices that address unmet clinical needs in acute neurological care. |
|
To identify strategies with tactics that enable POC testing (medical testing at or near the site of care) to effectively improve outcomes in emergencies, disasters, and public health crises, especially where community infrastructure is compromised. | To define the need for and value of multivalent diagnostic platforms for health care providers in managing febrile illness in pediatric patients within the constraints of a resource-limited setting. |
| Stakeholders/participants | Clinicians and other health care providers who provide acute neurological care including emergency medical technicians, emergency department personnel, neurointensivists, and neurosurgeons. | Government, industry, the research community, and health care providers. Participants were respondents from a survey addressed to a list of all categorical STI clinics (N = 700 clinics) throughout the United States. | Professional POC technology developers and users, academicians, researchers, emergency and disaster care experts, public health, urban search and rescue teams, and other first responders. | Health care system: Partnering with the Research Institute for Tropical Medicine, Philippines. Health care facilities spanning the entire breadth of the health system. |
| Assessment methods used | A primary needs assessment was done through survey mechanisms and direct interview. Results from surveys and interviews were compared with mortality/morbidity statistics gathered from the CDC and WHO. Finally, project priorities were compared with funding activities inferred from priorities in recent solicitations and analysis for topic areas for recently awarded grants as reported through grants.gov. | Based on findings from focus groups that identified the need for, and perception of, qualities imperative for an ideal new STI POC test. The survey contained 1) demographics, including sex, country, profession, and primary practice; 2) currently available POC tests and their barriers to use; 3) future ideal POC tests for STIs, including priority pathogens for development and economic factors; and 4) build-your-own POC test—preference of POC test for STIs with different levels of sensitivity, specificity, turnaround time, and cost. The last part of the survey used the discrete choice experiment approach on the SurveyMonkey platform. | Multinational needs assessment conducted using several forms of media including focus groups, SurveyMonkey, direct interviews, and various forms of electronic communications, driven by logic model-critical path-feedback with visual logistics and trade-off survey questions. | Exploratory mixed-methods approach that incorporated qualitative and quantitative components with the goal of capturing descriptive detail and the range of views of all respondents. Quantitative data were entered into the CDC Epi Info database. χ2 tests of significance and t tests were used to explore bivariate relationships. To explore relationships among multiple variables, we used multivariate techniques such as analysis of variance between groups, correspondence analysis, principal components analysis, and cluster analysis. For qualitative data management, we used AtlasTI as our primary database. |
| Results/conclusions | Established a prioritization for the development of technologies that focus on detection of seizures and intracranial hemorrhage as diagnostic and prognostic indicators in acute neurological emergencies such as, but not limited to, hemorrhagic and ischemic stroke, cerebral contusions, and nonconvulsive seizures. | Chlamydia trachomatis was identified as the priority for development of a new STI POC test, followed by a herpes simplex virus POC test. Themes indicated for the ideal POC test included rapid turnaround (preferred >5 min but acceptable up to 20 min), ease-of-use, noninvasive, accurate (preferred sensitivity and specificity in the range of high 90s), Clinical Laboratory Improvement Amendments–waived, user-friendly (for both patients and staff), compact, durable, and sturdy.2 | Needs assessments helped establish that careful implementation of POC testing will facilitate evidence-based screening, triage, diagnosis, treatment, and follow-up monitoring of victims and patients while advancing performance in emergency and disaster medicine. | Developed understanding of the existing practices in the management of febrile illness in pediatric patients, including which existing diagnostic tools are used at various levels of the health system. Identified and characterized the desired and valued product attributes. |
CDC indicates Centers for Disease Control and Prevention; GHDx Center, Center to Advance POC Diagnostics for Global Health; WHO, World Health Organization.
Market Versus Need
Developers analyze market potential and determine economic viability to achieve high-impact implementation. However, clinical need is not synonymous with market demand. Proper CNA takes an all-encompassing approach to determine stakeholder needs, social implications, economic viability, and technical feasibility.
Importance of Stakeholders
Clinical needs assessment is a transdisciplinary process requiring input from multiple stakeholders with different priorities. Health care providers, patients, industry, government, clinic and hospital administrators, first responders, and all others who benefit from the solution to the clinical need are stakeholders. Ultimately, stakeholders determine the utility of biomedical devices. Emerging technologies should be founded on stakeholder needs to ensure rapid implementation and the greatest clinical impact. Ignoring stakeholder needs can result in failure to implement innovative ideas.7
Starting Premise/Research Question
The first step in a needs assessment is to start with a clear premise or a targeted question to focus the research and to organize the various factors relevant to the topic of interest.8 The premise should be hypothesis driven and based on literature review, some preliminary research, and personal experience in the field. After defining the premise, research questions for the CNA should be prioritized and distilled into a limited number of clear and concise objectives. Logic models, discussed later, can assist with framing questions as well as stating hypotheses and determining implementation and evaluation steps. The research questions chosen will drive the choice of methodology. The formulation of the research questions must also be weighed against available research resources including time, research capacity, and access to the right set of respondents.9
RESULTS
Examples of CNAs Performed by POCTRN Researchers
Results are described in Table 1.
DISCUSSION
A number of different models of CNAs exist, and by nature, these models need to be flexible and iterative. A CNA can be open-ended, without presupposing a particular technical solution, or can be an inquiry into specific attributes of a more general technology.1 A group wishing to develop a CNA should articulate a strategic focus based on group mission, strengths, weaknesses, and acceptance criteria.10
Developing Logical Frameworks for CNAs
Steps to developing a CNA may start with the development of a logic model that can be iteratively used to define and monitor success and to effectively communicate within the research group. It can help formulate and focus all the inputs, methods, and expected outputs for a needs assessment. A logic model, for example, can describe and define the flow of information from the inputs from focus groups that then can inform a quantitative survey to investigate the needs of clinicians for a simple POC diagnostic tool (Fig. 3).
FIGURE 3.
Logical framework for CNAs: logical frameworks can help clarify inputs, outputs, and outcomes required from a CNA and can be used as a method to develop and refine user needs assessments. The example here describes the logic model for a CNA for point-of-care tests for STIs for use in community clinics and other care settings.
Quantitative Versus Qualitative Approaches
As previously mentioned, the research questions drive the choice of methodology. Depending on the type of research question, one can pursue either a qualitative, quantitative, or a hybrid approach. There are many similarities between qualitative and quantitative methods. Both gather information in the form of numbers, text, or pictures. Data collection methods may be similar, and the primary data source could be individual respondents in focus groups or individual surveys. Quantitative methods are useful for generating numerical findings that are amenable to statistical manipulations that allow the researcher to arrive at generalizations, predictions, and estimations of events and causal explanations. Qualitative methodology, on the other hand, is more useful for understanding context, interpreting or explaining numbers and causal events, and considering the experiences or perspectives of the study respondents. Both methods are valid depending on the research questions to be answered. Data gathered with both approaches can complement each other and can result in insights that would be difficult to achieve with one approach alone.2,10
Observational studies, literature reviews, and focus groups are examples of qualitative tools.1,2,11–13 Focus groups should be performed in person, in small groups of interested stakeholders. Results from such studies are useful in formulating larger, more quantitative surveys. An example from a focus group survey analysis for POC tests for sexually transmitted infections (STIs) provided information as to what was important to clinicians, the potential users of a POC test.1 Themes raised as barriers for current POC tests included complexity, time to result, number of procedural steps, training requirement, difficulty in reading results, interruption of workflow, unreliability, and invasiveness.
Written or electronic surveys such as SurveyMonkey (surveymonkey.com), which measure and analyze quantitative questions, can also be valuable in gaining insight into the needs of the users of POC assays.2
A Warfare Analysis Laboratory Exercise, a type of quantitative tool, involves assembling a varied group of stakeholders, such as clinician-users (nurses, physician assistants, and health care workers), patients for whom tests would be used, test developers from industry, biomedical engineers, and regulatory officials, into one central conference center.14 The meeting room is equipped with a structured moderator, individual computers, and recording equipment. A survey using “choice questions” may require the potential buyer/user to choose between several desired qualities of a prospective product offering.15 Analysis of such answers when entered into a “design of experiments” statistical package can help manufacturers and retailers identify important product attributes and assign dollar values to them.16 Such choice modeling, a type of conditional logistic regression, results in a method that can estimate the probability of individuals making a particular choice from presented alternatives.
Respondents
Respondents need to be credible and knowledgeable on the research topic. Developing eligibility criteria is important to ensure that the data gathered will be valid and that recruitment of study participants will be consistent.
The appropriate sampling plan is dictated by the choice of methodology and the types of downstream data analysis conducted. Other factors that may influence the sampling plan include site selection, access to the relevant respondents, and the total potential area of use of a diagnostic. Casting a wide net is important. For example, to attempt to ensure universality, 1 survey for opinions about POCs for STIs started with a list of more than 700 clinics that included every categorical STI clinic in the United States as well as participants at 2 international meetings for STIs and women’s reproductive health.2
Training
Many researchers rely on interviewers to perform data collection. It is critical that interviewers are trained sufficiently to ensure data validity. If the research includes field work, real-time monitoring of data collection by such means as direct observation or double data entry could be used to address issues that come up in the field. In addition, field interviewers must have a clear protocol for management of data to facilitate smooth logistics and ensure respondent confidentiality and data integrity.
Data Analysis
Several methods for qualitative and quantitative data analysis are used depending on research goals and methods used in the needs assessment study.17,18 Data analysis and study design go hand in hand, and both should be considered before a CNA is undertaken. Regardless of the model one chooses, it is important to write a clear analysis plan at the beginning of the study to ensure that the interview guide or survey tool being developed will generate the kind of data consistent with planned downstream analysis methods.
For example, questions focusing on device design and critical pathogen targets can be formatted and analyzed as follows: (a) multiple-choice questions can be analyzed using the χ2 for goodness-of-fit test, (b) ranking questions can be analyzed using analysis of variance and Tukey multiple comparison test, and (c) trade-off questions can be analyzed using multivariate logistic regression to generate adjusted odds ratios and confidence intervals.4,19–21
CONCLUSIONS
Policymakers, administrators, and technology developers all benefit from a better understanding of the needs of various stakeholders. Many tools and methods that aid in the development and conduct of CNAs are available. Selecting the most appropriate tool and CNA approach depends on the type of diagnostic device, the extent of a potential market, existing knowledge about a market and/or a class of diagnostics, and the degree of innovation present in the diagnostic device or its application. Diagnostics developers, especially those outside established diagnostics R&D enterprises, should seriously consider performing iterative CNAs at the beginning and throughout the development process. In addition, there should be a greater focus on policy requirements and education about CNAs among grant funders, investors, professional associations, regulators, and academics.
Acknowledgments
Financial disclosure: The authors received no specific funding for this article. Bernhard H. Weigl, Gerald Kost, Charlotte A. Gaydos, and Fred R. Beyette, Jr, gratefully acknowledge center funding from the National Institute of Biomedical Imaging and Bioengineering (grant nos. U54EB007958, U54-EB007954, U54-EB007959, and U54–EB007949) under the Point-of-Care Technologies Research Network.
Footnotes
The authors declare no conflict of interest.
Author contributions: Bernhard H. Weigl, Gerald Kost, Charlotte A. Gaydos, Fred R. Beyette, Jr, Stephanie Sabourin, and John Haller contributed equally to the writing and editing of the article. Anne Rompalo, Tala de los Santos, and Jason T. McMullan contributed to the section entitled, “Examples of CNAs performed by POCTRN researchers.”
REFERENCES
- 1.Hsieh YH, Hogan MT, Barnes M, et al. Perceptions of an ideal point-of-care test for sexually transmitted infections—a qualitative study of focus group discussions with medical providers. PLoS One. 2010;5(11):e14144. doi: 10.1371/journal.pone.0014144. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Hsieh YH, Gaydos CA, Hogan MT, et al. What qualities are most important to making a point of care test desirable for clinicians and others offering sexually transmitted infection testing? PLoS One. 2011;6(4):e19263. doi: 10.1371/journal.pone.0019263. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Heron MP, Hoyert DL, Murphy SL, et al. Deaths: final data for 2006. Natl Vital Stat Rep. 2009;57(14):1–60. [PubMed] [Google Scholar]
- 4.Hoyert DL, Heron MP, Murphy SL, et al. Deaths: final data for 2003. Natl Vital Stat Rep. 2006;54(13):1–120. [PubMed] [Google Scholar]
- 5.Heron MP. Deaths: leading causes for 2004. Natl Vital Stat Rep. 2007;56(5):1–95. [PubMed] [Google Scholar]
- 6.Mathers C, Boerma T, Ma Fat D. Global Burden of Disease: 2004 Update. Geneva, Switzerland: World Health Organization; 2008. [Google Scholar]
- 7.DeLorenzo RJ, Waterhouse EJ, Towne AR, et al. Persistent nonconvulsive status epilepticus after the control of convulsive status epilepticus. Epilepsia. 2007;39(8):833–840. doi: 10.1111/j.1528-1157.1998.tb01177.x. [DOI] [PubMed] [Google Scholar]
- 8.Ling MM, Ricks C, Lea P. Multiplexing molecular diagnostics and immunoassays using emerging microarray technologies. Expert Rev Mol Diagn. 2007;7(1):87–98. doi: 10.1586/14737159.7.1.87. [DOI] [PubMed] [Google Scholar]
- 9.Procop GW. Molecular diagnostics for the detection and characterization of microbial pathogens. Clin Infect Dis. 2007;45(suppl 2):S99–S111. doi: 10.1086/519259. [DOI] [PubMed] [Google Scholar]
- 10.Peeling RW, Mabey D. Point-of-care tests for diagnosing infections in the developing world. Clin Microbiol Infect. 2010;16(8):1062–1069. doi: 10.1111/j.1469-0691.2010.03279.x. [DOI] [PubMed] [Google Scholar]
- 11.Espinoza H, Sequeira M, Domingo G, et al. Management of the HIV epidemic in Nicaragua: the need to improve information systems and access to affordable diagnostics. Bull World Health Organ. 2011;89(8):619–620. doi: 10.2471/BLT.11.086124. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Gerlach J, Sequeira M, Alvarado V, et al. Cost analysis of centralized viral load testing for antiretroviral therapy monitoring in Nicaragua, a low-HIV prevalence, low-resource setting. J Int AIDS Soc. 2010;13:43. doi: 10.1186/1758-2652-13-43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Kuzel AJ. Sampling in qualitative inquiry. In: Crabtree BF, Miller WL, editors. Doing Qualitative Research. Newbury Park, CA: Sage Publications; 1992. pp. 33–46. [Google Scholar]
- 14.Nolen JM. The WALEX process. Johns Hopkins APL Technical Digest. 2000;21(2):225–230. [Google Scholar]
- 15.Farrar S, Ryan M, Ross D, et al. Using discrete choice modelling in priority setting: an application to clinical service developments. Soc Sci Med. 2000;50(1):63–75. doi: 10.1016/s0277-9536(99)00268-3. [DOI] [PubMed] [Google Scholar]
- 16.Lancsar E, Donaldson C. Discrete choice experiments in health economics: distinguishing between the method and its application. Eur J Health Econ. 2005;6(4):314–316. doi: 10.1007/s10198-005-0304-3. [DOI] [PubMed] [Google Scholar]
- 17.Brock TK, Mecozzi DM, Sumner S, et al. Evidence-based point-of-care tests and device designs for disaster preparedness. Am J Disaster Med. 2010;5(5):285–294. [PMC free article] [PubMed] [Google Scholar]
- 18.Kost GJ, Hale KN, Brock TK, et al. Point-of-care testing for disasters: needs assessment, strategic planning, and future design. Clin Lab Med. 2009;29(3):583–605. doi: 10.1016/j.cll.2009.07.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Mecozzi DM, Brock TK, Tran NK, et al. Evidence-based point-of-care device design for emergency and disaster care. Point Care. 2010;9(2):65–69. doi: 10.1097/POC.0b013e3181d9d47a. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Hale KN, Brock TK, Mecozzi D, et al. Point-of-care testing for disaster and pandemic preparedness. In: Price CP, St John A, Kricka LJ, editors. Point-of-Care Testing: Needs, Opportunity, and Innovation. 3rd ed. Washington, DC: American Association for Clinical Chemistry Press; 2010. pp. 355–371. [Google Scholar]
- 21.Kost GJ, Sakaguchi A, Curtis C, et al. Enhancing crisis standards of care using innovative point-of-care testing. Am J Disaster Med. 2011;6(6):351–368. doi: 10.5055/ajdm.2011.0074. [DOI] [PMC free article] [PubMed] [Google Scholar]