Methods for assessment of innovative medical technologies during early stages of development

Abstract

Conventional Health Technology Assessment (HTA) is usually conducted at a point in time at which the development of the respective technology may no longer be influenced. By this time developers and/or purchasers may have misinvested resources. Thus the demand for Technology Assessment (TA) which incorporates appropriate methods during early development stages of a technology becomes apparent. Against this health political background, the present report describes methods for a development-accompanying assessment of innovative medical technologies. Furthermore, international research programmes set out to identify or apply such methods will be outlined. A systematic literature search as well as an extensive manual literature search are carried out in order to obtain literature and information. The greatest units of the identified methods consist of assessment concepts, decision support methods, modelling approaches and methods focusing on users and their knowledge. Additionally, several general-purpose concepts have been identified. The identified research programmes INNO-HTA and MATCH (Multidisciplinary-Assessment-of-Technology-Centre-for-Healthcare) are to be seen as pilot projects which so far have not been able to generate final results. MATCH focuses almost entirely on the incorporation of the user-perspective regarding the development of non-pharmaceutical technologies, whereas INNO-HTA is basically concerned with the identification and possible advancement of methods for the early, socially-oriented technology assessment. Most references offer only very vague descriptions of the respective method and the application of greatly differing methods seldom exceeds the character of a pilot implementation. A standardisation much less an institutionalisation of development-accompanying assessment cannot be recognized. It must be noted that there is no singular method with which development-accompanying assessment should be carried out. Instead, a technology and evaluation specific method selection seem to be necessary as medical innovations are diverse and none of the methods are exhaustive. Because of a variety of issues (e. g. ideal time of evaluation, lack of data and uncertainty of data) a development-accompanying assessment should not replace a comprehensive HTA, but rather form a possible preceding step in a multi-staged HTA-process. A final appraisal of the methods for development-accompanying assessment cannot be made based on the available sources. However, the present review may serve as a starting point for further development and application of these methods as well as further examination of the concept of development-accompanying assessment. There is a substantial need for further research concerning the application, validation and comparison of the various methods for development-accompanying assessment.

Executive Summary

1. Health political background

The conventional methods spectrum of Health Technology Assessment (HTA) focuses on a medical and economic evaluation and is predominantly based on data yielded by clinical studies and possibly additional observations from routine use. It is usually applied at a point in time when the development of the respective technology has advanced to such an extent that it can no longer be directly influenced. Even in early stages of development existing technology-related information (which is usually not published) is not utilised. The demand for a development-accompanying assessment during early stages of development, that is to say during the phase of development and/or preparation for the diffusion of a technology, is becoming more obvious. It could thus, at a very early state, yield important information on potentially relevant developments to purchasers, developers and patients. This in turn could allow for the promotion of the development of promising technologies on the one hand or the modification or even cessation of the development of disadvantageous technologies on the other hand. From a methodological point of view development-accompanying technology assessment constitutes a challenge because ‘hard data’ (e. g. clinical data obtained from randomized controlled studies (RCT) of comparable technologies) must be combined with developmental data and must then, via modelling tools, be turned into prognoses of utility, necessity, economic viability etc. Development-accompanying assessment would not replace a comprehensive HTA. Instead it would be the first step of a multi-staged HTA-process, in which unpromising technologies are eliminated or changed early so better technologies reach the market. The objective of the present report is to identify and describe applied methods for the development-accompanying assessment of technologies as well as to outline international programmes for the identification or application of these methods and to discuss their significance for the German context.

2. Scientific background

As defined by the Office for Technology Assessment (OTA) of the USA, health care technologies include drugs, devices, medical and surgical procedures as well as organisational and supportive systems in which such care is provided. New technologies are increasingly submitted to HTA. The objective is to support decisions in policy making and practice based on the best possible evidence. Thus, HTA results are used for the approval of new technologies, for example, or their inclusion in the benefits catalogues of health insurance funds, for decisions on further application of already established technologies, for the coordination of biomedical research activities and for resource allocation (e. g. investments made by hospitals). The combination of limited resources (funding and staffing) available for the production of HTA-reports and the steadily increasing number of technologies pushing onto the market presents a challenge regarding prioritization, meaning the decision for which one of the technologies that are ready for an assessment to be evaluated next. For example, through so called Horizon Scanning (HS), a greater formalization of the selection process is intended, making use of data which inform about the socio-economic relevance of the subject-matter and allow conclusions about the impact of a potential HTA-report. Furthermore, against the background of limited financial resources, the mass of new technologies pushing onto the market and the demand for timely evidence-based information as a basis for health political decisions, rapid HTA has emerged, which accounts for the tight time frame of political decisions, for instance. In this way scientifically funded decisions can be made in a timely manner, albeit concerning a very tightly focused issue, while otherwise decisions would have to be made based on lower evidence levels or delayed by the time consuming completion of a comprehensive HTA-report. Like HS, rapid-HTA does not primarily aim at adjusting technology development. Both sensibly complement conventional HTA, however they are no solution to the problem of being able to assess innovative technology in an early stage of the development, i. e. clearly before the diffusion process. Development-accompanying HTA on the other hand is aimed at appraising aspects like safety, efficacy, demand and cost effectiveness of a new development as soon as possible, in order to have a guiding function even during development as opposed to conventional reactive approaches.

3. Research questions

In the present report drafted on behalf of the German Agency for Health Technology Assessment (Deutsche Agentur für Health Technology Assessment (DAHTA)) of the German Institute for Medical Documentation and Information (Deutsches Institut für Medizinische Dokumentation und Information (DIMDI) the research questions are as follows:

• Which methods exist for processing partial questions of HTA in the early development stage of an innovative health-related technology?

• How are these methods appraised according to the identified references?

• Which research programmes exist for the identification, application or further development of the respective methods?

4. Methods

In order to obtain literature and information on the subject a systematic literature search is conducted by DAHTA in 29 data bases during April and May 2007 in accordance with current requirements and in agreement with the authors regarding the search strategy. The time frame of the search begins in 2002 and the search is limited to the languages of German and English. The selection of the literature identified by the systematic literature search is accomplished in three steps based on predefined inclusion and exclusion criteria. In a first review only the titles of the references are analysed, and in a second review the abstracts are analysed, while lastly the full texts are considered. In order to obtain further basic information, it is necessary to conduct an extensive manual search in addition to the usual systematic literature search. Relevant institutions in the countries to be investigated as well as respective helpful documents are identified via the internet (internet presences and the respective results are noted in the appendix). In doing so, the criteria of the manual search partially deviate from the DIMDI-search (e. g. the publication date). In an initial search several data bases are investigated (the search terms and respective hits are noted in the appendix). In a second search the internet pages of various HTA-organisations are investigated for projects or programmes as well as documents concerning development-accompanying assessment of medical technologies. Additionally an internet search is carried out using the keywords ‘constructive’ and ‘formative’ (with a high count of hits, the results are limited using the additions ‘assessment’, ‘technology assessment’, ‘health technology assessment’ or ‘evaluation’). The documents identified in the literature search have to be partially excluded from the results, but can be used for introductory parts or the discussion. Where necessary for comprehension, the references delineated in the results are preceded by a brief introduction into the respective method, given such information is available. This is done because the references identified in the literature search are often methodologically intricate or incomplete, but are still supposed to be presented in respect to the intended broad overview.

5. Results

The greatest units of the identified approaches consist of methods for decision support (e. g. the aid of experts), modelling approaches and methods that focus on users and their knowledge. Besides these methods several concepts for development-accompanying assessment are found (Constructive Technology Assessment (CTA)), Iterative Economic Evaluation and evaluation frameworks for information technologies (e. g. Clinical, Human and Organizational, Educational, Administrative, Technical and Societal evaluation framework (CHEATS)). Within these concepts a whole spectrum of methods can be applied rather than a single method. CTA is primarily utilised in the Netherlands to assess non-pharmaceutical technologies (early), whereas CHEATS is deployed for the evaluation of information and communication technologies in health care. The concept of iterative economic evaluation is to accompany the technology development from the early developmental phase up to the introduction into practice. Among the methods for decision support are Analytic Hierarchy Process (AHP), Stated-Preference (SP)-methods, expert systems and Fuzzy-Logic. AHP and SP-methods make an assessment possible by comparing alternatives. Expert systems are knowledge-based or computer systems intended to make professionally funded decisions (comparable to those made by human experts). The necessary knowledge for the problem solving is acquired from actual professionals. Fuzzy-Logic also imitates human decision making, but it has the particular ability of developing precise solutions from imprecise data. Modelling methods are mostly based upon sophisticated mathematics (e. g. Bayesian methods). An important position in this group is filled by Decision Analytic models (e. g. Markov-models), which are already partially applied in conventional HTA, e. g. for assessing economic aspects. Pharmacocinetic and pharmacodynamic (PK/PD) modelling naturally only refers to pharmaceuticals. User Centered Design (UCD) offers the greatest variety of methods. It bases the development process on information about the user and should be started as early on in the development process as possible. Ideally this approach is already applied in the concept stage, where the idea underlying the product is formulated. The category of ‘other methods’ includes methods which cannot be classed with one of the four large groups (failure and reliability analysis, real-options analysis (ROA), pre-protocol research and tracker-trials). Failure mode and effects analysis (FMEA) identifies failures and contributes to the improvement of the respective system. ROA coming from the financial world is primarily employed to appraise the advantages from an economical point of view. Pre-Protocol-Research is used to incorporate early data from lower evidence levels while Tracker-Trials try to equip elements of RCT with an adaptability in regard to ongoing change of the considered technology. Both of the identified research programmes INNO-HTA and MATCH (Multidisciplinary-Assessment-of-Technology-Centre-for-Healthcare) have to be considered as pilot projects. MATCH is almost entirely focused on the integration of the user perspective into the development of non-pharmaceutical technologies, whereas INNO-HTA is basically concerned with the identification and further development of methods for early, socially-oriented technology assessment.

6. Discussion

The present report is to be considered as a descriptive overview of methods for development-accompanying assessment of innovative medical technologies. The information content of the sources utilisable for the present report is predominantly meagre. Nevertheless, in order to enable a first overview of potentially applicable methods, the existing sources are rather generously included as better sources are lacking.

Development-accompanying assessment of innovative technologies struggles with three inherent and interrelated difficulties: the ideal timing for an evaluation, the availability of sufficient data and the uncertainty regarding the use of this data. An early assessment yields the advantage of still being able to influence the development of a technology in a simple and inexpensive manner. The compiled data however are to be considered only as preliminary or indicative, as they are subject to a rather great uncertainty and may even become irrelevant due to subsequent changes in the technology. In contrast, the data of a later assessment are associated with greater certainty; influencing the technology on the other hand may be far more difficult and expensive or actually impossible. In this way an assessment may be in vain, if it is carried out too early. However, most authors agree thatan assessment should be conducted early. If data do exist, they may be difficult to obtain, may become obsolete soon and may be subject to an increased probability of error, which cannot be completely eliminated by any methodological approach. If decisions are made based on these data there is a risk of recommending technologies that subsequently prove to be disadvantageous or rejecting technologies which finally prove to be advantageous. Development-accompanying assessment could possibly have an adverse effect on innovation, if another hurdle is erected besides market approval and inclusion in benefits catalogues, thus prolonging and impeding the process of developing an innovation all the way up to actual use.

As the utilisation of innovations is influenced by several factors, it seems to be important, to assess technologies before ethical restraints ensue, the respective technology substantially alters the cost structure or before it has considerable effects on organizational structures inside the health care system. Furthermore early assessments are expected to yield cost advantages. The cited assessment concepts (e. g. CTA) lastly remain very non-committal; even so they depict a framework on which aspects can be examined and how to do so. They do not however point out new methods, but they employ the techniques illustrated in the present report, which are also already applied in other fields of research. The differences between the methods of decision support concern the generation of knowledge and know-how. Expert systems for example leave decisions to heuristics and computers, established by experts. With AHP on the other hand, decision processes are also broken down. The knowledge however comes from experts like potential users. Modelling techniques are already partially used in conventional HTA, but in development-accompanying assessment different sources are used for the generation of data, e. g. interviewing experts. Methods for ascertaining the user perspective seem to be a useful means of estimating user needs, desires and demand. As is the case with FMEA and ROA it is still debatable to what extent these methods can be employed in a socially-oriented assessment. Pre-Protocol-Research and Tracker-Trials do not greatly differ from conventional HTA; up to what extent they will be able to atone for randomized controlled trials (RCT) at least regarding the development of non-pharmaceutical technologies has not yet been resolved. Regarding the methods for development-accompanying assessment the criteria to be evaluated are not standardised and depend on the individual case. Also there is no general consensus about the aspects which should be considered in a development-accompanying assessment. Many of the cited references do not describe the respective method in a detailed and comprehensible manner, but often stay very imprecise. Predominantly it remains unclear which data is used for the assessment. Occasionally it is only mentioned which type of technology a method is tested on, while a description of how this is done is left out. Lastly even an evaluation and validation on whether the development-accompanying assessment correctly predicted the future (e. g. whether the diffusion and the assumed success actually occurred) was also always missing. Analysing the results of the literature search, it has to be noted that many methods are primarily used by the manufacturers and the assessment is not institutionalized (e. g. CA and methods for ascertaining the user perspective). Because of this, social aspects are seldom considered. Although there are already two research programmes (MATCH and INNO-HTA) considering methods for the development-accompanying assessment of innovative technologies, these activities are still a long way from reaching the extent of collaboration and standardisation which have by now developed in conventional HTA. MATCH is a strictly national programme, while universities and research facilities from different countries collaborate for INNO-HTA.

7. Conclusions

In spite of a broad literature search, both the number of hits as well as the content of information in the identified references are insufficient for a final appraisal of the methods; nevertheless the present overview may serve as a starting point for further development and application of these methods as well as a further examination of the concept of development-accompanying assessment. Both of the identified research programmes (MATCH and INNO-HTA) are to be considered as pilot projects in the sector of development-accompanying assessment. Concluding publications are not yet available during the editing of the present report. A standardisation of development-accompanying technology assessment is not perceivable as of now. It has to be noted that a development-accompanying assessment will, due to a variety of problems (e. g. ideal timing of evaluation, lack of data and uncertain data), mostly not replace a comprehensive HTA, but rather form a preceding step in a multi-staged HTA-process.

There is no singular technique for the completion of an appraisal. Instead, a technology and evaluation specific selection of methods seems to be necessary as the innovations to be assessed are very diverse and none of the methods are exhaustive. Heretofore development-accompanying technology assessment lacks an institutionalization; such would be best in an existing Horizon-Scanning-programme where appropriate professional knowledge is already present as would be the opportunity of conducting an early assessment in sequence to identifying a promising technology. The implementation of development-accompanying assessment methods requires multidisciplinary teams since many methods are mathematically complex and a multitude of aspects needs consideration. In contrast to many other countries, Germany does not yet have a Horizon-Scanning-program, hence an addition to the existing HTA-programs of the Institute for Quality and Cost Efficiency in Health Care (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen ((IQWiG)) and DIMDI seems to be most feasible. The authors recognize a strong need for further research concerning application, validation and comparison of the different methods for development-accompanying assessment.