Adaptation of the technology readiness levels for impact assessment in implementation sciences: The TRL-IS checklist

Abstract

Background

Intervention development is a critical process in implementation research. There are key stages involved in the process to design, pilot, demonstrate and release a technology or an intervention. The Technology Readiness Level (TRL) is a globally accepted instrument for assessing the maturity of research development. However, the original levels do not fit all, and some adjustments are required for its applicability in implementation sciences.

Aims

This study aimed to gather the prior knowledge base on TRL in public and population health research; to develop a standard definition of readiness, and to adapt and validate the TRL to an implementation science context (TRL-IS).

Materials and methods

A Mixed methods approach has been followed in this study. A scoping review using the PRISMA extension (PRISMA-ScR) informed a nominal expert panel for developing a standard definition of readiness and to modify the TRL following an ontoterminology approach. Then the maturity of six practical case study examples were rated by ten researchers using the modified TRL to estimate inter-rater reliability, and a group of experts provided final content and face validity and feasibility.

This mixed methods study included 1) a scoping review to examine the current literature and develop a knowledge base, identify knowledge gaps and to clarify concepts; 2) the development of a standard definition of ‘Readiness’ and related terms; and 3) adaptation of the TRL to implementation science and development of a checklist to rate the maturity of applications.

A standard definition of readiness and related terms was produced by the core team, and an international nominal group (n = 30) was conducted to discuss and validate the definition and terms, and the location of ‘Readiness’ in the initiation and early development phases of implementation.

Following feedback from the nominal group, the development of the TRL-IS was finalised and a TRL-IS rating checklist was developed to rate the maturity of applications. The TRL-IS checklist was tested using six cases based on real world studies on implementation research.

The inter-rater reliability of the TRL-IS was evaluated by ten raters and finally six raters evaluated the content and face validity, and feasibility, of the TRL-IS checklist using the System Usability Scale (SUS).

Results

Few papers (n = 11) utilised the TRL to evaluate the readiness of readiness of health and social science implementation research. The main changes in the adaption of the TRL-IS included the removal of laboratory testing, limiting the use of “operational” environment and a clearer distinction between level 6 (pilot in a relevant environment) and 7 (demonstration in the real world prior to release). The adapted version was considered relevant by the expert panel. The TRL-IS checklist showed evidence of good inter-rater reliability (ICC = 0.90 with 95 % confident interval = 0.74–0.98, p < .001) and provides a consistent metric.

Conclusions

In spite of recommendations made by national and international research funding agencies, few health and social science implementation studies include the TRL as part of their evaluation protocol. The TRL-IS offers a high degree of conceptual clarity between scientific maturity phases or readiness levels, and good reliability among raters of varying experience. This study highlights that adoption of the TRL-IS framework in implementation sciences will bolster the scientific robustness and comparability of research maturity in this domain.

1. Introduction

Implementation research in health is a multidisciplinary field that focuses on the methods and tools needed to promote the systematic uptake of research findings and other evidence-based practices into routine healthcare and public health settings [1]. It addresses the complex interplay of factors that influence the successful integration of interventions into real-world contexts, aiming to bridge the gap between research and practice [2].

Health implementation research plays a crucial role in improving healthcare outcomes by identifying effective strategies for integrating evidence-based interventions, treatments, and policies into healthcare settings [3]. For example, the implementation of Electronic Health Records to improve patient care and healthcare management [4].

In social research, the focus of implementation is on optimising the delivery of social services and support systems [5]. For instance, implementation of community-based nutrition programs to address food insecurity and promote healthy eating habits within underserved populations to improve overall health outcomes and reduce disparities [6].

The development of applications such as interventions, tools and strategies is a key part of implementation research [7]. The development process starts with the generation of a concept, the making of a model or prototype, and the analysis of its feasibility, piloting and demonstration. These stages of development should be clearly described in any implementation project. Different approaches have been suggested to guide the development of interventions (e.g., Re-AIM Framework, Behaviour Change Wheel, Participatory Action Research) [[8], [9], [10]]. However, these frameworks do not inherently provide a measure for quantitative evaluation of intervention development processes. Additionally, they do not explicitly identify essential steps to determine if there are missing components in the intervention development process. Addressing these limitations will lead to more robust frameworks that effectively address the complexities of health and social science implementation research.

One of the earliest frameworks for the description of the key stages in the development process is the Technology Readiness Level (TRL) system. The TRL is a globally accepted tool for measuring progress and supporting development from blue sky research to full implementation in an operational or real-world setting [11]. It uses criteria to assess the current readiness level of the technology and determines if it is ready to progress to the next level. It has the advantage of a simple metric to indicate the maturity of a technology. TRL was initially developed by NASA in the 1970s as a method to assess the maturity of new technologies and components for complex systems The original seven-level TRL were redefined in 1995, and an additional two levels were added (levels 8–9) [11].

The TRL has been progressively adopted in a wide variety of fields outside of the aerospace context, such as defence, machinery, energy systems, manufacturing, machine learning, and electronics. The original scope and language has evolved to suit field-specific needs [[12], [13], [14]]. TRLs are relatively generic and need to be adapted to the specific context of research project development. In 2014 the European Commission adapted the TRL for scientific research for the first time for their Horizon 2020 Work Programme (2014–2020 programming period) and the European Regional Development Fund 2014–2020 [15,16]. In the health sector TRL has been adapted for the development of pharmaceuticals [17], medical products (e.g., treatment models, clinical service delivery strategies) [18], and information technologies in healthcare [19].

Identifying the readiness of implementation projects informs decision makers about the progress of projects towards maturity, facilitates funding decisions and resource allocation, as well as the design of risk mitigation plans. Regular assessment of the maturity of a project or program is considered best practice for program management through the use of defined processes, a knowledge-based approach and readiness standards [20].

The generic TRL has faced criticism for potentially subjective readiness interpretations and its inability to measure technology's readiness within human competencies, usability, acceptability, and applicability of the technology to enhance user experience (among others) [[11], [12], [13], [14]]. Furthermore, the growing literature and wide spread of the TRL has not been accompanied with a standard definition of Readiness and the related terms. Even when this term is defined (Table 1) there is considerable variability across research teams particularly when applied to implementation research.

Table 1.

Definitions of the term “readiness”.

Source	Readiness Definition
Oxford Languages [21]	The state of being fully prepared for something
European Commission [22]	The development or maturity of a research and its readiness for the market uptake
Nuclear Decommissioning Authority [23]	This refers to time. Specifically it means ready for operations at the present time.
US Department of Energy [24]	The extent to which a technology is suited for deployment in a real operational environment
US Government Accountability Office [20]	The readiness of new technologies or new applications of existing technologies (sometimes referred to as heritage technologies) to be incorporated into a system or program

The use of a common frame of reference and language, along with defined metrics, can facilitate dialogue across health disciplines, sectors, departments, and organisations. It also assists in identifying, tracking, and monitoring potential concerns or impacts on implementation, Previous research has identified the need for a TRL adapted to a health and social context [25].

The aims of this study were 1) to review the existing literature about the use of TRL to measure the maturity of public health applications; 2) to develop a consensus definition of readiness and the related terms useable in implementation sciences, and 3) to adapt and validate the TRL for its use in implementation research.

This study forms part of a broader project developing a Global Impact Analytics Framework (GIAF) [26].

2. Materials and methods

This mixed methods study combines a scoping review and expert knowledge using an ontoterminology approach to define readiness and the adaptation of the technology readiness level to implementation sciences, the development of the TRL-IS checklist, and the evaluation of its psychometric properties using practical real-world cases.

2.1. Phase 1 – scoping review

The scoping review was conducted to identify the current knowledge base of TRLs in implementation sciences. We followed the five stages of Arksey and O'Malley's methodological framework: the research question; identifying relevant studies, study selection, charting the data, collating, summarising and reporting the results [27]. The PRISMA extension for scoping reviews (PRISMA-ScR) methods and results checklist was followed to report the scoping review process and results [28].

The scoping review was undertaken in June 2022 using Scopus database and Google Scholar. Scopus was selected as it is a multidisciplinary database and recognised as a principal search system [29]. Google Scholar was added to include searches for relevant grey literature [30]. The research core team decided on a simple and pragmatic search strategy based on knowledge and experience within the subject area [31]. The search terms were: TITLE-ABS-KEY ({technology readiness level} AND health), with no restrictions on the publication date or type of publication. The search was limited to publications in English language, and health, medicine, and social sciences subject areas. Inclusion criteria were studies that applied technology readiness levels in a health, medicine or social science context and exclusion criteria were studies that assessed organisational, community or human readiness to implement or accept and use technology. Peer review papers, conference papers and research report were included in the selection process. Due to the substantial number of results in Google Scholar (n = 10,600), only the first six webpages, sorted by relevance, were searched for eligible publications (n = 60).

The initial title and abstract screening was performed independently by one of the authors (CW), an experienced reviewer. Single screening has been demonstrated to be a robust method for experienced reviewers [32,33] and is useful for scoping reviews aimed at establishing the knowledge base in a new area. Studies that did not meet the inclusion criteria were excluded. Full-text review with application of inclusion/exclusion criteria was applied to the selected articles. Variables were extracted from the selected articles and charted to gather characteristics of the studies including: author/s, year of publication, country, study title, study design, participant characteristics, characteristics of the application, technology readiness levels, and reliability and validity assessments. Extracted data were entered and collated in an Excel spreadsheet and descriptively analysed.

2.2. Phase 2 – adaptation of the TRL framework to implementation sciences

The ontoterminology approach followed in this study provides explicit and standardised readiness concepts and definitions together with a classification of readiness domains as part of a comprehensive classification system of impact analysis (Global Impact Analytics Framework – GIAF [26]. The TRL domains were mapped to the to the Initiation (pre-implementation), and Maturity (early implementation) phases of implementation [34].

Then the adapted Readiness Levels were reviewed and refined by an expert panel using a nominal group technique [35] to build consensus on concepts, terms and definitions and validate components of the taxonomy and glossary. The expert panel participants were selected and invited to participate based on their diverse fields of expertise, organisational affiliations, experiences and perspectives relevant to health and social implementation research. The participants were experts in implementation sciences including psychologists and psychiatrists, epidemiologists, health economists and health planners.

The nominal group work was completed in two rounds using videoconferencing. The total group across the two regions was n = 25 (European = 12 and Australian = 13) with subject matter experts (SMEs) (Research team n = 4) and a facilitator, observers (n = 4) and a notetaker.

The first round of nominal groups took place between November and December 2020. It included five sessions and provided consensus on key definitions related to Readiness as part of the process of impact analysis (Supplementary File 1). After each session, attendees were sent a copy of the meeting minutes and the revised/refined definitions to allow time for reflection and further comment.

The second round of nominal groups took place between June to July 2022, it comprised three sessions with 11 SMEs as a subset of the group that participated in the first round, with relatively equal gender representation (female n = 5). It focused on the taxonomy of impact analysis and discussed the location of readiness and the different levels of TRL within the taxonomy. It also revised the definitions of the TRL and other relevant terms.

2.3. Validation of the adapted readiness levels

A case study approach was used to test and validate the adapted TRL framework. The case studies were pragmatically selected from ongoing real-world mental health and implementation studies, encompassing studies at different stages of maturity. Permission to utilise information from these studies was obtained from the funding organisations.

The information derived from the six selected case studies was presented to raters in two rounds. The first round included four raters experienced in implementation research plus three naïve researchers (n = 7). The results of this first round were used to improve the summary of the case studies and the TRL-IS checklist.

The second round of rating included the four experienced raters and six naïve raters (n = 10). The raters independently assessed the readiness level of each case study and allocated a TRL-IS level. Intraclass correlation (ICC) analysis was used to analyse inter-rater reliability.

ICC estimates and 95 % confident intervals were calculated based on a mean-rating, absolute-agreement, 2-way mixed-effects model. The following general guideline values were used to evaluate the level of reliability:

Values less than 0.5 are indicative of poor reliability, values between 0.5 and 0.75 indicate moderate reliability, values between 0.75 and 0.9 indicate good reliability, and values greater than 0.90 indicate excellent reliability (Koo & Li, 2016, p. 161). The IBM SPSS statistical package (Version 27) was used in this analysis.

Face and content validity [36] of the TRL-IS were assessed by the 2022 expert panel. The expert panel reviewed the domains, sub-domains and definitions of the TRL-IS. In determining the relevance and representativeness of TRL-IS domains, the expert panel considered several key criteria and considerations, including:

• •The extent to which the domains aligned with the intended construct being measured;
• •The consensus among subject matter experts regarding the adequacy of each domain, ensuring that the content domain was adequately represented;

These considerations and criteria collectively contributed to the relevance and representativeness of the TRL-IS domains, ensuring their validity and applicability within the context of the assessment tool [37].

In addition, the usefulness of the TRL-IS checklist was evaluated using the System Usability Scale (SUS), a standardised 10-item scale of negatively and positively worded questions scored on a 5-point Likert scale [38]. Minor changes were made to the wording to suit the context of the TRL-IS, which has no effect on the resulting scores. For example, question 1 I think that I would like to use this system frequently was modified to I think that I would like to use the TRL-IS checklist frequently. By assigning a rating of 1–5 to indicate their level of agreement or disagreement with each question, an aggregate score can be computed. Scores were normalised as per original scoring instructions [39]. The average SUS score is 68, with scores above considered above average and scores below, below average. Six naïve raters and five SMEs completed the SUS (n = 11). The SUS can be used with very small samples and the results are reliable.

2.4. Ethics considerations

Ethics approval to conduct the nominal groups was obtained from the University Ethics Committee (Protocol: 2020/768) and participants gave written informed consent. Participants also completed a disclosure of interests declaration prior to taking part in the nominal groups. No potential conflicts of interest were raised.

3. Results

3.1. Scoping review

The Scopus database search output was 20 document results including 15 journal articles/reviews and five conference/workshop papers. These results illustrate that the exploration of technology readiness assessment is a relatively recent subject in health, medicine, and social research, with discussions only emerging at scientific conferences after 2006. Analysis of the publication years of the journal articles/reviews reinforces this observation, as only one paper was published until 2017, with twelve publications in the last five years (2018–2022) (see Fig. 1). The Google Scholar search returned five relevant publications, consisting of a review from 2016 and four articles published between 2019 and 2022. Notably, the types of publications varied between the two database searches. The Scopus results encompassed journal articles and a conference paper, while the Google Scholar results included a policy brief and a research report.

Fig. 1.

Publication trend of included papers (n = 11).

After applying the inclusion/exclusion criteria to the full text review, 11 studies were selected and included in the charting and descriptive analysis process (Fig. 1, Fig. 2). Of these, four are review articles [[40], [41], [42], [43]], four are research articles [19,[44], [45], [46]], one is a science-policy brief [47], one is a conference paper [48], and one is a research report [18]. Excluding the reviews, two studies were located in Spain, two in France, and one each in USA, Ecuador, and Guatemala.

Fig. 2.

PRISMA flowchart – scoping review.

Selected studies most frequently referenced or used the US Department of Energy version of the TRL [40,42,43], and the HORIZON 2020 version [19,45,48]. Other versions referenced or used are US Department of Health and Human Services [18], US Department of Defense [47], TRL scale for development of innovative medical devices and drugs [44], and TRL for medical machine learning [41]. One study did not reference a TRL [46].

3.2. Description of selected studies

The selected studies can be grouped into five categories:

3.2.1. Health monitoring technologies

The utilisation of Technology Readiness Levels (TRLs) in the context of health, medicine, and social science studies is particularly prominent in the development and review of health monitoring technologies. Specifically, five studies, namely Cruz et al. [40], Fattal et al. [46], Jahfari et al. [41], Lapierre et al. [42], and Liu et al. [43], have focused on this area, demonstrating an emphasis on the advancement and assessment of health monitoring technologies. Of these studies, four are comprehensive reviews that aim to enhance understanding of the current state of knowledge. Concurrently, the fifth study places specific emphasis on the usability and accessibility of health-related robotics, examining the practical applicability and integration of robotics in the healthcare and medical monitoring sphere.

3.2.2. Digital technology to improve health services

Two studies have concentrated on leveraging digital technology to enhance health services, specifically targeting mental health service planning and the creation of a semi-automated service directory [19,45]. These studies highlight the potential of digital technology in improving healthcare delivery and accessibility and also demonstrate the systematic use of Technology Readiness Levels (TRL) to develop and rigorously test the technology.

3.2.3. Medical device development

In the study by Bastogne [44], the Quality by Design (QbD) approach is applied to the risk assessment in medical device development, offering a proactive engineering approach to the development process. This approach aims to address the lack of experience with innovative materials and technologies, introducing a new Quality by Design (QbD) paradigm indexed on the technological readiness level (TRL) of the innovative product. The mapped QbD approach onto TRLs 3, 4, and 5 in medical device development offers a framework for systematic risk assessment and proactive engineering approach.

3.2.4. Social impact start-ups

The study by Boburg & Mazariegos [47] explains the process and results of the 2021 National Innovation Prize (PNI) in Guatemala, which aimed to adopt and apply Technology Readiness Level (TRL) criteria to identify science, technology, and innovation (STI) start-ups with potential for social impact. The study illustrates the use of TRL criteria for emerging technologies, demonstrating the practical application of the TRL scale in identifying STI start-ups with potential for social impact.

3.2.5. Alternative approaches to readiness scale development – research/knowledge products

Two studies developed alternative readiness levels for measuring the maturity of higher education research products and medical research knowledge products (KPs) respectively [18,48]. Of the two studies that proposed an alternative TRL, one adapted the TRL for medical research knowledge products (e.g., treatment models, clinical service delivery strategies) [18]. The Knowledge Readiness Level (KRL) assesses the maturity of “knowledge resulting from research with potential to improve individual or public health” [18] (p. 1). The KRL uses a two-step process, first the knowledge product is assigned to one of three groups of KRLs that best describe maturity; foundation 1–3, applications 4–6, or real-world context 7–9, then it is assigned to the appropriate level of maturity within that phase.

The study by Cobos et al. [48] presents the development of a comprehensive model for assessing the maturity of research projects in Higher Education Institutions (HEI) within low- and middle-income countries. The model integrates Technology Readiness Levels (TRL) with the Commercial Readiness Index, Investment Readiness Level, Systems Readiness Level, and Human Readiness Level to address the limitations of the traditional TRL approach.

The TRL level is assessed using an evaluation instrument of over 100 questions, and the scores are summed for each parameter achieved to provide an overall score out of 100. The NBR ISO 16290 approach (based on the TRL methodology), a calculator approach, a weighted approach and a percentual approach are used to provide a comprehensive analysis of the preparation level of research projects.

3.3. Definition of readiness

Only two studies provided a definition of readiness, the definitions are provided below:

• •A knowledge product's stage of development toward implementable improvements in real-world practices or processes [18].
• •The level of preparedness for the application of a new scientific knowledge for commercialisation or generalized use in the real world [19].

3.4. Technology readiness levels

Table 2 shows the TRLs of the selected studies. The TRLs ranged from 1 to 9, with the highest levels attributed to health monitoring technology [43] and social impact start-ups [47]. The lowest levels (apart from the adapted TRLs) were applied to medical device development [44] and machine learning for cardiovascular outcomes [41].

Table 2.

TRLs of selected studies.

Author	Technology	TRL used or referenced	Adapted Y/N	Psychometric or validity testing	Rating	How TRL assigned
Jahfari et al. [41]	Review of machine learning for cardiovascular outcomes (technology)	Medical machine learning	N	N	3–5	Review authors assigned TRL ratings
Bastogne [44]	Medical device development (medical)	Medical devices and drugs	N		3	Author of article/researcher assigned TRL rating based on laboratory tests
Lapierre et al. [42]	Review of health monitoring technology (falls)	US Department of Energy	N	N	Detection M = 4.54 Locating system = 8	Review authors assigned TRL ratings
Cruz et al. [40]	Review of health monitoring technology (frailty)	US Department of Energy	N	N	Detection M = 5.88 Detection and monitoring M = 5.80 Intervention = 7	Review authors assigned TRL ratings
Fattal et al. [46]	Health monitoring technology (robot)	Unknown	N	N	5–6	Authors of article/researchers assigned TRL ratings
Liu et al. [43]	Review of health monitoring technology (medical conditions)	US Department of Energy	N	N	56 % = 5–6 44 % = 7–9	Review authors assigned TRL ratings
Romero-Lopez-Alberca et al. [19]	Health service technology – semi-automated digital service directory	HORIZON 2020	N	N	7	A panel of 23 domain experts and a group of 68 end users assigned TRL ratings
Chung et al. [45]	Health service technology – self-organising map network	HORIZON 2020	N	N	7	A group of 13 domain experts in mental health systems planning and research assigned TRL ratings
Boburg & Mazariegos [47]	Social impact start-up competition winners (technology)	US Department of Defense	N	N	8–9	Applicants' self- evaluation, internal National Secretariat of Science and Technology (SENACYT) innovation team, external National judges (subject matter experts) and an international judge for each category assigned TRL ratings
Engel et al. [18]	Medical research knowledge products (medical)	Knowledge Readiness Level (KRL) Framework for Medical Research	Y	Validity and Reliability – inter-rater reliability and Delphi technique	2–7	Authors/researchers and 30 mid- to high-level research program managers from the USAMRMC assigned TRL ratings
Cobos et al. [48]	Higher Education social and technical research projects for commercialisation (unknown)	US Department of Defense + *Commercial Readiness Index *Systems Readiness Level *Investment Readiness Level *Human Preparedness Level	Y	Validity – expert panel	1–2 (depending on approach)	Project manager/s and a group of experts formed with academic researchers from higher education institutions assigned TRL ratings

3.5. Standardisation of technology readiness levels

Excluding the literature reviews, two of the seven studies described a TRL validation process (Table 2). Both studies adapted the TRL framework as mentioned above, to medical research knowledge products and to social and technical research projects based in Higher Education Institutions. One used an expert panel [48], and the other used inter-rater reliability and a Delphi panel technique [18] to validate the adapted TRL and KRL.

Almost all of the selected studies included technology (hardware or software) or a material/physical product (e.g., medical device). Cobos et al.’s [48] study applied their alternative TRL to research projects in human health and social sciences fields such as administration, education and arts, but no details about these projects were available. Only one study assessed the readiness of clinical human services knowledge products (such as treatment models, clinical service delivery strategies, conceptual and analytical tools, health indicators or quality metrics, practice guidelines, training procedures, manuals) [18]. None of the selected studies assessed the readiness of non-technology or non-clinical/non-medical related health services or social implementation research including health promotion strategies, preventive programs, educational modules, mental health screening tools, therapeutic interventions, changes to service delivery and organisation, policy or legislation change, or organisational change strategies. This review highlighted a significant gap; the need for a suitable TRL to assess readiness or maturity of health and social applications.

3.6. Summary of the prior knowledge base

Use of TRLs to assess the readiness of implementation research is a relatively new field. It is important that health or social research applications are evidence-informed and ideally should be tested and validated before being demonstrated in a real-world environment. However, the existing technology-based versions of TRLs are not fit for purpose. Very few implementation science studies include the TRL as part of their evaluation protocol, as demonstrated in this scoping review, despite its importance in implementation research and recommendations made on its use by major funding agencies such as the HORIZON Program in Europe. With substantial global investment in health and social research, especially during the recent pandemic, understanding readiness and maturity is crucial. This understanding aids funding decisions and helps address priority gaps in health and social research.

3.7. Phase 2 – definition and adaptation of TRL to the implementation research context

The final definition of “Readiness” agreed with the expert panel is: The level of preparedness of an application of the emerging scientific knowledge to be used in the real world and for its’ release, marketing, commercialisation or open access (see Supplementary File 1).

Following the first round of expert consultations, readiness is considered a bridging entity between the two first phases of the process of implementation. Pre-readiness comprises the levels of development happening during the initiation phase (levels 1 to 6), and Readiness includes the three levels happening in the real-world environment as the first steps of early implementation or maturity (levels 7 to 9).

In the second round of expert consultations, the domains and subdomains of Readiness were discussed, revised and refined by the expert panel until consensus was reached. This included a broader definition of “application” to include policies, plans, strategies, and other public applications apart from clinical interventions. Two words used in product development were removed from this version of TRL: “laboratory” and “operational”. Laboratory has a different meaning in implementation sciences than in technology and product development; and the word “operational” does not add meaning to the use of “real world” in implementation sciences. A clearer differentiation between “proof of concept” and “prototype”, and between “pilot” and “demonstration” applicable to implementation sciences was added to the TRL-IS.

In addition, the TRL was adapted to the implementation phases, and a better definition of the context of the different levels, their functions and descriptions were adapted to implementation research (Fig. 3). TRL bridges the initiation and maturity phases in implementation sciences in the Global Impact Analytics Framework (GIAF).

Fig. 3.

Map of the technology readiness levels in implementation sciences (TRL-IS) according to their grouping and implementation phases (adapted from [13]).

As with TRLs for technology and products, the TRL-IS broadly progresses from the prior knowledge base to pilot/feasibility testing, demonstration and release to full implementation. Although the TRL-IS is presented in a linear fashion, implementation research can frequently be an iterative process, as feedback is received from stakeholders and applications are refined and tested [18]. Table 3 illustrates the TRL-IS levels with a brief description of each step.

Table 3.

TRL-IS description of levels.

Level		Description
1	Basic Principles	This level relates to the generation of new concepts or new uses for existing applications. It includes the formulation of the idea and its preliminary potential relevance for implementation. Activities are performed to create the knowledge base underpinning the application. There is verifiable information of the compilation and synthesis of existing knowledge prior to the implementation: previous quantitative and qualitative research (discovery and corroboration) and/or context and/or expert and experiential knowledge.
2	Concept Formulated	The concept is formulated. The foundational knowledge is peer reviewed, critiqued and revised as part of the scientific process for external validation. Preferably it should include a logic map. In level 2 the concept is formulated through either: - submission for an alternative peer review and a validation process (e.g., nominal group, or expert panel has critiqued it and agreed). - OR it has been submitted or published in a peer reviewed journal.
3	Proof of concept	Proof of concept is part of the early development of an application involving the design of a conceptual model that confirms the principles of the application and its workability, as the basis for subsequent activities (e.g., prototyping, piloting, demonstration). This early stage of development includes methods to validate the potential use of the application such as the prior Knowledge Base, experts' feedback, exploratory surveys, previous laboratory/experimental studies, secondary analysis or relevant databases, modelling, simulation. Activities at this level include: - co-create with key experts and stakeholders to examine the prior knowledge base and use it to produce the prototype and to identify weaknesses and risks. - refining the proof of concept and the components to develop the preliminary prototype.
4	Prototype	A working model or preliminary version of the application (concept, process, product) has been developed and tested in an experimental or simulated environment. The final prototype of the application is ready for the next step. This level may also include the completion of a manual for use. Activities at this level include identifying weaknesses and risks of the final prototype. Changes of the prototype may be incorporated but are not substantial.
5	Validation of Prototype	This level relates to validating the prototype in a relevant environment. At this level, the focus is on initial or preliminary demonstration and evidence of standardisation. Examples include the feasibility of the application with the target audience. Activities at this level may include tests to provide evidence the application may perform as expected in the eventual real-world setting with the target audience.
6	Test in Relevant Environment	This level involves testing the application in a relevant operational environment closely representing the actual intended operational setting–typically this step has been referred to as the ‘pilot’ stage. Activities at this level focus on whether the application can work, for whom, how and in what context or conditions.
7	Demonstration in real world environment	At this level there has been an implementation of the application with the defined target audience in a real-world setting (i.e., in the intended settings and by the targeted people). The final application incorporates the final amendments, follows external validation (generalisation) and can be fully compared with usual practice or existing tools in the real-world.
8	Pre-Release	This level indicates all the preliminary steps have been undertaken to release and commercialise (if applicable) the final application. This is a pre-release stage to prepare for the application to be ready and available to the market. Actions and activities to make the application ready and available to the market (e.g., gaining a license, registration, steps for commercialisation) have started.
9	Release	This level is the release stage to the open market; the application has been proven under the full range of conditions and is ready to be marketed and made available to end users.

3.8. Pre-readiness

The first 6 levels of the TRL-IS closely mirror existing TRLs although the definitions for each level differ to reflect the emphasis on developing and validating proof of concept and a prototype or “beta” version of the application with subject matter experts and/or key stakeholders, then demonstrating or testing the application in contexts progressively approaching a real-world setting (Fig. 3).

Level 1 research is foundational and relates to the summary of the discovery and corroboration scientific research that generates the prior knowledge base leading to potential future application. This level relates to the generation of new uses for existing applications. Examples include literature or scoping reviews, descriptive or exploratory studies, and prior laboratory research related to discovery and corroboration.

Level 2 provides a standard formulation of the concept, application or model which is gradually refined. The concept or application is speculative at this stage, is based on previous evidence and reasoning, and points to a practical intervention, tool or strategy from the initial emerging scientific knowledge. Examples include analytical or paper studies (review) that may support the observations made in TRL-IS 1, development of a logic map or registration of a protocol.

Level 3 research validates the potential use (workability) of the application into a proof of concept model. Examples include the use of expert knowledge (for example nominal/focus groups), modelling and simulation to produce a theoretical or preliminary model of the application. Active development of a prototype is initiated and may involve co-design or co-creation with experts or stakeholders.

Level 4 is concerned with the completion of the prototype to determine if it works, and if so, how and for whom. This level may also include the performance metrics instruments for outcome measurement. Activities at this level may include identifying weaknesses and risks and refining the prototype following testing.

Level 5 includes analysis aimed at validating the prototype in a relevant environment. At this level, the main focus is on the analysis of feasibility of the application (relevance, acceptability, applicability, practicality, efficiency and value). Activities at this level may include prototype testing with a small group of people to provide evidence the application may perform as expected in the eventual real world setting with the target audience. Typically, modifications are made following testing.

Level 6 research concerns all the activities related to piloting the application in a “relevant” environment which closely represents the actual intended real world setting. Activities at this level focus on whether the application can work, for whom, how and in what context or conditions.

3.9. Readiness

Levels 7–9 research occurs in the early implementation phase and focus on demonstration of the application in a real-world environment with the intended target audience, preparation for the application to be made available for end users, and release of the application to the open market.

TRL-IS 7 involves implementation of the application with the defined target audience in a real-world setting. It facilitates the analysis of the efficiency, and it is an opportunity for external validation and generalisation of the application of the emerging scientific knowledge and comparison with usual practice in the real world.

TRL-IS 8 is reached when the demonstration is completed and consists of activities to prepare for the application to be ready and available to the market. This pre-release step involves activities such as applying for licenses, registration, patents, gaining approval from regulatory bodies, and establishing intellectual property and copyright agreements. These steps prepare the application for public use or commercialisation.

TRL 9 is the final release step; the application has been proven under the full range of conditions and is ready to be marketed and made available to end users.

Table 4 shows a comparison of existing TRLs and the adapted TRL-IS for implementation science. The final right-hand column shows completion criteria for each level of the TRL-IS.

Table 4.

Comparison of existing technology readiness levels and adapted TRL-IS for process of implementation science in health and social research.

TRL Levels	NASA [49]	Australian Defence [50]	European Union (HORIZON 2020) [22]	Pharmaceuticals [51]	Medical Informatics [51]	Adapted TRL TRL-IS	TRL-IS Completion Criteria
1	Basic principles observed and reported. Scientific knowledge generated underpinning hardware technology concepts/applications.	Basic research. Initial scientific research has been conducted. Principles are qualitatively postulated and observed. Focus is on new discovery rather than applications.	Basic principles observed.	Maintain scientific awareness; tech watch. Scientific literature reviews and market surveys initiated and assessed.	Identified potential medical solution to mission need. Defined data & knowledge representation Issues	Basic Principles: Prior scientific research has been conducted, and associations or effects have been observed and reported, and potential outcomes and implications have been identified.	Structured activities to compile the prior knowledge base (PKB) were undertaken.
2	Technology concept and/or application formulated. Invention begins, practical applications are identified, but are speculative. Neither an experimental proof nor detailed analysis is available to support the conjecture.	Applied research. Initial practical applications are identified. Potential of material or process to solve a problem, satisfy a need, or find application is confirmed.	Technology concept formulated.	Research ideas and protocols developed. Hypothesis(es) generated.	System concepts documented. Schema defined. Data and knowledge representation issues defined.	Concept formulated: Practical application is identified and the potential for the application to solve a problem or satisfy a need is confirmed.	The synthesis and formulation of the foundational knowledge base has been completed and has been/will be submitted for peer review or published. .
3	Analytical and experimental proof of concept of critical function and/or characteristic. Research and development is initiated, including analytical and laboratory studies to validate predictions regarding the technology.	Critical function, i.e., proof of concept established. Applied research advances and early stage development begins. Studies and laboratory measurements validate analytical predictions of separate elements of the technology.	Experimental proof of concept.	Hypothesis testing and initial proof-of-concept (PoC) demonstrated in limited number of in vitro and in vivo models.	Data and knowledge representation schema modeled	Proof of concept: The early stage of development which includes methods to validate the potential use of the application such as nominal/focus groups, laboratory study, modelling, simulation etc.	Proof of Concept completed. Is it workable?
4	Component and/or breadboard validation in laboratory environment. A low-fidelity system/component breadboard is built and operated to demonstrate basic functionality in a laboratory environment.	Laboratory testing/validation of alpha prototype component/process. Design, development and lab testing of components/processes. Results provide evidence that performance targets may be attainable based on projected or modeled systems.	Technology validated in the laboratory.	PoC and safety of candidate drug formulations or biologic/vaccine constructs are demonstrated in defined laboratory/animal model(s)	Prototype produced. HW/SW pieces work together. Models use real data/knowledge.	Prototype: a working model or preliminary version of the application (concept, process, product) has been developed and tested in an experimental or simulated environment.	Prototype completed.
5	Component and/or brassboard validation in relevant environment. A medium-fidelity component brassboard, with realistic support elements, is built and operated for validation in a relevant environment so as to demonstrate overall performance in critical areas.	Laboratory testing of integrated/semi-integrated system. System Component and/or process validation is achieved in a relevant environment.	Technology validated in relevant environment (industrially relevant environment in the case of key enabling technologies).	Preclinical studies, including GLP animal safety & toxicity	Models are implemented into data/knowledge system & tested in lab environment. Actual interfaces specified.	Validation of prototype: Validation of the prototype application (including feasibility) has been performed in a relevant environment, and results provide evidence the application may perform as expected within the real-world environment/context.	Pilot started: Prototype validated, feasibility tested. Is it feasible?
6	System/sub-system model or prototype demonstration in a relevant environment. A high-fidelity prototype of the system/subsystem that adequately addresses all critical scaling issues is built and tested in a relevant environment to demonstrate operational performance under critical environmental conditions.	Prototype system verified. System/process prototype demonstration in an operational environment (beta prototype system level).	Technology demonstrated in relevant environment (industrially relevant environment in the case of key enabling technologies).	Phase 1 clinical trials completed, data support proceeding to Phase 2 clinical trials.	System tested with interfaces & support systems in relevant or simulated operational environment. Configuration Management Approach Developed.	Test in relevant environment: Testing of the application in a simulated or somewhat realistic setting similar to the actual intended operational setting.	Pilot completed. Testing in a relevant operational environment completed and documented. Does it need changes?
7	System prototype demonstration in an operational environment. A high-fidelity prototype or engineering unit that adequately addresses all critical scaling issues is built and functions in the actual operational environment and platform (ground, airborne, or space).	Integrated pilot system demonstrated. System/process prototype demonstration in an operational environment (integrated pilot system level).	System prototype demonstration in operational environment.	Phase 2 clinical trials completed. Phase 3 clinical study plan approved.	System is operationally integrated and tested with target applications in operational environment with end users.	Demonstration in real world environment: Application has been demonstrated in a real world environment with the defined target audience.	Evidence the application is implemented and being used. Is it ready for general use?
8	Actual system completed and “flight qualified” through test and demonstration. The final product in its final configuration is successfully demonstrated through test and analysis for its intended operational environment and platform (ground, airborne, or space). If necessary, life testing has been completed.	System incorporated in commercial design. Actual system/process completed and qualified through test and demonstration- (pre-commercial demonstration).	System complete and qualified.	Phase 3 clinical trials completed. D- Approval of New Drug Application	Development Test & Evaluation (DT&E) of the hardware/software system in its intended environment. Demonstrated it meets design specifications. Validated in several operational environments.	Pre-release: Actions and activities to make the application ready and available to the market (e.g., gaining a license, registration, steps for commercialisation) have been undertaken.	Pre-release prepared - preliminary steps started to prepare for release for public use. Has the approval process started?
9	Actual system “flight proven” through successful mission operations. The final product is successfully operated in an actual mission.	System proven and ready for full commercial deployment. Actual system proven through successful operations in operating environment, and ready for full commercial deployment.	Actual system proven in operational environment (competitive manufacturing in the case of key enabling technologies; or in space).	Post marketing studies/surveillance. Post-marketing studies may be required.	Product successfully used as part of Initial Operational Test and Evaluation (IOT&E). Logistical Demonstration successfully conducted.	Release: Application is ready to be made available to end users (e.g., license or patent is approved).	Release completed. Application is released to other organisations/commercialised. Is it approved for commercialisation? Has implementation and surveillance started?

3.10. Validation of the TRL-IS

Based on the literature review and the 2022 expert panel discussion, a checklist of TRL-IS was developed and tested in a series of practical cases by two groups of raters (first group n = 7, second group n = 10) according to their experience in implementation research.

3.11. Reliability and validity

An intra-class correlation analysis was conducted to test inter-coder reliability, which showed moderate reliability, ICC = 0.70 with 95 % confident interval = 0.18–0.95, p = .010. However, based on the 95 % CI, the level of reliability should be regarded as poor.

Following the second round of rating, the intra-class correlation analysis showed good reliability, ICC = 0.90 with 95 % confident interval = 0.74–0.98, p < .001 (reliability moderate to excellent).

In terms of face validity of the TRL-IS, the 2022 expert panel confirmed the TRL-IS domains and definitions were clear and unambiguous. The expert panel assessed content validity by reviewing the relevance and representativeness of the TRL-IS domains, sub-domains and definitions, and determined they are relevant, reasonable and representative of the content being measured.

3.12. Feasibility

The Systems Usability Scale total mean score of all raters (n = 11) was 68, placing the perceptions of usability of the TRL-IS at the 50th percentile (average). Overall, 55 % (6/11) of all raters scored the usability of the TRL-IS as above average. According to rating groups, 80 % (4/5) of experienced raters perceived the usability of the TRL-IS as above average, whilst 33 % (2/6) of naïve raters scored the usability above average. The lowest scored SUS questions related to learnability (Q7), and confidence in use (Q9).

4. Discussion

To the best of our knowledge, this is the first study to adapt the TRL to implementation sciences, using real world case studies to test reliability, validity and feasibility. The TRL-IS showed evidence of good inter-rater reliability and consistency across raters with different levels of expertise utilising this instrument. Face and content validity was confirmed by expert panel review. TRL-IS offers a high degree of conceptual clarity between scientific maturity or readiness levels in a health and social context.

4.1. Need for an adapted TRL for implementation science

Previous research also identified a need for and adapted the TRL for assessing the readiness of medical devices, pharmaceuticals [51], clinical practices and processes [18], and digital tools [40]. Our earlier work indicates that these adapted TRL are complex to use and require further adaptation for practical use in implementation research [19,40,45].

Increased clarity has been addressed in the adapted TRL-IS which includes a detailed description and multiple research-type specific examples at each level. The generic TRL has been criticised for simplifying the research development process to nine steps [12]. Héder argued that the TRL scale's concreteness and sophistication decreased as its use expanded beyond its original context, primarily space programs. Customising the levels to specific disciplines is crucial before application [52].

4.2. Incorporation of stakeholders and end-users in TRL-IS

Implementation research occurs in complex environments [53,54]. The effectiveness of interventions can be affected by how they are implemented in specific contexts, cultures, and settings [55]. Furthermore, the way the intervention is perceived by those delivering and receiving it can also impact its effectiveness [56].

Taking this into account, the TRL-IS incorporates stakeholders and end-users to the concept, design, adaptation, planning and conduct of implementation research. For example, TRL-IS level 3 (proof of concept) encourages co-design with experts and end-users (engagement) to develop a model that will fit their needs and is workable in their context. Apart from the main levels, the TRL-IS hierarchical taxonomy also include sub-domains and child categories. TRL-IS 5 and 6 involve piloting the application, including validation and testing in operational environments with small user samples, using feedback loops to make modifications, revisions, and refinements. For example, feasibility is a key component of the validation at Level 5.

4.3. Roadmap to plan and assess implementation research

The TRL-IS provides a roadmap to guide researchers and monitor the incremental stages of research development and validation. It can be used either retrospectively or prospectively for planning or to assess the level of an application's development or maturity using a standardised framework. It allows decision-makers to monitor progress and evaluate investment and resource allocation [13]. Confirmation that an application has undergone testing, revision, and demonstration to be ‘ready’ for real-world use provides credible information for organisations, decision makers, and policymakers to make evidence-informed decisions about implementation, integration, and adoption.

5. Strengths and limitations

The TRL-IS is specifically designed to address the unique challenges and considerations present in implementation research, making it more relevant and applicable to these fields. Unlike the traditional TRL framework, the TRL-IS incorporates social considerations (co-design, pilot testing with end users) that are paramount in health and social science research, ensuring that interventions are not only technically ready but also socially acceptable. Adaptation of the TRL to implementation sciences promotes standardisation and communication between different stakeholders, including researchers, policymakers and industry professionals. This collaboration facilitates knowledge sharing and fosters the development and implementation of advanced health and social applications. By adapting and adopting a TRL framework, the implementation research field can ensure thorough testing and validation of applications before implementation. This approach reduces risk and improves end user safety, satisfaction and outcomes.

Although the TRL-IS reached a good level of inter-rater reliability, the SUS ratings indicate that orientation and training prior to using the TRL-IS checklist is likely to improve consistency between raters and perceptions of usability. The naïve raters had little or no experience in implementation research which may have impacted their ratings of usability (learnability and low confidence in use). Ideally, orientation would include framework familiarisation – a comprehensive overview of the TRL-IS framework, its components, and the specific criteria for assessing application readiness in health and social care settings. Training ensures that raters have a clear understanding of the assessment criteria, including the specific parameters for each readiness level, to minimise subjectivity and ensure consistent evaluations.

The scoping review was limited by only using two databases and some relevant papers may have been excluded. Although scoping reviews use a systematic approach to identify, screen and chart studies, not all the features of a systematic review are applied. For example, quality and risk of bias appraisal was not conducted in this review, and we recognise that the included papers may be of variable quality [57].

Expert panels, while valuable for providing insights and recommendations, are susceptible to various biases that can influence their decision-making and recommendations. To mitigate these risks of bias, we established clear guidelines for decision-making, encouraged diverse perspectives, and promoted open discussion. We also aimed for diversity among background and gender to increase the generalisability of the findings. Actively addressing biases can help to minimise their impact, but biases may never be completely excluded.

Evaluating readiness in the initiation and maturity phases of implementation research typically requires consensus among evaluators [13,58,59]. Discussions to reach consensus about maturity can be ambiguous or arbitrary without a framework to provide a standard language and understanding [13]. The TRL framework has been criticised for its subjectivity and potentially conflicting perspectives or bias of those conducting the assessment [13,60]. While the TRL-IS is open to a degree of subjectivity, the detailed descriptions and examples at each level may overcome this challenge given the high level of reliability across raters.

A number of alternative TRL versions have been suggested in the health sector. However, the feasibility and usability have not been tested prior to this study. In any case, our findings should be corroborated by different groups and in different contexts. Next steps should include cultural validation, exploring the cross-cultural applicability and validity of the TRL-IS framework to ensure its effectiveness across diverse cultural and social contexts. Additionally, the next steps in validating or refining the TRL-IS framework may involve empirical validation, expert consensus building, pilot testing, and standardisation.

6. Conclusions

Complex system and technology development requires a common frame of reference and a common language to communicate maturity, progress, and risks. The TRL has gained attention as a reference framework to assess the level of development of any application in science by major national and international funding organisations. However, the adaptations needed to be effectively used in implementation research remains a matter of debate. This study presents a practical solution based on its application to real world cases and expert knowledge, and it incorporates an analysis of its feasibility. The use of TRL in different areas of health (e.g., development of pharmaceuticals, medical devices, public health) may require further harmonisation in the coming future.

Funding disclosure

C.M. received funding in the form of a pre-doctoral grant from the Generalitat de Catalunya (PIF-Salut grant, code SLT017/20/000138).

Ethics declaration

• This study was reviewed and approved by University of Canberra Human Research Ethics Committee, with the approval number: [Protocol: 2020/768].

• All participants provided informed consent to participate in the study.

CRediT authorship contribution statement

Luis Salvador-Carulla: Writing – review & editing, Writing – original draft, Resources, Methodology, Investigation, Conceptualization. Cindy Woods: Writing – review & editing, Writing – original draft, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Carlota de Miquel: Writing – review & editing, Writing – original draft, Validation, Methodology, Data curation. Sue Lukersmith: Writing – review & editing, Writing – original draft, Validation, Supervision, Project administration, Methodology, Investigation, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

List of abbreviations

CI

Confidence interval

GIAF

Global Impact Analytics Framework

IBM

International Business Machines Corporation

ICC

Intraclass correlation

ISO

International Organization for Standardization

KP

Knowledge product

KRL

Knowledge Readiness Level

SPSS

Statistical Package for the Social Sciences

SME

Subject matter expert

SUS

System Usability Scale

TRL

Technology Readiness Level

TRL-IS

Technology Readiness Level – Implementation Science

UK

United Kingdom

US

United States of America

Appendix A. Supplementary data

The following is the Supplementary data to this article: