Table 2.

Definitions of the research phases adapted from the MRC guideline for complex interventions [16] used to classify studies

Research Phase	Definition
Development phase	Studies in the development phase are those that investigate intervention design-related outcomes (satisfaction, feasibility, usability) before the intervention has reached a deployable state of development. Also included are theoretical and modeling studies or reports limited to describing the technology or user interactions with it.
Piloting phase	Studies in the piloting phase are those that investigate intervention design-related outcomes when it is a question of refining the intervention after it has reached a relatively complete stage of development. User-related outcomes (behavior change, resource use, clinical outcomes, quality of life) are often measured in the same study. Feasibility and pilot studies that feature user-related measures are differentiated from full-scale evaluations (below) if their outcomes are less important (eg, process outcomes), sample size is small, or a less rigorous study design is used. Some studies reported the adaptation of an existing technology (eg, video-conferencing for telemedicine) for a particular disease, using a case study format where patient outcomes are described. Although these studies do not involve a program of development, they were categorized as feasibility and piloting studies because they report user-related outcomes.
Evaluation phase	Studies in the evaluation phase are those that evaluate important user-related outcomes that use one of the more rigorous available study design options and have a large sample size.
Implementation phase	Studies in the implementation phase are those that evaluate user-related outcomes for an intervention that is well established (eg, in use for more than 2 years) or for which a full-scale evaluation has been published. As many implementation efforts are not reported, it was expected that this phase would have low representation.