E-Health Interventions as Complex Interventions: Improving the Quality of Methods of Assessment

Abstract

E-health interventions require rigorous evaluation, preferably within a randomized trial. Health technology assessment and complex intervention methodologies exist, but are often not used in health informatics research. We propose a framework of methodologic issues which should be considered for every e-health intervention.

Background

Electronic health records (EHR) and computerized decision supports systems (CDSS) are excellent examples of the evaluation problems in health informatics, where 2 major problems persist after years of research. First, the interventions are often enormously complex, meaning multifaceted, multidisciplinary, involving multiple technologies, systems and partners. Second, the typical evaluation methods used are not sufficiently robust to adequately address the key evaluation domains of validity (did it really work?), generalizability (can it work elsewhere?) and cost-effectiveness (is it worth the cost?).

Health technology assessment (HTA) is increasingly embracing the additional challenges of complex interventions, although no CONSORT methods guidance statement exists yet on this topic. Our objective is to encourage health informatics to embrace more rigorous evaluation using current principles of HTA and complex intervention research¹ and to suggest a methodologic framework that will assist in e-health clinical intervention evaluation.

Proposed Evaluation Framework

Traditional levels of evidence should hold sway for e-health interventions as for any clinical intervention or health technology assessment. Principles of a) randomization, b) blinding, c) adequate follow-up and d) monitoring of co-intervention, are key to validity. Each has challenges, notably related to cluster versus individual randomization, inability to blind clinicians and patients from the intervention, and the difficulties of follow-up and co-intervention given the necessary “real world” environment of the studies. Generalizability is also a challenge where interventions tend to be system- and site-specific thus necessitating that key technical, clinical and health system details are described. Cost-effectiveness analysis is recommended as part of any important e-health intervention trial since the simple demonstration of effectiveness is not sufficient to decide whether the intervention should become part of (funded) routine practice. In addition, it is recommended that these complex interventions be developed and evaluated iteratively and in stages – from assurance based on systematic review or cogent theory that the intervention could work, to identifying the important components of the intervention, to development of a feasible “best design” protocol, to carrying out a rigorous randomized trial, to promoting effective implementation of the intervention in practice. These additional considerations mandate that qualitative research supplements the quantitative and that process outcomes accompany clinical outcomes.