Abstract
Deviation from protocolized assessment times is commonplace in pragmatic randomized clinical trials. Working with a stakeholder advisory board for a Patient-Centered Outcomes Research Institute®-funded project on statistical methods for handling potential biases introduced by irregular assessment times, we identified reasons for off-schedule or missed assessments. We used the Consolidated Framework for Implementation Research 2.0 to organize our findings. We conjectured that timely completion of outcome assessments is a function of multiple determinants, only some related to participants’ health status. We identified potential determinants that can be modified during the protocol design stage and can be reassessed and mitigated during trial implementation stage. Research to more formally evaluate our findings is warranted as well as studies to evaluate multi-level strategies that reduce off-schedule or missed assessments.
Keyword: Implementation science, Determinants of off-schedule or missed assessments, Stakeholder perspectives
Main text
Despite pre-specification of outcome assessment times in pragmatic randomized trials, participant deviation from these times is commonplace. A key analytic concern is whether there are factors linked to the outcomes of interest that are driving these deviations. Off-schedule or completely missed assessments may be related to participants’ health status—such as poor health or feeling well enough to engage in other activities (e.g., work, school, caregiving). Differences between the health status at the observed assessment times and the health status at the pre-specified assessment times can result in overestimation or underestimation of the harms or benefits of study interventions [1].
The Patient-Centered Outcomes Research Institute® (PCORI®), a nonprofit organization established by the Affordable Care Act in 2010 in the United States, funded our group to develop and disseminate a statistical methodology to account for irregular (i.e., off-schedule or missed) assessment times in pragmatic randomized clinical trials. PCORI® values pragmatic randomized trials because they produce practical, actional evidence that can directly impact patient care and improve health outcomes in real-word settings. Irregular assessment times in these trials present a significant challenge for trialists in estimating unbiased treatment effects.
Our project focused on trials evaluating interventions to improve care for patients with chronic diseases and limited socioeconomic resources. Members of the investigator group (authors AA and JK) conducted four pragmatic randomized clinical trials: two enrolled adults with asthma (ARC [2]—NCT02086565 and HAP2 [3]—NCT 01972308), one enrolled children presenting with uncontrolled asthma to emergency departments (CHICAGO Plan [4]—NCT02319967), and the fourth enrolled adults hospitalized for heart failure, myocardial infarction, pneumonia, COPD, or sickle cell disease (PArTNER [5]—NCT02114515). All four pragmatic trials were designed to test interventions under real-world conditions and experienced substantial missingness and irregularity of assessment times (see Table 1 and Fig. 1); in ARC and HAP2, primary outcomes were assessed in clinic or by phone, depending on the preference of the participant, and in CHICAGO PLAN and PArTNER, primary outcomes were assessed by phone.
Table 1.
Missing and out-of-window assessments, by treatment group, for four studies
Fig. 1.
Distribution of actual vs. targeted assessment times for four studies. Treatment-specific distributions of the actual time of first (red), second (blue), third (green), and fourth (purple) scheduled assessments. Shaded regions reflect assessment windows
To provide first-hand insights into the problem, our project is informed by a diverse Stakeholder Advisory Board (SAB), consisting of two individuals from affected populations (i.e., people with asthma or other conditions relevant to the study protocol), clinicians, clinical trialists, implementation scientists, a qualitative researcher, and biostatisticians. Importantly, the researchers on the SAB have diverse expertise extending beyond the type of behavioral trials highlighted above, including substantial experience with trials outside the United States. The initial SAB activities were devoted to understanding, from the stakeholders’ perspective, what might cause patients to postpone or miss a pre-specified assessment. In a meeting of the entire SAB, members generated a list of reasons for off-schedule or missed assessments. The list reflected the diverse perspectives and experiences that each member brought to the elicitation process. Authors FKB and JDS created a coding tree to sort the reasons into discrete categories. The coding tree included codes for the five key domains (outer setting, inner setting, innovativeness of the trial, individuals involved, and implementation process) in the Consolidated Framework for Implementation Research 2.0 (CFIR2.0) [6] and subcodes describing who or what causes off-schedule or missed assessments in each of those domains. CFIR2.0 was selected because it provides a systematic approach for understanding the multiple domains (determinants) of study implementation, including various features of the study (e.g., pre-specified data collection schedule).
Figure 2 (top) displays the relationship among the domains, and Fig. 2 (bottom) provides examples of key constructs within each of the CFIR2.0 domains. Our insight from this process is that the timely completion of outcome assessments is a function of multiple determinants, only some of which are related to participants’ health status. Importantly, we identified determinants of irregular assessment times that can be modified during the protocol design stage (e.g., developing study protocols that are feasible for both participants and study implementers). At the protocol implementation stage, investigators can then reassess barriers and enact flexible mitigation plans to reduce the risk of irregular assessments (e.g., collection of outcomes via home visits, rather than depending on participant travel to clinics or study sites).
Fig. 2.
(Top) CFIR 2.0 domains to consider to optimize protocol adherence in a clinical trial. (Bottom) Examples of constructs within each CFIR 2.0 domain that affect protocol adherence in a clinical trials; stars indicate modifiable factors
Our preliminary observations stem from four localized behavioral trials conducted by researchers at two universities as well as the extensive experience of our SAB. We conjecture that our observations are generalizable to other pragmatic trials, including those that are geographically diverse, non-US-based and involve pharmaceutical treatments. Research to more formally evaluate this conjecture is warranted as well as studies to evaluate multi-level (e.g., participant- and study procedure-level) strategies that reduce off-schedule or missed assessments.
Acknowledgements
The authors would like to acknowledge Jan Cotton who was a patient member of the Stakeholder Advisory Board and helped create the list of reasons for off-schedule or missed assessments.
Authors’ contributions
Frances K. Barg, Daniel O. Scharfstein, and Justin D. Smith drafted the letter, and all other authors provided critical feedback.
Funding
Research reported in this research letter was funded through a Patient-Centered Outcomes Research Institute® (PCORI®) Award (ME-2021C3-24972). The views and statements presented in this research letter are solely the responsibility of the authors and do not necessarily represent the views of the PCORI®.
Data availability
All data and materials are in the letter.
Declarations
Ethics approval and consent to participate
N/A.
Consent for publication
All authors have consented to publication.
Competing interests
The authors declare no competing interests.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Pullenayegum EM, Scharfstein DO. Randomized trials with repeatedly measured outcomes: handling irregular and potentially informative assessment times. Epidemiol Rev. 2022Dec 21;44(1):121–37. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Apter AJ, Localio AR, Morales KH, Han X, Perez L, Mullen AN, et al. Home visits for uncontrolled asthma among low-income adults with patient portal access. Journal of Allergy and Clinical Immunology. 2019Sep;144(3):846–853.e11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Apter AJ, Perez L, Han X, Ndicu G, Localio A, Park H, et al. Patient advocates for low-income adults with moderate to severe asthma: a randomized clinical trial. The Journal of Allergy and Clinical Immunology: In Practice. 2020 Nov;8(10):3466–3473.e11. [DOI] [PMC free article] [PubMed]
- 4.Krishnan JA, Martin MA, Lohff C, Mosnaim GS, Margellos-Anast H, DeLisa JA, et al. Design of a pragmatic trial in minority children presenting to the emergency department with uncontrolled asthma: the CHICAGO Plan. Contemp Clin Trials. 2017Jun;57:10–22. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Prieto-Centurion V, Basu S, Bracken N, Calhoun E, Dickens C, DiDomenico RJ, et al. Design of the patient navigator to Reduce Readmissions (PArTNER) study: a pragmatic clinical effectiveness trial. Contemporary Clinical Trials Communications. 2019Sep;15: 100420. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implementation Sci. 2022Oct 29;17(1):75. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
All data and materials are in the letter.