Comparing and using prominent social accountability frameworks in medical education: moving from theory to implementation in Northern Ontario, Canada

Abstract

Background

Social accountability in medical education is conceptualized as a responsibility to respond to the needs of local populations and demonstrate impact of these activities. The objective of this study was to rigorously examine and compare social accountability theories, models, and frameworks to identify a theory-informed structure to understand and evaluate the impacts of medical education in Northern Ontario.

Methods

Using a narrative review methodology, prominent social accountability theories, models, and frameworks were identified. The research team extracted important constructs and relationships from the selected frameworks. The Theory Comparison and Selection Tool was used to compare the frameworks for fit and relevance.

Results

Eleven theories, models, and frameworks were identified for in-depth analysis and comparison. Two realist frameworks that considered community relationships in medical education and social accountability in health services received the highest scores. Frameworks focused on learning health systems, evaluating institutional social accountability, and implementing evidence-based practices also scored highly.

Conclusion

We used a systematic theory selection process to describe and compare social accountability constructs and frameworks to inform the development of a social accountability impact framework for the Northern Ontario School of Medicine. The research team examined important constructs, relationships, and outcomes, to select a framework that fits the aims of a specific project. Additional engagement will help determine how to combine, adapt, and implement framework components to use in a Northern Ontario framework.

Introduction

In the 1990’s, the World Health Organization (WHO) recommended that medical education aspire to be socially accountable.¹ Social accountability is touted as a way for medical education to improve the health of local communities, give voice to communities and health system partners, produce health system change agents, and make systems of inequity visible.^2–4 As education transforms to incorporate more person and community-oriented perspectives,³ medical schools and the graduates they produce are faced with ethical, political, and pragmatic pressures to be socially accountable.⁵ Over the past two decades, medical schools and academic health centres have begun to incorporate social accountability into their institutional missions.^2–4,6 Institutionalizing social accountability is considered to be the most effective, sustainable way to affect change in this area. through better governance and policy design, amplifying voices of marginalized groups, and responding to local communities’ needs policy should improve.⁷ Medical education accreditation bodies are promoting social accountability by requiring schools to report impacts on the populations and systems that they serve, although metrics must be contextualized by school and jurisdiction.⁸ As a result, the literature is replete with social accountability frameworks that researchers and schools use to guide, monitor, and evaluate their social accountability efforts.^2,8–14

, This collection of frameworks present inconsistent theoretical or conceptual underpinnings, which makes it difficult to understand how and why medical schools are responding to community and other stakeholder health needs.^3,15 The lack of clarity and coherence around what activities and outcomes constitute “social accountability” means that schools and organizations might invest resources and energy into activities that merely appear socially accountable. A lack of conceptual consistency is compounded by an emphasis on adapting frameworks and activities to local needs. While “contextualizing” is important and inherent to the responsiveness of socially accountable schools, it is challenging for educators, researchers, administrators, and health professionals to identify the most appropriate actions to take or impacts to assess without clear, specific guidance.

A lack of conceptual and terminological clarity and a disinclination for critical reflexivity ^16,17 likely contribute to mixed evidence of the population impacts of social accountability interventions,^18,19 particularly in medical education.¹² While academics, administrators, funders, communities—among others—demand more transparent demonstrations of impact,^8,20,21 there is little guidance on how to critically select, adapt, and implement social accountability frameworks and its concepts. A need for simple guidance to compare and select concepts, frameworks, and models for studying and implementing research in practice has previously been identified,^22,23 although not in a medical education setting. The objective of this study was to rigorously examine and compare prominent social accountability models and frameworks to identify key conceptual foundations to understand and evaluate the impacts of medical education.

The results of this work will inform the development of a social accountability impact framework that aims to help stakeholders understand and report the progress toward social accountability by a health or academic institution. Our team will apply this framework in a case study for the Northern Ontario School of Medicine (NOSM). NOSM pioneered the social accountability movement among Canadian medical schools by committing to respond to the needs of Northern Ontario residents in its institutional mandate. Almost 20 years into its mandate, the medical school aims to rigorously assess its progress toward its socially accountable goals, which should be guided by an appropriate framework. This framework must engage with the unique and dynamic social, cultural, clinical characteristics of the region it serves while aligning with national and international directions for social accountability (e.g., accreditation).

Methods

Eligibility and selection of frameworks for appraisal

The research team followed a narrative critical review approach²⁴ using a non-comprehensive search to identify the most significant theories, models, or frameworks (TMFs) that could be used to evaluate the downstream impacts of medical education (e.g., “social accountability”). Specifically, the goal of identifying frameworks that could be adapted or otherwise inform the development of a NOSM-specific social accountability framework. First, we examined a list of TMFs and associated references identified in a narrative review by two members of the research team.²⁵ The team also used results of an internal environmental scan about medical education research units that included a scan of Canadian Faculty of Medicine websites to identify social accountability frameworks used by other medical school as well as engagement with local health and education stakeholders. Once this list was populated, the team conducted a search in Google Scholar to identify existing published reviews that examined social accountability evaluation frameworks, using the keywords of social accountability, medical education, health systems, evaluation, and review. The search was limited to English-language articles published during the previous 10 years (1 January 2009 to March 5, 2020) to narrow the results. Reference lists of returned reviews were examined to identify potential TMFs that had not been captured in earlier searches.

Articles were reviewed in full-text if they described the initial development or application of a social accountability TMF; TMFs were considered eligible for comparison if they 1) aligned with national and international definitions of social accountability in medical education (“standardization”), 2) was developed or pilot-tested in a population, setting, or system similar to NOSM (“contextualization”), and 3) identified health system outcomes and impacts of system-level interventions. Two members of the research team met to determine the applicability of the references to the NOSM context and select the final TMFs for appraisal.

Data extraction, description, and analysis

After a full-text review of all included articles, one researcher extracted the origins of the TMF and the authors’ description of social accountability. The key TMF components including the TMF constructs and relationships, the key actors and stakeholders, and the implementation context were extracted by a researcher and organized into tables. TMFs were categorized using an implementation science taxonomy,²⁶ whether the framework was used to 1) guide the process of social accountability (“process models”) in medical education, 2) explain how academic institutions can lead to social accountability outcomes (“determinant models”), or 3) evaluate the implementation of socially accountable education through health system outcomes (“evaluation models”). This classification scheme was helpful for understanding the intended purpose of a TMF. Another research team member validated the data extraction and classification.

We used the Theory Comparison and Selection Tool (T-CaST)²⁷ to systematically and transparently compare and select among multiple TMFs. The T-CaST is designed to support teams involved with the research and practice of implementing evidence-informed new practices by helping team members select an appropriate TMF from a candidate list.^23,26 To the research team’s knowledge, no comprehensive candidate list of social accountability TMFs existed, and given its broad use in public administration, academia, sustainable development among others,⁷ it is questionable whether such a list could capture the “full-spectrum”²⁸ of constructs, relationships, and outcomes relevant to medical education and health services. In the absence of a comprehensive list of social accountability frameworks, the research team used a narrative review²⁵ and an environmental scan to identify potentially relevant TMFs for consideration. The purpose of this study is to support medical education research and practice in designing, implementing, and evaluating social accountability activities, specifically to implement in the Northern Ontario and NOSM contexts, the research team determined that T-CaST criteria are transferable to this study.

One researcher independently populated the information page of the T-CaST²² that outlines the important project information including research questions, study design and analysis approach, and a priori identified constructs. The research team added an additional field “relevance to NOSM context” that allowed the researchers to describe elements that were important in the development of a contextualized framework. This outline was validated with the co-authors before beginning the appraisal. Two researchers independently scored each selected framework (i.e., 0 – poor fit, 1 – moderate fit, 2 – good fit) and extracted data for each criterion within the four theory selection domains (usability, testability, applicability, and acceptability). Any conflicting scores and comments were resolved through discussion between the two reviewers, or the broader research team. All criteria were weighted equally to produce final scores for each TMF included in the analysis.

The final step of the T-CaST is to select a TMF, to either use directly, adapt the TMF, or to combine multiple TMFs to improve the fit with the study objective.²² The researchers aggregated their comments, discussed any conflicts in assessments, and iteratively adjusted the quantitative ranking of the TMFs. The TMFs were sorted according to the total and adjusted average scores from the T-CaST to identify what TMF(s) is most important to assess the impacts of NOSM in Northern Ontario and its approaches to social accountability.

Ethics approval and consent to participate

This study did not require ethics approval as it is an analysis of published research.

Results

Selection of TMFs for comparison

Table 1 (Appendix A) describes how each TMF was identified for the review and how they were interpreted in the NOSM context. Through the narrative review and keyword search, we identified two published systematic reviews^12,29 in medical education, two realist reviews of social accountability health services interventions in low and middle-income countries,^30,31 one realist review of community relationships in medical education,³² and a literature review of social accountability from a public sector perspective.¹⁸

Two TMFs ^10,33 were identified in the narrative review, ²⁵ as well as the reference list of systematic review identified in the keyword search.²⁹ Three additional TMFs were solely present in the Emadzadeh 2016 systematic review reference list.^9,11,14 A reference¹⁸ returned in the narrative review findings and the keyword search produced one TMF.³⁴ Two realist frameworks^31,32 were identified in the keyword search. Their relevance for the NOSM context is captured in Table 1, with specific consideration for our assessment criteria. From the environmental scan, two concepts (the “learning health system” and the “health performance measurement”) were identified by health system stakeholders that are priorities in ongoing health system integration and evaluation.³⁵ Consequently, the researcher team selected two recent articles that described a learning health system³⁶ and health system performance measurement system),³⁷ which were validated by local health system stakeholders. The scan of Canadian medical school websites identified an additional TMF.³⁸ In total, eleven TMFs were considered in this theory appraisal.

Comparison: characteristics of included articles

Table 2 (Appendix A) describes the characteristics of included TMFs. Four TMFs focused on evaluating the progress or impacts of social accountability,^9,10,33,38 two TMFs aimed to describe or guide the process of implementing social accountability,^11,14 three TMFs intended to identify the constructs and relationships that are important to achieving predetermined outcomes.^34,36,37 The two realist frameworks^31,32 present the contexts, mechanisms, and outcomes of how certain programs or interventions work. Realist frameworks differ from the other frameworks because they have been developed using critical realism perspectives, which acknowledges the unobservable forces that contribute to empirical outcomes. As a result, realist frameworks include context-mechanism-outcome configurations to depict what interventions or programs work, for whom, and in what context.³⁹ The realist frameworks did not neatly fit into the taxonomy of eligible frameworks and the research team determined they were important for assessment because the context-mechanism-outcomes foundation addresses each of the taxonomy categories (i.e., process, determinant, evaluation frameworks).

Comparison: social accountability meanings, constructs, relationships

The main conceptual components for each of the TMFs considered in this analysis are presented in Table 2 (Appendix A). Most articles (n = 7) provided an explicit definition of social accountability, which usually captured the obligation that health and/or education institutions have to respond to external stakeholders’ needs in their operations,^{7,12–14,17,37,42} and many cited the WHO 1995 definition of social accountability. In most articles, the needs of underserved communities were emphasized, sometimes as the only needs for consideration, although some TMFs mentioned the needs of health care providers, policymakers, health administrators and funders, and academic institutions.

Four TMFs did not explicitly define social accountability,^32,34,36,37 though addressed similar ideas in the articles. For example, the realist framework that looked at community relationships and medical education³² described social accountability as a discourse in which relationships occurred, and the learning health system article³⁶ described “shared accountability” as a collaborative system of governance.

Four TMFs^9–11,33 presented similar constructs within their TMF, particularly focusing on activities, outputs, and outcomes of medical education institutions. The two realist frameworks^31,32 also considered specific activities and outcomes, and additionally articulate how and under what circumstances these activities affect important outcomes. Relevant social accountability values (e.g., equity, quality, relevance) and processes are key components of five TMFs,^{14,34,36–38} although there are differences in how the TMFs report and operationalize these values.

Most TMFs represented actors (i.e., individuals), entities (i.e., institutions and organizations), and often systems from medical education, and four TMFs^31,34,36,37 did not explicitly reference medical education or schools. All TMFs identified multiple levels of analysis, although the levels of granularity differed. For example, some TMFs would indicate that the smallest (“micro”) level of analysis would involve individual physicians¹⁴ or individual actors,³² while other TMFs suggested that the most granular unit of analysis was an organizational unit or department.³⁶

Comparison: framework appraisal

Table 1 (Appendix A) summarizes comments from the T-CaST appraisal process according to each categories of assessment (i.e., Useability, Testability, Applicability, Acceptability)²² as part of the interpretation of each framework in the NOSM context. The complete scoring for each of the frameworks is included in Appendix B. Most TMFs were considered somewhat-to-highly useable because the constructs and relationships that were defined were deemed face-valid by the reviewers, although there were varying levels of details describing the relationships between constructs. Most of the TMFs did not reference classical theories (e.g., theories of behaviour change) to justify their selections of constructs or relationships. Few TMFs reported a step-by-step process for implementation.

Some TMFs have been tested in empirical settings,^10,34 although most used a descriptive approach to the development of a framework or required further assessment. The TMFs that reported systematic evidence synthesis as part of their development^{31,32,34,36,37} also contributed to theory development by systematically integrating theories and empirical data from published literature.

One TMF considered implementation outcomes (e.g., acceptability) as part of a pilot study.¹⁰ Five TMFs^{10,14,33,34,38} suggest specific data collection methods to implement the TMF in practice. All TMFs identified the different analytic levels that were relevant for their framework. It is unclear whether the TMFs^31,34,36,37 that focus on health services but not medical education are generalizable outside of their discipline, or whether they can be adapted to include educational constructs.

Selection and action: combine, adapt, use

According to the T-CaST, two of the three TMFs with the highest average overall scores were realist frameworks: one that investigated the relationships between communities and medical education,³² and another that described social accountability interventions in low-income settings, respectively.³¹ THEnet framework for social accountability shared the second highest score.¹⁰ When ranking the TMFs based on a subset of priority criteria, the EPIS framework³⁴ and a framework describing a Canadian learning health system³⁶ shared the third highest-rank with THEnet framework. Table 3 (Appendix A) reflects the overall and adjusted scores for the highest-ranking TMFs.

The researchers determined no single framework adequately captured the system learning while also representing the complexity and influence of stakeholder relationships in medical education. Ellaway’s realist analysis³² focused on the relationship(s) between medical schools (and its members) with communities (and their members), synthesizing how medical education activities can lead to individual and system outcomes. Adapting this framework was needed to add important concepts (e.g., system learning, other stakeholders) that were outside the scope of the original framework, clarify definitions to make processes and outcomes more measurable, and align with current discourses on social accountability in medical education. The reasons for combining these frameworks are described in Table 3 (Appendix A).

Discussion

This study compared eleven well-known frameworks in social accountability in medical education and health services research. This approach and research findings will support researchers and medical schools to engage in social accountability planning and evaluation in a more rigorous and intentional way by leveraging the conceptual and empirical work of others. This work will also inform a school-specific social accountability impact framework at NOSM. Guided by a theory selection tool (T-CaST), the research team aimed to identify framework(s) that embedded continuous improvement in an evaluation of the impacts of socially accountable medical education, with specific attention to health equity and underserved population. The researchers did not identify a single TMF that met these requirements. Subsequently, the research team examined the characteristics, strengths, and challenges of the high scoring TMFs to combine frameworks to investigate social accountability and its impacts in Northern Ontario.

With the conceptual ambiguity of social accountability and complexity in enacting social accountability, TMFs can explain “how and why” certain social accountability activities “address the priority health concerns” of the communities that certain institutions or organizations intend to serve. They can provide consistent language and practical guidance for moving from intention into action.^21,40,41 Using an implementation science taxonomy,²⁶ the most common type of TMF included in this analysis were evaluation frameworks that provide structure for evaluating social accountability activities. Other TMFs focused on describing the process of being socially accountable,^11,14 or understanding the determinants of social accountability.^34,36,37 The realist frameworks fell outside of this classification because realist inquiry focuses on what works, for whom, and under what circumstances,^39,41 and interestingly, these received the highest scores through the research team’s assessment process. The research team considered these frameworks highly testable, useable, and applicable because they comprehensively described the settings and relationships that would lead to specific outcomes, consistent with the social accountability. The research team might have perceived increased value in the realist approaches because the theoretical basis aims to provide explanatory power to complex, social interventions. However, the knowledge gaps around risk management, authority, accountability, complex understandings, and required resources might complicate the implementation of such a framework. Evaluation TMFs detailed specific social accountability outcome measures for medical education, but lacked the explanatory power to address how and why those outcomes occur. TMFs that identified the determinants or processes of social accountability emphasized the importance of adapting activities to specific contexts, but lacked guidance to do this in practice. A social accountability impact framework based in realist theory would be “full-spectrum”—the framework could address the planning, implementation, and continued assessment of social accountability activities,^23,28 and it could explore the complexity of relationships and activities in medical education.^32,41 These characteristics are important to advance the science that links medical education activities to downstream health and system outcomes.

The present study shares a similar aim as a recently published narrative review⁴² in which authors identified high-level themes and indicators to evaluate social accountability at medical schools. Our study also examines the concepts and relationships across different social accountability frameworks, while also identifying incongruent or divergent themes. The findings are complementary: Barber’s 2020 review identified critical components of an evaluation framework for social accountability based on common themes across four prominent frameworks. Our present review compares and contrasts different conceptualizations of social accountability among a larger set of frameworks, with the aim of adapting or tailoring to a specific medical school context.

There are some limitations in our approach. This research study did not exhaustively identify all social accountability TMFs in the published literature. The research team determined that given the multidisciplinary nature of the concept and existing syntheses,^12,18,29 that a systematic appraisal of notable TMFs would offer a rigorous foundation for a local NOSM impact framework. While the T-CaST offers a simple and transparent process for comparing and selecting implementation TMFs, some of the appraisal criteria (e.g., does the TMF focus on relevant implementation outcomes? Is the TMF familiar to key stakeholders?) might guide researchers toward conventional implementation TMFs³⁴ than those specific to medical education, for example. The theory selection process identified five TMFs with critical elements of social accountability; however, combining five frameworks, might create unnecessary complexity and redundancy⁴³ in using, testing, and validating a new framework. The research team will use key informant interviews with health system and medical education stakeholders to elicit a deeper understanding of how to represent and analyze different contextual elements (e.g., partnerships, settings, priorities) using existing data systems, and identify further areas of investigation.

Conclusion

In this study, the research team systematically described and compared important concepts, relationships, actors, and levels of analysis across prominent social accountability theories, models, and frameworks. A theory comparison and selection tool from implementation science helped the team identify five frameworks that explicitly address contextual considerations, provide explanatory power, and identify outcomes for assessment. This approach generated a conceptual foundation to describe and evaluate impacts of medical education that reflects the current understanding of social accountability from research evidence as well as the priorities and local contexts for the Northern Ontario School of Medicine. The research team will use these findings to guide interviews with local stakeholders and thought leaders to further develop this impact framework for the Northern Ontario School of Medicine.

Appendix A.

Table 1.

List of TMFs considered in this analysis, and their interpretation in the NOSM context according to useability, testability, applicability, and acceptability

	Theory/model/ framework citation	Search strategy identification	NOSM Useability	NOSM Testability	NOSM Applicability	NOSM Acceptability
1	Conceptualization, Production, Useability (CPU)33	Narrative review; keyword search	Boelen’s World Health Organization Definition (from which this framework emerges) is currently the definition of social accountability used by NOSM. Well-known (the foundation of social accountability evaluation) by stakeholders. No diagram. Aims to promote implementation through accreditation. Appendix B describes potential measurement strategies	Does not propose testable hypotheses but face-valid relationships and contributed to theory development	Challenging to populate given current data availability/accessibility, somewhat of a checkbox approach, more appropriate for accreditation (across school comparisons) Does not indicate implementation outcome, but clearly outlines relevant considerations for health professional education institutions to assess social accountability.	Definition and interpretations accepted by NOSM faculty and staff, and broader group of health professional education stakeholders.
2	THENet framework10	Narrative review; keyword search	Offers an implementation strategy for the Boelen CPU model; later publications depict a visual model describing relationships (Preston et al., 2016). NOSM is looking to align with existing data collection and strategic plan.	Pilot study tested acceptability and feasibility, and describes face valid constructs. Used in empirical studies (as described in reference). NOSM conducted pilot test to assess feasibility and acceptability of framework.	The framework requires collection of new data specifically for this purpose including qualitative and quantitative data, which requires significant resources. Methods for implementing framework were clearly outlined at three levels: individual school, network, and system level	NOSM participated in the pilot testing of this framework and is an active participant with THENet community, an international collaboration of health professional leaders.
3	ASPIRE criteria9	Keyword search	Checklist approach is face-valid and simple to understand; but “implementation in practice” is not relevant, because applicants can justify their social accountability through open text application form. Narrative examples of social accountability are useful to support NOSM reporting, but are not sufficient for evaluation or quality improvement.	Framework is narrative, which supports contextualization, but challenging for NOSM to compare itself over time or between institutions. Not testable hypotheses or empirical studies as this was not original intent of checklist	Narrative approach and high-level interpretation of social accountability make it hard to use as assessment tool due to subjectivity. Framework has not been used in research investigations, and individuals or schools are required to provide their own evidence/interpretation of each criterion. Unclear whether certain evidence is assessed as more valuable than others.	NOSM stakeholders are familiar with ASPIRE from an individual physician, faculty member, or program perspective, but would not consider it an evaluation or improvement mechanism. Unclear of application outside current context.
4	CARE model11	Environmental scan	CARE model depicts an organizational structure for implementing social accountability, though the constructs described (Clinical Activity, Advocacy, Research, and Education and training) don’t include the theoretical and philosophical underpinnings. Helpful for processes of social accountability, but tough to evaluate.	Not clear how model could be testable, although authors note that this structural approach could be compared to other institutions. Organizational relationships are face-valid.	The CARE model identifies an approach to implementing social accountability (through sub-committee structures) which is relevant for medical schools and organization levels, but not other socio-ecological levels. There are no explicit outcomes to measure its impacts.	The University of Saskatchewan has a similar population, geographic context and training focus as NOSM, and the types of activities (clinical activity, advocacy, research, and education) are important when evaluating social accountability at NOSM. Probably not specific enough for ongoing evaluation. Unclear of application outside of current context.
5	Collaborative E2 framework38	Environmental scan	This framework includes a set of social accountability “lenses” that incorporate important constructs in social accountability, and then includes phases to implement these lenses in a medical school and its units. These are important for discussing and evaluating impacts, although no specific indicators or metrics assigned to the lenses.	Authors of framework describe five implementation steps to integrate social accountability lenses at a department/school level, but do not elaborate on how the lenses relate to each other to demonstrate social accountability.	NOSM identified a need for a framework that includes indicators of social accountability that can be tracked over time (for continuous improvement), at multiple levels (e.g., individual, school, system), and leveraging existing data, which is not be defined in Collaborative E2 framework. Authors agree that agree that the framework should be evaluated in other settings. A specific method (deliberative engagement) is used to guide implementation and assessment of the implementation by internal and external stakeholders.	The engagement approach and different “lenses” for social accountability might be transferrable for NOSM. Specifically, the model might help with program assessment or the development of new structures at the school, but unclear of application outside of current context.
6	EPIS framework34	Keyword search	EPIS is useful for capturing multiple levels of how context can affect the implementation of a new program/intervention, and identifying interactions and feedback that influence the effectiveness. This framework does not describe constructs or ideas specific to social accountability in health professional education.	These categories can guide a “systems approach” to a NOSM social accountability framework, but further specification is needed to examine how and in what ways health professional education communicates with health systems, and how to measure or characterize this. Face-valid description of constructs and relationships, but other tools are recommended to measure/evaluate outcomes according to EPIS framework constructs	This framework offers specific process and outcome indicators, and clear relationships between concepts. This framework can provide important considerations for contextualizing social accountability at NOSM, across time and levels of context, but needs to be adapted to health professional education context.	An implementation science framework is not familiar to administrators or staff at NOSM, although concepts might be.
7	AIDER framework14	Environmental scan	Important concepts are included in this model, has built upon Boelen’s CPU framework and THENet, though not described in relationship to evaluating impacts. Case studies describe how framework applied by different stakeholders in different settings. Framework emphasizes community participation in social accountability processes and incorporates systems-thinking through iterative and continuous improvement. The model depicts the process of implementing socially accountable practices	Case studies offer examples of framework in practice, though what outcomes the framework aims to achieve are not clear. Description of how the concepts relate to each other is helpful, could be tested empirically. While the framework encourages ongoing monitoring, there is not a clear measurement strategy to assess the process outcomes and impacts of this approach.	Emphasis on systems thinking and identifying priority health concerns of underserved communities is important for NOSM framework. Examples of the framework in action are provided through project cases, which doesn’t include specific evaluation goals or measures. The framework is set up to facilitate socially accountable interactions through participatory action or design-based research, to influence change in a complex (health) system. Framework focuses on physicians and system-level interventions.	The concepts and the frameworks that informed AIDER development are well-known and accepted at NOSM, but not the specific framework itself. Unclear of application outside of current context.
8	Learning Health System36	Environmental scan	This framework describes relevant constructs/concepts for social accountability and feedback loops/system. Different from other frameworks in that social (shared) accountability is a pillar, not an outcome. Shared accountability is included as a core value, defined as a system of governance that allows stakeholders to share accountability for learning health system strategies, standards, and outcomes.	The diagram and narrative describe the values, pillars, processes, and outcomes of the framework, which could be tested, but not clear in terms of implementation process. Additionally, the lack of health professional education would mean adaptation. This framework is a culmination of published frameworks of learning health systems, but there is no supportive evidence for impacts of this model.	The values, pillars, enablers/accelerators, learning cycles, and outcomes resonate with current priorities and directions of Northern Ontario (and provincial/national) health systems. There is no explicit consideration of health professional education, which means specific process and outcome measures would need to be defined separately. The framework does not offer insight into how to implement these relationships, structures, and processes to achieve outcomes. Several examples of existing learning health systems are provided.	Framework is well-known among health system partners, though not clear how and in what ways to position health professional education as upstream concept (or pillar, processes or outcome).
9	Integrated health care performance measurement framework37	Environmental scan	Similar to learning health systems, this particular framework represents a specific interpretation of how different perspectives (patients, clinical, system) outcomes relate to overall health system goals of equity and access. Primary stakeholders identified include health care professionals, governments, and health agencies who focus on performance measurement to assess progress and aid improvement. The framework does not include indicators and authors note is not best suited to focus on “current problems”. Emphasis on equity as an outcome, no mention of social accountability. Provides in-depth conceptual clarity, though does not identify health professional education processes or outcomes in framework	This framework depicts clear relationships between concepts, which are likely testable. The purpose of the framework is to support performance measurement, quality improvement, and policy, though does not include health professional education as upstream determinant. This theoretically grounded framework includes examples of how the constructs can be populated with measureable indicators, although there is no empirical evidence of its use.	This framework emphasizes performance measurement, heavily quantifiable indicators, though recognizes that more experiential characteristics are important. The goals of this framework involve achieving equity and population health, which requires system adaptation according to community needs. This framework aligns with health system evaluation frameworks used by Northern Ontario health system stakeholders in performance measurement. It is unclear who is responsible for implementing the framework.	While there are a variety of performance measurement frameworks available in the Ontario context, this example reflects integrated whole-system outcomes and measures. Many of the example indicators do not include context-specific measures, and might be better suited for country-level comparisons, instead of smaller health systems. Unclear how the different stakeholders contribute to the measurement and implementation of the framework.
10	Realist framework of social accountability in health services31	Keyword search	This realist framework identified the contexts, mechanisms, and outcomes that promoted health provider response to social accountability interventions. This framework provided specific details about “downstream health outcomes” that could be relevant for medical education, but no explicit mention of education.	The realist perspective aims to produce testable context-mechanism-outcome configurations. This framework identifies three groups of outcomes (receptivity, responsiveness, relations) and hypothesizes the circumstances and mechanisms that could lead to these outcomes.	This framework suggests multiple types of data collection that could be used in the evaluation of context-mechanism-outcome at multiple levels. Without explicit recognition of health professional education contexts in these configurations, adaptations to this framework and further testing needed. The framework conceptualizes social accountability through health service provider responsiveness, and acknowledges the health system context, and broader socio-political context. The outcomes considered are receptivity, responsiveness, and accountability relations, which will in turn lead to health impacts.	The complexity of this framework might be implementation at NOSM more resource-intensive. The testability and specific configurations proposed are helpful for understanding how social accountability concepts relate to create impacts. Unclear how medical education fits in (as an intervention and/or through contextual elements).
11	Realist framework of the relationships between communities and medical education32	Keyword search	By identifying contexts, mechanisms, and outcomes for community engaged education, this framework provides detailed insight into the “downstream” impacts of medical education, while also considering the determinants of such outcomes. This framework clearly alludes to NOSM’s model of community engaged education, though it focuses on education and learning, with little consideration of research or improvement. Provides a clear network diagram of ways that communities and health professionals, but doesn’t include the mechanisms and outcomes in diagram (only described in text).	Similar to the previous realist framework, configurations are intended to be testable. This framework list specific mechanisms and outcomes, and depicts contexts of health professional education and community relationships, the specific configurations and tests would need to be further defined by NOSM. Authors also identify opportunities for improving research by adding a critical lens, and better description and measurement of communities’ needs and outcomes.	Although this framework does not explicitly reference social accountability and accountability processes and outcomes, NOSM had several stakeholders involved in the framework development. How the framework can be implemented or contexts-mechanisms-outcomes assessed using data is not clear.	The concepts and relationships specified in this framework are familiar to NOSM stakeholders (given their role in the development of it, and community-engaged education model at NOSM), though the implementation of this framework would likely be resource-intensive.

	Theory/model/ framework citation	Search strategy identification	NOSM Useability	NOSM Testability	NOSM Applicability	NOSM Acceptability
1	Conceptualization, Production, Useability (CPU)33	Narrative review; keyword search	Boelen’s World Health Organization Definition (from which this framework emerges) is currently the definition of social accountability used by NOSM.	No clear, explanatory pathways between constructs. Example indicators are identified.	Challenging to populate given current data availability/accessibility, somewhat of a checkbox approach, more appropriate for accreditation (across school comparisons)	Definition and interpretations accepted by NOSM faculty and staff.
2	THENet framework10	Narrative review; keyword search	NOSM is looking to align with existing data collection and strategic plan.	Pilot testing of framework was conducted at NOSM to assess feasibility and acceptability of framework.	The framework requires collection of new data specifically for this purpose including qualitative and quantitative data, which requires significant resources.	NOSM participated in the pilot testing of this framework and is an active participant with THENet community.
3	ASPIRE criteria9	Keyword search	Narrative examples of social accountability are useful to support reporting, but are not sufficient for evaluation or quality improvement.	Framework is narrative, which supports contextualization, but challenging for NOSM to compare itself over time or between institutions.	Narrative approach and high-level interpretation of social accountability make it hard to use as assessment tool due to subjectivity.	NOSM stakeholders are familiar with ASPIRE from an individual physician, faculty member, or program perspective, but would not consider it an evaluation or improvement mechanism.
4	CARE model11	Environmental scan	This framework outlines types of approaches that different sub-committees need to consider (clinical activity, advocacy, research, and education) within a faculty of medicine, but doesn’t specify constructs specific to accountability in action.	Not clear how model could be testable, although authors note that this structural approach could be compared to other institutions.	The CARE model identifies an approach to implementing social accountability (through sub-committee structures) which is relevant for medical schools and organization levels, but not other socio-ecological levels.	The University of Saskatchewan has a similar population, geographic context and training focus as NOSM, and the types of activities (clinical activity, advocacy, research, and education) are important when evaluating social accountability at NOSM. Probably not specific enough for ongoing evaluation.
5	Collaborative E2 framework38	Environmental scan	Social accountability lenses provide in-depth insight into constructs that are important for discussing and evaluating impacts.	Authors of framework describe five implementation steps to integrate social accountability lenses at a department/school level, but do not elaborate on how the lenses relate to each other to demonstrate social accountability.	NOSM identified a need for a framework that includes indicators of social accountability that can be tracked over time (for continuous improvement), at multiple levels (e.g., individual, school, system), and leveraging existing data, which is not be defined in Collaborative E2 framework.	The engagement approach and different “lenses” for social accountability might be transferrable for NOSM. Specifically, the model might help with program assessment or the development of new structures at the school.
6	EPIS framework34	Keyword search	EPIS is useful for capturing multiple levels of how context can affect the implementation of a new program/intervention, and identifying interactions and feedback that influence the effectiveness. This framework does not describe constructs or ideas specific to social accountability in health professional education.	These categories can guide a “systems approach” to a NOSM social accountability framework, but further specification is needed to examine how and in what ways health professional education communicates with health systems, and how to measure or characterize this.	This framework offers specific process and outcome indicators, and clear relationships between concepts. This framework can provide important considerations for contextualizing social accountability at NOSM, across time, but needs to be adapted to health professional education context.	An implementation science framework is not familiar to administrators or staff at NOSM, although concepts might be.
7	AIDER framework14	Environmental scan	Important concepts are included in this model, has built upon Boelen’s CPU framework and THENet, though not described in relationship to evaluating impacts. Case studies describe how framework applied by different stakeholders in different settings.	Case studies offer examples of framework in practice, though what outcomes the framework aims to achieve are not clear. Description of how the concepts relate to each other is helpful, could be tested in research.	Emphasis on systems thinking and identifying priority health concerns of underserved communities is important for NOSM framework. Examples of the framework in action are provided through project cases, which doesn’t include specific evaluation goals or measures.	The concepts and the frameworks that informed AIDER development are well-known and accepted at NOSM, but not the specific framework itself.
8	Learning Health System36	Environmental scan	This framework describes relevant constructs/concepts for social accountability and feedback loops/system. Different from other frameworks in that social (shared) accountability is a pillar, not an outcome.	The diagram and narrative describe the values, pillars, processes, and outcomes of the framework, which could be tested, but not clear in terms of implementation process. Additionally, the lack of health professional education would mean adaptation.	The values, pillars, enablers/accelerators, learning cycles, and outcomes resonate with current priorities and directions of Northern Ontario (and provincial/national) health systems. There is no explicit consideration of health professional education, which means specific process and outcome measures would need to be defined separately.	Framework is well-known among health system partners, though not clear how and in what ways to position health professional education as upstream concept (or pillar, processes or outcome)
9	Integrated health care performance measurement framework37	Environmental scan	Similar to learning health systems, this particular framework represents a specific interpretation of how different perspectives (patients, clinical, system) outcomes relate to overall health system goals of equity and access. Provides in-depth conceptual clarity, though does not identify health professional education processes or outcomes in framework.	This framework depicts clear relationships between concepts, which are likely testable. The purpose of the framework is to support performance measurement, quality improvement, and policy, though does not include health professional education as upstream determinant.	This framework aligns with health system evaluation frameworks used by Northern Ontario health system stakeholders in performance measurement. Because authors indicate they are describing a measurement system, it’s unclear who is responsible for implementing the framework.	While there are a variety of performance measurement frameworks available in the Ontario context, this example reflects integrated whole-system outcomes and measures. Many of the example indicators do not include context-specific measures, and might be better suited for country-level comparisons, instead of smaller health systems.
10	Realist framework of social accountability in health services31	Keyword search	This particular review developed a realist framework that identified the contexts, mechanisms, and outcomes that promoted health provider response to social accountability interventions. This framework provided specific details about “downstream health outcomes” that could be relevant for medical education, but no explicit mention of education.	The realist perspective aims to produce testable context-mechanism-outcome configurations. This framework identifies three groups of outcomes (receptivity, responsiveness, relations) and hypothesizes the circumstances and mechanisms that could lead to these outcomes.	This framework suggests multiple types of data collection that could be used in the evaluation of context-mechanism-outcome at multiple levels. Without explicit recognition of health professional education contexts in these configurations, adaptations to this framework and further testing needed.	The complexity of this framework might be implementation at NOSM more resource-intensive. The testability and specific configurations proposed are helpful for understanding how social accountability concepts relate to create impacts.
11	Realist framework of the relationships between communities and medical education32	Keyword search	By identifying contexts, mechanisms, and outcomes for community engaged education, this framework provides detailed insight into the “downstream” impacts of medical education, while also considering the determinants of such outcomes. This framework clearly alludes to NOSM’s model of community engaged education, a	Similar to the previous realist framework, configurations are intended to be testable. This framework list specific mechanisms and outcomes, and depicts contexts of health professional education and community relationships, the specific configurations and tests would need to be further defined by NOSM.	Although this framework does not explicitly reference social accountability and accountability processes and outcomes, NOSM had several stakeholders involved in the framework development. How data is collected to use the metrics identified in the framework is not clear.	The concepts and relationships specified in this framework are familiar to NOSM stakeholders (given their role in the development of it, and community-engaged education model at NOSM), though the implementation of this framework would likely be resource-intensive.

Table 2.

Characteristics of models assessed

	Theory/model/framework citation	Year	Geographic focus	Framework type	Objective of framework	Definition of social accountability	Constructs (actions) and relationships	Actors (stakeholders)	Context (level and scale)	Methods of framework development
1	Conceptualization, Production, Useability (CPU)33	First published in 2009, indicators added in 2012	International	Evaluation	Provide clarity surrounding what social accountability means for a medical school, and providing a way to measure its progress	School tailors all of its services and activities towards addressing priority health needs in collaboration with key stakeholders (government, policy makers, etc)	Conceptualization (design of the educational system/institution), Production (implementation and design of curriculum and activities), Utilization (the final outputs of the institution)	Accreditation bodies; policy-makers, academic institutions, communities, health administrators, health professionals	Health professional education: school-level and globally (accreditation standards)	Boelen’s 1995 WHO report, along with accreditation standards from other governing bodies
2	THENet framework10	Published and pilot tested in 2013	International	Evaluation	To critically reflect on health professional education progress towards social accountability, to evaluate the effectiveness of strategies designed to meet this goal and to facilitate learning from each other as strategies are shared	Health professional education graduates to lead health system change through partnerships with health sector, policy makers and communities to solve priority health needs. Social accountability is the alignment of research, education, and service to match local and underserved communities’ priority health needs	Conceptually based on CPU model, then practically implemented using tools driven by three questions: How does the school work (conceptualization)?; What do we do (production)?; What difference do we make (usability)?	Actors in health professional education institutions (e.g., learners, staff faculty), community partners, governments, health sector leaders	Emphasizes diverse socio-cultural contexts, usually medical schools who work with underserved or rural communities. Considers the macro-level (environment), meso-level (school-level), and micro-level (people in health and education institutions)	An international collaboration, Training for Health Equity Network operationalized CPU framework with statements, indicators, and measurement tools, then piloted among five medical schools
3	ASPIRE criteria9	Introduced by the Associated for Medical Education in Europe in 2012	International	Evaluation	To demonstrate progress toward social accountability, contextualized to the setting of the school	Medical schools must engage, partner with, and respond to the needs of their communities and/or regions and/or nation	Four criteria that are assessed: Organization and function; Education of doctors, dentist, and veterinary practitioners (admissions, programs, faculty development); Research activities; Contribution to health services and health service partnerships in community/region (graduates, partnerships)	Medical education institutions, health systems, communities, health profession governing bodies	Members of a health professional education institution	The Association for Medical Education in Europe identified a panel of international experts to come to consensus on benchmarks of school-level social accountability in 2012
4	CARE model11	Published in 2011	Canada and international	Process	To offer support for identifying the priority health concerns of local, regional, national, and international communities, while making health systems more responsive and socially accountable	Educational institutions must serve the health of their surrounding communities and calling on them to collaborate with governments, health care organizations, and the public in identifying, prioritizing, and responding to health issues [WHO definition]	Clinical activity, Advocacy, Research, and Education and training	Local, regional, national, international communities and their corresponding health systems; work is done through sub-committees of the Social Accountability committee in the College of Medicine (involve faculty, staff and students)	A College of Medicine in an academic institution	Operational and strategic planning at the University of Saskatchewan
5	Collaborative E2 framework38	Published in 2016	Canada	Evaluation	To support evaluation and enhancement of the social accountability of a wide variety of operational components within faculties of medicine, including clinical and basic science departments, medical education divisions, research offices, healthcare delivery programs, and the whole school	Medical schools must direct their education, research and service activities towards addressing the priority health concerns of the community, region, or nation they have a mandate to serve [WHO definition]	Four social accountability lenses: Diversity, Inclusion and Cultural Responsiveness; Equity; Community / Stakeholder Engagement and Partnering; and Justice-Fairness and Sustainability	Affected individuals, social groups, communities, and collaborative partners, health professional education (learners, members of Faculty of Medicine)	Within Faculties of Medicine, including clinical and basic science departments, medical education divisions, research offices and healthcare delivery programs. Framework also designed for use by medical schools as-a-whole	Deliberative engagement methodology to operationalize strategic priorities of Faculty of Medicine at Dalhousie University
6	EPIS framework34	First published in 2011, with systematic review published in 2019 documenting its uses	USA and international	Determinant	To specify variables hypothesized to play important roles in achieving effective implementation of evidence based practices	No definition of social accountability. Research and service communities will work together effectively to address the challenges of translating scientific potential into improved public health	Exploration, Adoption/Preparation, Implementation, and Sustainment: Outer context (socio-political, networks, funding); inner context (organizational characteristics, individual adopters, leadership); interconnections; Innovation/system fit and innovation/organization fit; innovation characteristics and intervention developers	Service system, organizations, research group, other stakeholders to identify needs of patients, clients, and communities	Moving evidence into practice in health services. Framework developed for youth and adult mental health services	Literature review and interpretation of select implementation science theories and frameworks
7	AIDER framework14	Published in 2013	Canada	Process	To guide physicians and institutions in their practice of collaborative patient- and family-centered care’ and to capture education and community participation in the process	Physicians and medical institutions must direct their research, services and education activities to adequately address health inequities	Assess, Inquire, Deliver, Educate and Respond; Community participation is at the centre of the model.	Physicians, medical institutions, and education; underserved stakeholders will be used to refer to groups who are disadvantaged by social hierarchies, thus leading to less access to healthcare and poorer health. Other academic and health system partners	Framework developed through institutional strategic priorities, aims to change individual physician practice	Review of select social accountability frameworks and addition of systems thinking
	Learning Health System36	Published in 2019	Canada	Determinant	To characterize the key components of a learning health system to support implementation in the Canadian context	No definition of social accountability. “Shared accountability” is a well-designed system of governance will allow stakeholders to share accountability for learning health system strategies, policies, standards, and outcomes	Core values: accessibility, adaptability, cooperative and participatory leadership, equity, fairness, governance, inclusiveness, person focused, privacy, scientific integrity, solidarity, transparency, and value in healthcare Learning processes: data to knowledge, knowledge to performance, performance to data Outcomes: population health, health system costs, patient experience, provider experience	Healthcare providers, administrators, policy-makers, patients, community members, researchers, industry partners or other experts/actors within health ecosystems	Local, regional, provincial, and national health ecosystems in Canada. Learning health systems can exist at the micro-level (organizations and units), meso-level (service continuums), and macro-level (decisions about planning and performance)	An interdisciplinary team at the Institut national d’excellence en santé et en services sociaux in Quebec, Canada used a scoping review of the scientific and grey literature on learning health systems, regular team discussions over a 14-month period, and consultations with Canadian and international experts to inform the final product.
	Integrated health care performance measurement framework37	Published in 2020	High income countries	Determinant	To establish a comprehensive, conceptually grounded healthcare system performance measurement system	No definition of social accountability. Public reporting of healthcare system performance is seen as a lever to improve performance and an imperative to accurately, fairly and meaningfully measure and report comparative information	Five measurable constructs: patients’ needs and expectations; healthcare resources and structures; receipt and experience of healthcare services; healthcare processes, functions and context; and healthcare outcomes Ten derived constructs of performance coverage; accessibility; appropriateness; safety; effectiveness; productivity; efficiency; adaptability; sustainability; resilience. Two overarching derived constructs: population health impact and equity	Clinicians, patients, policy-makers, administrators, public, governments, funders, quality improvement agencies	Healthcare delivery systems	An eight-phase framework development approach to collect the range and distribution of concepts
	Realist framework of social accountability in health services31	Initial framework published in 2013 realist review, with revised framework published in 2017	Low and middle income countries	Realist	To understand how social accountability initiatives influence health provider responsiveness to citizens’ demands	Collective action by citizens to encourage responsiveness by public social services (also known as “external responsibility”)	Outcomes include provider receptivity, responsiveness, and accountability relations. Contexts and mechanisms: Provider’s perceptions and expectations of health service users Providers’ perceptions of the legitimacy of citizen groups Providers’ feelings of support, safety and appreciation Providers’ fear of repercussions from influential third parties Providers’ feelings of moral obligation Providers’ self-perceived capacity and identity	Framework focuses on health workers and their behaviour. Emphasizes the importance of service users and citizen/community groups (and how service users are perceived, as a patient, client, consumer, etc). Also includes governmental and political actors, health committees, district health boards, non-governmental organizations, civil society organizations, and their networks.	Three levels: 1.Micro-level: social accountability initiative – characteristics of initiative 2. Meso-level: structure, culture, practices of health system 3. Macro-level: legal, social, political, economic context and norms	A realist review to investigate how, and under what circumstances, do social accountability interventions produce outcomes in the frontlines of health service provision
	Realist framework of the relationships between communities and medical education32	Protocol published in 2013, final review published in 2015	International	Realist	To identify how different relationships with host communities impact medical education, identifying key factors, dependencies and their contextual binding	Not explicitly defined; but the role of communities is important in medical education to produce doctors who can meet community needs. Social accountability is a discourse that motivates community relationships.	Context: health professional education programme activities are the key focus of community relationships. Medical learners experience programs and activities through community relationships and connect learners with community stakeholders. Mechanisms: Through community relationships, learners will change (or enrich) their values, knowledge, attitudes, beliefs about that community; Teachers might better prepare students for choosing and working in underserved communities; Programme leaders will respond to community or institutional priorities and goals; Community members value community relationships with medical education because they perceive active contribution to community wellbeing. Outcomes: Learner awareness, compassion, and empathy; programme expectations; improved health services; actual and perceived power relationships between medical schools and communities.	This framework highlighted the inconsistencies around how communities are defined in medical education literature. Communities were defined geographically, by setting, by recipient status, by populations, etc. Community members could be clinical, civic, agencies, or citizens. From the medical school side, relationships were enacted by the school as a whole, programme leaders, and individuals or groups of faculty or students. Relationships tended to be between entities of similar sizes and abstraction.	Contexts, mechanisms, and outcomes occur at four different levels: 1. System or societal level 2. School or community level 3. Programme or agency level 4. Individual participant level	A realist review to synthesize empirical and non-empirical evidence on the ways that community is linked to medical education

Table 3.

T-CaST high ranking frameworks and assessment

Framework number and name	2	6	8	10	11
	THENet framework	EPIS framework	Learning Health System	Realist framework of social accountability in health services	Realist framework of the relationship between communities and medical education
Average score (total score/number of characteristics)	1.75	1.63	1.63	1.75	1.81
Average adjusted score (adjusted score/number of characteristics)	1.67	1.67	1.67	1.83	1.75
How will you apply the information from this tool? (e.g., Which TMF(s) did you select? What is your rationale for selecting the TMF(s)?	Strengths: Familiar to medical education stakeholders, in particular northern and rural stakeholders Helpful, explicit implementation guide (processes and outcome measures) and logic model Pilot tested, refined, revised Challenges: Unclear of how to embed continuous learning and ongoing monitoring and improvement Research study oriented, would prefer nimble and embedded approach Adapt and combine: An explicit implementation plan, with metrics and processes to collect data, are important to align with final framework. This framework has international recognition, and the collaboration includes Northern Ontario School of Medicine. Requires more explicit, precise description of responsiveness, and more emphasis on accountability processes.	Strengths: Well- established implementation framework with empirical evidence Provides significant resources to design, implement, and evaluate theory and process in public sector settings Challenges: Implementation framework does not specifically consider learning outcomes or formalized educational processes Adapt and combine: This implementation framework intends to support researchers and practitioners in implementing evidence-based innovations. It will be helpful to examine how specific social accountability interventions or innovations are implemented, and how that might affect outcomes for different stakeholders (at different levels).	Strengths: Familiar language of “learning health system” in academic medicine and health services Identifies values, processes, outcomes that are key for continuously improving health system Challenges: Does not consider education or training of health professionals in processes or outcomes Need to examine how/if values of social accountability and shared accountability compare Smallest unit of analysis is organization (not individual) Adapt and combine: This conceptual framework explicitly describes the structures, processes, and outcomes to enable and embed continuous improvement in health systems, and offers ways to align distinct “eco-systems” to bring value to health systems and people. This framework offers insight to optimize organizational and system learning, wish an emphasis on creating and sharing evidence. There is no explicit consideration of health professional education or training.	Strengths: Realist framework provides underlying theory of change to support improvements Includes individual-level Challenges: Particularly developed in low-income health services settings Does not include an element about medical education Adapt and combine: This framework describes the steps of accountability, which none of the other frameworks have considered. This is helpful for categorizing social accountability activities and intervention, and understanding the “object of change.” Focus is mainly on relationships between communities and health service providers at the organization-level.	Strengths: Realist framework provides underlying theory of change to support improvements Lists specific outcomes relevant to medical education, “communities”, and other stakeholders Highlights importance of defining “communities”, relationships, and understanding power dynamics Developed by Northern Ontario leaders Challenges: Complex network analysis might be challenging to align with outcomes and measure Role of organization or system learning not explicitly captured Adapt and combine: The network diagram depicts relationships between actors and activities, which describes the multiple levels of context. The description of mechanisms for different actors will help understand why certain outcome patterns occurred. How to monitor and rapidly learn in this network is not clear according to this model.

Appendix B.

Table 4.

Complete T-CaST scoring, consensus by two reviewers

	T/M/F Number and Name	1	2	3	4	5	6	7	8	9	10	11
		Conceptualization, Production, Useability (CPU)	THENet framework	ASPIRE criteria	CARE model	Collaborative E2 framework	EPIS framework	AIDER framework	Learning Health System	Integrated health care performance measurement framework	Realist framework of social accountability in health services	Realist framework of the relationship between communities and medical education
	T/M/F Characteristic
Priority	Usability
X	a. TMF includes relevant constructs (e.g., self efficacy; climate)	1	2	2	1	2	1	2	1	1	1	1
X	b. Key stakeholders (e.g., researchers; clinicians; funders) are able to understand, apply, and operationalize TMF	2	1	2	2	2	2	2	2	1	1	2
x	c. TMF has a clear and useful figure depicting included constructs and relationships among them.	0	1	0	2	0	2	2	2	2	2	2
X	d. TMF provides a step-by-step approach for applying it.	1	2	2	0	2	2	1	1	2	2	1
X	e. TMF provides methods for promoting implementation in practice	1	2	1	1	1	2	1	2	1	2	1
X	f. TMF provides an explanation of how included constructs influence implementation and/or each other	1	1	1	1	1	2	2	2	2	2	2
	Testability
X	a. TMF proposes testable hypotheses.	0	2	0	2	1	2	1	1	2	2	2
X	b. TMF includes meaningful, face-valid explanations of proposed relationships.	1	1	1	2	2	2	1	2	2	2	2
	c. TMF contributes to an evidence base and/or theory development because it has been used in empirical studies	2	2	0	0	1	2	1	2	1	2	2
	Applicability
	a. TMF focuses on a relevant implementation outcome (e.g., fidelity; acceptability).	1	2	0	2	1	1	1	1	2	2	2
X	b. A particular method (e.g., interviews; surveys; focus groups; chart review) can be used with TMF.	2	2	0	1	1	1	2	2	2	2	2
X	c. TMF addresses a relevant analytic level (e.g., individual; organizational; community).	2	2	2	1	2	2	1	2	2	2	2
	d. TMF has been used in a relevant population (e.g., children; adults with serious mental illness) and/or conditions (e.g., attention deficit hyperactivity disorder; cancer).	2	2	2	2	2	2	2	2	1	1	2
X	e. TMF is generalizable to other disciplines (e.g., education; health services; social work), settings (e.g., schools; hospitals; communitybased organizations), and/or populations (e.g., children; adults with serious mental illness).	2	2	1	2	2	1	1	1	1	2	2
	Acceptability
X	a. TMF is familiar to key stakeholders (e.g., researchers; scholars; clinicians; funders).	2	2	2	1	1	1	1	2	1	2	2
	b. TMF comes from a particular discipline (e.g., education; health services; social work).	2	2	2	2	2	1	2	1	1	1	2
	Total Score	22	28	18	22	23	26	23	26	24	28	29
	Number of Characteristics	16	16	16	16	16	16	16	16	16	16	16
	Average score (total score/number of characteristics)	1.38	1.75	1.13	1.38	1.44	1.63	1.44	1.63	1.5	1.75	1.81
	Adjusted score	15	20	14	16	17	20	17	20	19	22	21
	Adjusted number of characteristics	12	12	12	12	12	12	12	12	12	12	12
	Average adjusted score (total score/number of characteristics)	1.25	1.67	1.17	1.33	1.42	1.67	1.42	1.67	1.58	1.83	1.75

Conflicts of interest

The authors have no competing interests to declare.

Funding

This study was funded as part of Brianne Wood’s postdoctoral fellowship at the Northern Ontario School of Medicine.

Author contributions

Brianne Wood and Erin Cameron conceptualized the study. All team members participated in the data collection, analysis, and interpretation. Brianne Wood drafted the first version of the manuscript, and all authors provided critical feedback that shaped this current work and future research for this project. All authors approved this version of the manuscript.

Data and material availability :

The authors confirm that the data supporting the findings of this study are available within the article.