Inter-institutional data-driven education research: consensus values, principles, and recommendations to guide the ethical sharing of administrative education data in the Canadian medical education research context

Abstract

Background

Administrative data are generated when educating, licensing, and regulating future physicians but these data are rarely used beyond their pre-specified purposes. The capacity necessary for sensitive and responsive oversight that supports the sharing of administrative medical education data across institutions for research purposes needs to be developed.

Method

A pan-Canadian consensus-building project was undertaken to develop agreement on the goals, benefits, risks, values, and principles that should underpin inter-institutional data-driven medical education research in Canada. A survey of key literature, consultations with various stakeholders and five successive knowledge synthesis workshops informed this project. Propositions were developed, driving subsequent discussions until collective agreement was distilled.

Results

Consensus coalesced around six key principles: establishing clear purposes, rationale, and methodology for inter-institutional data-driven research a priori; informed consent from data generators in education systems is non-negotiable; multi-institutional data sharing requires special governance; data governance should be guided by data sovereignty; data use should be guided by an identified set of shared values; and best practices in research data-management should be applied.

Conclusion

We recommend establishing a representative governance body, engaging trusted data facility, and adherence to extant data management policies when sharing administrative medical education data for research purposes in Canada.

Introduction

Throughout training, a vast amount of data on future physicians is generated - admissions information, assessments, attendance records, professionalism reports, logs of clinical activities, certification results, and much, much more – which are held by numerous institutions, including medical schools, licensing bodies, regulatory authorities, and health human resource agencies. While regularly collected, these data are rarely used beyond their pre-specified purposes (e.g., to assess competency). Recently, however, there have been calls to make better use of these data to create new insights for educational practice.¹ These calls are further amplified with widespread curricular reforms currently underway.^2,3 Previous stand-alone investigations using administrative education data have been effective at evaluating training efficacy in relation to educational,⁴ professional,⁵ and patient outcomes.⁶ Accordingly, the need for a structural foundation that makes inter-institutional data-driven research in medical education more feasible and sustainable is recognized.

The challenge here is not in methodological or technological expertise. Rather, it relates to the need for building the governance conditions that are pre-requisites for this type of research: Scholars of higher education have highlighted the ethical, legal, privacy, and autonomy-related issues inherent to this type of work.^7-9 Without similar consideration for these issues in the context of medical education, advances in data-driven research in our field is fraught with risks. Institutions must understand when it is safe to share data; learners should recognize the potential benefit or harm data-driven research poses to them; and researchers must have clarity on how to engage in this research in a respectful and sustainable manner. In this regard, the capacity for sensitive and responsive oversight needs to be developed. Without this, our field will lag and increase the potential for unreflective and harmful work. To this end, we recently engaged a comprehensive group of Canadian medical education stakeholders and data stewards to advance collective thinking and build agreement. Our goal was to generate consensus recommendations and principles for governance that can support the collection, sharing, and use of administrative medical education data for the purposes of research.

Method

Our process of building consensus recommendations and principles for governance was informed by 1) key literature concerned with data ethics for education, 2) a set of pre-workshop consultations, 3) a series of five successive 3-hour knowledge synthesis workshops held online between March 25^th and April 21^st, 2021, and 4) a set of post-workshop consultations. At each stage, we developed a set of propositions concerning the risks, benefits, values, and principles that should underpin the ethical conduct of inter-institutional data-driven medical education research in Canada. Throughout the consensus-building process, we solicited ideas and feedback on our propositions from representatives of the Association of Faculties of Medicine of Canada (AFMC), Black Medical Students Association of Canada (BMSAC), Canadian Federation of Medical Students (CFMS), Canadian Medical Protective Association (CMPA), College of Family Physicians of Canada (CFPC), Federation of Medical Regulatory Authorities of Canada (FMRAC), Indigenous Physicians Association of Canada (IPAC), Medical Council of Canada (MCC), Resident Doctors of Canada (RDOCs), Royal College Physicians and Surgeons of Canada (RCPSC), and the 17 Canadian medical schools, compiling descriptive field notes pertaining to salient discussions as we progressed. Invitations to participate were extended directly from the consensus team to these organizations as well as to individuals in our professional network who are publicly engaged in relevant data science research. Across the consultations and workshops, we solicited ideas and feedback on pertinent propositions, and compiled field notes pertaining to the participants’ responses and discussions. At the conclusion of each session, we reviewed and categorized these notes according to relevant themes, questions, and issues; a descriptive process aimed at capturing what was said by participants in the workshop. As notes coalesced, consensus ideas were identified, as were those ideas for which consensus remained contentious, ambiguous, or unclear. These ideas were then used by the series facilitators to drive discussions across subsequent sessions; such that, over the course of the series a shared set of goals, benefits, risks, values, recommendations, and principles for governance were distilled.

Literature foundations

We relied on key literature describing the ethical and governance issues in general education and data science.^7,10,11 Regan and Jesse’s (2018) framework for contemplating technology and big data in general education was notably relevant. This framework describes ethical concerns associated with information privacy anonymity, surveillance, autonomy, non-discrimination, and ownership of education information⁸. We oriented ourselves to each of these concepts at the outset, using them as a starting point for our pre-workshop consultations.

Pre-workshop consultations

Prior to our series of five pan-Canadian workshops, we engaged in consultations with data custodians, research scientists, and learner advocacy groups from across Canada. This involved independent discussions with representatives of the AFMC, BMSAC, CFMS, CMPA, CFPC, FMRAC, IPAC, MCC, RCPSC, and prominent Canadian data science scholars from multiple disciplines within and beyond medical education.

The workshop series

The workshops were attended by more than 50 individuals including representatives of the AFMC, BMSAC, CFMS, CFPC, FMRAC, MCC, RCPSC, CAPER, and members the education scholarship communities associated with Canada’s 17 medical training institutions. Workshops were each three hours in length. The series began by prompting consideration for the ethical concerns and sensitizing considerations determined through the literature and pre-workshop consultations. The first four workshops were augmented by invited presentations from experts in the fields of ethics, higher education, library science, and epidemiology/health services; each of whom had either unique perspective on the ethical challenges of inter-institutional education data sharing or had made considerable progress in data sharing within their own field. In each workshop, participants engaged in facilitated activities geared towards the development and refinement of consensus agreement. These activities involved reflective breakout group exercises and large group dialogue. As we progressed, discussions aimed to foster ideation and promote understanding of the perspectives of all stakeholders. Participants were provided orienting concepts and relevant readings that would prepare them for each workshop. Notably, each session was attended by an observer group, composed of representatives of relevant physician trainee advocacy organizations. This group closed each session by reflecting on the workshop’s progress and sharing their perspectives during a dedicated roundtable activity. This reflection ensured that a clear orientation pertaining to the trainee perspective were integrated into the final agreement.

Post-workshop consultations

Prior to its final presentation, as a comprehensive form of member checking, draft iterations of this document were reviewed by representatives of the AFMC, BMSAC, CFMS, CFPC, FMRAC, RCPSC, Canadian Association for Medical Education (CAME), Society of Rural Physicians of Canada (SRPC), and Resident Doctors of Canada (R-DOCS). We engaged in a final round synthesis to reconcile and incorporate the final articulation of consensus in response to this feedback.

Outcomes

Sensitizing considerations

Through the pre-workshop consultations, we identified three salient themes regarding inter-institutional data-driven medical education research in Canada:

1. The challenge of balancing institutional mandates. Consultants highlighted tensions that may exist within an organization’s own missions (e.g., promoting quality improvement versus protecting constituents from harm), and across the mandates of different institutions (e.g., curricular improvement versus identifying potential risks to patients).

2. The logistical obstacles associated with sharing data across institutions. Consultants pointed to the importance of establishing processes for harmonizing data fields across organizations, building effective data sharing agreements, and conducting research activities within reasonable timelines.

3. The need to differentiate our considerations for admissions and assessment data currently held and used by data stewards from those pertaining to socio-demographic data (e.g., race, gender, ethnicity) that are not currently collected within the context of Canadian medical education. Consultants emphasized strongly that the implications of data exchange shift when socio-demographic data are included. Although critical for the advancement of effective equity, diversity, and inclusivity mandates, many noted that the collection and analysis of these data is challenged with the tensions regarding the ethical use and misuse of data.

These initial considerations informed the structure and nature of the workshop series, which gave way to following findings that were ultimately member-checked through post-workshop consultation.

Shared goals and perspectives

Stakeholders and data stewards represented in the consensus-building project included organizations involved in physician training, physician licensure, physician certification, physician regulation, training program accreditation, and advocacy for physicians and physicians-in-training. While each has a separate function, a shared goal was recognized among them:

A self-regulating health human resource and Canadian healthcare system that is effective in constantly meeting the evolving healthcare needs of all persons who access it, as well as the professional aspirations of its physician constituents.

They also agreed that medical education plays an important role in meeting this goal; that education data has the potential to support knowledge generation that can enhance education practice and policy; and that there exists a potential for data-driven research to contribute to the adoption of education policy and practice that promotes harm and accentuates social inequity.

Benefits of inter-institutional data-driven education research

It was agreed that the benefits of inter-institutional data-driven education include positive impacts on education program evaluation efforts, social accountability missions, and education research quality.

Risks of inter-institutional data-driven education research

It was agreed that the pursuit of the presumed benefits may see data used in ways that expose individuals and institutions to risks. The group agreed upon a set of procedural risks, which include the contravention of institutional commitments to privacy and confidentiality, informed consent, and promoting data sovereignty. The group also agreed upon a set of outcome risks, which include the use of data-driven research to discriminate physicians-in-training and physicians on the basis of personal or social identity characteristics, promote a culture of surveillance, draw harmful misinterpretations of findings, and proliferate inter-institutional comparisons that negatively impact institutions with respect to public perception. These risks were identified as relevant regardless of the intent of the scholarship.

Shared values

The consensus-building process highlighted that a key goal for all participants is to establish trust between stakeholders and researchers. Values that the participating medical education stakeholders and data custodians agreed were fundamental to building this trust include commitments to informed consent, appropriate data collection, appropriate research purposes, privacy and anonymity, autonomy, and non-discrimination (Table 1).

Table 1.

Descriptions of the shared values for inter-institutional data-driven education research in Canada

Informed Consent	The collection and sharing of data about an individual should take place with the knowledge of the individual. This value includes providing impacted individuals with appropriate notice of data uses, a transparent view of research practices, and the ability to choose to participate.
Appropriate Data Collection	Monitoring is a key feature of institutional accountability and fundamental to education assessment practices. Accordingly, physicians-in-training and physicians understand that their progress will be subject to data collection. However, the collection of data on physicians-in-training and physicians should be limited to those metrics that support the core missions of the relevant institution.
Appropriate Research Purposes	The pursuit of data-driven education research in an opportunistic manner and/or simply for the purposes of leveraging available infrastructure and/or enhancing research productivity carries with it heightened possibilities that data may be misused, misinterpreted, or mismanaged (i.e., as a function of the articulated procedural and outcome risks). In this regard, the pursuit of opportunistic data innovation may detract from addressing prevalent and firmly established issues. It may also contribute to a culture of research productivity that does not lead to meaningful changes in education practice or policy.
Privacy and Anonymity	Individuals should remain anonymous or obscure within research datasets. In this regard, the amount of information collected and shared should be minimized to that which is required for the particular purpose. Almost all institutions and jurisdictions have policies in place to protect the privacy of individuals. Data-driven research should abide by and incorporate the policies and practices relevant to the institutions and jurisdictions involved in the project.
Autonomy	Data-driven research should not be used to limit the options or developmental possibilities of learners. Predictive analytics may be used to constrain an individual’s ability to govern their own education, by leading or nudging people in certain directions. Data driven research should not influence learners or communities in a way that constrains their ability to pursue thoughtful and deliberate choice.
Non-discrimination	Data need to be used in ways that do not discriminate or perpetuate inequity. Analyses that make predictions about individuals based on constellations of their information can perpetuate prejudices, profiling, and discrimination while also accentuating social stratification.

Principles for governance

With the shared goal, perspectives, benefits, risks, and values agreed upon and articulated, we reached consensus on a set of principles for governance when engaging in the collection, sharing, and use of administrative medical education data across institutions for the purposes of research.

1. Researchers should establish clear a priori purposes, rationale, and methodology for data sharing. While these can be broad and cover various scholarly activities, the types of associations, linkages, methodologies, and the provisions for protection of confidentiality, privacy, and minimization of harm should be articulated before research commences.

2. Informed consent on the part of those who generate data in education systems including learners, faculty, staff, and organizations is a non-negotiable aspect of research. In the case of meta-informed consent (i.e., pertaining to a range of different potential types of secondary use as determined by a relevant data sharing agreement), this should be time-limited. Time-limited meta-consent provides opportunity to respect the autonomy of data generators with information about risks and benefits resulting from new data created through data linkage as well as to the confirm the continued validity of data in the system.

3. Research oversight bodies should include knowledgeable representation from institutions, communities, and groups which hold meaningful relationships with the data. Notably, the noted risks are not evenly distributed among all populations. Relevant communities with a history of marginalization should be given special consideration in the contemplation of the risks.

4. Governance should be guided by respect for data sovereignty (i.e., the rules/policies/laws of the community from which data are collected). In the Canadian context, this is particularly relevant to notions of Indigenous autonomy from post-colonial states. Representatives with meaningful sovereignty relationships to the data should be engaged in the creation, review, and refinement of data sharing agreements prior to any data sharing. This involvement should be maintained throughout the entire research process, including during the interpretation of results and dissemination of outcomes, ensuring that the research has meaningful benefit and/or does not perpetuate harms for affected communities. This recommendation aligns with those outlined in the Tri-Agency Research Data Management Policy,¹² which contemplates Chapter 9 of the Tri-Council Policy Statement 2 (2022) on Research Involving First nations, Inuit, and Metis Peoples of Canada¹³ and the First Nations Principles of Ownership, Control, Access, and Possession.¹⁴

5. Governance structures should review proposed uses of data against the criteria of respect for the foundational values (i.e., autonomy, informed consent, privacy and anonymity, appropriate data collection, appropriate purpose, non-discrimination), scientific merit, and potential for beneficial impact. Regardless of the intended activity of the data sharing (e.g., program evaluation, research), the risks are the same and still relevant.

6. Governance structures should be vigilant in ensuring data are used in accordance with the principles, processes, and specific policies established in data sharing agreements and approved research practices. Governance should promote data management plans aligned with best practices including established policies for research data management (e.g., Tri-Agency Research Data Management Policy).¹²

Recommendations

On the basis of this consensus, three recommendations for any proposed data-driven research study are offered:

First, a governance body should be established to oversee data sharing agreements and activity. This body should include representation of the perspectives of learners, physicians, data stewards, regulators, education institutions, researchers, and community members including knowledgeable representatives of marginalized and under-represented communities, as is appropriate for the data shared and the research questions pursued. Additionally, this body may provide additional ethical review beyond local review and, accordingly, may also include subject matter experts in data management and privacy legislation. The governance body should be legitimately empowered to enact data sharing that aligns with the rules, policies, and laws of the sovereign jurisdictions in which the data have been collected.

Secondly, a trusted data management facility should be engaged in the technical procedures of data linking, applying the processes approved by the governance body. This facility may administer de novo data sharing agreements, assume existing ones where governance and legislation approve, and work with existing data managers where relevant. The facility should be responsible for quality assurance of the data, reporting to the governance body on safety of data and adherence to protocols. This facility should also ensure and oversee, as necessary, the disaggregation and return of data, as well as data forgetting (i.e., expiration of data). Such a facility may or may not have a role in data archiving.

Third and lastly, inter-institutional education data should be managed according to best practice standards.^12-14 Data shared for the purposes of analysis should minimize the use of individual and institutional identifiers. There may be situations wherein the governance permits inclusion of this information; depending on the nature of the research question, the assent of the participating stakeholders, and the risks involved.

Implications

Inter-institutional data-driven research directed towards the improvement of medical education requires a strong structural foundation. Data sharing creates new opportunities, but also amplifies risks. Given the myriad potential uses of administrative education data and institutional actors potentially involved, a single policy may not be feasible to protect and promote ethical research. Thus, we present here principle-based recommendations that can support data-driven education research in Canada, offering a feasible approach to mitigating the identified risks. While each Canadian data steward and medical education stakeholder has a unique organizational and jurisdictional reality that dictates the data management processes that they must consider in pursuit of research evidence that supports their missions, this consensus presents a set of values that can guide institutions and researchers when they work together.

Operationalization at a pan-Canadian level or even in smaller collaborations will be challenging. That is indeed the point of this exercise – data sharing is not an easy task. It is necessary to consider and navigate the challenges that have meaningful impact on stakeholders. This consensus statement represents a starting point from which this type of work may be pursued. We recognize that the context of inquiry will have considerable influence on the way in which the principles and recommendations are enacted. These ideas may underpin formal relationships and serve as the basis for data sharing agreements that mediate inter-institutional education research; or they may guide the development of statements of reflexivity that serve to make transparent the ways in which data were managed and research was conducted. The hope is, regardless of the context, that researchers and those who manage and oversee education data are attentive to the core values as they embark on their inquiry. We believe that firm adherence to the principles can guide implementation in each unique context.

One salient idea that emerged during this consensus project is that the risks of inter-institutional data-sharing involving socio-demographic data in the Canadian context are very high. This concern is not surprising given the numerous examples in other contexts but also historical misuses of research data to perpetuate inequity.^15-17 Participants highlighted the importance of these data to address institutional mandates for social accountability and inquiry that advances diversity, inclusivity, and equity. Learner and advocacy group stakeholders noted the continuing potential for harm, stigma, and inquiry that may not benefit those represented by the data. We must acknowledge the potential that pro-social research benefits may be offset by oppressive outcomes at all stages of the data-driven research continuum - from research question through to the interpretation of findings. One potential resolution for this tension is to focus inquiry on the impact of education policies and procedures represented in the data rather than on the individuals. Our consensus coalesced on the importance of centering the system.⁸

Conclusion

While this consensus originates in a Canadian context, it has relevance to other medical education communities. The number of medical education data sharing collaborations continue to increase with prominent consortiums and groups in the US, UK, and Canada.^4,18-20 However, formal knowledge on how to develop these collaborations remains limited. A consensus approach that engages all concerned stakeholders is one way to identify the necessary contextual knowledge for collaboration. While some of our recommendations may not be relevant to all contexts, we believe the presented values and principles resonate with good practice for any ethical and impactful inquiry. It is our hope that other collaborations can use this consensus to guide their own research.

Funding Statement

Funding

This work draws on research supported by the Social Sciences and Humanities Research Council; the Education Investigator Award from the Department of Family & Community Medicine and the Temerty Chair in Learner Assessment in the Temerty Faculty of Medicine at the University of Toronto; Department of Family Medicine at McMaster University.

Conflict of Interest

None.

Data Sharing Statement

Information regarding the consensus building process including agendas, workbooks, presentations, and supporting documents are openly available at www.dataconnection.ca in both English and French versions.

Contributors

LG and KK designed and supervised all aspects of the project. BB, RLK, KM, and AY acted as facilitators for the workshop. AC also acted as a facilitator for the workshop series, with special attention to the inputs of the observer group of trainee stakeholders. All authors (LG, KK, BB, RLK, KM, AY, AC) participated in synthesis and interpretation of information gathered through the workshop series, contributed to the critical revision of the paper, approved the final manuscript for publication, and have agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.