Transparency in the secondary use of health data: assessing the status quo of guidance and best practices

Abstract

We evaluated what guidance exists in the literature to improve the transparency of studies that make secondary use of health data. To find peer-reviewed papers, we searched PubMed and Google Scholar. To find institutional documents, we used our personal expertise to draft a list of health organizations and searched their websites. We quantitatively and qualitatively coded different types of research transparency: registration, methods reporting, results reporting, data sharing and code sharing. We found 56 documents that provide recommendations to improve the transparency of studies making secondary use of health data, mainly in relation to study registration (n = 27) and/or methods reporting (n = 39). Only three documents made recommendations on data sharing or code sharing. Recommendations for study registration and methods reporting mainly came in the form of structured documents like registration templates and reporting guidelines. Aside from the recommendations aimed directly at researchers, we also found recommendations aimed at the wider research community, typically on how to improve research infrastructure. Limitations or challenges of improving transparency were rarely mentioned, highlighting the need for more nuance in providing transparency guidance for studies that make secondary use of health data.

1. Introduction

Health data have become increasingly accessible to researchers with the advent of large databases providing routine patient data from electronic health records (e.g. OpenSafely, OpenPrescribing, Clinical Practice Research Datalink (CPRD), German Portal for Medical Research Data (FDPG)). The secondary use of health data (SU/HD) for research purposes may yield valuable knowledge, but sometimes high risk of bias or even fraud may arise [1–6]. Because the datasets are not tailor-made to research studies, researchers typically need to inspect the data before being able to develop a sensible analysis plan. However, inspecting the data provides researchers with information about the variables of interest, thereby potentially biasing the statistical analyses [7]. Aside from that, routinely collected health data may be more prone than clinical trial data to selection bias because proper randomization cannot typically be achieved [8,9], and to measurement error because of differences in the data entry and classification procedures among health organizations [10]. Furthermore, it can be challenging to identify all available SU/HD studies relevant to a given research question, making it difficult to properly review and synthesize the literature.

The analytical complexity and the potential for bias in SU/HD studies highlight the need for more transparency, which would benefit science in two ways. First, it becomes easier to identify SU/HD studies and thus to reduce bias in the review and synthesis of such studies. Second, it becomes easier to identify biases in individual studies, and to prevent (via researchers engaging less in questionable research practices) and correct (via follow-up studies or correction notices) these biases. Moreover, as secondary use of patient data increasingly works with broad consent or opt-out models, transparency about the studies conducted also plays an important role in building and maintaining social trust in this form of patient data use [11]. In a broad consent and opt-out model, patients no longer consent to the individual secondary use studies with their patient data but are only informed about study-wide objectives, risks and governance of secondary use. In these situations, it is even more important that society is informed about which studies are being conducted.

Important pillars of research transparency are registration, methods and results reporting and data and code sharing [12]. Registration (also called preregistration because it should take place before data analysis; see [13,14]) refers to the documentation of research plans (e.g. hypotheses and/or analyses) before research outcomes are known [15]. This documentation typically occurs in a specific registration repository like https://clinicaltrials.gov or https://osf.io. Registration allows readers of a scientific paper to assess what the research plan was and whether the author(s) conducted a study as planned. This documentation can help identify potentially biased, data-driven decisions the authors might have made during or after running the study. Moreover, registration of the existence of specific studies has the advantage of making transparent what studies are out there, and thus informing the public, informing researchers working on similar topics, and potentially preventing publication bias [16]. A recent study in the Swedish context shows that only 0.5% of SU/HD studies are prospectively registered [17].

Methods reporting refers to the public documentation of the research design and methodology of a scientific study once it is completed and the results are known. This typically occurs in the methods section of a research paper. Transparent methods reporting allows readers of a paper to assess whether the study was carried out in line with the registration (assuming a registration is available and sufficiently clear), and potentially rerun and verify analyses or perform other replications [18–20]. In the case of SU/HD studies, methods reporting typically involves a detailed description of the handling of data and the statistical analyses performed.

Results reporting refers to the documentation of the outcomes of a scientific study. It is transparent if a result is reported for all the planned analyses, and unplanned analyses are presented as unplanned. Results transparency is important because omitting certain results (e.g. because they are not statistically significant) biases the scientific literature [21,22].

Sharing refers to the distribution of the data and code of a study, which can be ‘open’ or ‘controlled’. Open data and open code are available to anyone with access to the Internet. Controlled data and controlled code are available to bona fide researchers but come with restrictions such as a confidentiality agreement. Controlled sharing is customary for data that are sensitive, which typically is the case for electronic health data. In general, sharing is transparent if it allows readers to redo the study’s analyses on the original data. Data and code sharing are seen as some of the most important transparency practices in biomedicine [23]. In the context of SU/HD, control over the data typically lies with the registry or database that provides the data, not with the researchers themselves. Transparency therefore does not necessarily mean providing access to the data but means providing information as to how to access the data from the data provider (if access is possible at all). For example, researchers could provide information about the specifics of the data use agreement they had in place with the data provider, the use-and-access criteria applied by the patient registry or an explanation of why access to the data to third parties is not feasible or allowed. Note that sharing is sometimes called on-sharing in the context of secondary use of data because, per definition, the data have already been shared before [24].

The importance of research transparency is already well acknowledged in the realm of clinical trials [25], as is evidenced by the large collection of authoritative guidance documents regarding registration (e.g. [26]; article III.L.1 of [27]; article 35 of [28]), reporting (e.g. [27,29,30]; article 36 of [28]) and sharing (e.g. article III.L.2 of [27]), as well as legal requirements and infrastructure such as clinical trial registries (e.g. https://clinicaltrials.gov, https://www.clinicaltrialsregister.eu).

While there is a large amount of guidance and infrastructure available from international organizations, journals, funders and research institutions to improve the transparency of clinical trials, the guidance and infrastructure in the area of SU/HD seem less developed. In recent years, some important repositories have taken root that aim to provide health data to researchers (e.g. OpenSafely, Clinical Research Practice Datalink, European Health Data Space). However, to our knowledge, there has been no assessment of peer-reviewed literature or institutional documents regarding guidance for improving the transparency of studies using such data. The current review involves such an assessment and includes a quantitative and qualitative analysis. In the quantitative analysis, we count the number of papers that include guidance and the prevalence of the different types of guidance provided. This allows us to identify areas of focus and expose gaps in the existing literature. In the qualitative analysis, we synthesize the content of the guidance, providing context about the robustness and potential impact of the recommendations. Together, these analyses provide a complete picture of the available guidance aimed at improving the transparency of SU/HD studies, making it possible to prioritize the development and adjustments of specific types of guidance in the future.

1.1. Terminology used in this study

1.1.1. Health data

According to the European Data Protection Supervisor [31] health data refer to personal information that relates to the health status of a person and includes medical data as well as administrative and financial information about health. Health data can stem from routine clinical processes as well as from patient-reported outcome (PRO) measures [32].

1.1.2. Secondary use of health data

Researchers have reported some confusion about what secondary use means (Joint Action Towards the European Health Data Space [33]). We follow the World Health Organization [34] by defining the secondary use of health data as the processing of health data for purposes other than the initial purposes for which the data were collected. Even though health data can have many secondary uses [35] we only focus on its use for biomedical research. A largely synonymous term that has gained traction in recent years is ‘real-world data’. Real-world data are typically used to refer to health data that are not derived from clinical trials but during routine clinical practice [36].

1.1.3. Transparency

We use transparency in the context of scientific research, by focusing on registration, reporting and sharing. However, transparency in the context of SU/HD is often also used to mean transparency with regard to the patient (i.e. whether the patient knows what happens with their personal data) [37,38]. The ethical and legal debate on whether patients should be informed about every secondary use project involving their patient data to decide whether to give their consent is not addressed in this paper (but see [11]).

2. Methods

The study design was registered on 20 July 2023 on the Open Science Framework at https://osf.io/7864h. The raw data and analysed data used in this study can be found at https://osf.io/2nup4. Note that we also preregistered an assessment of the transparency guidance on patient registry websites, but in hindsight we realized that providing guidance is not one of the main goals of such websites. As such, we do not present results from that part of the preregistration in this paper. Additionally, we decided not to pursue our preregistered analysis on dataset registration (i.e. publicly registering that and how one is using a specific dataset). We did so because we could not find any references to dataset registration in the first batch of about 30 documents and decided it would not be worth the extra coding effort to further pursue it. An overview of all deviations from our preregistration can be found at https://osf.io/m4ehx.

2.1. Sample selection

To find documents that potentially discuss transparency in SU/HD studies, we used the following search term combination on PubMed: (‘guidance’ OR ‘best practice*’ OR ‘guideline*’ OR ‘recommendation*’ OR ‘road map’ OR ‘position paper’) AND (‘secondary use’ OR ‘secondary data’ OR ‘reuse’ OR ‘database stud*’ OR ‘real-world data’ OR ‘real-world evidence’ OR ‘registry data’) AND (‘transparen*’ OR ‘registration’ OR ‘reporting’ OR ‘sharing’). This PubMed search retrieved 954 documents (see https://osf.io/bz9mr) on 20 July 2023, before our registration. We also did a search of these keyword combinations on Google Scholar, where we added the term ‘health’ to restrict our search to the secondary use of health data.

Prior to registration, we conducted the search process using Google Scholar. Unfortunately, search results for Google Scholar are not reproducible [39]. We decided to include Google Scholar despite the disadvantage of irreproducibility because it is the most comprehensive [40,41] and most used [42,43] source of scientific literature. Moreover, Google Scholar is especially useful for exploratory searches like ours [44]. To identify any missed documents that may be relevant, we used the snowball method and searched the references section of the documents included based on the initial screening.

We also wanted to include documents from health institutions with a relevance for SU/HD studies. Based on our own expertise and an overview provided by Burns et al. [45] we selected a set of (inter)national health institutions that had previously published transparency guidance in the context of clinical trials (as these institutions often extend their guidance frameworks to cover secondary use of health data), and a set of learned societies specifically revolving around SU/HD studies. We then looked on their websites for any documents that may conceivably include transparency guidance for SU/HD studies.

From all identified documents (both peer reviewed and institutional), we selected documents relevant to our research question (post-registration) in the following way. First, we screened the title and abstract (for documents found via Google Scholar and PubMed) or the title (for documents found via the snowball method) of a document and assessed whether the document was likely to contain guidance for any of the transparency aspects: registration, methods reporting, results reporting, data sharing and code sharing. We assessed that this would be the case for documents that state:

• —that they provide guidance for SU/HD studies on one or more transparency aspects;
• —that they provide general guidance for SU/HD studies;
• —that they discuss one or more transparency aspects in the context of SU/HD studies; and
• —that they discuss SU/HD studies generally.

In sum, our set of included documents involved documents that based on the title and/or abstract potentially included transparency guidance for SU/HD. A PRISMA flow diagram of our search and selection process can be found in electronic supplementary material, figure S1. The full set of peer-reviewed papers and institutional documents can be found at https://osf.io/ednwx and https://osf.io/gajxt, respectively.

2.2. Analysis

We employed both a qualitative as well as a quantitative approach. The qualitative approach consisted of a thematic analysis [46] using MAXQDA [47] in which we retrieved relevant sections from the peer-reviewed papers and institutional documents. In the first stage of coding, our approach was primarily deductive as we screened documents for text relevant to one of the transparency themes: registration, methods reporting, results reporting, data sharing and code sharing. We also extracted texts that highlighted the main goal of the documents, and any additional texts that we deemed potentially useful in writing our paper.

In the second, inductive stage of coding, we identified subthemes within the a priori selected themes. We went over all the extracted texts from a given theme and categorized each text based on content. This led to a more granular understanding of guidance in the main transparency areas.

The quantitative approach consisted of counting the number of documents that included one or more texts regarding each of the themes we used in the deductive stage, and all the subthemes identified in the inductive stage of our thematic analysis. We distinguished three ways in which the themes were included in a document. We defined a piece of text as a ‘call’ if the authors claimed that transparency should be improved in a particular area, without providing an argument for why this should be the case. We defined a piece of text as a ‘justification’ if the authors did make an argument supporting a statement for more transparency in a particular area. Finally, we defined a piece of text as a ‘recommendation’ when the authors made a recommendation for how to improve the transparency in a particular area. If a document contained multiple calls for a transparency practice, we coded this as one call because calls do not qualitatively differ from one another like justifications and recommendations do (e.g. there could be multiple justifications for why we need more registration of SU/SH studies). As such, a document could have a maximum of five calls, one for each of the transparency practices, but in theory unlimited justifications and recommendations.

In some cases, we initially coded a text excerpt as a general call because authors did not directly explain why more transparency would be beneficial. However, justifications were often provided earlier or later in the text. In those cases, we recoded the ‘call’ to a ‘justification’. Consequently, our dataset does not include documents with a call and a justification. Note that a justification does not automatically imply a call, and a recommendation does not automatically imply a justification or a call. For example, it could be that a paper states that a health organization has called for more registration and then provides recommendations, which does not mean that the authors in the paper call for more registration.

3. Results

Below we present the quantitative and qualitative results in narrative form. In addition, we present the quantitative results in tabular form in table 1, and visually in figure 1, and the qualitative results in tabular form in table 2 (registration), table 3 (reporting) and table 4 (sharing), and visually in figure 2 (justifications) and figure 3 (recommendations).

Table 1.

Total number of calls (C), justifications (J) and recommendations (R) in the peer-reviewed literature and institutional documents.

	peer-reviewed literature (n = 116)			institutional documents (n = 23)
	C	J	R	C	J	R
registration	4	24	112	2	1	55
methods reporting	3	24	147	1	1	19
results reporting	4	6	26	0	1	1
data sharing	2	9	1	1	0	1
code sharing	2	9	4	2	0	1
total	15	72	290	6	3	77

Figure 1.

Number of papers with at least one call, justification or recommendation regarding the five transparency elements in the peer-reviewed literature (top) and institutional documents (bottom).

Table 2.

Qualitative analysis of justifications and recommendations on registration.

	explanation	example quote
justifications
publication bias	registration can prevent or at least allows identification of publication bias because it provides a record of studies that are not published	’How can investigators presenting an RWE study dispel the suspicion that they ran multiple similar studies and analyses, but published only the one that gave a positive result? One answer is to adopt institutional and corporate policies on RWE studies that provide a similar level of rigor to policies on the conduct of RCTs. Such a policy may set the definition of RWE studies, mandate posting an outline protocol for each study on an appropriate forum’ [48]
questionable research practices	registration can prevent or at least allows identification of questionable research practices because registration allows third parties to compare the plans to the actual study	‘Because research plans and hypotheses are specified before the results are known, pre-registration reduces the potential for cognitive biases to lead to p-hacking, selective reporting, and HARK-ing’ [49]
informative conclusions	registration allows third parties to better assess the study and its results when the study has been completed	‘[T]ransparency improves the ability of decision-makers to assess the quality and validity of a study by giving them a deeper understanding of why and how the research was conducted and whether the results reflect pre-established questions and methods’ [7]
prior knowledge	registration makes clear what prior knowledge authors had about the dataset that could have biased their decisions	‘Bias resulting from retrospective selection can be serious, especially when selecting external data and key analysis features, when the external control results are already known. Pre-specification is therefore an essential pre-requisite when using external control data’ [50]
recommendations
deviations	transparently discuss any deviations made from the registration in the final paper	‘We recommend that researchers be transparent about their ex ante analytic plans, provide justification for subsequent changes in analytic models, and report out the results of their ex ante analytic plan as well as the results from its modifications’ [51]
what to register:
— basic information	provide basic information about the study, like the study rationale and the main hypotheses	‘Please provide the hypotheses of your secondary data analysis. Make sure they are specific and testable, and make it clear what your statistical framework is (e.g., Bayesian inference, NHST). In case your hypothesis is directional, do not forget to state the direction. Please also provide a rationale for each hypothesis’ [52]
— data source	provide information about the data source/study population	‘First, registration would require researchers to prespecify their data source(s) along with sample inclusion and exclusion criteria allowing preanalytic evaluation of whether the study is representative of the patient population using the medical product. For any data source, quality-control measures to ensure data integrity would be proactively described’ [53]
— prior knowledge	provide information about any knowledge the authors have about the data	‘To increase transparency about potential biases arising from knowledge of the data, researchers could routinely report all prior data access in a pre-registration. This would ideally include evidence from an independent gatekeeper (e.g., a data guardian of the study) stating whether data and relevant variables were accessed by each co-author’ [49]
— analysis plan	provide detailed information about the planned statistical analyses	‘Describe details of sensitivity analysis that will be performed to confirm the robustness of the results of analysis. Sensitivity analysis is especially important in pharmacoepidemiological studies with databases because the results of analysis tend to vary significantly depending on study design such as definition of exposure, outcome, covariates, etc. Describe all previously planned sensitivity analyses and ensure that these are differentiated from additional interim sensitivity analyses’ [54]

Table 3.

Qualitative analysis of justifications and recommendations on reporting.

	explanation	example
methods reporting
justifications
informative conclusions	methods reporting provides context with which third parties can interpret the drawn conclusions in a less biased manner	‘Data users should agree to make the methods and results of their secondary analyses publicly available not only through scientific publications (that may or may not be prepared and, if prepared, that may or may not be accepted for publication) but also by depositing them in a repository and making them discoverable. This will be important to provide further examples of effective data sharing and allow any conclusions from secondary use to be examined by others’ [55]
reproducibility	providing information about the methods/statistical analyses helps other researchers to redo the analysis using the same dataset to see whether the results are consistent	‘Because of the lack of standardization in secondary data analytics, complete transparency is critically important in the reporting of analytic approaches and all coding details. This will allow reproduction of analyses, replication of findings using different data sources, and ultimately greater confidence in such analyses, possibly approaching the trust we place in highly controlled clinical trials’ [56]
replicability	providing information about the methods/statistical analyses helps other researchers to redo the analysis using a different dataset to see whether the results are consistent	‘This system would enable regulators to repeat the exact same study and change assumptions or definitions in the design and statistical analysis either through submission of data or by providing access to the data’ [57]
recommendations
guidelines	use existing reporting guidelines to write the methods section of your research papers	‘Several guidelines have been developed to enhance reporting, such as Strengthening the Reporting of Observational Studies in Epidemiology (STROBE), the Reporting of studies Conducted using Observational Routinely-collected Data (RECORD) statement, and its extension for pharmacoepidemiology studies (RECORD-PE). Interested researchers should always consult these guidelines for reporting of their studies’ [58]
what to report:
— overview study design	provide a descriptive or visual overview of the basic design elements that make up your study	‘Reporting on overall study design should include a figure that contains 1st and 2nd order temporal anchors and depicts their relation to each other’ [59]
— data source/study population	provide any relevant details about where the data came from and how they were managed	‘Describe the nature of dataset(s) used. In particular: The purpose of the dataset—e.g. observational research registry, national audit programme, administrative dataset (linked to financial remuneration or service delivery). This should include details of the funding of the dataset, and the organization(s) responsible for the administration and oversight’ [60]
— prior knowledge	provide any prior knowledge about the data that the authors may have had before doing the statistical analyses	‘Authors should describe the extent to which the investigators had access to the database population used to create the study population’ [61]
— exposures/predictors	provide information about the factor, condition or intervention that may impact the health outcomes of interest	‘Reporting on exposure definition should include: The type of exposure that is captured or measured, e.g. drug versus procedure, new use, incident, prevalent, cumulative, time-varying’ [59]
— comparators/control sample	provide information about the groups or conditions against which the exposure group is compared to evaluate the impact of the exposure on the health outcome of interest	‘When compared with another TCM exposure/intervention, it is necessary to evaluate whether the evidence base for the efficacy and safety of the control is sufficient’ [62]
— confounders/covariates	provide information about the variables that are (potentially) associated with both the exposure and the health outcome of interest	‘Discuss the potential for confounding, both measured and unmeasured, and how this was assessed and addressed’ [51]
— outcomes	provide information about health-related endpoints or events of interest	‘Discuss how outcomes were measured and how classification bias was addressed’ [51]
— statistical analysis	provide detailed information about the statistical methods used to draw inferences about the variables of interest	‘Describe the methods used to evaluate whether the assumptions have been met’ [63]
results reporting
justifications
informative conclusions	effective reporting of results helps make more accurate conclusions of the results of a study	‘The results of all analyses that are conducted (e.g., matched, unmatched, adjusted, and unadjusted) results should be reported. Presentation of the unadjusted results helps to demonstrate the robustness of the chosen method of analysis; matched or adjusted results that differ substantially from the unmatched/unadjusted can reduce confidence in the matched/adjusted trends observed’ [64]
publication bias	also presenting non-statistically significant results helps to prevent the literature from being disproportionally filled with ‘positive’ studies	‘The lack of RW study protocol registration and reporting of results can potentially lead to significant bias in reporting positive/selective results, as studies that do not produce the expected data will probably not be completed or submitted for peer review. RW studies, regardless of the origin of RW data, need to be registered in a manner equivalent to that of clinical trials’ [65]
recommendations
patient characteristics	provide detailed information about the characteristics of the patients in the sample	‘The rationale for reporting on characteristics of the study population is described in numerous other reporting guidance documents. This includes items such as an attrition table (showing patient numbers as eligibility criteria are applied), baseline characteristics of the derived population, as well as the number and timing of outcomes of interest. It allows the investigator and reviewers to describe and assess whether the frequency of a derived variable is consistent with expectation (e.g., that the outcome incidence or a covariate prevalence looks approximately correct). The same rationale applies in studies that develop or use derived information from NLP and ML algorithms’ [66]
sensitivity analyses	provide more than one analysis so that others can assess the robustness of the results	‘Report univariate and multivariate results in an unbiased and complete fashion such that the benefits and risks of all comparators reflect “fair balance” ’ [67]
effect sizes	provide effect sizes alongside the statistical significance of the results	‘second, we recommend researchers report effect sizes as many associations may be statistically, but not practically, significant when analyzing large sample sizes. In doing so, we may need to adjust our collective expectations of what effect sizes to expect, and which ones to treat as substantial’ [68]
interpretation	provide a cautious discussion of the results in light of the research questions and/or hypotheses	‘Discuss the potential for confounding by indication, contraindication or disease severity or selection bias (healthy adherer/sick stopper) as alternative explanations for the study findings when relevant’ [63]

Table 4.

Qualitative analysis of justifications and recommendations on sharing.

	explanation	example quote
data sharing
justifications
informative conclusions	shared data, including metadata, allows third parties to make better assessments of the findings based on the data	‘The disclosure of research-related information is also fundamental to improving the transparency of studies, especially the availability of raw data, which enables readers to assess the authenticity and reliability of the findings’ [69]
computational reproducibility	data sharing (and code sharing) allows third parties to redo the analyses in the study to see whether the same results are found	‘For full analytic reproducibility, sharing of code and data is encouraged. However, there are often privacy and intellectual property considerations that prevent sharing of data, data derivatives, or code’ [66]
sensitivity analyses	data sharing allows third parties to do analyses that differ from the analyses in the original study, allowing an assessment of the reliability of the results	‘Ideally reviewers of submitted evidence, including HTA bodies or independent review groups, would also have access to the data and analytical code to ensure the replicability of the submitted results and assess the impact of alternative analytical decisions or data on the resulting estimate(s). However, there remain substantial governance, technical and practical challenges to sharing data, including a lack of in-house expertise in many HTA agencies’ [70]
code sharing
justifications
informative conclusions	shared code allows third parties to see the exact analyses that were conducted, allowing a better interpretation of the results	‘Finally, the analyses conducted on secondary data are commonly more complex than those applied to simpler experimental designs. Methods sections in high-impact journals are often highly condensed or hidden at the end of an article, which can make it difficult or even impossible to assess which analyses exactly were performed. To address this issue, we recommend authors always publish the full analytic code, even when the raw data cannot be directly shared’ [68]
computational reproducibility	shared code allows third parties to redo the analyses in the study to see whether the same results are found	‘Irreproducibility can be mitigated by sharing raw and processed data and codes, assuming no privacy is compromised in this process. For replicability, given that RWD are not generated from controlled trials and every data set may has its own unique data characteristics, complete replicability can be difficult or even infeasible. Nevertheless, detailed documentation of data characteristics and pre-processing, pre-registration of analysis procedures, and adherence to open science principles (e.g., code repositories) are critical for replicating findings on different RWD datasets, assuming they come from the same underlying population’ [71]

In contrast to tables 2 and 3, this table does not include recommendations because we could not find enough recommendations to do a thematical analysis.

Figure 2.

Overview of the justifications in peer-reviewed literature and institutional documents for improving the transparency in studies making secondary use of health data. Note that the sizes of the areas represent the number of times we found these justifications in the literature.

Figure 3.

Overview of the recommendations in peer-reviewed literature and institutional documents for improving the transparency in studies making secondary use of health data. Note that the sizes of the areas represent the number of times we found these recommendations in the literature.

Our sample of peer-reviewed literature (which was slightly different from our preregistered sample; see https://osf.io/m4ehx) that included guidance on transparency for SU/HD studies contained 116 papers (36 first found through Google Scholar, 57 first found through PubMed and 23 found through the snowball method by checking the 4745 references of the papers found through Google Scholar and PubMed). We extracted 606 text excerpts from the peer-reviewed papers. The quantitative data are summarized per paper at https://osf.io/z7hvg, including links to the papers and their numbers of citations so that readers can easily identify papers that might be relevant to them. An overview of all the texts we extracted from our analyses can be found at https://osf.io/ednwx.

Our sample of 21 institutional documents was slightly different from our preregistered sample; see https://osf.io/m4ehx. We extracted 130 text excerpts from these 21 documents. The quantitative data are summarized per paper at https://osf.io/rmvjp, including links to the documents. An overview of all the texts we extracted from our analyses can be found at https://osf.io/gajxt.

3.1. Guidance in the peer-reviewed literature

3.1.1. Registration

Among the 116 papers, we found four papers with a call for more registrations of SU/HD studies, 18 with a justification for more registrations of SU/HD studies (of which four had multiple justifications) and 19 with one or more recommendations on how to register SU/HD studies. In total, we found 112 recommendations, with three papers providing most (61/112) of those (25 in [52], 12 in [72] and 24 in [58]). These three papers involved structured templates that researchers can use to register SU/HD studies. We counted each individual item of these templates as a separate recommendation, but we excluded subitems. For example, for Wang et al. [59] we coded the reporting item ‘Methods used for confounder adjustment’ as a separate recommendation but did not code the subitems in which the method for confounder adjustment was discussed for different potential statistical analyses.

3.1.1.1. Justifications

The main justification for registering SU/HD studies was that registration would prevent questionable research practices like p-hacking, selective or retrospective reporting and HARKing (hypothesizing after the results are known). Several authors argued that the large scale [52,73] and widespread availability of health datasets [7,52] make analyses based on existing health data more susceptible to biases because of the large number of analysis options and possible prior knowledge of the data. Another commonly mentioned justification for registration was that it would allow someone to identify publication bias or potentially prevent publication bias. This is because an overview of all published and non-published studies allows the research community to identify non-published studies and request the results to be made available. The authors of one study claimed that publication bias might be more severe for SU/HD studies because journals may have less expertise in evaluating such studies [7]. Finally, a paper mentioned that registration could be helpful in drafting ethical review board submissions and informed consent forms [74] and justified more registrations of SU/SH studies on that ground.

3.1.1.2. Recommendations for researchers

Most often, recommendations on how to register SU/HD were about which study elements need to be specified in a registration. Many elements were mentioned but in the context of SU/HD most emphasis was placed on registering the data source and the statistical choices. Registering the authors’ prior knowledge of the data was emphasized strongly by Baldwin et al. [49], Orsini et al. [7] and Van den Akker et al. [52], but was not mentioned in two extensive papers presenting registration guidance [72,75]. Several papers simply provided a list of platforms where authors can preregister their SU/HD study, where references to clinicaltrials.gov and the electronic Register of Post-Authorization Studies were most common [7,74,76,77]. Finally, several papers stated that deviations from registrations should be transparently disclosed, preferably including the timing of and justification for the change [7,49,51,53,67,70,78].

3.1.1.3. Recommendations for institutions

We also found several recommendations in the literature where guidance was provided to institutions on how to improve the infrastructure surrounding registration. For example, Orsini et al. [7] discussed that embargoing registrations could preserve intellectual property and prevent scooping. Zarin et al. [73] and Wang et al. [72] focused more on registration templates, where the former argued for creating balanced templates that take into account both comprehensiveness (providing more information about the study) and simplicity (yielding higher uptake of registration of SU/HD studies), and the latter argued for the integration of templates with HD registries.

3.1.1.4. Limitations

Some authors also provided points of concern with relation to the registration of SU/HD studies, although these discussions were often limited. Orsini et al. [7] stated that registration does not guarantee high-quality studies, and Zarin et al. [73] argued that registration of SU/HD studies likely has limited impact unless any recommendations or policies can be legally enforced. Dhruva et al. [53] agreed with the point about enforcement and called for a mandate for registration of SU/HD studies as the 2004 ICMJE policy did for clinical trials [79].

3.1.2. Reporting

We found three papers with a call for methods reporting, four with a call for results reporting, 16 with a justification for methods reporting, six with a justification for results reporting, 32 with one or more recommendations for how to best report the methods of a SU/HD study and 13 with one or more recommendations for how to best report the results of a SU/HD study. In total, we found 147 recommendations for methods reporting, and 26 recommendations for results reporting. Five papers provided more than 10 recommendations (13 in [61]; 15 in [62]; 13 in [63]; 50 in [59]; 11 in [66]).

3.1.2.1. Justifications

Justifications for better methods reporting typically came in two shapes: (i) good reporting provides the information necessary to reproduce (i.e. redo the analyses on the same dataset) or replicate (i.e. do the same analyses on a different dataset) results from SU/HD studies [56,59,76,80] and (ii) good reporting makes it easier to assess the results of the study itself by other researchers or peer reviewers [48,55,60,74,81].

Several authors justified results reporting by claiming that it is a good way to provide information to patients, enhancing trust and facilitating informed decisions [65,82,83]. Others mentioned that reporting the results of all conducted analyses would decrease publication bias [48,65].

3.1.2.2. Recommendations for researchers

The main recommendations for improved methods reporting came in the shape of formal reporting guidelines, in which authors presented lists of relevant study elements that are important to include in research reports. Some papers presented a guideline themselves [59,61,63,84], while others merely advised to adhere to such guidelines [48,58,84–86]. The Reporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) [84] and Wang et al. [59] had a general goal to improve reporting in SU/HD studies. Other guidelines were more specific: RECORD-PE [63] is dedicated to improve reporting in pharmacoepidemiologic research, Patorno et al. [87] discussed RECORD in light of the reporting of diabetes research and Chai et al. [62] focused on traditional Chinese medicine.

Recommendations for results reporting were limited because many authors simply state that results should be reported and why, not necessarily how (though an elaborate list of different elements to consider can be found in the supplementary materials of [59]). Berger et al. [76] stated that not only medical journals can be used to report results, but also publicly available websites. Hersh et al. [88], Roche et al. [64] and Wang et al. [66] emphasized that sensitivity analyses can be useful when presenting results because many analysis options in secondary health data give rise to many different interpretations of the data.

3.1.2.3. Recommendations for institutions

We also found some institutional recommendations regarding reporting. For example, while the value of reporting guidelines was echoed by Khachfe et al. [89] they emphasized that such guidelines should be included in manuscript submission and editorial processes for them to be effective. In a similar way as Dhruva et al. [53] did for registration, they stated that the ICMJE could play a mandating role in this regard. In addition, Khachfe et al. [89] argued that more domain-specific checklists should be drafted, and such checklists could already be integrated into educational modules. Finally, Bate [90] mentioned that reporting guidelines for unstructured data like social media data are lacking and that meta-research on the impact of the guidelines is desirable.

3.1.2.4. Limitations

A limited number of concerns or critiques were raised. Both Kent et al. [70] and Orsini et al. [7] warn that adhering to reporting guidelines does not necessarily support reproduction, and that they are not necessarily a sign of high research quality.

3.1.3. Sharing

Guidance on sharing the data and sharing the code of SU/HD studies was sparse in the peer-reviewed literature. Two papers included a call for more data sharing and more code sharing, eight papers included a justification for more data sharing and seven papers for more or better code sharing. We found one recommendation in a paper with regard to data sharing [91], and four recommendations in one paper [80] with regard to code sharing.

3.1.3.1. Justifications

As justifications for data and code sharing, we found that it would allow computational reproducibility checks [68,71,72] and robustness checks [70,57]. Both have to do with redoing the analysis but the goal of computational reproducibility checks is to see whether one arrives at the same outcome using the same parameters as in the original analysis, and the goal of robustness checks is to see whether one arrives at the same outcome using slightly different parameters than in the original analysis. Some authors [77] also mentioned that access to data and code is necessary for third parties to redo the analysis on a different dataset.

3.1.3.2. Recommendations for researchers

Regarding code sharing, recommendations include a modular programming approach, where code is separated into independent and interchangeable modules, version control systems and a standardization of common analytical approaches [80]. Herrett et al. [92] point to code repositories for electronic health record research. In these repositories, users can share their methods (metadata, code) so that others can use it or modify it. An example of such a repository is the HDR UK Phenotype Library (https://phenotypes.healthdatagateway.org/).

3.1.3.3. Recommendations for institutions

Then et al. [24] would like to see that the conditions of data sharing are stated more clearly when data users make agreements with data providers. Kent et al. [70] discussed a broader conception of data sharing and argued that data and code availability would be useful not only for other researchers but also for health technology assessment (HTA) bodies or independent review groups.

3.1.3.4. Limitations

Most discussions about re-sharing data came with caveats, which could be technical, practical or moral. Many authors cautioned that data sharing is not always allowed because of privacy reasons embedded in data transfer agreements [59,66]. Many add that information like codebooks or verbal descriptions of the data is necessary for other researchers to effectively re-use the data [59,66]. To alleviate the concerns surrounding privacy and data transfer agreements, one paper [91] argues in favour of broad consent, where patients provide their consent not only for the original study but also for studies after that, or dynamic consent, where patients can make granular decisions about who can use their data and when.

3.2. Guidance in institutional documents

3.2.1. Registration

Of the 21 institutional documents, two included a call for more or better registration of SU/HD studies, one included a justification for more or better registration and seven included one or more recommendations. In total, we found 55 recommendations for the registration of SU/HD studies. The registration recommendations were mainly found in three papers: seven in [93]; 15 in [94]; and 24 in [54]. Because we found hardly any justifications, recommendations for institutions and limitations in our set of institutional documents, we only discuss the recommendations below.

3.2.2.1. Recommendations for researchers

The Japanese Pharmaceuticals and Medical Devices Agency (2014) established a ‘Committee for preparation of guidelines on conducting pharmacoepidemiological studies’ to provide a list of elements that would be good to include in registrations of SU/HD studies. Similarly, the German Society for Epidemiology (2008) based their list of to-register elements on a working group, the AGENS Working Group for the Survey and Utilization of Secondary Data, which in turn was strongly based on the Good Epidemiological Practice report, which has been available since 2000 and has seen many revisions, the most recent one in 2019 [95]. Other documents typically refer to external sources when making recommendations. For example, Health Canada [94] stated a list of elements that researchers would do well to register based on European Network of Centres for Pharmacoepidemiology and Pharmacovigilance [96] and International Society of Pharmacoepidemiology [97], and the Council for International Organizations of Medical Sciences [98] refers to Berger et al. [76]. The specific elements recommended to be included in registrations overlapped greatly with those specified in the peer-reviewed literature.

3.2.2. Reporting

For reporting, we found one call and one justification for methods reporting, and one justification for results reporting. We found 19 recommendations for methods reporting in seven documents, and just one recommendation for results reporting. We only discuss the recommendations in more detail below.

3.2.2.1. Recommendations for researchers

The Japanese Pharmaceuticals and Medical Devices Agency (2014) and the German Society for Epidemiology [93] provided several recommendations for methods reporting, most of which revolve around specifying the data source and summarizing the study design. We did not locate recommendations for results reporting aside from a listing of possible publication sites [98].

3.2.3. Sharing

Guidance for data sharing and code sharing was scarce, with only one call for data sharing, two calls for code sharing and one recommendation for both. We only discuss the recommendations in more detail below.

3.2.3.1. Recommendations for researchers

While the Council for International Organizations of Medical Sciences (2024) did call for more code sharing and Health Canada [94] called for more data sharing, they did not provide any recommendations on how to effectively engage in these practices. Only the German Society for Epidemiology (2008) came with some advice: data sharing should only take place with permission of the data owner, but code sharing could take place independently.

4. Conclusions and discussion

In this study, we found that 44 (out of 116) peer-reviewed papers and 12 (out of 21) institutional documents discussed transparency in studies that make secondary use of health data (SU/HD). The justifications for increased transparency included the prevention of questionable research practices, the facilitation of more informative conclusions and the enhancement of reproducibility and replicability. Recommendations for increased transparency were primarily presented in structured documents like registration templates and reporting guidelines. These documents provide guidance on the study elements to describe in registrations and research papers. For registrations, guidance documents are mainly recommended to provide detailed information about the data source, the planned statistical analyses and any prior knowledge about the data that authors may have. For methods reporting, guidance documents primarily recommended to provide detailed descriptions of the study design, data sources, the variables used in the analysis and the statistical analysis itself. Recommendations on results reporting highlighted the importance of presenting all conducted analyses, including non-significant results, providing detailed sample characteristics and running sensitivity analyses. Guidance was limited in the context of data and code sharing. Instead, practical and privacy concerns that could prevent sharing were noted frequently.

In the context of registration and reporting, we found that the available guidance often emphasized that communication about the data source is vital. Indeed, some authors argued that existing data, being more voluminous and accessible, provide more scope for researcher biases to creep in, and therefore a higher need for transparency. Based on these arguments, we believe that it would be better if discussions about the transparency of SU/HD studies would not be limited to methods reporting and results reporting, like is typically done for clinical trials, but would also involve a separate category of dataset reporting (i.e. providing detailed information about the data source, and the variables in the dataset). Formalizing these three separate categories of reporting could help researchers become more aware of the need to be transparent in all three of these areas.

As data and code sharing are crucial for reproducibility and transparency, the limited guidance in these areas represents a significant gap that needs to be addressed. In the case of data sharing, researchers may not know how to be transparent because the data owner may have placed restrictions on sharing, or it is unclear who is allowed to share and what is allowed to be shared. Further research should prioritize developing detailed and actionable guidelines for data and code sharing in SU/HD studies and how to align these steps with current consent processes.

When recommendations were provided for improving transparency, we often noted that little attention was given to the caveats or concerns related to these recommendations. This is surprising because improving the transparency of a research study is not as straightforward as it may seem. For example, drafting a transparent manuscript typically requires more time and effort than drafting a non-transparent one [99–101]. This can impact researchers at any career stage but is particularly relevant for early career researchers, who rely heavily on producing output quickly to attain their desired academic careers [102]. Future guidance on improving the transparency of SU/HD studies could discuss the benefits and costs of transparency. When researchers are more aware of the complexities of transparent practices upfront, they may be more likely to continue to engage with transparency practices in the future. If they are faced with challenges during or after improving the transparency of their papers, they may become disgruntled and steer clear of these practices from then on. Research into the day-to-day work processes of researchers may shed more light on this.

Finally, the majority of transparency guidance we found came from papers in the scientific literature. Guidance in institutional documents was relatively sparse, which is important to know because researchers may be more likely to turn to organizations like the WHO or the EMA for guidance than to the peer-reviewed literature. More institutional guidance would align with broader trends in biomedical research that underscore the importance of clear, reproducible and robust study methodologies to maintain public trust and scientific integrity. In March 2024, a new law came into force in Germany, the Health Data Utilization Act (Gesundheitsdatennutzungsgesetz, GDNG), which regulates the secondary use of health data and, in a separate paragraph, makes both the registration of the corresponding studies in WHO-recognized registers and results reporting mandatory (Bundesministerium für Gesundheit [103]). One month later, the European Parliament adopted the provisional agreement on the European Health Data Space (EHDS) Regulation [104]. The EHDS will provide researchers, innovators and industry with access to a large health dataset. These developments show that the secondary use of health data is becoming more and more embedded into the scientific ecosystem, highlighting the importance of guidance.

4.1. Limitations

Our review of institutional documents was less comprehensive compared to that of the peer-reviewed literature. While we aimed to include a representative sample of institutional guidance documents, the subjective selection process based on our own expertise may have inadvertently overlooked key documents from health organizations, regulatory bodies or other relevant entities. This limitation means that our findings might not fully capture the institutional perspective on transparency practices, potentially overlooking valuable insights and recommendations that could have influenced our conclusions.

Relatedly, one could argue that our distinction between peer-reviewed literature and institutional documents was somewhat arbitrary, potentially leading us to underestimate the availability of guidance in institutional documents. Indeed, some peer-reviewed papers involved initiatives or collaborations of formal organizations like the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) [59,76,105]. That being said, drawing a line between (more top-down) institutional initiatives and (more bottom-up) initiatives by researchers is hard as it is often difficult to assess the formality of scientific collaborations (e.g. in case of the RECORD initiative [61], the RECORD-PE initiative [63] and the STaRT-RWE initiative [72]). To allow readers to draw their own conclusions about our sample selection for both peer-reviewed literature and institutional documents, a full list of documents in our sample can be found at https://osf.io/ednwx (peer-reviewed papers) and https://osf.io/gajxt (institutional documents).

Another point to take into consideration when interpreting our results is the subjectivity of the decisions we made throughout the research process. One example lies in our choice of inclusion criteria for guidance documents. While carefully chosen, we might have missed important papers that use different terminology or focus on specific aspects of transparency not covered by our search strategy. Another example of subjectivity in our research choices lies in the nature of our thematic analysis and coding process. Because this process is inherently subjective, our prior knowledge and experience could have introduced bias in our judgements. Although we used established qualitative analysis methods to mitigate this risk, it remains a potential limitation that could affect the reliability of our results. All our quantitative and qualitative codes can be found on the OSF repository of this project: https://osf.io/2nup4. Readers interested in finding specific guidance on any of the transparency topics can find a quantitative summary of the analysed articles at https://osf.io/ap7e8, including links to the documents and the numbers of citations.

4.2. General conclusion

Our study highlights substantial efforts in the academic community to enhance transparency in SU/HD studies. To bridge the gap between peer-reviewed recommendations and institutional practices, health organizations could integrate the existing bottom-up initiatives into their formal guidelines. Future research could focus on developing standardized, enforceable guidelines for data and code sharing, while addressing practical and privacy concerns. Additionally, meta-research evaluating the implementation of transparency practices in SU/HD and the impact of transparency practices on research quality, health outcomes and public trust would be desirable.

Ethics

This work did not require ethical approval from a human subject or animal welfare committee.

Data accessibility

All our data are available on the OSF repository of our project at https://osf.io/2nup4. The preregistration is available at https://osf.io/7864h. A document with deviations from our preregistration is available at https://osf.io/2nup4.

Supplementary material is available online [106].

Declaration of AI use

We have not used AI-assisted technologies in creating this article.

Authors’ contributions

O.R.v.d.A.: conceptualization, data curation, formal analysis, investigation, methodology, project administration, writing—original draft; R.T.T.: methodology, writing—review and editing; J.P.A.I.: methodology, writing—review and editing; S.G.S.: methodology, writing—review and editing; D.S.: conceptualization, funding acquisition, methodology, supervision, writing—review and editing.

All authors gave final approval for publication and agreed to be held accountable for the work performed therein.

Conflict of interest declaration

We declare we have no competing interests.

Funding

This study is part of the project HiGHmed, which is funded by the Medical Informatics Initiative of the German Federal Ministry of Education and Research (funding number 01ZZ2302E).