Critically appraising the cass report: methodological flaws and unsupported claims

Abstract

Background

The Cass Review aimed to provide recommendations for the delivery of services for gender diverse children and young people in England. The final product of this project, the Cass report, relied on commissioned research output, including quantitative and qualitative primary research as well as seven systematic reviews, to inform its recommendations and conclusions.

Methods

We critically evaluated the Cass report and the research that was commissioned to inform it. To evaluate the Risk of Bias within the seven systematic reviews commissioned by the Cass Review, we applied the ROBIS tool – a domain-based assessment of risk of bias within systematic reviews. It focuses on four domains (i) study eligibility criteria, (ii) identification and selection of studies, (iii) data collection and study appraisal, and (iv) synthesis and findings. To maintain rigour, the ROBIS tool was applied to each systematic review by two independent assessors, within Covidence, with conflicts resolved by an additional two independent assessors. We also conducted a detailed critical evaluation of the methods used in the survey of gender services for young people in Europe, the two quantitative studies of health records, and the qualitative study on the experience of gender dysphoria among young people and the claims made in the Cass report based on these studies.

Results

Using the ROBIS tool, we identified a high risk of bias in each of the systematic reviews driven by unexplained protocol deviations, ambiguous eligibility criteria, inadequate study identification, and the failure to integrate consideration of these limitations into the conclusions derived from the evidence syntheses. We also identified methodological flaws and unsubstantiated claims in the primary research that suggest a double standard in the quality of evidence produced for the Cass report compared to quality appraisal in the systematic reviews.

Conclusions

We discuss these issues in relation to how evidence regarding gender affirming care is framed, the wider political context, and the future for gender affirming care. The Cass report’s recommendations, given its methodological flaws and misrepresentation of evidence, warrant critical scrutiny to ensure ethical and effective support for gender-diverse youth.

Background

“The Cass Review” [1] was commissioned by National Health Service (NHS) England and NHS Improvement “to make recommendations on the services provided to children and young people who are exploring their gender identity or experiencing gender incongruence” [2, 3]. The Cass Review (formally the “Independent Review of Gender Identity Services for Children and Young People”) and its interim components received widespread international attention throughout the four years during which the process took place (2020–2024), during which time the NHS closed its Gender Identity Development Services (GIDS) to be replaced with regional centres on the advice of the Cass Review’s 2022 Interim Report [4].

The final report (“the Cass report”) generated by the review was informed by (i) seven systematic reviews, (ii) a survey of gender services for young people in Europe, (iii) two quantitative studies of health records, and (iv) a qualitative study on the experience of gender dysphoria among young people. The commissioned survey of gender services [5] and reviews were published as open-access articles in the British Medical Journal’s (BMJ) Archives of Disease in Childhood [6–13]. The other primary research was published within the Cass report but has not yet been published in any peer-reviewed journal.

On March 12th, 2024, the first response to the final Cass report from the NHS was to change its interim service specification for specialist gender dysphoria services for children and young people to reflect “the NHS policy position that puberty suppressing hormones are not available to children and young people for gender incongruence / gender dysphoria because there is not sufficient evidence of safety and clinical effectiveness” [14]. Following this, the then Conservative government introduced an “emergency ban” on the prescription of puberty blockers to under-18 s on May 29th, 2024, just before parliament was dissolved ahead of a general election, citing the Cass report as justification. Since then, circumstances surrounding its implementation in the UK have evolved further. The health secretary in the subsequent Labour government indicated that the puberty blocker ban would be made permanent [15], and legal efforts to overturn it have failed [16]. In response to rising concerns and the ban of puberty blockers the British Medical Association (BMA) released a statement citing a pre-print of our methodological critique of the Cass report [17] and a report by the Integrity Project at Yale Law School [18]. The statement explained that, following a Council vote, the BMA now intends to undertake an independent evaluation of the Cass report with a special focus on the methodology that underpin it [19]. Despite concerns raised in the BMJ about this action [20], there appears to be an increasing desire in the UK and internationally to examine the Cass report and critically consider how it should be guiding policies for transgender healthcare.

In this paper, we report our critique of the methodologies used to synthesise and generate evidence to inform the Cass report. We then discuss how the Cass report draws several flawed conclusions from the evidence that it reviewed and commissioned. We also point to additional criticisms that have been made by other researchers regarding claims made in the Cass report. For an overview of the strength of evidence for gender affirming care (GAC) for young people, see Budge and colleagues [21].

Methods

We applied the Risk of Bias in Systematic Reviews tool [ROBIS] [22] to evaluate the risk of bias within the seven systematic reviews commissioned to inform the Cass report. The ROBIS is a domain-based assessment of risk of bias within systematic reviews, conducted in three phases. The optional first phase assesses the relevance and applicability of the review to the research question. Phase 2 focuses on four domains (i) study eligibility criteria, (ii) identification and selection of studies, (iii) data collection and study appraisal, and (iv) synthesis and findings. This phase focuses on identifying concerns in the review conduct. Each domain has a set of signalling questions to support the judgement within each domain of either low, high, or unclear risk of bias. Phase 3 involves an overall assessment of the risk of bias for the review, supported by a series of signalling questions. We followed the ROBIS guidance document available here which explains the considerations for making a judgment on the being a risk of bias for each domain and signalling questions to guide these judgments [23]. We applied Phase 2 and Phase 3 of the ROBIS to the assessment of the risk of bias within the seven systematic reviews commissioned to inform the Cass report, considering an assessment of relevance was not necessary because we did not have a research question that we were intending to answer using these systematic reviews – we simply wanted to evaluate their risk of bias systematically so that their evidential value as part of the Cass Review could be clarified. There was just one protocol that pertained to all seven of these reviews, which was published on PROSPERO [24]. We also reviewed this, and its revisions, and compared them to the published systematic reviews, to inform the ROBIS assessment.

To maintain rigour, the ROBIS tool was applied to each systematic review by two independent assessors, within Covidence [25]. Prior to the submission of these ROBIS assessments for consensus, the ROBIS assessors met online to reflect on and agree on the interpretation of each signalling question and ensure a consistent approach across all reviews. An additional two independent assessors resolved any conflicts and agreed the final decisions for each signalling question, leading to a judgment of low, high or unclear risk of bias for each domain, and an overall risk of bias judgment for each systematic review.

In addition, we undertook a review of the Cass report document, including the primary research reported within it that was not published in peer-reviewed journals. Specifically, we conducted a detailed critical evaluation of the methods used in the survey of gender services for young people in Europe, in the two quantitative studies of health records, as well as in the qualitative study on the experience of gender dysphoria among young people and the claims made in the Cass report based on these studies.

Results

For each of the four domains that comprise the ROBIS tool, our analyses resulted in all seven reviews being assigned an overall rating of a high risk of bias due to methodological limitations and a failure to adequately address these limitations in their interpretations and conclusions (see the Supplementary Material).

Domain 1: Study eligibility criteria

In this domain, all seven reviews [6–13] were considered at high risk of bias because, as we explain below, there are significant deviations from the protocol’s eligibility criteria, the eligibility criteria excluded study types relevant to the research questions, and the eligibility criteria are, in most cases, unacceptably ambiguous.

There are two unexplained deviations from the PROSPERO protocol [24] that guided our judgments for each of the systematic reviews in this domain. First, six of the seven reviews [6, 7, 10–13] excluded non-English sources and grey literature (including but not limited to dissertations, white papers, and government reports). Therefore, relevant studies may have been unreasonably excluded from these reviews.

Second, qualitative research was excluded from all of the reviews despite being part of the protocol’s inclusion criteria, and regardless of several of the reviews’ research questions having been investigated using rigorous qualitative methods e.g., [25–27].

Given these deviations, and others outlined in the following sections, when the protocol was updated in January 2023 to record the completion of screening against the eligibility criteria, the opportunity should have been taken to record and explain deviations from the original criteria. Best practice guidelines for systematic reviews require that such deviations are described and justified [28].

Finally, the review focused on social transition [6] excluded studies in which social transition was not treated “as an exposure”, and therefore excluded Olson et al. [29] and Rae et al. [30]. However, the authors included five other studies from the same ongoing project (the ‘TransYouth Project’) ignoring that the same design limitations would also apply to these studies. In particular, the authors may not have recognised that a finding of no difference in outcomes between the transgender group and the cisgender control group indicates something different from a finding of no difference between a socially-transitioned transgender group and a transgender group who were denied social transition. It is highly likely that this transgender group will differ significantly in terms of experiences, composition and presentation from a cisgender control group. For these reasons, we believe the authors were not justified in excluding Olson et al. [31] and Rae et al. [28] and that doing so is evidence of unacceptably ambiguous application of the exclusion criteria in this systematic review. See Table 1 for a summary of the ROBIS decisions for this domain.

Table 1.

Summary of ROBIS Evaluations for Domain 1: study eligibility criteria^a

Systematic Review	Did the review adhere to pre-defined objectives and eligibility criteria?	Were the eligibility criteria appropriate for the review question?	Were eligibility criteria unambiguous?	Were any restrictions in eligibility criteria based on study characteristics appropriate?	Were any restrictions in eligibility criteria based on sources of information appropriate?	Concerns regarding specification of study eligibility criteria
1	N	PY	PN	N	N	High
2	N	N	N	N	N	High
3	N	Y	N	Y	PY	High
4	PN	PY	PY	N	N	High
5	PY	Y	N	PY	N	High
6	N	N	PN	Y	PN	High
7	N	PN	N	PN	PN	High

^aFor ease of reading, we have assigned a number to each systematic review that was evaluated using the ROBIS as follows:

1: Impact of social transition in relation to gender for children and adolescents: a systematic review

2: Psychosocial support interventions for children and adolescents experiencing gender dysphoria or incongruence: a systematic review

3: Clinical guidelines for children and adolescents experiencing gender dysphoria or incongruence: a systematic review of guideline quality

4: Interventions to suppress puberty in adolescents experiencing gender dysphoria or incongruence: a systematic review

5: Characteristics of children and adolescents referred to specialist gender services: a systematic review

6: Masculinising and feminising hormone interventions for adolescents experiencing gender dysphoria or incongruence: a systematic review

7: Care pathways of children and adolescents referred to specialist gender services: a systematic review

Domain 2: Identification and selection of studies

A single search strategy was used, unmodified, for all the reviews. This calls into question the appropriateness of the search strategy and its applicability to each specific systematic review given their distinct research questions. Additionally, in six of the reviews, there is no evidence that the searches included grey literature [6, 7, 10–13]. For these reasons, all seven systematic reviews were considered at high risk of bias for this domain as the search strategy likely failed to identify all relevant studies [6–13]. See Table 2 for a summary of the ROBIS decisions for this domain.

Table 2.

Summary of ROBIS Evaluations for Domain 2: identification and selection of studies^a

	Did the search include an appropriate range of databases /electronic sources for published and unpublished reports?	Were methods additional to database searching used to identify relevant reports?	Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?	Were restrictions based on date, publication format, or language appropriate?	Were efforts made to minimise error in selection of studies?	Concerns regarding methods used to identify and/or select studies
1	N	Y	Y	N	Y	High
2	N	PY	PN	N	PY	High
3	Y	Y	Y	Y	Y	High
4	N	PY	PN	N	PY	High
5	N	N	Y	N	Y	High
6	PN	Y	PN	N	Y	High
7	PN	PY	PN	N	PN	High

^aFor ease of reading, we have assigned a number to each systematic review that was evaluated using the ROBIS as follows:

1: Impact of social transition in relation to gender for children and adolescents: a systematic review

2: Psychosocial support interventions for children and adolescents experiencing gender dysphoria or incongruence: a systematic review

3: Clinical guidelines for children and adolescents experiencing gender dysphoria or incongruence: a systematic review of guideline quality

4: Interventions to suppress puberty in adolescents experiencing gender dysphoria or incongruence: a systematic review

5: Characteristics of children and adolescents referred to specialist gender services: a systematic review

6: Masculinising and feminising hormone interventions for adolescents experiencing gender dysphoria or incongruence: a systematic review

7: Care pathways of children and adolescents referred to specialist gender services: a systematic review

Domain 3: Data collection and study appraisal

In this domain, we identified concerns across all of the systematic reviews due to the lack of clarity regarding whether the inclusion of studies for synthesis depended upon how well these studies were reported. None of the systematic reviews provided information as to how missing data was sought from study authors, or about problems with the application, or lack, of study appraisal.

There were several issues with how study appraisal was conducted in the systematic reviews on psychosocial support intervention [6], clinical guidelines [7, 30], social transition [5], puberty suppression [11], and HRT [12].

In the systematic review of psychosocial support interventions for children and adolescents experiencing gender dysphoria or incongruence [7], the Mixed Methods Appraisal Tool (MMAT; [29]) was used to appraise the quality of the primary studies. The reviewers categorised study quality as low, medium, or high. This is not recommended by the authors of the MMAT, who actively discourage the use of an overall score of quality [32].

In the review of clinical guidelines [8, 33], which was not described in the protocol, an appropriate study appraisal tool – the AGREE-II [34] – was used. However, its reliability in this context is questionable since several other systematic reviews have applied this tool to some of the same studies and arrived at quite different conclusions; some also criticised the usefulness of AGREE-II for GAC guidelines [35–37]. Notably, the most restrictive guidelines were rated highest in this systematic review, without a clear justification related to how these guidelines were more robustly developed [8, 33]. In reviews by other authors, it was the guidelines for more affirmative models of care that were judged to be of higher quality [35–37].

Three of the reviews [6, 12, 13] used an adapted version of the Newcastle–Ottawa Quality Assessment Scale (NOS; [38]) instead of the MMAT – a deviation from the protocol that was not explained or clearly reported. The NOS has been criticised [39] and the use of an adapted version negates previous attempts to validate the NOS. One of the systematic reviews [6] cites a paper by Stang and colleagues [39] to support their use of the NOS, despite this paper arguing against the use of the NOS for systematic reviews. These authors have previously noted this practice in other publications and called it out as a major quotation error [40]. The systematic review authors provide no rationale for the threshold scores used to categorise the quality of studies. It is generally accepted that this use of a single score is unacceptable in the assessment of risk of bias of individual studies within systematic reviews [41, 42]. Single scores do not capture the nuances of risk of bias, making them difficult to interpret. Additionally, the NOS is considered a quality appraisal scale, but within systematic reviews exploring the effectiveness of interventions it is recommended that a domain-based risk of bias assessment tool should be used instead of a quality appraisal tool [43]. The ROBINS-I is an example of a more suitable tool [44].

For these reasons, each systematic review was considered at high risk of bias in this domain, except for the review of clinical guidelines, which was considered at low risk of bias overall as there were only minor concerns with efforts to minimise errors in data collection and the tool used for study appraisal is widely regarded as appropriate. See Table 3 for a summary of the ROBIS decisions for this domain.

Table 3.

Summary of ROBIS Evaluations for Domain 3: data collection and study appraisal^a

	Were efforts made to minimise error in data collection?	Were sufficient study characteristics available for both review authors and readers to be able to interpret the results?	Were all relevant study results collected for use in the synthesis?	Was methodological quality formally assessed using appropriate criteria?	Were efforts made to minimise error in risk of quality assessment?	Concerns regarding methods used to collect data and appraise studies
1	NI	Y	Y	N	PY	High
2	PY	Y	PN	PN	Y	High
3	PY	Y	Y	PY	Y	Low
4	PY	PY	N	N	Y	High
5	PY	Y	PN	N	N	High
6	PY	Y	PN	N	Y	High
7	PY	Y	PY	N	N	High

^aFor ease of reading, we have assigned a number to each systematic review that was evaluated using the ROBIS as follows:

1: Impact of social transition in relation to gender for children and adolescents: a systematic review

2: Psychosocial support interventions for children and adolescents experiencing gender dysphoria or incongruence: a systematic review

3: Clinical guidelines for children and adolescents experiencing gender dysphoria or incongruence: a systematic review of guideline quality

4: Interventions to suppress puberty in adolescents experiencing gender dysphoria or incongruence: a systematic review

5: Characteristics of children and adolescents referred to specialist gender services: a systematic review

6: Masculinising and feminising hormone interventions for adolescents experiencing gender dysphoria or incongruence: a systematic review

7: Care pathways of children and adolescents referred to specialist gender services: a systematic review

Domain 4: Synthesis and findings

For two of the systematic reviews [12, 13], there were concerns about bias in how the evidence was synthesised because of the inappropriate exclusion of studies deemed to be “low quality” according to the adapted NOS. Using this approach, the authors excluded 48% and 36% of studies for puberty blockers and hormone replacement therapy (HRT), respectively. This practice is not recommended in systematic reviews unless explicitly pre-specified in a protocol with a clear and reasonable rationale [45], which was not evident in these cases. Instead, the narrative syntheses should have included all studies and integrated observations regarding study quality into the analyses, similar to how a sensitivity analysis would treat study quality in a meta-analysis [46]. In the reviews on the characteristics of children and young people referred for GAC [10] and on care pathways for this population [9], there was no assessment of study quality or risk of bias, so their conclusions could not take these issues into account, which is considered a source of bias for this domain.

In the synthesis section in the review on HRT [13], the authors conclude that low, or even moderate, quality evidence indicates that there is insufficient evidence for recommending GAC practices. For example, the authors argue that no conclusions can be drawn regarding any relationships between HRT and psychological health. However, this minimises their own data where one study showed an improvement in gender dysphoria [47], one showed an improvement in body satisfaction [48], four studies showed a reduction in depression-related outcomes [47–50], three studies showed an improvement in anxiety-related outcomes [47, 48, 50], and three studies indicated a clear decrease in self-harm and suicide attempts [49–51]. So, five distinct studies on 415 trans youth show positive psychological outcomes, and no study demonstrated “consistent” evidence for harm. To avoid reliance on vote-counting of these studies, and the limitations of this as a basic method of evidence synthesis, we can look to a recent meta-analysis that found benefits of HRT for young people experiencing gender dysphoria for several psychological outcomes including gender dysphoria, depression, and global function, though there was low certainty regarding the evidence [52]. Therefore, it appears that the body of evidence, despite its limitations, is at least suggestive of a benefit of HRT for mental health among adolescents experiencing gender dysphoria. At a minimum, the lack of evidence for harm resulting from carefully prescribed HRT should have been clearly identified by these authors. Had this been done, the authors might still have concluded that more research is required to elucidate the impacts of GAC, but it would have been less likely that their conclusions would have implied that GAC is harmful. See Table 4 for a summary of the ROBIS decisions for this domain.

Table 4.

Summary of ROBIS Evaluations for Domain 4: synthesis and findings^a

	Did the synthesis include all studies that it should?	Were all pre-defined analyses reported or departures explained?	Was the synthesis appropriate given the nature and similarity in the research questions, study designs and outcomes across included studies?	Was between-study variation (heterogeneity) minimal or addressed in the synthesis?	Were the findings robust, e.g. as demonstrated through funnel plot or sensitivity analyses?	Were biases in primary studies minimal or addressed in the synthesis?
1	Y	N	N	Y	N	N
2	Y	PN	N	Y	N	N
3	Y	N	Y	Y	PN	N
4	N	N	N	Y	N	N
5	NI	N	N	N	N	N
6	N	N	PY	Y	N	PN
7	PN	N	PY	PY	N	N

^aFor ease of reading, we have assigned a number to each systematic review that was evaluated using the ROBIS as follows:

1: Impact of social transition in relation to gender for children and adolescents: a systematic review

2: Psychosocial support interventions for children and adolescents experiencing gender dysphoria or incongruence: a systematic review

3: Clinical guidelines for children and adolescents experiencing gender dysphoria or incongruence: a systematic review of guideline quality

4: Interventions to suppress puberty in adolescents experiencing gender dysphoria or incongruence: a systematic review

5: Characteristics of children and adolescents referred to specialist gender services: a systematic review

6: Masculinising and feminising hormone interventions for adolescents experiencing gender dysphoria or incongruence: a systematic review

7: Care pathways of children and adolescents referred to specialist gender services: a systematic review

For more detail on why all the systematic reviews were considered at high risk of bias for the synthesis and findings domain, see Table 5.

Table 5.

Overall ROBIS Evaluations and Rationales^a

	Did the interpretation of findings address all of the concerns identified in Domains 1 to 4?	Was the relevance of identified studies to the review’s research question appropriately considered?	Did the reviewers avoid emphasising results on the basis of their statistical significance?	Risk of bias in the review	Rationale for concerns
1	N	N	Y	High	Areas of concern in each domain include: [D1] lack of detail in the protocol making it difficult to determine whether decisions regarding study eligibility were pre-specified or made post-hoc; non-English studies, grey literature and qualitative studies were excluded; studies in which social transition was not treated “as an exposure” were excluded, and therefore relevant studies related to the ‘Trans Youth Project’ were likely excluded; [D2] unmodified singular search strategy used across all seven reviews; no attempt to search for sources outside of traditional published articles indexed within the searched databases; [D3] use of an inappropriate tool (NOS) for study appraisal that deviates from the protocol without justification; [D4] no pre-defined analysis plan; studies inappropriately excluded from the synthesis that were assessed as “low risk of bias”, based on overall scores. Studies. Overall, there is a high risk of bias due to concerns identified in all four domains which were not addressed by the authors in the discussion, limitations or conclusions
2	N	PN	NI	High	Areas of concern in each domain include: [D1] lack of detail in the protocol making it difficult to determine whether decisions regarding study eligibility were pre-specified or made post-hoc; non-English studies, grey literature and qualitative studies were excluded; [D2] unmodified singular search strategy used across all seven reviews; no attempt to search for sources outside of traditional published articles indexed within the searched databases; [D3] inappropriate use of overall quality scores during study appraisal; [D4] no pre-defined analysis plan; insufficient synthesis methods used. Overall, there is a high risk of bias due to concerns identified in all four domains which were not addressed by the authors in the discussion, limitations or conclusions
3	N	Y	Y	High	Areas of concern in each domain include: [D1] lack of detail in the protocol making it difficult to determine whether decisions regarding study eligibility were pre-specified or made post-hoc; grey literature excluded; the selective inclusion of some non-English studies without justification, deviating from the protocol; [D2] unmodified singular search strategy used across all seven reviews; no attempt to search for sources outside of traditional published articles indexed within the searched databases; [D4] no pre-defined analysis plan; an apparent overemphasis on non-English studies in the discussion and final recommendations. Overall, there is a high risk of bias due to concerns identified in three of the four domains which were not addressed by the authors in the discussion, limitations or conclusions. Of particular concern is the apparent post-hoc decision to include non-English studies that were then overemphasised in the synthesis and recommendations
4	N	PN	NI	High	Areas of concern in each domain include: [D1] lack of detail in the protocol making it difficult to determine whether decisions regarding study eligibility were pre-specified or made post-hoc; non-English studies, grey literature and qualitative studies were excluded; [D2] unmodified singular search strategy used across all seven reviews; no attempt to search for sources outside of traditional published articles indexed within the searched databases; [D3] use of an inappropriate tool (NOS) for study appraisal that deviates from the protocol without justification; [D4] no pre-defined analysis plan; studies inappropriately excluded from the synthesis that were assessed as “low risk of bias”, based on overall scores. Overall, there is a high risk of bias due to concerns across all four domains which are not addressed by the authors in the discussion, limitations or conclusions
5	N	N	Y	High	Areas of concern in each domain include: [D1] lack of detail in the protocol making it difficult to determine whether decisions regarding study eligibility were pre-specified or made post-hoc; non-English studies, grey literature and qualitative studies were excluded; [D2] unmodified singular search strategy used across all seven reviews; no attempt to search for sources outside of traditional published articles indexed within the searched databases; [D3] no attempt to appraise the included studies which deviates from the protocol, without justification; [D4] no pre-defined analysis plan; bias not assessed and thus not addressed in the synthesis; insufficient synthesis. Overall, there is a high risk of bias due to concerns across all four domains which are not addressed by the authors in the discussion, limitations or conclusions
6	N	PN	PY	High	Areas of concern in each domain include: [D1] lack of detail in the protocol making it difficult to determine whether decisions regarding study eligibility were pre-specified or made post-hoc; non-English studies, grey literature and qualitative studies were excluded; [D2] unmodified singular search strategy used across all seven reviews; no attempt to search for sources outside of traditional published articles indexed within the searched databases; [D3] use of an inappropriate tool (NOS) for study appraisal that deviates from the protocol without justification; [D4] no pre-defined analysis plan; studies inappropriately excluded from the synthesis that were assessed as “low risk of bias”, based on overall scores; studies excluded from analysis dur to missing data (with no apparent attempt to source or address missing data). Overall, there is a high risk of bias due to concerns across all four domains which are not addressed by the authors in the discussion, limitations or conclusions
7	N	PN	Y	High	Areas of concern in each domain include: [D1] lack of detail in the protocol making it difficult to determine whether decisions regarding study eligibility were pre-specified or made post-hoc; non-English studies, grey literature and qualitative studies were excluded; [D2] unmodified singular search strategy used across all seven reviews; no attempt to search for sources outside of traditional published articles indexed within the searched databases; [D3] no attempt to appraise the included studies which deviates from the protocol, without justification; [D4] no pre-defined analysis plan; bias not assessed and thus not addressed in the synthesis; selective emphasis on studies; no apparent attempt to source missing data. Overall, there is a high risk of bias due to concerns across all four domains which are not addressed by the authors in the discussion, limitations or conclusions

^aFor ease of reading, we have assigned a number to each systematic review that was evaluated using the ROBIS as follows:

1: Impact of social transition in relation to gender for children and adolescents: a systematic review

2: Psychosocial support interventions for children and adolescents experiencing gender dysphoria or incongruence: a systematic review

3: Clinical guidelines for children and adolescents experiencing gender dysphoria or incongruence: a systematic review of guideline quality

4: Interventions to suppress puberty in adolescents experiencing gender dysphoria or incongruence: a systematic review

5: Characteristics of children and adolescents referred to specialist gender services: a systematic review

6: Masculinising and feminising hormone interventions for adolescents experiencing gender dysphoria or incongruence: a systematic review

7: Care pathways of children and adolescents referred to specialist gender services: a systematic review

The primary research and related claims in the cass review’s final report

In our critical evaluation of the Cass report, we found several instances of insufficiently evidenced claims being used to inform its recommendations. We observed serious methodological deficiencies with the primary research commissioned by the Cass Review and used to support several of the Cass report’s claims (see for example 10.70 and 10.71; 1 pp 146). We briefly describe select examples here (for more, see [18, 53]). These claims are selected as they pertain to key questions that were central to the motivation for commissioning the Cass Review [2].

In the cohort study (see appendix 5 of the Cass report), the authors aimed “to estimate for people aged 18 and under with gender dysphoria: changes in incidence and prevalence over time”. They did not account for changing acceptance, stigma, diagnostic criteria, and clinical coding available to GPs (i.e., from DSM-IV to DSM-V, ICD-10 to ICD-11; [54, 55]), or clinical guidelines, which may alter sampling over time and bias gender dysphoria prevalence estimates (see “chronology bias” or “surveillance bias”; [56, 57]). The authors fail to demonstrate that the observed increase is either unexpected or of concern, yet the assumption of both underpins the Cass Review and its commissioning. Regarding the co-occurrence of “Autistic Spectrum Disorder” (ASD) and gender dysphoria, the authors conclude that this has increased, without appropriate statistical tests (e.g., time trend analysis; [58]) or consideration of changes in the visibility and diagnosis of ASD, despite also warning of large confidence intervals. The authors also claim there was a two-phase growth in referrals for GAC, with an “acceleration” in 2015, without justifying this by statistically modelling the claim and comparing it to alternative models.

In an analysis of changes in patient profile, the Cass report states that “the exponential increase in numbers within a 5-year timeframe is very much faster than would be expected for the normal evolution of acceptance of a minority group” (1; pp 118). There are three problems with this argument. First, there is no such thing as a “normal”, passive “evolution” of acceptance toward a minority group, because discrimination against minority groups is calcified into de jure policy. Second, when that policy changes, exponential changes are well-precedented. The proportion of queer Americans that are married has increased exponentially since 2004, when the first state legalised gay marriage [59]. In 2015 alone, the proportion of same-sex couples that were married increased from 38 to 45% a mere two months after gay marriage was legalized nationwide [59]. The proportion of American adults who self-identify as queer has more than doubled in the last decade, representing an estimated increase of over 11 million people [60]. As such, the claim that increased social acceptance cannot explain increased referrals for gender-affirming care is ill-considered, as it neglects the complex relationship between increased social acceptance, referrals, and that any increase from near-zero will appear to be exponential [61]. Further, the authors made use of several flawed datasets and analyses, including double counting in referral data (see Fig. 11: Child and Adolescent Referrals for Gender Dysphoria (UK, GIDS), 2010/11 to 2021/22; 1; pp 85), referral trajectories that are over seven years old (see Fig. 15: Number of referrals over time by country; 1; pp 88), and personal communications without associated methodology (see Fig. 16: Referrals to the National Gender Clinic for children and young people in Norway; 1; pp 89).

The authors further argue that increasing social acceptance is not an adequate explanation for the current demographics of trans youth because it does not explain “the switch from birth-registered males to birth-registered females” is “unlike trans presentations in any prior historical period” (1; pp 26). There are two major problems with this assertion. First, existing scholarship has explained this “shift” within the context of increasing social acceptance: in “prior historical period[s]”, parents brought their children to gender clinics seeking to abolish or “fix” their gender non-conformity, and gender non-conformity was more stigmatised among boys than girls, but now, parents bring their children to gender clinics seeking to support their child’s transition—but gender non-conformity is still more stigmatised among boys and girls [62–64]. The paper that the Cass Review authors pull data from to make this point in Table 10 [65] actually makes this argument: “At present, the reasons for this shift in the sex ratio among adolescents are not clear, but may include less stigma for birth-assigned girls who are behaviourally masculine compared to birth-assigned boys who are behaviourally feminine, 4 which makes it easier to “come out” as transgender and to seek out mental health care and biomedical treatment (pg. 3). The second problem is that even ignoring that the “greater acceptance” theory can explain the “shift”, we would need to be confident that the evidence from “prior historical periods” was sufficiently representative of transgender youth to be comparable, which it was not [60], particularly under the guidelines for quality that the authors adopt.

On detransition, the authors claim that there is a “suggestion that the numbers are increasing” (1; pp 33), without providing evidence, whilst also failing to cite major studies on detransition (e.g., [66]).

On social transition, the authors state that “others consider that it makes it more likely that a child’s gender dysphoria, which might have resolved at puberty, has an altered trajectory potentially, culminating in life-long medical intervention” (1; pp 31). Despite their own review concluding that the small volume and low quality of current research makes it difficult to assess the impact of social transition [6], and that young people report reduced gender dysphoria and feeling more comfortable in themselves after socially transitioning (1; pp 159), the Cass report recommends partial rather than full transition for prepubertal children to prevent an altered “developmental trajectory” without evidence supporting this practice.

We also found inaccurate communication of participant quotes from the primary qualitative research to support the Cass report’s claims. For example, one participant is quoted as saying “there’s not only one route or one set way to transition or be trans. They might want just hormones, or just surgery, people are different with different experiences, presentations, and bodies. It’s fine for that to be the case, it’s okay to have different plans for your medical transition.” (1; pp 147). While it appears this participant is advocating for increased availability of medical care options for trans people, this quote directly informs 10.81, which suggests “it is important to inform people that medical transition is not the only option and that choosing not to go down that route does not invalidate their identity” (1; pp 147). The report then reframes this response as evidence for the need to reduce the number of medical transitions.

In general, in the qualitative research study reported in the Cass report, there is a lack of appropriate information about the overall methods, especially recruitment and data collection methods, and no discussion of reflexivity or positionality (where the researcher stands in relationship to those they are interviewing), which has consequences for the data interpretation and is an important indicator of rigour in qualitative research of healthcare [67].

Discussion

The Cass Review has had a significant impact on the delivery of GAC for young people in the UK, and on discourse regarding GAC internationally. However, as we and others have shown, the systematic reviews and primary research commissioned to inform its final report have significant limitations and the report itself contains several unsupported claims that undermine its recommendations. If the goal was to conduct a thorough overview of all extant knowledge on the subject, these limitations—apart from being incongruent with best practices in the absence of justification—obstruct that goal. These issues have not prevented proponents of the Cass report from praising its application of EBM whilst criticising existing literature as “substandard” [68]. This view is based upon an inappropriate use of a paternalistic lens, regarding GAC as quasi-psychiatric care, and upon an inappropriate methodological lens which downplays the value of high-quality observational data, issues that we discuss below as they are also apparent in the Cass report.

Beyond what is captured by our application of the ROBIS tool, there are several other ways in which these systematic reviews deviate from best practice. Of particular concern is the lack of a separate protocol detailing explicit, pre-specified methods for each review, resulting in a lack of transparency and reproducibility. The consequent pattern of deviations from this singular protocol’s plan for quality assessment across the reviews focusing on interventions is particularly striking. It is notable that the combination of using the NOS instead of the MMAT, altering how it is scored, and then excluding evidence on the basis of this altered score only applied to the reviews covering arguably the three most controversial topics that the Cass report addressed—puberty blockers, HRT, and social transition. The fact that these decisions were deviations from the protocol, and that justifications for them were not provided, raises concerns about cherry-picking.

Another type of evidence that the protocol indicated would be included in the reviews, but which was subsequently excluded, is qualitative research. Given that the Cass report presents qualitative data (e.g., anecdotes, quotations, community claims, and its own qualitative study) as evidence and purports to value stakeholder input, the fact that qualitative studies were excluded not only impedes the comprehensiveness of the overall project but could be considered evidence of a double standard towards the commissioned and synthesised evidence that informed the Cass report. The lack of consideration for how rigorous qualitative studies are designed, combined with the aforementioned exclusion of qualitative research from the systematic reviews, and the use of single, and sometimes misrepresented, quotes from participants to support much broader conclusions in the Cass report, demonstrates a misunderstanding of qualitative evidence and of the valuable insights about GAC that have been generated through qualitative research (see [69]).

Furthermore, the protocol does not adhere to the PRISMA-P guidelines [70] by failing to document changes to the protocol or how they would be recorded (item 4), specifying ambiguous eligibility criteria (item 8), failing to disclose how the search strategy was developed and the expertise of the searcher (item 10), omitting a description of whether or how information missing from included studies would be obtained (item 11), and did not discuss how meta-biases (e.g., publication bias, outcome reporting bias) may have affected the reviews (item 16). Ideally, an adequately detailed protocol should have been written for each individual systematic review and submitted for peer review.

Another deviation from best practice concerns the composition of the review team. Both the Cochrane Handbook [43] and the Institute of Medicine [71] recommend including content area experts on the review team. The systematic review team did include “a former Clinical Psychologist at the Tavistock Gender Identity Development Service” [24]. Despite this, there is still a distinct lack of content expertise among the authors on many of the issues examined by the systematic reviews. Ideally, there would also be input to systematic reviews from those affected by the topic; there is evidence that this practice is becoming common [72].

Finally, it is considered best practice by Cochrane [73] to rerun searches that are more than 12 months old and to screen the results for eligibility. The search for these systematic reviews was conducted in May 2021 and updated in April 2022 and was therefore 24 months old on publication. This means that the most recent eligible research is not included in the analyses for these reviews. Not only were these systematic reviews out of date and conducted in a manner that is likely to have biased their conclusions, but their necessity is also questionable in some cases. Several previous studies had addressed similar research questions regarding puberty suppression and HRT, for example, and by systematically reviewing the same evidence, they came to more positive conclusions regarding their usefulness for gender diverse children and young people [74–76].

Despite these deviations from best practice in evidence synthesis, the Cass report is positioned as being grounded in the principles of Evidence-based Medicine (EBM). As part of this argument, the Cass report sets out that randomised controlled trials (RCTs) are the gold standard to assess the efficacy of gender-affirming care (GAC), leading to the implication that the “research protocol” mandated for those accessing puberty blockers will be an RCT (e.g., 1 pp 177). Indeed, the benefit of RCTs lies in their high internal validity, achieved through the randomisation process which reduces biases related to confounding factors [77]. However, the external validity of RCTs is often criticised [78], both regarding generalisability and applicability to real-life settings [79], and within EBM, it is recognised as counterproductive to rely solely on RCTs, at the expense of other potentially valuable sources of data [78]. In transgender medicine, individuals willingly recruited to an RCT are likely not representative of the broader population, and the homogeneous treatments in an RCT do not easily translate to individualised care in clinical practice [80]. Furthermore, the coercive nature of access to puberty blockers potentially being contingent on consenting to participation in research is also of great ethical concern.

There are also concerns with the internal validity of an RCT in this context. The causal agent in an RCT comes from the contrast between treatment and control groups to determine a treatment’s effect [79]. However, in GAC, blinding is impossible due to the obvious effects of puberty blockers or HRT, likely causing control group participants to feel resentful demoralisation [80]. This can bias responses of the controls, or lead them to self-destructive behaviours, shifting the causal link from the treatment effect to the knowledge of group assignment. Differential attrition is also likely, as youths with supportive families, better socioeconomic status, or living in areas with a better availability of GAC may leave the study if in the control group, or not participate at all [80].

Additionally, HRT takes time to show effects [81] and various interventions may be needed at different times based on individual needs. This requires long-term follow-up in RCTs. However, the benefits of randomisation diminish over time, leading to biases similar to observational studies [77]. This would be especially true in transgender medicine, where affirmed youths’ life trajectories differ from those without access to GAC, due to the experience of living in their authentic gender and differential exposure to discrimination. Participants might also access other types of GAC at different rates, adding more confounding factors. Thus, over the necessary duration to assess the efficacy of HRT on wellbeing, the groups would likely diverge enough to lose the benefits of randomisation, reducing the RCT’s internal validity to that of a well-conducted observational study (which would not present the same ethical issues).

Finally, and perhaps more fundamentally, evaluating the efficacy of GAC based on psychosocial well-being alone is misguided. The primary goal of GAC is to prevent or induce the appearance of certain physical characteristics, and their physiological efficacy is undisputed. Mental health benefits are a logical consequence of living authentically [82] and we noted previously that studies included in the Cass review found positive effects of HRT on gender dysphoria, depression, and anxiety [47–50]. However, advocating for RCTs with mental health outcomes frames transness as a quasi-psychiatric condition, a distress to be alleviated by the most evidenced-based methods. This contradicts the depathologisation of transness (e.g., the ICD-11 moving Gender Incongruence from the “Mental and behavioural disorders” chapter to the new “Conditions related to sexual health” chapter [83] and its recognition as an issue of bodily autonomy and human rights [84, 85]. Improved well-being does not come from the physiological action of hormones, which might be adequately isolated by an RCT, but from a combination of factors contributing to increased congruence. Proposing RCTs with a mental health outcome thus shows, at best, a profound misunderstanding of transness and a naïve understanding of RCTs always being the gold standard in EBM.

Recognising and supporting the authenticity and competence of transgender young people is an important aspect of the provision of high-quality care [86]. However, the Cass report emphasises their distress, rather than their treatment wishes: the report describes them as “children with gender dysphoria and/or gender-related distress” (1 pp 52) and then emphasises the resolution of this distress as the main goal of interventions. Further, we contend that the commentary on developmental trajectories frames early social transition and detransition through a pathologising lens that leaves little room for the possibility that the formation of gender identity is non-linear and may be experienced positively [87]. Framed in this way, GAC becomes one of several treatment options for a quasi-psychiatric condition, rather than the authentic preference of competent individuals (note that Gillick competence is still applicable, without special limits, to under-16 s seeking GAC in England and Wales, after the Court of Appeal quashed the High Court judgement that set restrictions specifically for GAC; Bell v. Tavistock, 2021). Given that transgender people have a care need rather than a disease and seek actualisation of their identities as opposed to a cure, this paternalistic lens is inappropriate and pathologising [82]. Moreover, such a lens is also generally inappropriate in psychiatric care, where patient autonomy should be supported wherever possible [88].

GAC should instead be considered through a similar lens as reproductive healthcare, akin to how healthcare providers and the public think about contraception, HRT, or fertility treatment [82]. Reproductive care requires not just the absence of illness, but “a state of physical, emotional, mental, and social well-being in relation to all aspects of sexuality and reproduction” [89]. All individuals have the right to make decisions regarding their own reproductive care and must have access to services that support that right. Having a young person with gender dysphoria undergo their natal puberty is not a neutral or desirable act just because it is a natural occurrence, in the same way that continuing an unwanted pregnancy or having intrusive menopausal symptoms should not be considered the default option. Aside from supporting self-determination, intervention in such situations is also required to support emotional, mental and social wellbeing. The presence of treatment side effects, and the possibility that competent individuals may later seek to change their trajectory, are not unique to GAC and should not be used to justify exceptionalism when compared to reproductive healthcare. By failing to use a reproductive healthcare lens, the Cass Review risks creating an environment where non-affirming alternatives can be undertaken contrary to competent patients’ wishes, where unethical controlled studies can be performed, and in which the utility of observational and cohort studies is downplayed [80].

The future application of EBM in Gender Affirming Healthcare

The Cass report’s editorial argues the importance of EBM in supporting clinicians with the everyday concerns and unknowns of practice. Central to EBM are the three pillars: best available evidence, the values and preferences of those accessing care, and clinical expertise. It is therefore helpful to consider what the best available evidence could look like, in an approach tailored to the context of GAC; the view and preferences of gender diverse children and young people, and those who support them; and the clinical expertise of healthcare providers who deliver GAC.

The consideration of values and preferences have been historically absent in trans care—with its continued exclusion rooted in a legacy of pathologisation. Even among other cohorts that remain pathologised, efforts are increasingly made to value “experts by experience”, including in the development of clinical guidelines [90]. Effective co-production requires community involvement at every stage, not just superficially, as is common in NHS England initiatives [91]. In a good example of this, Ziegler led two reviews of clinical practice guidelines for adults and for children in primary care for which the broader team included both members of the trans community and primary care GAC providers [36, 37].

In their response to criticisms following the Review, the Cass team attempted to justify their criticism of cohort studies: “the same level of rigour should be expected when looking at the best treatment approaches for this population as for any other population so as not to perpetuate the disadvantaged position this group have been placed in when looking for information on treatment options” [92]. The Cass team fail to mention that the majority of strong treatment recommendations in healthcare are based upon low or very low-quality evidence [93] or that, as elaborated above, cohort studies may be best suited for producing evidence that can best inform claims about GAC.

The Cass report does not consider all the best available evidence regarding GAC for children and young people. It applies generic standards of evidence rather than considering what is the best possible evidence in this context given methodological, practical, and ethical constraints. It is our view that the best possible evidence regarding GAC is produced when the engagement and trust of participants is maximised through community involvement and clear communication [94, 95] when these participants are diverse and followed longitudinally in rigorous observational designs [80, 96], and when the outcomes measured are those considered important by gender diverse children and young people, and those who support them, including the clinical expertise of healthcare providers who deliver GAC, using culturally appropriate and valid measures [97–99]. These recommendations are consistent with the methodological standards for validity, generalisability, and patient-centredness set out by the Patient-Centred Outcomes Research Institute [100] and capture the three pillars of EBM.

Strengths and Limitations

The Cass Review has been conducted amidst a fraught political context characterised by repeated controversies around the social and legal rights of trans people. This risks skewing the way in which appropriate healthcare can be delivered to this vulnerable group. Our analysis was undertaken in this climate, with an intent to approach a review of the report from a methodological perspective. Accusations of ideologically driven viewpoints on trans healthcare are highly prevalent in the UK and internationally. While many of our authors have been publicly critical of the Cass report and some have significant experience in working with trans youth in different capacities and/or are trans themselves, our use of a validated tool, the ROBIS tool [22] is a strength of our critique as it allowed us to take a systematic approach to analysing the risk of bias and potential methodological flaws of the systematic reviews that were the foundation of the Cass Review. We deemed the ROBIS tool as the most suitable tool for this critical appraisal, as it is an effective tool to assess risk of bias within systematic reviews [101], it is supported by Cochrane and recommended in Overview of Reviews [102] and has a wide application across systematic review topics and approaches [22]. The AMSTAR-2 [103] was also considered but it is only suitable for reviews of effectiveness including RCT’s and non-RCT’s and the systematic reviews in question vary in their approaches [101]. Our critique could have been strengthened through greater access to the detail regarding the justifications for methodological, analytical, and interpretative decisions throughout the process of the Cass Review.

Conclusions

We have demonstrated that the Cass report’s application of EBM to GAC for children and young people is deeply flawed. Our critical analysis reveals significant methodological problems in the commissioned systematic reviews and primary research that undermine the validity of the Cass report’s recommendations. During our review of the report and supplementary primary research, we found insufficient statistical rigor, unreliable datasets, claims presented without evidence, and misrepresentation of quotes from primary research participants. These flaws highlight a potential double standard present throughout the review and its subsequent recommendations, where evidence for gender-affirming care is held to a higher standard than the evidence used to support many of the report’s recommendations. Considering this, and the Cass report’s poor understanding of transgender identities and experiences, it is vital to question the integrity and validity of the Review’s recommendations and the appropriateness of basing health policy on them. To uphold its commitment to evidence-based medicine, future gender-affirming care research must generate robust observational data, involve transgender communities, and prioritise patient-centred outcomes, ensuring validity, generalisability, and cultural relevance.

Abbreviations

AGREE-II

Appraisal of Guidelines for Research and Evaluation II

AMSTAR

A MeaSurement Tool to Assess Systematic Reviews

ASD

Autistic Spectrum Disorder

BMJ

British Medical Journal

EBM

Evidence-Based Medicine

GAC

Gender Affirming Care

GIDS

Gender Identity Development Services

HRT

Hormone Replacement Therapy

MMAT

Mixed Methods Appraisal Tool

NOS

Newcastle-Ottawa Scale

NHS

National Health Service

PRISMA-P

Preferred Reporting Items for Systematic review and Meta-Analysis Protocols

RCTs

Randomised Controlled Trials

ROBINS-I

Risk of Bias in Non-Randomized Studies—of Interventions

ROBIS

Risk of Bias in Systematic Reviews

Authors’ contributions

According to the CRediT taxonomy, C.N. led project administration, conceptualisation, methodology, investigation, data curation, formal analysis, visualization, and resources. A.S., D.C., D.M.G., É.Q., E.K., J.P., J.G., M.O., N.K., Q.M.L., R.B.G., S.C., T.E.W., and V.S. contributed to conceptualization, data curation, formal analysis, investigation, methodology, writing, and reviewing the manuscript. A.A., D.S., F.A., J.H., R.H., M.W. contributed to writing and reviewing the manuscript.

Funding

Not applicable.

Data availability

Data is provided within the manuscript or supplementary information files.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

Maxence Ouafik declares unpaid volunteering with TransKids Belgium, membership of WPATH (with no specific role inside the organisation) and being provider of gender affirming care. Ryan Biskupovic Goulding and Sibéal Coll are Committee Members & Co-Leads of the Medical Advocacy Subcommittee for Trans Healthcare Action (Ireland). Alex Ashman declares unpaid volunteering with Royal College of Surgeons of England Pride in Surgery Forum. Jo Hartland is an LGBTQ + activist associated with GLADD via the “UK Medical School charter on so-called conversion therapy”. Dr Hartland holds no formal role with GLADD. They are not part of the board or membership but have been associated in the media. Chris Noone is an unpaid board member of the National LGBT Federation (Ireland). Ryan Biskupovic Goulding, Sibéal Coll, John Gilmore, and Chris Noone are members of the Professional Association for Transgender Healthcare in Ireland. None of the organisations listed here had a role in authoring of this manuscript. We know of no other interests to declare. The views of the individual authors do not necessarily represent the views of the organisations by whom they are employed or otherwise associated.