Search for resonances decaying to an anomalous jet and a Higgs boson in proton–proton collisions at $\sqrt{s}=13\text{Te}\text{V}$

Abstract

This paper presents a search for new physics through the process where a massive particle, X, decays into a Higgs boson and a second particle, Y. The Higgs boson subsequently decays into a bottom quark–antiquark pair, which is reconstructed as a single large-radius jet. The decay products of Yare also assumed to produce a single large-radius jet. The identification of the Yparticle is enhanced by computing the anomaly score of its candidate jet using an autoencoder, which measures deviations from typical quark- or gluon-induced jets. This allows a simultaneous search for multiple Ydecay scenarios within a single analysis. In the main benchmark process, Yis a scalar particle that decays into a Wboson pair. Two other scalar Ydecay processes are also considered as benchmarks: decays to a light quark–antiquark pair, and decays to a top quark–antiquark pair. A fourth benchmark process considers Yas a hadronically decaying top quark, arising from the decay of a vector-like quark into a top quark and a Higgs boson. Data recorded by the CMS experiment at a center-of-mass energy of 13$\text{Te}\text{V}$ in 2016–2018, corresponding to an integrated luminosity of 138${\text{fb}}^{-1}$, are analyzed. The search covers Xmasses between 1.4 and 3.0$\text{Te}\text{V}$ and Ymasses between 90 and 400$\text{Ge}\text{V}$, with all simulated signals produced in the narrow-width approximation. No significant excess above the standard model background expectation is observed. The most stringent upper limits to date are placed on benchmark signal cross sections for various masses of X and Y particles

Introduction

The standard model (SM) of particle physics [1–3] is an exceptionally successful theory, demonstrating remarkable agreement across the whole range of experimental observations. Despite its success, there are indications that point toward the existence of physics beyond the SM (BSM). These include, among others, the unknown nature of dark matter [4], and the insufficient violation of the combined charge conjugation and parity symmetry to explain the observed baryon asymmetry of the universe [5].

To address such questions, a multitude of BSM theoretical models, such as supersymmetry [6, 7], large or warped extra dimensions [8–10], and those incorporating additional scalar or gauge particles [11] have been proposed. A common feature of these theoretical models is the prediction of new particles. Therefore, one avenue of exploration for BSM physics involves resonance searches, which aim to reconstruct the four-momentum of a new particle by analyzing its decay products. Recent advancements in machine-learning techniques have further motivated the development of inclusive, model-independent searches capable of probing multiple decay scenarios without reliance on specific assumptions [12]. Signatures involving the Higgs boson are well motivated in BSM searches, as new particles may preferentially couple to it [13, 14].

We search for processes of the form $\text{X}\to \text{HY},$ where Xis a heavy BSM resonance, His the 125$\text{Ge}\text{V}$ Higgs boson, and Ycan be either an SM or a BSM particle. We focus on the case where Hdecays to a bottom quark–antiquark pair ($\text{b}\overline{\text{b}}$), its most probable decay mode, and Ygives rise to a hadronic jet. The analysis targets the regime where Xis sufficiently massive that both Hand Yare produced with large Lorentz boosts, causing their decay products to be collimated and reconstructed as large-radius jets clustered with a distance parameter of 0.8 [15, 16]. An illustration of the process of interest is shown in Fig. 1. The analysis uses proton-proton ($\text{p}$) collision data at $\sqrt{s}=13\text{Te}\text{V}$ collected by the CMS experiment at the CERN LHC in 2016–2018, and corresponding to an integrated luminosity of 138${\text{fb}}^{-1}$.

Fig. 1.

An illustration showing the signal targeted by this analysis. The final state consists of a large-radius jet originating from Hdecaying to $\text{b}\overline{\text{b}}$and another large-radius jet originating from the decay of a second particle, Y

We evaluate the results in four benchmark signal scenarios. In the main scenario, Yis a scalar particle decaying to a VVpair, where V(Wor Zboson) decays hadronically. We choose the $\text{WW}$ decay as the proxy for this process; however, similar sensitivity is expected also for the $\text{ZZ}$ or $\text{WZ}$ decays. The decay allows for one or both vector bosons to be off-shell. Two additional scalar particle decay modes are considered, with Ydecaying either into a light-flavored quark–antiquark pair ($\text{q}\overline{\text{q}}$) or a top quark–antiquark pair ($\text{t}\overline{\text{t}}$), in which both top quarks decay fully hadronically. In the fourth scenario, Yis a top quark decaying fully hadronically. These four scenarios span a range of jet substructures, enabling the evaluation of the analysis performance across diverse signal signatures. The first three benchmark scenarios are motivated by theoretical models, such as the next-to-minimal supersymmetric extension of the SM (NMSSM) [17], in which cascade decays involving more than one new particle become possible. Reference [13] provides an overview of CMS searches for the $\text{pp}\to S^{\prime }\to \text{H}+\text{S}$ process, where ${\text{S}}^{\prime }$ and Srepresent new scalar particles and Sis assumed to decay to $\text{b}\overline{\text{b}}$. However, depending on the NMSSM parameters, other Sdecay modes may be favored, motivating searches targeting alternative Sdecays. The presented analysis thus complements Ref. [13] by exploring benchmark models with different Sdecay modes. The fourth signal scenario is motivated by models predicting the existence of vector-like quarks [18].

This work builds on a generic dijet anomaly search performed by the CMS Collaboration [19], but it specifies one particle and its decay mode ($\text{H}\to \text{b}\overline{\text{b}}$). This enables the application of more specific event selection criteria, enhancing the sensitivity of the analysis, while retaining a degree of generality in the characterization of the second particle.

The ParticleNet  [20] jet tagging algorithm, is applied to Hcandidate jets to assign scores indicating consistency with the decay of a massive resonance into $\text{b}\overline{\text{b}}$. The ParticleNet score is used to define “Pass” and “Fail” analysis regions. An autoencoder (AE) [19] is applied to Ycandidate jets to assign anomaly scores, with higher scores indicating deviation from jets typical of multijet production via quantum chromodynamics (QCD). This approach enables the identification of jets with unusual substructure, potentially arising from BSM decays, without relying on specific signal models. The anomaly score is used to define measurement (MR) and control (CR) analysis regions. Thus, the analysis uses four orthogonal regions, with the search performed in MR Pass. The specific selection criteria for these regions are defined in Sect. 4.

The search is carried out in a two-dimensional (2D) plane defined by the mass of the Ycandidate jet, $M_{\text{j}}^{\text{Y}}$, and the dijet invariant mass of the Yand Hcandidate jets, $M_{\text{jj}}$. For a signal process, the two variables correspond to the masses of Yand X, denoted as $M_{\text{Y}}$ and $M_{\text{X}}$, respectively. As a result, the analysis searches for a localized excess of events in the $M_{\text{jj}}$-$M_{\text{j}}^{\text{Y}}$ plane.

The paper is organized as follows. After the description of the CMS detector and event reconstruction in Sect. 2, we describe signal and background processes in Sect. 3. The event selection is discussed in Sect. 4, followed by the description of the statistical model in Sect. 5 and systematic uncertainties in Sect. 6. The results are presented in Sect. 7, followed by a summary of the paper in Sect. 8.

The CMS detector and event reconstruction

The CMS apparatus [21] is a multipurpose, nearly hermetic detector, designed to trigger on [22–24] and identify electrons, muons, photons, and charged and neutral hadrons [25–27]. Its central feature is a superconducting solenoid of 6$\text{m}$ internal diameter, providing a magnetic field of 3.8$\text{T}$. Within the solenoid volume are a silicon pixel and strip tracker, a lead tungstate crystal electromagnetic calorimeter (ECAL), and a brass and scintillator hadron calorimeter (HCAL), each composed of a barrel and two endcap sections. Forward calorimeters extend the pseudorapidity $(\eta )$ coverage provided by the barrel and endcap detectors. Muons are reconstructed using gas-ionization detectors embedded in the steel flux-return yoke outside the solenoid. More detailed descriptions of the CMS detector, together with a definition of the coordinate system used and the relevant kinematic variables, can be found in Refs. [21, 28].

Events of interest are selected using a two-tiered trigger system. The first level, composed of custom hardware processors, uses information from the calorimeters and muon detectors to select events at a rate of around 100$\text{kHz}$ within a fixed latency of about 4$\mu \text{s}$  [22]. The second level, known as the high-level trigger, consists of a farm of processors running a version of the full event reconstruction software optimized for fast processing, and reduces the event rate to a few kHz before data storage [23, 24].

A particle-flow algorithm [29] aims to reconstruct and identify each individual particle in an event, with an optimized combination of information from the various elements of the CMS detector. The energy of photons is obtained from the ECAL measurement. The energy of electrons is determined from a combination of the electron momentum at the primary interaction vertex as determined by the tracker, the energy of the corresponding ECAL cluster, and the energy sum of all bremsstrahlung photons spatially compatible with originating from the electron track. The energy of muons is obtained from the curvature of the corresponding track. The energy of charged hadrons is determined from a combination of their momentum measured in the tracker and the matching ECAL and HCAL energy deposits, corrected for the response function of the calorimeters to hadronic showers. Finally, the energy of neutral hadrons is obtained from the corresponding corrected ECAL and HCAL energies.

For each event, hadronic jets are clustered from these reconstructed particles using the infrared- and collinear-safe anti-$k_{\text{T}}$ algorithm [15, 16] with a radius parameter of either 0.4 (AK4 jets) or 0.8 (AK8 jets). Jet momentum is determined as the vectorial sum of all particle momenta in the jet, and is found from simulation to be, on average, within 5–10% of the true momentum over the whole transverse momentum ($p_{\text{T}}$) spectrum and detector acceptance. Additional $\text{p}$ interactions within the same or nearby bunch crossings (pileup) can contribute additional tracks and calorimetric energy depositions to the jet momentum. The pileup-per-particle identification algorithm [30, 31] is used to mitigate the effect of pileup at the reconstructed particle level, making use of local shape information, event pileup properties, and tracking information. A local shape variable is defined, which distinguishes between collinear and soft diffuse distributions of other particles surrounding the particle under consideration. The former is attributed to particles originating from the hard scatter and the latter to particles originating from pileup interactions. Charged particles identified to be originating from pileup vertices are discarded. For each neutral particle, this variable is computed using the surrounding charged particles compatible with the primary interaction vertex within the tracker acceptance $(|\eta |<2.5),$ and using both charged and neutral particles in the region outside of the tracker coverage. The momenta of the neutral particles are then rescaled according to their probability to originate from the primary interaction vertex deduced from the local shape variable, superseding the need for jet-based pileup corrections [30].

For AK8 jets, masses are computed after applying grooming [32] techniques, which remove soft, wide-angle and collinear radiation from the jets, in order to mitigate the effects of contamination from initial-state radiation, the underlying event, and multiple hadron scattering. The trimming algorithm [33] uses a subjet size parameter of 0.3 and a radiation fraction parameter $z=0.1,$ which determines the minimum $p_{\text{T}}$ fraction that the reclustered jet constituents need to have in order not to be removed. The mass of the resultant jet is referred to as its “trimmed mass”. The “soft-drop mass” of the jet is obtained as follows. The constituents of the AK8 jets are reclustered using the Cambridge–Aachen algorithm [34, 35]. Then, the “modified mass drop tagger” algorithm [36, 37], also known as the “soft drop” algorithm, is applied. The algorithm is applied with angular exponent $\beta =0,$ soft cutoff threshold $z_{\text{cut}}<0.1,$ and characteristic radius $R_0=0.8$ [38], which corresponds to the AK8 jet clustering distance parameter.

Signal and background processes

The main benchmark decay mode in this paper is $\text{X}\to \text{HY}\to \text{b}\overline{\text{b}}{\text{W}}^{+}{\text{W}}^{-}$, where Xand Yare scalar particles, and where both Wbosons decay hadronically. It is simulated at leading order (LO) using the MadGraph 5_amc@nlo 2.6.5 [39] event generator and the NMSSM model [17, 40]. The samples are generated for 42 different combinations of $M_{\text{X}}$ and $M_{\text{Y}}$ values, with $M_{\text{X}}$ ranging from 1400 to 3000$\text{Ge}\text{V}$, and $M_{\text{Y}}$ ranging from 90 to 400$\text{Ge}\text{V}$. The range of $M_{\text{X}}$ is chosen to ensure full trigger efficiency at low mass and sufficient background event yields for the background estimation procedure at high mass, while the $M_{\text{Y}}$ values are selected to maintain validity of the Lorentz-boosted regime across the $M_{\text{X}}$ spectrum. Two other scalar decay modes, $\text{Y}\to \text{q}\overline{\text{q}}$ (with $\text{u}\overline{\text{u}}$ used as a proxy) and $\text{Y}\to \text{t}\overline{\text{t}},$ are also simulated at LO with the NMSSM model using MadGraph 5_amc@nlo 2.6.5, each with a fixed $M_{\text{Y}}$ (200 and 400$\text{Ge}\text{V}$ , respectively) and the same range of $M_{\text{X}}$ values as the main benchmark. The above scalar benchmark processes are modeled assuming production via gluon-gluon fusion. The $\text{Y}\to \text{bq}{\overline{\text{q}}}^{\prime }$ decay mode is also considered, motivated by models predicting vector-like quarks [18]. In this scenario, Xis identified as a vector-like quark and Yas the top quark. Here, the anomalous jet originates from the fully hadronic decay of the top quark, providing an example that such jets do not necessarily have to come from a BSM particle. This scenario is simulated at LO for six different $M_{\text{X}}$ values with $M_{\text{Y}}$ set to the mass of the top quark. In the rest of the paper, we differentiate between the signal models by specifying the decay mode of Y. All generated signal samples assume the narrow-width approximation for Xand Y.

The SM background consists of $\text{t}\overline{\text{t}}$production, QCD multijet events, $\text{W}+\text{jets}$ and $\text{Z}+\text{jets}$ processes, and Hproduction, with and without an associated vector boson.

The $\text{t}\overline{\text{t}}$events in all-hadronic or lepton(s)+jets final states are modeled using powheg 2.0 [41–44], at next-to-LO (NLO). The simulated $\text{t}\overline{\text{t}}$event yields are normalized using a cross section of 834$\text{pb}$, calculated at next-to-NLO (NNLO) in QCD with soft gluon resummation at next-to-next-to-leading logarithmic precision [45]. The theoretical uncertainty on this cross section is approximately 5%, combining scale, parton distribution function (PDF), and top mass uncertainties in quadrature.

The QCD multijet, $\text{W}+\text{jets}$, and $\text{Z}+\text{jets}$ event samples are simulated at LO using the MadGraph 5_amc@nlo event generator. These backgrounds have nonresonant distributions in the analysis observables and are thus jointly estimated using data from the sidebands. Their simulated samples are only used to develop the tools for the analysis. The gluon-gluon fusion Hboson production process is simulated using the MiNLO event generator [46]. The powheg generator [47, 48] is used to model Hproduction via vector boson fusion and associated production with a Wor Zboson at NLO accuracy.

All samples are generated using the NNPDF3.1 [49] NNLO PDFs from the lhapdf6 PDF library [50]. The showering and hadronization of partons are simulated with pythia 8.240 [51] and the CP5 underlying event tune [52].

All generated events are processed through a simulation of the CMS detector based on Geant4  [53]. The effects of pileup are modeled assuming a total inelastic $\text{p}$ cross section of 69.2$\text{mb}$ [54] by overlapping simulated minimum bias events with the hard-scattering process. All simulated event samples are weighted to match the distribution of pileup vertices observed in the data.

Event selection

The online selection uses triggers based on significant jet activity. Different trigger thresholds were used in 2016 (2017 and 2018) as follows. One trigger criterion required a single AK8 jet with $p_{\text{T}}>450$ (500)$\text{Ge}\text{V}$. A second trigger required that $H_{\text{T}}$, the scalar sum of the $p_{\text{T}}$ of all AK4 jets with $p_{\text{T}}>30\text{Ge}\text{V}$ and $|\eta |<2.5,$ must be greater than 900 (1050)$\text{Ge}\text{V}$. Additionally, in 2017–2018, a third trigger algorithm required an AK8 jet with a trimmed mass $>30$ $\text{Ge}\text{V}$ along with $p_{\text{T}}>400\text{Ge}\text{V}.$ The combined logical OR of all the triggers is used. The trigger selection is fully efficient with respect to the offline selection.

The offline selection requires the two leading (in $p_{\text{T}}$) AK8 jets to have $p_{\text{T}}>300\text{Ge}\text{V}$ and $|\eta |<2.4.$ A requirement is also placed on the invariant mass of the two jets, $M_{\text{jj}}>1300\text{Ge}\text{V},$ where $M_{\text{jj}}$ is computed from the jet four-vectors, including the jet mass. A selection is also placed on the difference in $\eta$ between the jets, $|\Delta {\eta }_{\text{jj}}|<1.3,$ in order to further suppress the QCD multijet background, which is characterized by larger jet separation. Finally, the Hcandidate jet, defined below, is required to have its soft-drop mass in the 100–150$\text{Ge}\text{V}$ range.

The ParticleNet algorithm [55] is used to discriminate AK8 jets consistent with the decay of a massive particle into $\text{b}\overline{\text{b}}$against the background from jets originating from QCD multijet production. A mass-decorrelated version of ParticleNet  [56] is used to ensure that background-enriched regions can be defined using jets with low ParticleNet scores while maintaining the same background mass distributions in the regions passing the ParticleNet selection. The jet with the higher ParticleNet score is considered to be the Hcandidate jet and the other to be the Ycandidate jet. The soft-drop mass of the Ycandidate jet ($M_{\text{j}}^{\text{Y}}$) is used as a second observable alongside $M_{\text{jj}}$.

An AE neural network [57] is used to quantify the anomaly score of the Ycandidate jet. An AE consists of two parts. The first part, the “encoder”, compresses an input into a latent representation of reduced dimensionality. The second part, the “decoder”, reverses this process and attempts to reconstruct the original input from the reduced representation. We choose to represent jets as $32\times 32$ pixel images [58], covering a region of $\eta$–$\varphi$ space with width $\Delta \eta =\Delta \varphi =1.2$ centered on the jet axis, where $\varphi$ is the azimuthal angle in radians. The intensity of each pixel is determined by the sum of energies from the constituent particles within the given $\eta$–$\varphi$ region. We define the anomaly score as the mean squared difference between the pixel intensities of the input image and the reconstructed output. This AE architecture was previously used in Ref. [19] as part of the “Tag N’ Train” method (as discussed in Appendix A.2 of Ref. [19]). The training is performed on simulated QCD jets, allowing the AE to compress these jets with minimal information loss. However, for non-QCD jets (from signal, $\text{t}\overline{\text{t}}$production, etc.), which exhibit different substructure patterns, this procedure leads to poor compression and reconstruction of the original jet. This enables effective discrimination of non-QCD jets.

Each event is categorized into one of four groups based on the ParticleNet score of the Hcandidate jet (pass or fail) and the anomaly score of the Ycandidate jet (control region or measurement region): CR Pass, CR Fail, MR Pass, and MR Fail. The CR is used to validate the background estimation method, described below, whereas the MR is used to extract the signal. The Fail regions are used to help estimate the shape of the QCD multijet background in their corresponding Pass regions.

A ParticleNet working point corresponding to a $\approx 60\%$ signal efficiency (defined as the fraction of signal jets passing the selection), as measured on the main benchmark with $M_{\text{X}}=2000\text{Ge}\text{V}$ and $M_{\text{Y}}=250\text{Ge}\text{V},$ and a $\approx 0.5\%$ mistag rate (defined as the fraction of multijet background jets passing the selection) defines the boundary between the Pass and Fail tagging categories.

An anomaly score working point with $\approx 85\%$ signal efficiency and 30% mistag rate on the same benchmark defines the MR region. The CR consists of jets that fail the anomaly score MR working point. A lower bound on the anomaly score is applied so that the CR remains similar to the MR, which helps with the validation of the background estimation method. As a result, about 10% of signal and 30% of QCD jets are assigned to the CR.

The ParticleNet efficiency for correctly identifying $\text{H}\to \text{b}\overline{\text{b}}$ jets is calibrated in observed data using a sample of jets originating from fragmentation of a gluon to $\text{b}\overline{\text{b}}$, which are similar to $\text{H}\to \text{b}\overline{\text{b}}$ jets. Such jets are selected from the observed data using a boosted decision tree classifier, such that their ensemble ParticleNet score resembles that of $\text{H}\to \text{b}\overline{\text{b}}$ jets [59]. A systematic uncertainty is assigned to account for the sensitivity of the results to changes in the boosted decision tree requirement applied to the jets. The data-to-simulation correction factors for the ParticleNet signal selection efficiency range from 0.93–1.05, with uncertainties up to 15%, depending on the jet $p_{\text{T}}$ and data-taking period. The ParticleNet efficiency for misidentifying jets from the $\text{t}\overline{\text{t}}$background as $\text{H}\to \text{b}\overline{\text{b}}$ jets is constrained directly in the fit to data by allowing the yield in the Pass and Fail regions to vary while keeping the total yield constant.

The data-to-simulation scale factors for the efficiency of the anomaly score selection are calculated using the Lund plane reweighting method [60], separately for each signal process and for the $\text{t}\overline{\text{t}}$background. The method involves applying per-jet event weights based on their distribution in the Lund plane, a 2D representation defined by the logarithms of splitting angle and $p_{\text{T}}$. These weights are used to adjust the simulated radiation pattern so that the density in the Lund plane matches that observed in the data. It was found that the shapes of the two observables, $M_{\text{j}}^{\text{Y}}$ and $M_{\text{jj}}$, do not change significantly with the application of per-jet weights. This allows applying an overall correction to the anomaly tagging efficiency instead of per-jet event reweighting. The corrections are calculated as the ratio of the anomaly tagging efficiency before and after per-jet reweighting. They are found to be compatible with unity, with 10–20% uncertainty.

Statistical model

The analysis searches for a localized signal in the 2D $M_{\text{jj}}$-$M_{\text{j}}^{\text{Y}}$ plane. The $M_{\text{jj}}$ ($M_{\text{j}}^{\text{Y}}$) axis spans the 1300–3000 (40–500)$\text{Ge}\text{V}$ range. For higher masses, the bin widths are increased in order to ensure sufficient population of each bin. As a result, all the bins in the MR Fail have 8 or more observed events. This helps to reduce the per-bin statistical uncertainties in the QCD multijet background in the Pass region. The $M_{\text{jj}}$-$M_{\text{j}}^{\text{Y}}$ distributions for the background and signal predictions are simultaneously fitted to data in either the MR Pass and MR Fail, or CR Pass and CR Fail. The contributions from different data-taking periods are added together in data and simulation to construct 2D histogram templates. The templates for the $\text{t}\overline{\text{t}}$and Hproduction background processes, and for the signal benchmarks, are obtained from simulation. A pass-to-fail ratio method, described in the following paragraphs, is used to model the nonresonant background.

The pass-to-fail ratio method is based on the ratio of $M_{\text{jj}}$-$M_{\text{j}}^{\text{Y}}$ distributions between ParticleNet Pass and Fail regions: $R_{\mathrm{P}/\mathrm{F}}(M_{\text{jj}},M_{\text{j}}^{\text{Y}})$. It is defined through the relation:

$$\begin{array}{c}\hfill n_{\text{P,nr}}(i)=n_{\text{F,nr}}(i)R_{\mathrm{P}/\mathrm{F}}(M_{\text{jj}},M_{\text{j}}^{\text{Y}}),\end{array}$$ 1

where $n_{\mathrm{F}(\mathrm{P}),\mathrm{nr}}(i)$ is the number of nonresonant events in bin i of the $M_{\text{jj}}$-$M_{\text{j}}^{\text{Y}}$ distribution in the Fail (Pass) region. Since the Fail region is dominated by nonresonant events, $n_{\text{F,nr}}(i)$ can be directly estimated from data by subtracting the relatively small simulated yields of other processes. The $R_{\mathrm{P}/\mathrm{F}}$  is a priori unknown and is determined during the fit of signal and background distributions to the data. It is modeled as a polynomial in $M_{\text{jj}}$ and $M_{\text{j}}^{\text{Y}}$ of order n and is defined as a linear combination of $M_{\text{jj}}^a{\text{M}}_{\text{jY}}^b$ terms where the exponents a and b satisfy $a+b\le n.$ Simulation shows no strong dependence of the ParticleNet mistag rate in either of the two observables, whereas the mistag rate of anomaly tagging is observed to rise with jet mass in the simulation. This makes ParticleNet tagging the preferred method for defining the Pass and Fail regions for background estimation, ensuring that polynomials with low n will be able to well describe $R_{\mathrm{P}/\mathrm{F}}$. A Fisher’s F-test [61] is used to determine the minimum polynomial order necessary and sufficient for the model. Starting from polynomials of order zero, the F-test is used to determine if the next higher order provides a significant improvement in the fit quality. The $R_{\mathrm{P}/\mathrm{F}}$  polynomial orders in the CR and MR are discussed in Sect. 7.

Systematic uncertainties, described in Sect. 6, are included in the statistical model by generating template histograms representing one standard deviation variations from the nominal simulation. These variations correspond to individual sources of uncertainty. Each is assigned a nuisance parameter in the likelihood function used for statistical interpretation, using a unit Gaussian distribution for shape uncertainties, and a log-normal distribution for normalization uncertainties [62]. These distributions change the simulated template shape according to the corresponding template uncertainty histograms.

Systematic uncertainties

Several sources of systematic uncertainty are considered, affecting both the shapes and yields of the signal and background templates. Data-to-simulation scale factors for the ParticleNet signal and Hboson background efficiency are calculated with the method discussed in Sect. 4. They have approximately 10% uncertainty. For the $\text{t}\overline{\text{t}}$background, the mistag rate correction is directly determined during the fit using the Pass and Fail regions. The scale factors for signal and $\text{t}\overline{\text{t}}$anomaly tagging efficiency, also discussed in Sect. 4, have uncertainties of about 15–30%. For the Hboson background, a 20% uncertainty is assigned to the anomaly tagging efficiency.

Initial- and final-state radiation uncertainties are evaluated separately by varying the renormalization scale used in the parton shower by factors of 0.5 and 2. Measured uncertainties on jet energy scale and resolution are propagated to the jets used in the analysis in order to evaluate their effects. Scale and resolution uncertainties are considered uncorrelated.

The PDF uncertainties are evaluated as the square root of differences of squared PDF Hessian eigenvector sets using NNPDF3.1 PDF sets. The uncertainties coming from the choice of renormalization and factorization scales are evaluated by separately varying each of the scales while keeping the other fixed to the nominal value, and also by scaling both by factors of 0.5 and 2.0. The effect of these variations is found to be negligible for signal acceptance and thus only the $\text{t}\overline{\text{t}}$and Hboson backgrounds are affected.

A 1.6% uncertainty in the total integrated luminosity is applied [63–65]. During the 2016–2017 data taking, a gradual shift in the timing of the inputs of the ECAL hardware level trigger in the region of $|\eta |>2$ caused a specific trigger inefficiency. Event-level weight corrections and their uncertainties are included in the simulation to account for this. A pileup correction is applied to simulated events based on the true number of interactions in the event. The corresponding uncertainty is generated by varying the inelastic $\text{p}$ cross section at 13$\text{Te}\text{V}$ (69.2$\text{mb}$) by 4.6% [54]. Finally, a 5% normalization uncertainty is applied to the $\text{t}\overline{\text{t}}$cross section.

The impact of each uncertainty source on the fitted signal strength parameter is estimated by varying the corresponding nuisance parameter within its postfit uncertainty and refitting the signal strength, using the $M_{\text{X}}=2200\text{Ge}\text{V}$ and $M_{\text{Y}}=250\text{Ge}\text{V}$ simulated signal sample as a representative test case. The uncertainties with the largest impacts, ranging from 5% to 8%, are associated with the efficiency of tagging anomalous jets, the efficiency of tagging $\text{H}\to \text{b}\overline{\text{b}}$ jets, parton showering, and the renormalization and factorization scales. Other uncertainties, such as those related to the jet energy scale and resolution, integrated luminosity, and pileup, have smaller effects, altering the signal strength by 3% or less.

Overall, the sensitivity of the analysis is statistically limited, due to the limited amount of the observed data.

Results

Maximum likelihood fits are performed with the CMS statistical analysis tool Combine [62] to estimate the background and extract the fitted signal. The fits are performed separately in the CR and MR. The F-test favors an $R_{\mathrm{P}/\mathrm{F}}$ polynomial order of $n=0(2)$ in the CR (MR) fit. Both fits pass the goodness-of-fit test, performed using the saturated model [66]. The p-value [67], indicating the level of agreement with the background-only hypothesis and the observed data, is found to be 0.59 (0.26) in the CR (MR) fit. The postfit distributions from the CR and MR fits are shown in Figs. 2 and 3, respectively.

Fig. 2.

The $M_{\text{jj}}$ (left) and $M_{\text{j}}^{\text{Y}}$ (right) projections showing the number of observed events per $\text{Ge}\text{V}$ (black markers) compared with the backgrounds estimated in the fit to the data (filled histograms) in the CR. Pass (upper) and Fail (lower) categories are shown. The high level of agreement between the model and the data in the Fail region arises because this region is used to constrain the background estimate. The lower panels show the “Pull” defined as $(\text{observed events}-\text{expected events})/\sqrt{{\sigma }_{\text{obs}}^2+{\sigma }_{\text{bkg}}^2},$ where ${\sigma }_{\text{obs}}$ and ${\sigma }_{\text{bkg}}$ are the total uncertainties in the observation and the background estimation, respectively

Fig. 3.

The $M_{\text{jj}}$ (left) and $M_{\text{j}}^{\text{Y}}$ (right) projections showing the number of observed events per $\text{Ge}\text{V}$ (black markers) compared with the backgrounds estimated in the fit to the data (filled histograms) in the MR. Pass (upper) and Fail (lower) categories are shown. The expected contribution from the signal benchmark with $M_{\text{X}}=2200\text{Ge}\text{V}$ and $M_{\text{Y}}=250\text{Ge}\text{V}$ is overlaid in the MR Pass, assuming a production cross section of 5$\text{fb}$. The high level of agreement between the model and the data in the Fail region arises because this region is used to constrain the background estimate. The lower panels show the “Pull” defined as $(\text{observed events}-\text{expected events})/\sqrt{{\sigma }_{\text{obs}}^2+{\sigma }_{\text{bkg}}^2},$ where ${\sigma }_{\text{obs}}$ and ${\sigma }_{\text{bkg}}$ are the total uncertainties in the observation and the background estimation, respectively

The joint likelihoods of the signal-plus-background $M_{\text{jj}}$-$M_{\text{j}}^{\text{Y}}$ distributions in the MR Pass and Fail regions are constructed for different signal hypotheses. The maximum likelihood fits are used to extract the observed signal strengths. Out of the 42 $\text{X}\to \text{H}(\text{b}\overline{\text{b}})\text{Y}(\text{WW})$ mass points considered, the $M_{\text{X}}=1600\text{Ge}\text{V}$ and $M_{\text{Y}}=90\text{Ge}\text{V}$ combination corresponds to the largest local significance of 2.1 standard deviations.

The same likelihoods are used to compute upper limits on the signal cross section with a modified frequentist approach, using the ${\text{CL}}_{\text{s}}$ criterion [68, 69] with the profile likelihood ratio used as the test statistic, in an asymptotic approximation [70].

Figure 4 shows the expected and observed limits at 95% confidence level (CL), assuming the $\text{Y}\to {\text{W}}^{+}{\text{W}}^{-}$ decay to an all-hadronic final state. These limits range from 0.3–19.1$\text{fb}$. In the upper left corner of the figures, where $M_{\text{X}}$ is low and $M_{\text{Y}}$ is high, we begin to exit the Lorentz-boosted regime. This results in the Ydecay products being reconstructed as separate jets, reducing the efficiency of the offline selection and leading to higher upper limits.

Fig. 4.

The expected (upper) and observed (lower) 95% confidence level upper limits on $\sigma (\text{p}\to \text{X}\to \text{HY})\mathcal{B}(\text{H}\to \text{b}\overline{\text{b}})\mathcal{B}(\text{Y}\to \text{WW}\to 2\text{q}2{\overline{\text{q}}}^{\prime })$ for different values of $M_{\text{X}}$ and $M_{\text{Y}}$. The limits have been evaluated in discrete steps corresponding to the centers of the boxes. The numbers in the boxes are given in fb

Comparable searches have been performed by the ATLAS Collaboration using both unsupervised [71, 72] and weakly supervised [73] anomaly detection techniques in two-body and dijet final states. The search in Ref. [71] targets several two-body final states, while Ref. [72] specifically focuses on the $\text{X}\to \text{H}(\text{b}\overline{\text{b}})\text{Y}$ topology, where Ydecays to a fully hadronic state, using anomalous-jet tagging. These unsupervised searches explore broader ranges of both $M_{\text{X}}$ and $M_{\text{Y}}$ than the analysis presented here, whereas the weakly supervised search [73] focuses on higher $M_{\text{X}}$ regions. Finally, a targeted ATLAS search for $\text{X}\to \text{HS},$ with $\text{H}\to \gamma \gamma$ and $\text{S}\to \text{b}\overline{\text{b}}$ [74], explores complementary lower-mass regions with $M_{\text{X}}<1\text{Te}\text{V}.$ Overall, these analyses provide complementary sensitivity across different mass ranges and final-state topologies. The presented results can be compared to the CMS search for resonances decaying into two jets [19], which reports expected model-independent limits on dijet resonances in the 50–600$\text{fb}$ range for $M_{\text{X}}$ values between 1.8 and 3.0$\text{Te}\text{V}$, depending on the method and $M_{\text{X}}$. The most sensitive method sets limits around 60$\text{fb}$ across this $M_{\text{X}}$ range. Reference [19] also provides limits on a process that consists of a Kaluza–Klein excitation of a Wboson (${\text{W}}_{\text{KK}}$) decaying into a radion and a Wboson [75, 76]. The radion then decays into two Wbosons, and all three Wbosons undergo hadronic decays. The comparison to our results is motivated by the similarity in jet substructure: both analyses involve a four-prong jet from the $\text{Y}\to {\text{W}}^{+}{\text{W}}^{-}$ decay and a two-prong jet originating from either Hor Wboson decay, where “prong” refers to the number of hard subjets within a large-radius jet. Although there are no directly overlapping mass points, we can compare the expected limits for a ${\text{W}}_{\text{KK}}$ boson mass of 3000$\text{Ge}\text{V}$ and a radion mass of 170$\text{Ge}\text{V}$ (22.1$\text{fb}$) with our limits for $M_{\text{X}}=3000\text{Ge}\text{V}$ and $M_{\text{Y}}=200\text{Ge}\text{V}$ (0.4$\text{fb}$). In all cases, our analysis sets more stringent limits, as expected, mostly because we require one of the jets to originate from $\text{H}\to \text{b}\overline{\text{b}}$. While this requirement makes the results less general, it enables the use of ParticleNet tagging, which provides most of the sensitivity in the analysis, with the autoencoder giving a modest additional improvement. Figure 5 shows 95% CL upper limits on the cross section for the three additional signal scenarios considered. Notably, the $\text{Y}\to \text{t}\overline{\text{t}}$ scenario constitutes the first search performed in the Lorentz-boosted regime for this process. The differences in the limits among the scenarios arise mainly from their distinct kinematic properties and the varying background levels in the corresponding regions of phase space, both influenced by the different assumed Ymasses.

Fig. 5.

The median expected (dashed line) 95% confidence level upper limits on the main and three alternative signal scenarios as a function of $M_{\text{X}}$. The inner (green) and outer (yellow) bands represent the regions containing 68% and 95%, respectively, of the distribution of limits expected under the background-only hypothesis. The solid dots indicate the observed limits, which are only evaluated at the simulated mass points

Tabulated results of this analysis are available on HEPData [77]. The HepData entry also includes the trained AE weights used for the anomaly score evaluation.

Summary

A search for beyond the standard model physics through the process where a resonant particle decays into a Higgs boson, H, and an additional particle Yhas been presented. The Hsubsequently decays into a bottom quark–antiquark pair, $\text{b}\overline{\text{b}}$, reconstructed as a large-radius jet in the Lorentz-boosted regime. The Hcandidate jets are tagged using the ParticleNet algorithm that is designed to recognize jets originating from a decay of a massive particle into $\text{b}\overline{\text{b}}$. The identification of the second particle, Y, is performed by computing the anomaly score of its candidate jet using an autoencoder, allowing the simultaneous search for multiple Y decay modes within a single analysis. This approach combines the targeted identification of Hdecays with model-independent anomaly detection technique, enabling a broad search for new physics. By combining the strong discrimination power of the Hboson decay to $\text{b}\overline{\text{b}}$with a model-independent anomaly detection for the second particle, this analysis achieves enhanced sensitivity for scenarios involving a Higgs boson in the decay chain, while maintaining applicability across a range of such models.

The analysis considers four benchmark models. The main benchmark assumes $\text{Y}\to {\text{W}}^{+}{\text{W}}^{-}$, with further hadronic decays of the Wbosons. It is simulated for $M_{\text{X}}$ values within 1400–3000$\text{Ge}\text{V}$ and $M_{\text{Y}}$ values within 90–400$\text{Ge}\text{V}$, covering 42 signal hypotheses. No significant excess above the standard model background expectations is observed. Upper limits on benchmark signal cross sections at 95% confidence level are set on the main benchmark model in the 0.3–15.8$\text{fb}$ range. Additionally, exclusion limits are calculated for three alternative benchmark models, assuming $\text{Y}\to \text{u}\overline{\text{u}}$, $\text{Y}\to \text{t}\overline{\text{t}}$, or $\text{Y}\to \text{bq}{\overline{\text{q}}}^{\prime }$ decay modes, for which the most stringent limits to date are achieved.

Funding

We congratulate our colleagues in the CERN accelerator departments for the excellent performance of the LHC and thank the technical and administrative staffs at CERN and at other CMS institutes for their contributions to the success of the CMS effort. In addition, we gratefully acknowledge the computing centers and personnel of the Worldwide LHC Computing Grid and other centers for delivering so effectively the computing infrastructure essential to our analyses. Finally, we acknowledge the enduring support for the construction and operation of the LHC, the CMS detector, and the supporting computing infrastructure provided by the following funding agencies: SC (Armenia), BMBWF and FWF (Austria); FNRS and FWO (Belgium); CNPq, CAPES, FAPERJ, FAPERGS, and FAPESP (Brazil); MES and BNSF (Bulgaria); CERN; CAS, MoST, and NSFC (China); MINCIENCIAS (Colombia); MSES and CSF (Croatia); RIF (Cyprus); SENESCYT (Ecuador); ERC PRG, TARISTU24-TK10 and MoER TK202 (Estonia); Academy of Finland, MEC, and HIP (Finland); CEA and CNRS/IN2P3 (France); SRNSF (Georgia); BMFTR, DFG, and HGF (Germany); GSRI (Greece); NKFIH (Hungary); DAE and DST (India); IPM (Iran); SFI (Ireland); INFN (Italy); MSIT and NRF (Republic of Korea); MES (Latvia); LMTLT (Lithuania); MOE and UM (Malaysia); BUAP, CINVESTAV, CONACYT, LNS, SEP, and UASLP-FAI (Mexico); MOS (Montenegro); MBIE (New Zealand); PAEC (Pakistan); MES, NSC, and NAWA (Poland); FCT (Portugal); MESTD (Serbia); MICIU/AEI and PCTI (Spain); MOSTR (Sri Lanka); Swiss Funding Agencies (Switzerland); MST (Taipei); MHESI and NSTDA (Thailand); TUBITAK and TENMAK (Türkiye); NASU (Ukraine); STFC (United Kingdom); DOE and NSF (USA). Individuals have received support from the Marie-Curie program and the European Research Council and Horizon 2020 Grant, contract Nos. 675440, 724704, 752730, 758316, 765710, 824093, 101115353, 101002207, 101001205, and COST Action CA16108 (European Union); the Leventis Foundation; the Alfred P. Sloan Foundation; the Alexander von Humboldt Foundation; the Science Committee, project no. 22rl-037 (Armenia); the Fonds pour la Formation à la Recherche dans l’Industrie et dans l’Agriculture (FRIA-Belgium); the Beijing Municipal Science and Technology Commission, No. Z191100007219010, the Fundamental Research Funds for the Central Universities, the Ministry of Science and Technology of China under Grant No. 2023YFA1605804, and the Natural Science Foundation of China under Grant No. 12061141002 (China); the Ministry of Education, Youth and Sports (MEYS) of the Czech Republic; the Shota Rustaveli National Science Foundation, grant FR-22-985 (Georgia); the Deutsche Forschungsgemeinschaft (DFG), among others, under Germany’s Excellence Strategy – EXC 2121 “Quantum Universe” – 390833306, and under project number 400140256 - GRK2497; the Hellenic Foundation for Research and Innovation (HFRI), Project Number 2288 (Greece); the Hungarian Academy of Sciences, the New National Excellence Program - ÚNKP, the NKFIH research grants K 131991, K 133046, K 138136, K 143460, K 143477, K 146913, K 146914, K 147048, 2020-2.2.1-ED-2021-00181, TKP2021-NKTA-64, and 2021-4.1.2-NEMZ_KI-2024-00036 (Hungary); the Council of Science and Industrial Research, India; ICSC – National Research Center for High Performance Computing, Big Data and Quantum Computing, FAIR – Future Artificial Intelligence Research, and CUP I53D23001070006 (Mission 4 Component 1), funded by the NextGenerationEU program (Italy); the Latvian Council of Science; the Ministry of Education and Science, project no. 2022/WK/14, and the National Science Center, contracts Opus 2021/41/B/ST2/01369, 2021/43/B/ST2/01552, 2023/49/B/ST2/03273, and the NAWA contract BPN/PPO/2021/1/00011 (Poland); the Fundação para a Ciência e a Tecnologia, grant CEECIND/01334/2018 (Portugal); the National Priorities Research Program by Qatar National Research Fund; MICIU/AEI/10.13039/501100011033, ERDF/EU, “European Union NextGenerationEU/PRTR”, and Programa Severo Ochoa del Principado de Asturias (Spain); the Chulalongkorn Academic into Its 2nd Century Project Advancement Project, and the National Science, Research and Innovation Fund via the Program Management Unit for Human Resources and Institutional Development, Research and Innovation, grant B39G680009 (Thailand); the Kavli Foundation; the Nvidia Corporation; the SuperMicro Corporation; the Welch Foundation, contract C-1845; and the Weston Havens Foundation (USA).

Data Availability Statement

Release and preservation of data used by the CMS Collaboration as the basis for publications is guided by the CMS data preservation, re-use and open access policy.

Code Availability Statement

The CMS core software is publicly available on GitHub (https://github.com/cms-sw/cmssw).

Declarations

Conflict of interest

The authors declare that they have no conflict of interest.