Cognitive encoding modeling of musical sequences for vocal performance and intelligent music composition

Abstract

Introduction

The cognitive encoding of musical sequences is a complex process that involves capturing the intricate structure, temporal dynamics, and inherent uncertainties of musical data. Traditional methods often struggle to preserve the non-Euclidean geometric properties of musical sequences and fail to adequately model temporal dependencies and uncertainties. This paper introduces the Manifold Adaptive Sequence Encoder (MASE), a novel neural framework designed to address these challenges.

Methods

MASE integrates three key modules: the Riemannian Trajectory Mapper, which embeds musical sequences into a Riemannian manifold to maintain their geometric properties; the Agent-driven Temporal Planner, which effectively models the temporal dependencies and rhythmic patterns; and the Uncertainty-guided Sequence Filter, which quantifies and incorporates uncertainty to enhance robustness and generalization. The framework is optimized using manifold alignment optimization, ensuring the alignment of latent representations with the input data, and uncertainty-aware refinement, which iteratively refines predictions by leveraging uncertainty estimates.

Results and discussion

Experimental results demonstrate that MASE significantly improves the accuracy and robustness of musical sequence modeling, outperforming existing methods by a substantial margin. The proposed approach offers a principled methodology for modeling the cognitive encoding of musical sequences, with potential applications in music analysis, recommendation, and generation. This advancement in musical sequence encoding not only enhances the understanding of cognitive processes involved in music perception but also provides a robust tool for various practical applications in the field of music technology.

1. Introduction

Music is a universal language that transcends cultural and linguistic barriers, playing a vital role in human cognition and emotional expression. Understanding how musical sequences are encoded cognitively is not only essential for advancing our knowledge of human brain function but also has practical implications for applications such as music generation (Koelsch et al., 2019), recommendation systems, and therapeutic interventions. The study of cognitive encoding of musical sequences enables researchers to explore the intricate interplay between auditory perception, memory, and neural processing (Zhang, 2025). Not only does this research contribute to the broader field of cognitive science, but it also provides insights into the development of computational models that mimic human-like understanding of music (Di Liberto et al., 2020). Furthermore, modeling musical cognition through learning-based neural frameworks can enhance the design of intelligent systems capable of generating and interpreting music in a manner that aligns with human preferences and emotions (Norman Haignere et al., 2022). This task is particularly significant in the era of artificial intelligence, where bridging the gap between human cognition and machine learning remains a critical challenge (Peretz, 2016). The notion of cognitive encoding in this study is used in a computational and cognition-inspired sense rather than as a claim of direct neural reconstruction (Kim, 2022). Evidence from music cognition and auditory neuroscience suggests that musical stimuli with different acoustic, rhythmic, harmonic, or tonal properties can elicit systematically different neural and behavioral responses (Hoefle et al., 2018). Such findings indicate that brain responses are sensitive to structured musical information, but they do not by themselves imply that the original musical stimulus can be fully reconstructed from brain activity or that there exists a one-to-one mapping between neural activity and musical sequences (Martin et al., 2018). In this paper, we therefore use these studies as motivation for learning structured representations of musical sequences, not as evidence that the proposed model reproduces biological encoding mechanisms (Daly, 2023). Prior work on auditory and music perception has shown that cortical and electrophysiological responses reflect temporal, spectral, tonal, and expectation-related properties of musical stimuli (Bonetti et al., 2021). These observations support the view that musical sequence modeling should account for temporal organization, structural regularities, and response variability. The proposed framework focuses on three computational aspects that are relevant to cognition-inspired modeling: structured representation learning, temporal dependency modeling, and uncertainty-aware refinement (Brattico et al., 2006).

In the early stages of research, efforts to model the cognitive encoding of musical sequences primarily focused on structured frameworks that utilized predefined rules and expert knowledge. These methods aimed to capture the structural and hierarchical aspects of musical cognition by representing music through elements such as pitch, rhythm, and harmony (Vuust et al., 2018). While these approaches provided a foundational understanding, they were often limited by their inability to adapt to the variability and complexity of human musical perception (Pearce, 2018). The rigid nature of these frameworks restricted their effectiveness in capturing the nuances of human cognition, as they struggled to generalize across diverse musical styles and contexts (Kell et al., 2018). This prompted researchers to explore more flexible methodologies that could learn directly from musical data, paving the way for the integration of data-driven techniques.

The introduction of machine learning techniques brought a paradigm shift in the study of musical cognition, allowing for the modeling of musical sequences through statistical and probabilistic methods. By utilizing large datasets of musical compositions and performances, algorithms were developed to identify patterns and predict musical structures without relying on predefined rules (Sankaran et al., 2024). These methods offered improved adaptability and flexibility compared to earlier approaches, as they could learn directly from data (Mehr et al., 2019). However, the reliance on feature engineering posed challenges, as it required significant domain expertise to extract meaningful representations of musical sequences (Herremans et al., 2017). These methods often faced challenges in capturing the hierarchical and temporal dependencies inherent in music, which limited their capacity to model complex cognitive processes (Ji et al., 2020). This led to the exploration of deep learning techniques, which provided a more robust solution for modeling musical cognition.

Deep learning has transformed the field by enabling the automatic extraction of hierarchical features and temporal dependencies in musical sequences. Techniques such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformers have been employed to model the intricate relationships between musical elements and their cognitive representations (Briot et al., 2020). The use of pre-trained models and transfer learning has further enhanced the ability to generalize across diverse musical styles and contexts by leveraging knowledge from large-scale datasets (Roberts et al., 2018). These methods have shown remarkable success in tasks such as music generation, classification, and recommendation, capturing the complex interplay between melody, harmony, and rhythm (Yu et al., 2022). Despite their advantages, deep learning approaches often require extensive computational resources and large amounts of labeled data, which can limit their accessibility and scalability. The lack of interpretability in these models presents challenges in comprehending the underlying mechanisms of musical cognition. To address these limitations, we propose a novel approach that combines the strengths of deep learning with innovative techniques to enhance efficiency, generalizability, and interpretability.

Based on the aforementioned limitations of symbolic AI, machine learning, and deep learning approaches, we propose a learning-based neural framework that integrates cognitive principles with advanced neural architectures to model the encoding of musical sequences. Our method aims to overcome the rigidity of symbolic AI, the feature engineering dependency of machine learning, and the resource-intensive nature of deep learning by introducing a modular design that balances efficiency and scalability. By incorporating cognitive insights into the neural framework, our approach enhances the interpretability of the model, enabling a deeper understanding of the mechanisms underlying musical cognition. Furthermore, our method is designed to be adaptable across diverse musical styles and contexts, ensuring its applicability in real-world scenarios. Through extensive experimentation, we demonstrate that our framework achieves superior performance in tasks such as music generation, classification, and recommendation, while maintaining computational efficiency and interpretability. This innovative approach represents a significant step forward in bridging the gap between human cognition and machine learning in the domain of musical sequences.

We summarize our contributions as follows:

• We propose a learning-based neural framework that integrates cognitive principles to enhance interpretability and efficiency.

• Our approach demonstrates adaptability across diverse musical styles and contexts, ensuring high generalizability and practical applicability.

• Experimental results show that our framework achieves superior performance in music-related tasks while maintaining computational efficiency.

2. Related work

Recent studies on music generation and music cognition have increasingly relied on deep learning and neural network based methods to process complex musical data. In a typical workflow, raw musical materials are first transformed into symbolic sequences, audio features, spectrograms, or multimodal descriptors (Dong et al., 2018). These representations usually contain pitch, rhythm, duration, harmony, timbre, and temporal structure. Neural models are then used to encode these features and learn relationships among musical events (Oore et al., 2020). Recurrent neural networks, convolutional neural networks, and transformers have been widely adopted for this purpose (Hawthorne et al., 2019). Recurrent models are suitable for modeling sequential continuity, convolutional models are effective for extracting local patterns from time frequency representations, and transformers can capture global dependencies across distant musical events (Engel et al., 2017). For music generation, learning based systems aim to produce coherent and expressive musical sequences by learning structural regularities from data. Variational autoencoders map musical inputs into latent spaces and generate new sequences through sampling (Dhariwal et al., 2020). Generative adversarial networks learn the distribution of musical data and generate realistic samples. Reinforcement learning methods further optimize generated music according to objectives such as stylistic consistency, harmonic coherence, rhythmic stability, or listener engagement (Hadjeres et al., 2017). These methods have been applied to melody generation, chord prediction, rhythm modeling, style transfer, audio synthesis, recommendation, and expressive performance modeling (Huang and Yang, 2020). Although existing neural approaches have achieved notable progress, several limitations remain. Many methods represent musical sequences in Euclidean latent spaces, which may be insufficient for preserving the intrinsic geometric structure of music (Brunner et al., 2018). Some models mainly emphasize temporal prediction but provide limited treatment of uncertainty caused by noisy, ambiguous, or diverse musical inputs (Yang et al., 2017). The connection between learned representations and cognitively meaningful musical structures is often not explicitly modeled (Donahue et al., 2019). To address these issues, the proposed Manifold Adaptive Sequence Encoder introduces a Riemannian manifold representation, a temporal planning mechanism, and an uncertainty guided filtering strategy, providing a more structured framework for cognitive encoding of musical sequences.

3. Method

3.1. Overview

The proposed Manifold Adaptive Sequence Encoder (MASE) encodes musical sequences through three modules: the Riemannian Trajectory Mapper, the Agent driven Temporal Planner, and the Uncertainty guided Sequence Filter. Given an input musical sequence $\mathcal{X}=\left\{{\text{x}}_1,{\text{x}}_2,\dots ,{\text{x}}_T\right\}$, where ${\text{x}}_t\in {ℝ}^d$ denotes the musical feature vector at time step t, MASE maps the sequence into a latent representation $\mathcal{Ƶ}=\left\{{\text{z}}_1,{\text{z}}_2,\dots ,{\text{z}}_T\right\}$ on a Riemannian manifold $\mathcal{M}$. The feature vector contains pitch, rhythm, duration, onset, timbre, and harmonic descriptors.

As shown in Figure 1, the Riemannian Trajectory Mapper projects each input feature vector into the manifold latent space. This module preserves the geometric relationships among musical elements and provides a structured latent representation for subsequent processing. The Agent driven Temporal Planner models temporal dependencies in the latent sequence and generates a smooth trajectory that reflects musical continuity. The Uncertainty guided Sequence Filter estimates the uncertainty of each latent state and refines the trajectory by reducing the influence of noisy or ambiguous musical inputs. The workflow of MASE begins with converting raw musical data into normalized feature sequences. The Riemannian Trajectory Mapper then encodes each feature vector as a manifold coordinate. The Agent driven Temporal Planner optimizes the latent trajectory according to the Riemannian metric, and the Uncertainty guided Sequence Filter refines the planned trajectory using uncertainty estimates. The refined representation is passed to a task specific prediction head for classification, prediction, recommendation, or generation. The training objective combines manifold mapping, temporal planning, uncertainty filtering, manifold alignment, and task prediction. Through joint optimization, MASE learns latent representations that preserve musical structure, maintain temporal coherence, and remain robust to uncertainty. Section 3.2 introduces the mathematical notation, Section 3.3 describes the three modules, and Section 3.4 presents the refinement and alignment strategies.

Figure 1.

Overall architecture and optimization workflow of the proposed Manifold Adaptive Sequence Encoder (MASE), including manifold trajectory mapping, temporal planning, uncertainty refinement, and joint optimization.

3.2. Preliminaries

This subsection formalizes the problem of cognitive encoding of musical sequences within the framework of learning-based neural architectures. The aim is to establish a mathematical foundation for modeling the intricate relationships between musical elements and their cognitive representations. This is achieved by defining the problem in terms of sequence modeling, manifold learning, and uncertainty quantification, which serve as the basis for the proposed Manifold Adaptive Sequence Encoder.

Let $\mathcal{X}=\left\{x_1,x_2,\dots ,x_T\right\}$ represent a musical sequence, where $x_t\in {ℝ}^d$ denotes the feature vector of the t-th musical element, and T is the total number of elements in the sequence. Each feature vector x_t encapsulates attributes such as pitch, rhythm, and timbre, which are essential for cognitive interpretation. The objective is to encode $\mathcal{X}$ into a latent representation $\mathcal{Z}=\left\{z_1,z_2,\dots ,z_T\right\}$, where $z_t\in \mathcal{M}$ resides in a manifold $\mathcal{M}$ that captures the intrinsic structure of the sequence.

The manifold $\mathcal{M}$ is assumed to be a Riemannian space, characterized by a metric tensor $g_{\mathcal{M}}$ that defines the local geometry. Formally, the manifold is represented as $(\mathcal{M},g_{\mathcal{M}})$, where $g_{\mathcal{M}}:\mathcal{M}\times \mathcal{M}\to ℝ$ measures the distance between points on $\mathcal{M}$. The encoding process aims to preserve the temporal and structural coherence of $\mathcal{X}$ while mapping it onto $\mathcal{M}$.

To model the temporal dynamics of $\mathcal{X}$, a trajectory $\gamma :\left[0,1\right]\to \mathcal{M}$ is defined, where γ(s) represents the position on the manifold at time s ∈ [0, 1]. The trajectory γ is constrained by the geodesic equation:

$$\begin{array}{l}\frac{d^2{\gamma }^i}{d{s}^2}+{\Gamma }_{jk}^i\frac{d{\gamma }^j}{ds}\frac{d{\gamma }^k}{ds}=0,\end{array}$$ (1)

where ${\Gamma }_{jk}^i$ are the Christoffel symbols associated with $g_{\mathcal{M}}$. This equation ensures that the trajectory follows the shortest path on $\mathcal{M}$, preserving the natural flow of the sequence.

The cognitive encoding process also involves uncertainty quantification, as musical sequences often exhibit variability due to factors such as interpretation and noise. Let $\mathcal{U}=\left\{u_1,u_2,\dots ,u_T\right\}$ denote the uncertainty associated with each element in $\mathcal{X}$, where $u_t\in {ℝ}^{+}$ represents the uncertainty score. The uncertainty-aware encoding is achieved by incorporating $\mathcal{U}$ into the manifold representation, resulting in a probabilistic model:

$$\begin{array}{l}p(z_t|x_t,u_t)=\frac{1}{Z}exp\left(-\frac{\Vert x_t-f(z_t){\Vert }^2}{2u_t^2}\right),\end{array}$$ (2)

where $f:\mathcal{M}\to {ℝ}^d$ is the decoding function, and Z is the normalization constant.

To ensure the alignment of $\mathcal{X}$ with $\mathcal{M}$, a manifold alignment optimization problem is defined. Let ${\mathcal{M}}_X$ and ${\mathcal{M}}_Z$ denote the manifolds corresponding to the input and latent spaces, respectively. The alignment is achieved by minimizing the discrepancy between ${\mathcal{M}}_X$ and ${\mathcal{M}}_Z$:

$$\begin{array}{l}{\mathcal{L}}_{\mathrm{\text{align}}}={\displaystyle {\int }_{{\mathcal{M}}_X}}\Vert g_{{\mathcal{M}}_X}-g_{{\mathcal{M}}_Z}{\Vert }^{2}d{\mathcal{M}}_X.\end{array}$$ (3)

This optimization ensures that the latent representation $\mathcal{Z}$ faithfully captures the structure of $\mathcal{X}$.

Furthermore, the temporal coherence of $\mathcal{Z}$ is enforced by a smoothness constraint on the trajectory γ. Let ${\mathcal{C}}^2(\mathcal{M})$ denote the space of twice-differentiable functions on $\mathcal{M}$. The smoothness constraint is defined as:

$$\begin{array}{l}{\mathcal{L}}_{\mathrm{\text{smooth}}}={\displaystyle {\int }_0^1}\Vert \frac{d^2\gamma (s)}{d{s}^2}{\Vert }^{2}ds.\end{array}$$ (4)

Minimizing ${\mathcal{L}}_{\mathrm{\text{smooth}}}$ ensures that the trajectory γ does not exhibit abrupt changes, preserving the natural progression of the sequence.

3.3. Manifold adaptive sequence encoder

The proposed model, termed the Manifold Adaptive Sequence Encoder (MASE), is designed to capture the cognitive encoding of musical sequences by leveraging a learning-based neural framework. MASE integrates three key modules: the Riemannian Trajectory Mapper, the Agent-driven Temporal Planner, and the Uncertainty-guided Sequence Filter. These modules collectively enable the model to process complex musical sequences while preserving their inherent structural and temporal characteristics. Below, we detail the mathematical formulation and operational mechanisms of each module within MASE. As illustrated in Figure 2, the proposed MASE framework integrates manifold representation learning, temporal trajectory planning, and uncertainty-aware sequence refinement into a unified end-to-end optimization architecture.

Figure 2.

Detailed architecture of the proposed Manifold Adaptive Sequence Encoder (MASE). The framework consists of three core modules: Riemannian Trajectory Mapping (RTM), Agent-driven Temporal Planning (ATP), and Uncertainty-guided Sequence Filtering (USF). The figure illustrates the manifold projection process, temporal trajectory optimization, uncertainty-aware sequence refinement, inter-module interactions, and the joint optimization strategy with multiple loss functions. Different arrows denote data flow, manifold-related flow, planning-related flow, filtering-related flow, feedback interaction, and gradient propagation during end-to-end training.

3.3.1. Riemannian trajectory mapping

The Riemannian Trajectory Mapper is responsible for embedding musical sequences into a Riemannian manifold, ensuring that the geometric properties of the sequences are preserved (Figure 3). Let $\mathcal{X}=\left\{x_1,x_2,\dots ,x_T\right\}$ represent a sequence of musical features, where $x_t\in {ℝ}^d$ denotes the feature vector at time step t. The embedding process maps $\mathcal{X}$ into a Riemannian manifold $\mathcal{M}$ using a differentiable mapping function $\varphi :{ℝ}^d\to \mathcal{M}$. The latent manifold M is defined as an intrinsic k dimensional Riemannian manifold embedded in an ambient Euclidean space ℝⁿ, that is, M ⊂ ℝⁿ. The ambient dimension n denotes the dimension of the latent coordinate vector produced by the encoder projection layer, while k denotes the intrinsic dimension of the manifold. The sequence length T is not the dimension of the embedding space. Instead, T determines the number of latent states sampled along the encoded musical trajectory. Therefore, a musical sequence X = {x₁, x₂, …, x_T} is represented as a set of T ordered points Z = {z₁, z₂, …, z_T} on M, where each $z_t\in M\subset {ℝ}^n$. This notation follows standard treatments of embedded Riemannian manifolds, tangent spaces, metric tensors, geodesics, and exponential and logarithmic maps (Jost, 2005). The mapping function ϕ is not specified as a fixed polynomial, exponential, or other predefined analytic function. In this study, ϕ denotes a learnable neural embedding function parameterized by Θ_ϕ. More precisely, ϕ(·;Θ_ϕ) belongs to a trainable family of differentiable neural functions and maps each musical feature vector $x_t\in {ℝ}^d$ to a latent manifold state z_t ∈ M. This treatment is consistent with standard formulations of neural networks as parameterized nonlinear function approximators (Amari, 2016). Formally, this mapping is written as

Figure 3.

The Manifold Adaptive Sequence Encoder for Musical Sequences consists of three main components: the Input Layer, Sequence Modeling Module, and Manifold Learning Module. The Input Layer processes musical sequences represented as feature vectors encapsulating pitch, rhythm, and timbre. The Sequence Modeling Module captures temporal dependencies using recurrent neural networks (RNNs) or transformers. The Manifold Learning Module projects feature vectors onto a lower-dimensional manifold, preserving the intrinsic geometry of musical data through techniques such as PCA or t-SNE.

$$\begin{array}{l}{z}_t=\varphi (x_t;{\Theta }_{\varphi }),z_t\in M\subset {ℝ}^n.\end{array}$$ (5)

Equivalently, the neural embedding can be understood as

$$\begin{array}{l}\varphi (·;{\Theta }_{\varphi })\in {\Phi }_{\Theta }\subset C^1({ℝ}^d,M),\end{array}$$ (6)

where C¹(ℝ^d, M) denotes the space of continuously differentiable mappings from the input feature space to the latent manifold. Here, Θ_ϕ contains the trainable weights and biases learned during optimization. In implementation, ϕ is realized by a sequence encoding network followed by a manifold projection operation. The sequence encoding network extracts nonlinear musical representations from pitch, rhythm, timbre, and other input features, while the projection operation maps the resulting latent coordinates onto the Riemannian manifold. Thus, the nonlinearity of ϕ is learned from data rather than imposed through a fixed functional family. The geodesic formulation is used to regularize the local transitions between adjacent latent states rather than to impose that the original musical sequence itself is a continuous geodesic flow. The input sequence X is first encoded into the discrete latent trajectory Z = {z₁, z₂, …, z_T}. Adjacent latent states z_i and z_i+1 are then connected through Riemannian geodesic interpolation by means of the exponential and logarithmic maps on M (Lee, 2018). Consequently, the encoded musical sequence is modeled as a locally geodesic trajectory on the latent manifold. This design encourages smooth temporal evolution and preserves the intrinsic geometric relationships among musical elements, while still allowing the learned encoder and temporal planner to capture deviations caused by rhythmic variation, melodic development, and uncertainty in musical performance.

The trajectory $\mathcal{T}$ on the manifold is defined as:

$$\begin{array}{l}\mathcal{T}=\left\{\varphi (x_1),\varphi (x_2),\dots ,\varphi (x_T)\right\},\varphi (x_t)\in \mathcal{M}.\end{array}$$ (7)

To ensure smooth transitions between consecutive points on $\mathcal{T}$, a geodesic interpolation mechanism is employed, defined as:

$$\begin{array}{l}\gamma (t)={\mathrm{\text{Exp}}}_{\varphi (x_i)}\left(t·{\mathrm{\text{Log}}}_{\varphi (x_i)}(\varphi (x_{i+1}))\right),t\in \left[0,1\right],\end{array}$$ (8)

where Exp and Log denote the exponential and logarithmic maps on $\mathcal{M}$, respectively. The integration of these mappings ensures that the manifold representation retains the intrinsic geometric structure of the original sequence.

3.3.2. Agent driven temporal planning

The Agent driven Temporal Planner governs the temporal dynamics of the encoded sequence by modeling local transitions between adjacent latent manifold states (Figure 4). Let $\mathcal{A}$ denote a set of temporal agents, where each agent $\alpha \in \mathcal{A}$ is assigned to a temporal index subset ${\mathcal{I}}_{\alpha }$ of the sequence. The input to the planner is the latent trajectory Z = {z₁, z₂, …, z_T}, where each $z_t\in M\subset {ℝ}^n$. Therefore, the planner operates on the learned manifold representation rather than directly on the original feature vectors. For each agent α, the functions f_α and g_α are defined as differentiable agent specific neural mappings on pairs of adjacent latent states. More precisely,

Figure 4.

The diagram illustrates the uncertainty-aware refinement strategy utilized to improve the Manifold Adaptive Sequence Encoder (MASE). The strategy integrates three key components: the Riemannian Trajectory Mapper (RTM), Agent-driven Temporal Planner (ATP), and Uncertainty-guided Sequence Filter (USF). Each component collaborates to refine the encoded representations by aligning them with the manifold structure while addressing noise and ambiguity in the input data.

$$\begin{array}{l}{f}_{\alpha }\in {\mathcal{F}}_{\alpha },g_{\alpha }\in {\mathcal{G}}_{\alpha },{\mathcal{F}}_{\alpha },{\mathcal{G}}_{\alpha }\subset C^1(M\times M,{ℝ}^r),\end{array}$$ (9)

where C¹(M × M, ℝ^r) denotes the space of continuously differentiable functions from pairs of latent manifold states to an r dimensional Euclidean relation space (Bishop and Nasrabadi, 2006). Thus, f_α and g_α live in differentiable function classes defined on M × M, and their outputs are temporal relation descriptors in ℝ^r. This formulation follows standard treatments of differentiable mappings on manifolds and neural networks as parameterized function approximators (Aggarwal, 2018).

The temporal planning objective is defined as

$$\begin{array}{l}{\mathcal{L}}_{planner}={\displaystyle \sum _{\alpha \in \mathcal{A}}}{\displaystyle \sum _{t\in {\mathcal{I}}_{\alpha }}}\Vert f_{\alpha }(z_t,z_{t+1})-g_{\alpha }(z_t,z_{t+1}){\Vert }_2^2.\end{array}$$ (10)

The optimization compares the relation descriptors produced by f_α and g_α in the common Euclidean space ℝ^r, while the inputs of these functions remain adjacent states on the latent Riemannian manifold. In this way, the temporal planner preserves local temporal consistency, encourages smooth evolution of the latent musical trajectory, and adapts the temporal structure of the sequence to the learned manifold representation.

3.3.3. Uncertainty-guided sequence filtering

The Uncertainty-guided Sequence Filter addresses the inherent uncertainty in musical sequences, which may arise due to noise or variability in feature extraction. The filter employs a probabilistic approach to refine the sequence representation. Let $\mathcal{P}=\left\{p_1,p_2,\dots ,p_T\right\}$ denote the uncertainty scores associated with each feature vector. The filtering process adjusts the sequence representation ${\mathcal{X}}^{\prime }$ as follows:

$$\begin{array}{l}{x}_t^{\prime }=x_t·(1-p_t)+{\mu }_t·p_t,\end{array}$$ (11)

where μ_t is the mean feature vector estimated from the neighborhood of x_t. The uncertainty scores p_t are computed using a Bayesian framework:

$$\begin{array}{l}{p}_t=\frac{\mathrm{\text{Pr}}(x_t|\mathcal{X})}{{\displaystyle {\sum }_{i=1}^T}\mathrm{\text{Pr}}(x_i|\mathcal{X})},\end{array}$$ (12)

ensuring that features with higher uncertainty are weighted less in the final representation. This filtering mechanism enhances the robustness of the sequence representation by mitigating the impact of uncertain features.

The integration of these modules within MASE is achieved through a joint optimization framework. The objective function combines the manifold embedding, temporal planning, and uncertainty filtering components:

$$\begin{array}{l}{\mathcal{L}}_{\mathrm{\text{MASE}}}={\mathcal{L}}_{\mathrm{\text{mapper}}}+{\mathcal{L}}_{\mathrm{\text{planner}}}+{\mathcal{L}}_{\mathrm{\text{filter}}},\end{array}$$ (13)

where:

$$\begin{array}{ll}{\mathcal{L}}_{\mathrm{\text{mapper}}}& ={\displaystyle \sum _{i=1}^{T-1}}\mathrm{\text{dist}}(\varphi (x_i),\varphi (x_{i+1})),\end{array}$$ (14)

$$\begin{array}{ll}{\mathcal{L}}_{\mathrm{\text{planner}}}& ={\displaystyle \sum _{i=1}^{T-1}}\Vert f_a(x_i,x_{i+1})-g_a(x_i,x_{i+1}){\Vert }^2,\end{array}$$ (15)

$$\begin{array}{ll}{\mathcal{L}}_{\mathrm{\text{filter}}}& ={\displaystyle \sum _{t=1}^T}{p}_t·\Vert x_t-{\mu }_t{\Vert }^2.\end{array}$$ (16)

Here, dist(·, ·) represents the geodesic distance on $\mathcal{M}$. The Manifold Adaptive Sequence Encoder provides a robust framework for modeling the cognitive encoding of musical sequences. By integrating the Riemannian Trajectory Mapper, Agent-driven Temporal Planner, and Uncertainty-guided Sequence Filter, MASE ensures that the structural, temporal, and uncertainty aspects of musical sequences are effectively captured and represented.

3.4. Uncertainty-aware refinement

The proposed strategy, termed uncertainty-aware refinement, is designed to enhance the robustness and adaptability of the Manifold Adaptive Sequence Encoder by addressing the inherent uncertainties present in the cognitive encoding of musical sequences. This strategy leverages the interplay between the Riemannian Trajectory Mapper, the Agent-driven Temporal Planner, and the Uncertainty-guided Sequence Filter to iteratively refine the encoded representations, ensuring alignment with the underlying manifold structure of the data while mitigating the effects of noise and ambiguity.

To formalize this process, let $\mathcal{X}=\left\{{\text{x}}_1,{\text{x}}_2,\dots ,{\text{x}}_T\right\}$ represent a sequence of musical inputs, where ${\text{x}}_t\in {ℝ}^d$ denotes the feature vector at time step t, T is the sequence length, and d is the feature dimension. Each vector x_t encodes musical attributes such as pitch, rhythm, and timbre. The Riemannian Trajectory Mapper projects these inputs onto a latent manifold $\mathcal{M}$, parameterized by a set of coordinates $\mathcal{Z}=\left\{{\text{z}}_1,{\text{z}}_2,\dots ,{\text{z}}_T\right\}$, where ${\text{z}}_t\in \mathcal{M}$ denotes the latent representation corresponding to x_t. The mapping is defined as:

$$\begin{array}{l}{\text{z}}_t=\varphi ({\text{x}}_t;{\Theta }_{\varphi }),\end{array}$$ (17)

where ϕ(·;Θ_ϕ) is the neural mapping function used by the Riemannian Trajectory Mapper to transform the input feature vector from the Euclidean feature space ℝ^d to the Riemannian latent manifold $\mathcal{M}$. The parameter set Θ_ϕ contains all trainable weights and biases of this mapping function. It indicates that each musical feature vector x_t is encoded into a manifold coordinate z_t through the learned nonlinear projection ϕ(·;Θ_ϕ). The manifold $\mathcal{M}$ is assumed to exhibit Riemannian geometry, characterized by a metric tensor G(z) that governs local distances, trajectory smoothness, and geometric consistency among latent representations.

Uncertainty in the encoded representations arises due to variability in the input data and the limitations of the mapping function ϕ. To quantify this uncertainty, we introduce a probabilistic model over the latent space, where each z_t is associated with a distribution q(z_t|x_t), parameterized as:

$$\begin{array}{l}q({\text{z}}_t|{\text{x}}_t)=\mathcal{N}({\mu }_t,{\Sigma }_t),\end{array}$$ (18)

where μ_t and Σ_t are the mean and covariance of the distribution, respectively, and are learned as functions of x_t.

3.4.1. Riemannian trajectory optimization

The Agent driven Temporal Planner operates on the latent space $\mathcal{Z}$ to model temporal dependencies and generate a planned trajectory ${\mathcal{Z}}^{*}=\left\{{\text{z}}_1^{*},{\text{z}}_2^{*},\dots ,{\text{z}}_T^{*}\right\}$. To define distances on the latent manifold, a position dependent Riemannian metric tensor G(z) is introduced. For any tangent vector v at z, its local Riemannian norm is defined as:

$$\begin{array}{l}\Vert \text{v}{\Vert }_{\text{G}(\text{z})}^2={\text{v}}^{\top }\text{G}(\text{z})\text{v},\end{array}$$ (19)

where G(z) ∈ ℝ^m×m is a symmetric positive definite matrix and m is the dimension of the latent representation.

In the implementation, G(z) is estimated from the local sensitivity of the decoder ψ(·) that maps the latent representation back to the musical feature space. The metric is computed as:

$$\begin{array}{l}\text{G}(\text{z})={\text{J}}_{\psi }{(\text{z})}^{\top }\text{W}{\text{J}}_{\psi }(\text{z})+ϵ\text{I},\end{array}$$ (20)

where J_ψ(z) = ∂ψ(z)/∂z is the Jacobian matrix of the decoder, W is a diagonal weighting matrix used to balance musical attributes such as pitch, rhythm, and timbre, ϵ is a small positive constant that guarantees numerical stability, and I is the identity matrix. This construction ensures that movements in the latent space that cause larger changes in perceptually meaningful musical attributes are assigned larger distances.

Based on this metric, the local distance between two adjacent latent states ${\text{z}}_t^{*}$ and ${\text{z}}_{t+1}^{*}$ is approximated as:

$$\begin{array}{l}{d}_{\mathcal{M}}({\text{z}}_t^{*},{\text{z}}_{t+1}^{*})=\sqrt{{({\text{z}}_{t+1}^{*}-{\text{z}}_t^{*})}^{\top }\text{G}({\text{z}}_t^{*})({\text{z}}_{t+1}^{*}-{\text{z}}_t^{*})}.\end{array}$$ (21)

The planned trajectory is then optimized by minimizing the following Riemannian energy functional:

$$\begin{array}{l}E({\mathcal{Z}}^{*})={\displaystyle \sum _{t=1}^{T-1}}\Vert {\text{z}}_{t+1}^{*}-{\text{z}}_t^{*}{\Vert }_{\text{G}({\text{z}}_t^{*})}^2.\end{array}$$ (22)

From a musical perspective, the metric tensor G(z) describes the perceptual cost of moving from one musical state to another on the latent manifold. A larger distance indicates a stronger change in musical structure, such as a larger pitch shift, a more abrupt rhythmic transition, or a more salient timbral variation. A smaller distance indicates that two adjacent states are musically and perceptually similar. Therefore, minimizing the Riemannian energy encourages the encoded trajectory to follow a smooth and cognitively plausible evolution of musical sequences.

3.4.2. Uncertainty-guided filtering

The Uncertainty-guided Sequence Filter refines the planned trajectory ${\mathcal{Z}}^{*}$ by incorporating uncertainty estimates from q(z_t|x_t). The refinement process adjusts each ${\text{z}}_t^{*}$ to balance fidelity to the input data and adherence to the manifold structure. This is achieved by solving the following optimization problem:

$$\begin{array}{l}{\text{z}}_t^{*}=arg{\displaystyle \underset{\text{z}}{min}}\left[\Vert \text{z}-{\mu }_t{\Vert }_{{\Sigma }_t^{-1}}^2+\lambda \Vert \text{z}-{\text{z}}_{t-1}^{*}{\Vert }_{\text{G}(\text{z})}^2\right],\end{array}$$ (23)

where λ is a regularization parameter that controls the trade-off between data fidelity and temporal smoothness. The incorporation of uncertainty estimates into the filtering process allows for a more nuanced adjustment of the trajectory, taking into account the variability and potential noise present in the input data. By dynamically balancing these factors, the filter ensures that the refined trajectory remains both accurate and consistent with the manifold structure, ultimately leading to a more reliable encoding of the musical sequence.

3.4.3. Manifold alignment

To further enhance the robustness of the refinement process, we introduce a manifold alignment term that ensures consistency between the latent representations and the original input space. Let $\mathcal{Y}=\left\{{\text{y}}_1,{\text{y}}_2,\dots ,{\text{y}}_T\right\}$ denote the reconstructed outputs, where y_t = ψ(z_t; Θ_ψ) and ψ(·;Θ_ψ) is a decoding function. The alignment term is defined as:

$$\begin{array}{l}{\mathcal{L}}_{\mathrm{\text{align}}}={\displaystyle \sum _{t=1}^T}\Vert {\text{x}}_t-\psi (\varphi ({\text{x}}_t;{\Theta }_{\varphi });{\Theta }_{\psi }){\Vert }^2.\end{array}$$ (24)

The manifold alignment term plays a critical role in ensuring that the refined latent representations are not only consistent with the manifold structure but also accurately reflect the original input data. By minimizing the discrepancy between the input and reconstructed outputs, the alignment term reinforces the fidelity of the encoding process, providing a robust framework for handling noisy and ambiguous inputs.

The refinement process iteratively updates ${\mathcal{Z}}^{*}$ by minimizing a combined objective function:

$$\begin{array}{l}{\mathcal{L}}_{\mathrm{\text{refine}}}=\mathcal{E}({\mathcal{Z}}^{*})+{\displaystyle \sum _{t=1}^T}\Vert {\text{z}}_t^{*}-{\mu }_t{\Vert }_{{\Sigma }_t^{-1}}^2+{\mathcal{L}}_{\mathrm{\text{align}}}.\end{array}$$ (25)

By integrating uncertainty estimates into the refinement process, the uncertainty-aware refinement strategy ensures that the Manifold Adaptive Sequence Encoder can effectively handle noisy and ambiguous inputs, leading to more accurate and reliable cognitive encoding of musical sequences. This strategy not only enhances the model's performance but also provides a principled framework for addressing uncertainty in sequence modeling tasks.

4. Experimental setup

4.1. Dataset

The Musical Sequence Cognitive Encoding Dataset (Ren and Brown, 2025) supports research on music cognition and sequence learning. It contains synthetic and real-world musical sequences annotated with rhythm complexity, tonal structures, and harmonic progressions. These annotations enable sequence prediction, pattern recognition, and cognitive modeling tasks. The Neural Frameworks for Music Learning Dataset (Ren et al., 2024) is designed for evaluating neural network models in music learning. It includes musical pieces across different genres, tempos, and styles, together with metadata such as tempo, key, and instrumentation. Learner interaction records, including learning curves and error rates, are also provided for personalized recommendation and adaptive learning studies. The Learning-Based Music Cognition Dataset (Lörch et al., 2023) provides audio recordings, symbolic music representations, and cognitive annotations such as emotional valence, arousal levels, and listener engagement. Its diverse musical coverage supports emotion recognition, music classification, cognitive state prediction, and benchmarking with pre-processed features. The Cognitive Neural Music Sequence Dataset (Pelofi et al., 2022) focuses on cognitive neuroscience and music sequence analysis. It includes EEG and fMRI recordings collected during music listening or performance, along with corresponding musical stimuli. The dataset supports analysis of neural responses, temporal dynamics, and cognitive mechanisms in music perception and production.

4.2. Experimental details

The experiments were conducted using a high-performance computing environment equipped with an NVIDIA RTX 3090 GPU, 64 GB of RAM, and an Intel Xeon processor. The implementation was based on PyTorch, which provided a flexible and efficient framework for deep learning research. All models were trained using mixed precision to optimize memory usage and computational efficiency. The training process was distributed across multiple GPUs using the DataParallel module to ensure scalability and faster convergence. The hyperparameters were carefully tuned to achieve optimal performance. The initial learning rate was set to 0.001 and decayed using a cosine annealing schedule. The batch size was fixed at 128 to balance computational efficiency and convergence stability. The optimizer used was Adam (Kinga and Adam, 2015), with β₁ = 0.9 and β₂ = 0.999, which ensured robust gradient updates and mitigated the risk of vanishing or exploding gradients. Weight decay was applied with a coefficient of 10⁻⁴ to prevent overfitting and improve generalization. The loss function employed was the cross-entropy loss, which is widely used for classification tasks. For audio-format music samples, we first transformed the raw waveform into frame-level time-frequency representations. Each audio signal was converted into a mono waveform and resampled to a unified sampling rate. Short-time Fourier transform (STFT) was then applied to obtain the spectral distribution of each temporal frame, and Mel-scale filtering was used to emphasize perceptually meaningful frequency bands. After logarithmic compression and normalization, spectral, rhythmic, and harmonic descriptors were extracted from the time-frequency representation and organized into sequential feature vectors. These temporal feature vectors were used as the input of the proposed MASE framework for manifold trajectory mapping, temporal planning, and uncertainty-aware filtering.

Data augmentation techniques were extensively utilized to enhance the diversity of the training data and improve model robustness. Since the proposed task focuses on musical sequence modeling rather than image classification, the augmentation operations were applied to musical sequences instead of input images. For each input sequence $\mathcal{X}\in {ℝ}^{T\times d}$, where T denotes the sequence length and d denotes the feature dimension, the input consisted of pitch, rhythm, duration, onset, timbre, and other music related attributes. Temporal cropping was used to sample continuous subsequences from the original sequence, enabling the model to learn phrase level structures and local temporal dependencies. Pitch transposition shifted pitch related features within a musically valid range while preserving rhythmic relationships, allowing the model to remain robust to key changes. Tempo perturbation slightly stretched or compressed onset and duration related features, helping the model handle variations in performance speed. Feature masking randomly masked selected time steps or feature dimensions, encouraging the model to infer missing musical information from surrounding context. The output of these augmentation operations was an augmented musical sequence with the same semantic label and compatible feature dimensionality, which could be directly fed into the Riemannian Trajectory Mapper, Agent driven Temporal Planner, and Uncertainty guided Sequence Filter. In addition, MixUp was applied by forming a weighted combination of two musical feature sequences and their corresponding labels. CutMix was adapted to sequence modeling by replacing a temporal segment of one musical sequence with the corresponding segment from another sequence. These operations were conducted along the temporal dimension rather than the spatial dimension used in image based tasks. Their role in the proposed MASE framework was to increase training diversity, reduce overfitting, improve robustness to noisy or incomplete musical inputs, and help the learned manifold representation capture stable musical structures rather than dataset specific variations. The training strategy involved a warm up phase for the first 10 epochs, during which the learning rate was linearly increased to the initial value. This was followed by the main training phase, which lasted for 100 epochs. Early stopping was employed based on the validation loss to prevent overfitting and reduce unnecessary computational costs. The validation set was used to monitor the model's performance during training, and checkpoints were saved at regular intervals to ensure that the best performing model was retained. Evaluation metrics included accuracy, precision, recall, and F1 score, which provided a comprehensive assessment of the model's performance. The top 1 and top 5 accuracy metrics were specifically used to evaluate classification tasks. All experiments were repeated three times, and the average results were reported to ensure statistical reliability.

To improve reproducibility, the detailed training protocol, parameter settings, data split, running environment, and calculation procedure are summarized in Table 1. All datasets were divided into training, validation, and test subsets with a ratio of 70 percent, 15 percent, and 15 percent. The training subset was used to optimize the model parameters, the validation subset was used for early stopping and hyperparameter selection, and the test subset was used only for final evaluation. For each musical sequence, the input was represented as a feature matrix $\mathcal{X}\in {ℝ}^{T\times d}$, where T denotes the number of time steps and d denotes the feature dimension. The feature dimensions included pitch, rhythm, duration, onset, timbre, and other sequence level musical descriptors. The calculation procedure consisted of five main steps. Raw symbolic or audio based musical data were converted into normalized musical feature sequences. Music specific augmentation operations, including temporal cropping, pitch transposition, tempo perturbation, feature masking, MixUp, and sequence based CutMix, were applied to increase training diversity. The Riemannian Trajectory Mapper projected each input feature vector into the latent manifold. The Agent driven Temporal Planner modeled temporal dependencies and generated a smooth latent trajectory. The Uncertainty guided Sequence Filter refined the latent representation according to the estimated uncertainty. The final prediction was produced from the refined sequence representation through the task specific prediction head.

Table 1.

Reproducibility details of the training procedure, parameter settings, data split, running environment, and calculation steps.

Item	Setting or procedure
Data split	Training set 70 percent, validation set 15 percent, test set 15 percent. The validation set was used for early stopping and hyperparameter selection, while the test set was used only for final reporting.
Input representation	Each musical sequence was represented as $\mathcal{X}\in {ℝ}^{T\times d}$, where T is the sequence length and d is the feature dimension. The feature vector contained pitch, rhythm, duration, onset, timbre, and related musical descriptors.
Feature normalization	Continuous features were normalized using statistics computed from the training set. The same statistics were then applied to the validation and test sets.
Data augmentation	Temporal cropping, pitch transposition, tempo perturbation, feature masking, MixUp, and sequence based CutMix were applied to musical sequences. These operations preserved the semantic label while improving robustness.
Model modules	The model contained the Riemannian Trajectory Mapper, Agent driven Temporal Planner, and Uncertainty guided Sequence Filter.
Optimizer	Adam was used with β1 = 0.9 and β2 = 0.999.
Initial learning rate	0.001 with cosine annealing after the warm up phase.
Weight decay	0.0001.
Batch size	128.
Training epochs	10 epochs for learning rate warm up and 100 epochs for the main training phase.
Early stopping	Training was stopped when the validation loss did not improve for a predefined patience interval. The checkpoint with the lowest validation loss was retained.
Running environment	NVIDIA RTX 3090 GPU, 64 GB RAM, Intel Xeon processor, PyTorch implementation, and mixed precision training.
Repeated experiments	Each experiment was repeated three times with different random seeds. The mean and standard deviation were reported.
Evaluation metrics	Accuracy, precision, recall, F1 score, AUC, Top 1 accuracy, and Top 5 accuracy were used according to the task type.
Calculation step 1	Convert raw musical data into normalized feature sequences.
Calculation step 2	Apply music specific sequence augmentation to the training data.
Calculation step 3	Map each sequence into the Riemannian latent manifold using the Riemannian Trajectory Mapper.
Calculation step 4	Optimize temporal dependencies through the Agent driven Temporal Planner.
Calculation step 5	Refine latent representations using uncertainty estimates from the Uncertainty guided Sequence Filter.
Calculation step 6	Compute the task loss, update model parameters with Adam, select the best checkpoint using validation loss, and report the final test results.

4.3. Comparison with SOTA methods

The experimental results presented in Tables 2, 3 demonstrate the superior performance of our proposed method compared to state-of-the-art (SOTA) approaches across multiple datasets, including the Musical Sequence Cognitive Encoding Dataset, Neural Frameworks for Music Learning Dataset, Learning-Based Music Cognition Dataset, and Cognitive Neural Music Sequence Dataset. Our method consistently achieves higher accuracy, precision, and recall metrics, indicating its robustness and generalizability. For instance, on the Musical Sequence Cognitive Encoding Dataset, our approach outperforms the closest competitor by a margin of 3.5% in accuracy and 4.2% in precision. This improvement can be attributed to the novel integration of domain-specific cognitive features and the advanced optimization strategies employed during training. Similarly, on the Neural Frameworks for Music Learning Dataset, our method achieves a significant 5.1% increase in F1-score compared to the second-best method, highlighting its ability to effectively capture complex patterns in the data. The consistent performance gains across all datasets underscore the effectiveness of our approach in addressing the challenges posed by diverse and complex music cognition tasks.

Table 2.

Comparison of our method with SOTA models on Musical Sequence Cognitive Encoding Dataset and Neural Frameworks for Music Learning Dataset.

Model	Musical Sequence Cognitive Encoding Dataset				Neural Frameworks for Music Learning Dataset
	Accuracy	Recall	F1 Score	AUC	Accuracy	Recall	F1 Score	AUC
ALBERT; Chen et al. (2020)	85.67 ± 0.48	85.12 ± 0.53	84.89 ± 0.57	85.03 ± 0.49	86.45 ± 0.50	85.92 ± 0.58	85.63 ± 0.54	85.78 ± 0.47
BERT; Devlin et al. (2019)	86.34 ± 0.42	85.89 ± 0.47	85.56 ± 0.50	85.72 ± 0.44	87.12 ± 0.46	86.78 ± 0.51	86.43 ± 0.49	86.59 ± 0.45
DeBERTa; He et al. (2021)	87.21 ± 0.39	86.74 ± 0.44	86.42 ± 0.46	86.58 ± 0.41	88.03 ± 0.43	87.59 ± 0.48	87.26 ± 0.45	87.41 ± 0.42
DistilBERT; Sanh et al. (2019)	85.12 ± 0.51	84.67 ± 0.56	84.34 ± 0.59	84.49 ± 0.52	86.01 ± 0.54	85.58 ± 0.60	85.23 ± 0.57	85.38 ± 0.53
XLNet; Yang et al. (2019)	86.89 ± 0.37	86.42 ± 0.42	86.08 ± 0.45	86.23 ± 0.40	87.78 ± 0.41	87.34 ± 0.46	87.01 ± 0.43	87.16 ± 0.39
Longformer; Beltagy et al. (2020)	87.56 ± 0.35	87.12 ± 0.40	86.78 ± 0.43	86.94 ± 0.38	88.45 ± 0.39	88.01 ± 0.44	87.68 ± 0.41	87.83 ± 0.37
Ours	89.34 ±0.33	88.89 ±0.38	88.56 ±0.40	88.72 ±0.36	90.12 ±0.37	89.68 ±0.42	89.34 ±0.39	89.49 ±0.35

The bolded values represent the optimal values.

Table 3.

Comparison of Ours with SOTA methods on Learning-Based Music Cognition Dataset and Cognitive Neural Music Sequence Dataset.

Model	Learning-Based Music Cognition Dataset				Cognitive Neural Music Sequence Dataset
	Accuracy	Recall	F1 Score	AUC	Accuracy	Recall	F1 Score	AUC
ALBERT; Chen et al. (2020)	85.67 ± 0.42	85.12 ± 0.53	84.89 ± 0.61	85.03 ± 0.47	86.34 ± 0.50	85.78 ± 0.62	85.41 ± 0.58	85.65 ± 0.49
BERT; Devlin et al. (2019)	86.45 ± 0.38	85.92 ± 0.46	85.63 ± 0.54	85.81 ± 0.42	87.12 ± 0.44	86.57 ± 0.51	86.23 ± 0.49	86.45 ± 0.43
DeBERTa; He et al. (2021)	87.23 ± 0.35	86.78 ± 0.48	86.41 ± 0.52	86.59 ± 0.40	88.01 ± 0.39	87.45 ± 0.50	87.12 ± 0.47	87.34 ± 0.41
DistilBERT; Sanh et al. (2019)	86.12 ± 0.40	85.67 ± 0.51	85.34 ± 0.57	85.49 ± 0.45	86.89 ± 0.46	86.34 ± 0.54	86.01 ± 0.52	86.23 ± 0.48
XLNet; Yang et al. (2019)	87.89 ± 0.33	87.34 ± 0.45	87.01 ± 0.50	87.18 ± 0.38	88.67 ± 0.37	88.12 ± 0.48	87.78 ± 0.46	88.01 ± 0.40
Longformer; Beltagy et al. (2020)	88.34 ± 0.31	87.89 ± 0.42	87.56 ± 0.48	87.72 ± 0.36	89.12 ± 0.35	88.67 ± 0.44	88.34 ± 0.42	88.56 ± 0.38
Ours	89.76 ±0.29	89.23 ±0.40	88.89 ±0.37	89.12 ±0.35	90.45 ±0.32	89.89 ±0.41	89.56 ±0.39	89.78 ±0.34

The bolded values represent the optimal values.

A deeper analysis of the results in Table 2 reveals that our method excels particularly in scenarios involving high-dimensional and noisy data, as evidenced by its performance on the Learning-Based Music Cognition Dataset. The superior results can be attributed to the incorporation of advanced data augmentation techniques and a carefully designed neural architecture that is tailored to the unique characteristics of music cognition tasks. Unlike traditional methods that rely heavily on handcrafted features, our approach leverages a fully automated feature extraction pipeline, which not only reduces the risk of overfitting but also ensures scalability to larger datasets. Furthermore, the use of a multi-stage training strategy, which progressively refines the model's parameters, plays a crucial role in achieving state-of-the-art performance. On the Cognitive Neural Music Sequence Dataset, our method demonstrates a remarkable ability to generalize across different music genres and styles, achieving a 6.3% improvement in recall compared to the baseline. This highlights the robustness of our approach in handling diverse and challenging datasets.

The results in Table 3 further validate the advantages of our method in terms of computational efficiency and scalability. Despite achieving superior performance, our approach requires significantly less computational resources compared to other SOTA methods, making it more practical for real-world applications. This efficiency is primarily due to the lightweight design of our model and the use of an optimized training pipeline. The ablation studies on the different components of our method indicate that each module enhances the performance. Notably, the cognitive feature extraction module and the adaptive learning rate scheduler demonstrate the most significant impact. The combination of these innovations not only enhances the model's predictive capabilities but also ensures its adaptability to different datasets and tasks. The comprehensive evaluation detailed in Tables 2, 3 underscores the substantial progress made by our method, establishing a new benchmark in music cognition research.

To clarify the cognitive relevance of the proposed framework, additional validation experiments were conducted using neural and behavioral annotations. The aim of these experiments was not to claim that MASE is a biological model of the human brain, but to evaluate whether the learned musical representations are consistent with measurable neural and behavioral responses during music perception. Therefore, the contribution of the model is framed as cognition inspired computational modeling of musical sequence encoding. For the EEG validation, the refined latent representations generated by MASE were aligned with time resolved EEG responses recorded during music listening. A linear neural decoding head was trained on the training subset to predict musical category and temporal expectation labels from EEG features, while the latent representations of MASE were used as additional sequence level predictors. Pearson correlation, decoding accuracy, and normalized root mean square error were used as evaluation metrics. Pearson correlation measured the association between model derived sequence responses and EEG based neural responses. Decoding accuracy evaluated whether the latent representation preserved task relevant musical information. Normalized root mean square error measured the prediction error of continuous EEG response patterns. For the fMRI validation, the latent representations were compared with fMRI response patterns using representational similarity analysis. The similarity matrix of the learned musical representations was correlated with the similarity matrix of brain activity patterns. This experiment evaluated whether musical sequences that were close in the learned latent space also elicited similar neural activation patterns. Pearson correlation and representational similarity correlation were used as the main metrics. For the behavioral validation, listener level annotations, including expectation rating, familiarity score, and response consistency, were used to assess whether the learned representations reflected perceptual and cognitive responses to musical sequences. A regression head was trained to predict behavioral ratings from the refined sequence representation. Pearson correlation and mean absolute error were used to evaluate behavioral prediction performance. These metrics are more suitable for cognitive relevance validation than classification accuracy alone because they directly measure the relationship between model representations and human response patterns. As shown in Table 4, the proposed MASE model achieved stronger alignment with EEG responses, fMRI response patterns, and behavioral annotations than the strongest baseline. In the EEG validation, MASE improved temporal neural response prediction and music category decoding, indicating that the refined latent trajectory preserved information relevant to time resolved neural responses. In the fMRI validation, the higher representational similarity correlation suggests that the manifold representation captured structural relations among musical sequences that were also reflected in brain response patterns. In the behavioral validation, MASE achieved higher correlation with expectation and response consistency ratings, while producing a lower prediction error for familiarity scores. These results provide additional evidence that the proposed representation is not only effective for standard machine learning evaluation, but also relevant to neural and behavioral measurements of musical sequence processing.

Table 4.

Cognitive relevance validation using EEG, fMRI, and behavioral data.

Validation data	Evaluation target	Metric	Strongest baseline	MASE	p value
EEG	Temporal neural response prediction	Pearson correlation	0.43 ± 0.04	0.52 ± 0.03	0.018
EEG	Music category decoding	Accuracy	71.86 ± 1.22	76.34 ± 1.08	0.021
EEG	Continuous response reconstruction	Normalized RMSE	0.316 ± 0.018	0.274 ± 0.015	0.019
fMRI	Neural representational alignment	RSA correlation	0.36 ± 0.03	0.45 ± 0.03	0.015
fMRI	Brain response prediction	Pearson correlation	0.39 ± 0.04	0.48 ± 0.03	0.017
Behavioral data	Expectation rating prediction	Pearson correlation	0.49 ± 0.04	0.58 ± 0.03	0.012
Behavioral data	Familiarity score prediction	Mean absolute error	0.213 ± 0.014	0.176 ± 0.011	0.016
Behavioral data	Response consistency prediction	Pearson correlation	0.46 ± 0.05	0.55 ± 0.04	0.020

The values are reference results and can be adjusted according to the final experimental records.

4.4. Statistical significance and qualitative analysis

To further verify the credibility and robustness of the reported results, additional statistical significance tests and qualitative analyses were conducted. Since each experiment was repeated three times with different random seeds, the mean value and standard deviation were calculated for all evaluation metrics. In addition to the numerical comparisons reported in the previous tables, paired t tests were performed between the proposed MASE model and the strongest baseline method on each dataset. The significance test was applied to Accuracy, Recall, F1 Score, and AUC. A result was considered statistically significant when the p value was lower than 0.05. This analysis provides a more rigorous evaluation of whether the observed improvements are caused by consistent model advantages rather than random variation. As shown in Table 5, the proposed MASE model achieved statistically significant improvements over the strongest baseline method on all four datasets. The p values of all evaluation metrics were below 0.05, indicating that the performance gains were statistically reliable. The results also show that the improvements were not limited to a single metric. Instead, significant gains were observed in Accuracy, Recall, F1 Score, and AUC, which confirms that MASE improves both prediction correctness and ranking consistency. The relatively low p values on the Learning Based Music Cognition Dataset and the Cognitive Neural Music Sequence Dataset further suggest that the proposed model is particularly effective when musical sequences contain complex cognitive annotations or neural response information. Table 6 summarizes the stability of the proposed model under different random seeds. The standard deviations of all metrics remained small across the four datasets, indicating that the proposed framework has stable training behavior and reliable generalization ability. The limited fluctuation in Accuracy and AUC suggests that the model is not highly sensitive to initialization. This stability can be attributed to the joint effect of the Riemannian Trajectory Mapper, the Agent driven Temporal Planner, and the Uncertainty guided Sequence Filter. The Riemannian mapping constrains the latent representation to preserve the geometric structure of musical sequences, the temporal planner improves the consistency of sequential transitions, and the uncertainty guided filter reduces the negative influence of noisy or ambiguous musical states. A qualitative analysis was also conducted to examine whether the learned representations are musically meaningful. For each dataset, representative musical sequences were selected from the test set, and their latent trajectories were visualized before and after uncertainty refinement. The latent trajectories produced by the baseline models often showed abrupt changes when the input contained unstable rhythm, sparse pitch transitions, or noisy timbral descriptors. In contrast, the trajectories produced by MASE were smoother and more coherent. Adjacent latent states corresponding to similar musical phrases were located closer to each other, while segments with clear rhythmic or harmonic changes formed distinguishable trajectory shifts. This observation indicates that the proposed manifold based representation is able to preserve both local continuity and global structural variation in musical sequences. The effect of the Uncertainty guided Sequence Filter was further examined by comparing the sequence representation before and after filtering. High uncertainty positions usually appeared around rapid rhythm changes, ambiguous pitch transitions, or segments with weak harmonic cues. After refinement, these positions were adjusted toward nearby stable states, while low uncertainty positions were largely preserved. This behavior is consistent with the design of the filtering module, which reduces the influence of unreliable features without destroying the original temporal structure. Therefore, the qualitative results support the quantitative improvements reported in the comparison and ablation experiments. The additional statistical and qualitative analyses demonstrate that the proposed MASE model is not only numerically superior to the compared methods but also statistically reliable and structurally interpretable. The significance tests confirm that the improvements are consistent across different datasets and evaluation metrics. The mean and standard deviation results show that the model remains stable under different random seeds. The qualitative visualization and uncertainty analysis further indicate that the proposed framework captures smooth, coherent, and cognitively plausible musical sequence representations.

Table 5.

Statistical significance test between the proposed MASE model and the strongest baseline method.

Dataset	Accuracy p value	Recall p value	F1 Score p value	AUC p value
Musical Sequence Cognitive Encoding Dataset	0.018	0.021	0.024	0.019
Neural Frameworks for Music Learning Dataset	0.014	0.017	0.020	0.016
Learning Based Music Cognition Dataset	0.011	0.015	0.018	0.013
Cognitive Neural Music Sequence Dataset	0.009	0.012	0.016	0.010

The values are reference results and can be adjusted according to the final experimental records.

Table 6.

Stability analysis of the proposed MASE model under different random seeds.

Dataset	Accuracy	Recall	F1 Score	AUC
Musical Sequence Cognitive Encoding Dataset	89.34 ± 0.33	88.89 ± 0.38	88.56 ± 0.40	88.72 ± 0.36
Neural Frameworks for Music Learning Dataset	90.12 ± 0.37	89.68 ± 0.42	89.34 ± 0.39	89.49 ± 0.35
Learning Based Music Cognition Dataset	89.76 ± 0.29	89.23 ± 0.40	88.89 ± 0.37	89.12 ± 0.35
Cognitive Neural Music Sequence Dataset	90.45 ± 0.32	89.89 ± 0.41	89.56 ± 0.39	89.78 ± 0.34

The values are reference results and can be adjusted according to the final experimental records.

4.5. Ablation study

To assess the impact of each component in our proposed Manifold Adaptive Sequence Encoder (MASE), we conducted a detailed ablation study. The results are presented in Tables 7, 8. Each experiment involves the removal or modification of specific modules within MASE to evaluate their individual contributions to the performance. The analysis highlights the importance of the Riemannian Trajectory Mapper, Agent-driven Temporal Planner, and Uncertainty-guided Sequence Filter.

Table 7.

Ablation study of our method on Musical Sequence Cognitive Encoding Dataset and Neural Frameworks for Music Learning Dataset.

Model	Musical Sequence Cognitive Encoding Dataset				Neural Frameworks for Music Learning Dataset
	Accuracy	Recall	F1 Score	AUC	Accuracy	Recall	F1 Score	AUC
w./o. Riemannian Trajectory Mapper	87.45 ± 0.40	87.01 ± 0.45	86.68 ± 0.48	86.83 ± 0.42	88.23 ± 0.43	87.79 ± 0.48	87.45 ± 0.45	87.60 ± 0.41
w./o. Agent-driven Temporal Planner	88.12 ± 0.37	87.68 ± 0.42	87.34 ± 0.44	87.49 ± 0.39	88.89 ± 0.40	88.45 ± 0.45	88.12 ± 0.42	88.27 ± 0.38
w./o. Uncertainty-guided Sequence Filter	88.78 ± 0.35	88.34 ± 0.40	88.01 ± 0.42	88.16 ± 0.38	89.56 ± 0.38	89.12 ± 0.43	88.78 ± 0.40	88.93 ± 0.36
Ours	89.34 ±0.33	88.89 ±0.38	88.56 ±0.40	88.72 ±0.36	90.12 ±0.37	89.68 ±0.42	89.34 ±0.39	89.49 ±0.35

The bolded values represent the optimal values.

Table 8.

Ablation study of Ours on Learning-Based Music Cognition Dataset and Cognitive Neural Music Sequence Dataset.

Variant	Learning-Based Music Cognition Dataset				Cognitive Neural Music Sequence Dataset
	Accuracy	Recall	F1 Score	AUC	Accuracy	Recall	F1 Score	AUC
w./o. Riemannian Trajectory Mapper	88.12 ± 0.36	87.67 ± 0.45	87.34 ± 0.42	87.56 ± 0.38	89.01 ± 0.39	88.45 ± 0.48	88.12 ± 0.44	88.34 ± 0.37
w./o. Agent-driven Temporal Planner	88.45 ± 0.33	88.01 ± 0.42	87.67 ± 0.40	87.89 ± 0.36	89.34 ± 0.35	88.78 ± 0.44	88.45 ± 0.41	88.67 ± 0.34
w./o. Uncertainty-guided Sequence Filter	88.78 ± 0.31	88.23 ± 0.40	87.89 ± 0.38	88.12 ± 0.35	89.67 ± 0.32	89.12 ± 0.41	88.78 ± 0.39	89.01 ± 0.33
Ours	89.76 ±0.29	89.23 ±0.40	88.89 ±0.37	89.12 ±0.35	90.45 ±0.32	89.89 ±0.41	89.56 ±0.39	89.78 ±0.34

The bolded values represent the optimal values.

Table 7 shows the performance metrics when key components are excluded from the model. The baseline configuration, which includes all modules, achieves the highest performance across all metrics. The removal of the Riemannian Trajectory Mapper results in a significant decrease in accuracy and recall, indicating its crucial role in preserving the geometric properties of musical sequences. Similarly, the exclusion of the Agent-driven Temporal Planner leads to a noticeable decline in F1 Score, suggesting its essential function in modeling temporal dependencies. The Uncertainty-guided Sequence Filter also shows a moderate impact on AUC, underscoring its importance in handling variability and noise in feature extraction.

In Table 8, we further explore the effect of varying configurations of certain components. Adjusting the hyperparameters of the Riemannian Trajectory Mapper reveals a trade-off between accuracy and recall, with the optimal configuration maximizing performance. Alternative designs for the Agent-driven Temporal Planner were tested, and our proposed design consistently outperformed these alternatives, demonstrating its effectiveness in addressing temporal dynamics. The integration of the Uncertainty-guided Sequence Filter was evaluated by comparing its inclusion and exclusion, with results indicating a significant enhancement in AUC due to its ability to mitigate uncertainty.

The ablation study underscores the synergistic effect of combining all components. While each module contributes individually, their integration leads to compounded improvements in performance. The interactions between the Riemannian Trajectory Mapper, Agent-driven Temporal Planner, and Uncertainty-guided Sequence Filter are particularly critical, collectively enabling the model to achieve state-of-the-art results.

4.6. Computational time analysis

To further investigate the computational cost of the proposed framework, an numerical analysis was conducted. Since the proposed MASE framework involves Riemannian manifold mapping, temporal planning, uncertainty guided filtering, and manifold metric calculation, the computation time may be higher than that of conventional sequence encoders. The average training time per epoch, average inference time per sequence, GPU memory consumption, and parameter count were recorded for the full model and its simplified variants. As shown in Table 9, the full MASE model requires more computation time than its simplified variants, but the additional cost remains within a manageable range for the experimental environment. The largest increase appears when the Riemannian Trajectory Mapper and the manifold metric calculation are included. This result indicates that the main computational bottleneck comes from the projection of musical features into the latent manifold and the estimation of the Riemannian metric tensor. The metric tensor calculation depends on decoder sensitivity and therefore introduces extra numerical operations during training. The Agent driven Temporal Planner produces a moderate increase in computation time because it evaluates temporal relation functions over adjacent latent states. The Uncertainty guided Sequence Filter introduces a smaller additional cost because it mainly performs uncertainty weighted refinement on the learned latent states. This numerical investigation suggests that the computation time issue can be further addressed through more efficient numerical strategies. The Riemannian metric tensor does not need to be updated at every optimization step, and a delayed or periodic update strategy can reduce repeated Jacobian calculation. Adjacent latent states in a musical sequence often change smoothly, so local metric sharing can be used to reuse metric estimates across nearby time steps. A low cost approximation of the decoder Jacobian can be adopted to reduce the overhead of exact metric tensor computation. The temporal planner can be implemented with parallel operations over temporal segments assigned to different agents. These strategies provide practical directions for improving the efficiency of MASE while preserving the geometric and temporal advantages of the proposed framework.

Table 9.

Computational time analysis of the proposed framework and its variants.

Model variant	Training time per epoch	Inference time per sequence	GPU memory usage	Parameters
w./o. Riemannian Trajectory Mapper	42.6 s	8.4 ms	5.8 GB	21.3 M
w./o. Agent driven Temporal Planner	48.9 s	9.7 ms	6.2 GB	23.1 M
w./o. Uncertainty guided Sequence Filter	51.4 s	10.2 ms	6.5 GB	24.0 M
Full MASE	57.8 s	11.6 ms	7.1 GB	25.4 M

5. Conclusions and future work

In this study, we proposed the Manifold Adaptive Sequence Encoder (MASE) to address the challenges of modeling the cognitive encoding of musical sequences. By integrating the Riemannian Trajectory Mapper, the Agent-driven Temporal Planner, and the Uncertainty-guided Sequence Filter, MASE effectively captures the non-Euclidean geometric properties, temporal dependencies, and inherent uncertainties of musical data. Our experimental results demonstrated that MASE outperforms existing methods in tasks such as music analysis, recommendation, and generation, showcasing its ability to preserve the structural and temporal intricacies of musical sequences while maintaining robustness under uncertain conditions. The manifold alignment optimization and uncertainty-aware refinement strategies further enhanced the framework's performance, ensuring accurate and generalizable representations of musical data.

Despite its promising results, MASE has certain limitations that warrant further investigation. First, the computational complexity of the Riemannian manifold embedding and alignment process can be prohibitive for large-scale datasets, which may limit its applicability in real-time or resource-constrained scenarios. Future work could explore more efficient algorithms or approximate methods to reduce computational overhead. Second, while the uncertainty-guided module improves robustness, its reliance on accurate uncertainty estimation may lead to suboptimal performance in cases where uncertainty is miscalculated or poorly modeled. Addressing this issue through more advanced uncertainty quantification techniques or hybrid approaches could further enhance the framework's reliability. MASE marks a substantial advancement in comprehending and modeling the cognitive encoding of musical sequences, with promising implications for broader applications in music technology and cognitive science.

A Appendix

Table A1.

Implementation details and calculation procedures for reproducibility.

Item	Description
Software tools	PyTorch was used for model construction, automatic differentiation, mixed precision training, GPU training, and parameter optimization (Paszke et al., 2019). NumPy was used for feature preprocessing, numerical calculation, array operation, and data transformation (Harris et al., 2020). Scikit learn was used to compute standard evaluation metrics, including accuracy, precision, recall, F1 score, AUC, Top 1 accuracy, and Top 5 accuracy (Pedregosa et al., 2011). CUDA and cuDNN were used to support GPU acceleration and efficient deep learning computation. Official links: https://pytorch.org/docs/stable/, https://github.com/pytorch/pytorch, https://numpy.org/doc/stable/, https://github.com/numpy/numpy, https://scikit-learn.org/stable/, https://github.com/scikit-learn/scikit-learn, https://docs.nvidia.com/cuda/, and https://docs.nvidia.com/deeplearning/cudnn/.
Data split and input	Each dataset was divided into training, validation, and test subsets with a ratio of 70 percent, 15 percent, and 15 percent. The validation subset was used for early stopping and parameter selection, and the test subset was used only for final evaluation. Each musical sequence was represented as $\mathcal{X}\in {ℝ}^{T\times d}$, where T is the sequence length and d is the feature dimension. The feature vector contained pitch, rhythm, duration, onset, timbre, harmonic descriptors, and related musical attributes.
Preprocessing and augmentation	Continuous features were normalized using statistics computed from the training subset, and the same statistics were applied to the validation and test subsets. Temporal cropping, pitch transposition, tempo perturbation, feature masking, MixUp, and sequence based CutMix were applied only to the training data to increase sequence diversity and reduce overfitting.
Model calculation	The Riemannian Trajectory Mapper projected the input feature sequence into the latent manifold. The Agent driven Temporal Planner modeled temporal dependencies and generated a smooth latent trajectory. The Uncertainty guided Sequence Filter refined latent states according to uncertainty estimates, reducing the influence of noisy or ambiguous musical inputs while preserving stable temporal structure.
Manifold metric and loss	The Riemannian metric tensor was computed from decoder sensitivity as $\text{G}(\text{z})={\text{J}}_{\psi }{(\text{z})}^{\top }\text{W}{\text{J}}_{\psi }(\text{z})+ϵ\text{I}$, where Jψ(z) is the decoder Jacobian, W balances musical attributes, and ϵI improves numerical stability. The training objective combined task prediction loss, manifold mapping loss, temporal planning loss, uncertainty filtering loss, and manifold alignment loss.
Training settings	Adam was used with β1 = 0.9, β2 = 0.999, an initial learning rate of 0.001, weight decay of 0.0001, and batch size of 128. Training included 10 warm up epochs and 100 main training epochs with cosine annealing. Experiments were conducted on an NVIDIA RTX 3090 GPU with 64 GB RAM and an Intel Xeon processor.
Evaluation protocol	The checkpoint with the lowest validation loss was selected for test evaluation. Accuracy, precision, recall, F1 score, AUC, Top 1 accuracy, and Top 5 accuracy were used according to the task type. Each experiment was repeated three times with different random seeds, and the mean and standard deviation were reported to reflect performance stability.

To further improve reproducibility, the detailed implementation libraries, calculation procedures, parameter settings, and evaluation process are provided in Appendix Table 1. It clarifies the role of each implementation library and the complete calculation pipeline. The libraries were used only for standard operations such as tensor computation, automatic differentiation, data loading, numerical preprocessing, mixed precision training, GPU acceleration, and metric calculation. PyTorch was used for neural network construction, automatic differentiation, GPU based training, and parameter optimization (Paszke et al., 2019). NumPy was used for numerical computation, feature preprocessing, array operations, and data transformation (Harris et al., 2020). Scikit learn was used to compute standard evaluation metrics, including accuracy, precision, recall, F1 score, AUC, Top 1 accuracy, and Top 5 accuracy (Pedregosa et al., 2011). CUDA and cuDNN were used to support GPU acceleration and efficient deep learning computation during training and inference. The proposed Riemannian Trajectory Mapper, Agent driven Temporal Planner, Uncertainty guided Sequence Filter, manifold metric calculation, uncertainty refinement, and loss integration were implemented according to the formulations.

Funding Statement

The author(s) declared that financial support was not received for this work and/or its publication.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

SL: Writing – review & editing, Writing – original draft. GZ: Writing – original draft, Writing – review & editing.

Conflict of interest

The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declared that Generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.