Self-supervised multimodal transformer for fine-grained detection of controlled perturbation events in piano performance

Abstract

The acquisition of piano performance skills relies on continuous practice and precise feedback, yet traditional manual evaluation is constrained by time costs and subjective variations, making it difficult to meet the demands of large-scale music education. This study proposes a self-supervised multimodal Transformer framework whose core contribution is the fusion across audio spectral features, symbolic MIDI representations, and a MIDI-derived spatial/kinematic proxy, demonstrating cross-modal attention’s ability to exploit heterogeneous representations under controlled conditions through adaptive fusion mechanisms. Since the MAESTRO dataset lacks video recordings, hand posture features are synthetically derived from MIDI parameters rather than captured from independent visual sensors, representing a kinematic proxy for validating multimodal fusion concepts under controlled conditions. The two-stage training strategy employs contrastive learning, masked prediction, and temporal reconstruction objectives to learn general-purpose music representations during the pretraining phase, and optimizes fine-grained detection capabilities for five error categories of pitch, timing, dynamics, touch, and pedal during the fine-tuning phase, significantly reducing dependence on large-scale annotated data. Experiments on the public MAESTRO dataset validated the substantial advantages of multimodal fusion over unimodal approaches, with the self-supervised pretraining strategy demonstrating stronger generalization capabilities under limited annotation scenarios. Difficulty-level comparison experiments confirmed the model’s robustness in complex performance contexts. The core contribution lies in demonstrating cross-modal attention’s ability to fuse heterogeneous representations across audio, symbolic MIDI, and a MIDI-derived spatial/kinematic proxy under controlled conditions; these findings do not imply that video-based hand pose observation would necessarily yield similar gains, which remains future work.

Introduction

The acquisition of piano performance skills requires persistent training and accurate feedback. Manual evaluation of these skills has some limitations in terms of the available time and the subjective nature of the assessment process, making it less feasible in meeting the demands of a broader musical education process. Technologies that help in the detection of errors enable the monitoring of musical factors in a computerized system¹. However, the finer qualities of performance errors suggest that small differences in timing or touches could affect musical expression², and the current standards of assessment have been questioned for perceptual validity³. While video-audio synchronization-enhanced masked generation models have achieved great progress in multimodal alignment technology⁴, the precision of unsupervised audio representation learning in fine-grained classification tasks still requires breakthrough^5,6. These technical challenges have driven researchers to explore more effective feature extraction and fusion methods.

Early methods relied on symbolic matching algorithms, and while spatial recurrent neural networks with attention improved fine-grained image feature extraction⁷, the expressiveness of unimodal information remained inherently limited. The use of Theme-conditioned Transformers demonstrated potential in modeling sequence tasks during symbolic music generation⁸, and the feature deep fusion models used in sentiment analysis tasks confirmed the effectiveness of collaboration between multiple modalities⁹. These findings suggest that it is likely that the limitations of individual modalities pushed the field towards fusion models.

Multimodal learning has created research opportunities for music analysis through integration of complementary information sources—a principle that motivates the present study’s multimodal architecture, with the explicit qualification that the third modality employed here constitutes a MIDI-derived structured proxy rather than an independently captured visual stream, a distinction maintained consistently throughout the experimental design and interpretation. Deep fusion approaches demonstrate complementarity between audio and visual modalities in genre classification tasks¹⁰, with large-scale audio-visual databases enabling systematic cross-modal investigation¹¹. Music gesture and sound separation studies establish that visual information encodes distinct value¹²; in general performance contexts, hand posture reflects biomechanical constraints governing finger positioning and hand trajectory, conveying spatial-temporal cues that distinguish articulation modes such as staccato versus legato execution and pedal coordination timing—cues complementary to spectral energy distributions in audio and discrete event sequences in MIDI representations. In the present study, this biomechanical rationale informs the design of a deterministically derived MIDI-based spatial/kinematic proxy rather than constituting an independently observed biomechanical channel, a distinction consistently maintained throughout the experimental design and interpretive framing that follows. Late fusion strategies validate deep learning benefits for emotion classification in music videos¹³, with systematic reviews highlighting cross-modal feature potential¹⁴. Transformer architectures advance multimodal integration through affective video-to-music generation¹⁵ and audio-visual correlation learning¹⁶. Audio detection applications demonstrate Transformer model applicability¹⁷, while layer-level analysis provides insights into self-supervised music representations¹⁸. Exploiting multimodal benefits under limited annotation constraints remains a fundamental challenge.

Self-supervised learning paradigms address data scarcity through representation learning without extensive manual annotations. Contrastive learning frameworks establish foundational approaches across audio domains: wav2vec 2.0 demonstrates effectiveness for speech representations¹⁹, while extensions to universal audio representations exhibit cross-task transferability⁶. Transformer-based generative models validate self-supervised pretraining feasibility in music contexts²⁰, with acoustic tokenizer pretraining significantly improving audio understanding performance²¹. Audio-language contrastive learning enables enhanced music search capabilities²², while masked autoencoders for spectrograms provide universal audio representations²³. Frame-level positive-negative masking optimizes music representation objectives²⁴, and large-scale self-supervised training facilitates acoustic music understanding²⁵. Speech domain techniques transfer effectively to music scenarios²⁶, with multi-level audio representation learning benefiting segmentation tasks²⁷ and audio mixscape augmentation strengthening model generalization²⁸. Contrastive self-supervision reveals intrinsic Transformer dynamics for musical features²⁹, while supervised contrastive learning with weak audio labels improves text matching³⁰. Collectively, these methodological advances provide a principled basis for constructing high-quality music representation models under annotation-constrained conditions.

However, three persistent challenges remain specifically for fine-grained controlled perturbation detection in piano performance contexts. Unimodal data does not adequately describe the multidimensional characteristics of performance, the requirement for large-scale annotations limits model scalability due to high annotation costs, and existing methods exhibit limited capability in differentiating fine-grained perturbation categories. Addressing these challenges requires a systematic investigation structured around three interrelated research objectives. The investigation examines whether multimodal integration of audio spectral features, MIDI symbolic information, and kinematic representations surpasses unimodal baselines in detecting five fine-grained perturbation categories (pitch, timing, dynamics, touch, pedal) under controlled experimental conditions. The study evaluates whether self-supervised pretraining through contrastive learning, masked prediction, and temporal reconstruction reduces annotation requirements while maintaining detection precision. The research assesses model generalization capability across performance complexity levels ranging from beginner-level repertoire to advanced technical demands. Experimental validation on the MAESTRO dataset employs controlled ablation experiments and difficulty-stratified analysis to provide empirical evidence addressing each objective.

To address these objectives, this research proposes a self-supervised multimodal Transformer model that integrates audio, MIDI, and MIDI-derived kinematic proxy data, achieving precise detection of five perturbation categories in the public MAESTRO dataset under controlled experimental conditions. Given the absence of video recordings in MAESTRO, the kinematic modality constitutes a MIDI-derived spatial/kinematic proxy through deterministic transformation of MIDI parameters rather than independent visual capture. The core contribution of this work is accordingly framed as: fusion across audio spectral features, symbolic MIDI representations, and this MIDI-derived spatial/kinematic proxy; demonstration of cross-modal attention’s ability to exploit heterogeneous representations under controlled conditions; and explicitly not as evidence that adding video-based hand pose observation would necessarily yield similar gains, which remains an open question for future validation with video-equipped datasets. In the proposed model, a two-step learning process is used, which includes self-supervised pre-training for learning universal representations of the musical data, followed by fine-tuning for error detection. The cross-modal attention mechanism helps the model adaptively integrate different information, and the temporal localization module improves the accuracy of perturbation boundary detection under controlled experimental conditions. Subject to validation on teacher-annotated student performance data, the proposed system constitutes a technically rigorous controlled benchmark whose demonstrated capabilities provide a promising foundation for future research into computer-assisted performance feedback in music education contexts, representing a new technical paradigm for multimodal self-supervised learning applied to fine-grained music performance analysis rather than a deployment-ready educational tool. It should be noted that because MAESTRO does not provide video recordings, the kinematic modality represents a MIDI-derived motion proxy rather than independently captured hand-pose observations; thus, the demonstrated multimodal fusion benefits should be interpreted as validating architectural concepts under controlled conditions, with future work requiring verification using genuinely independent visual data.

Methodology

Dataset and error sample generation

MAESTRO dataset and controllable perturbation strategy

This study employs MAESTRO v2.0.0 (MIDI and Audio Edited for Synchronous TRacks and Organization; (https://magenta.tensorflow.org/datasets/maestro) as the experimental foundation, which contains 1,282 complete piano performance recordings with a total duration of approximately 200 h, covering diverse musical styles and difficulty levels across ten years of International Piano-e-Competition recordings. The dataset provides high-quality audio (48 kHz sampling rate) and precisely aligned MIDI sequences, which gives the model an excellent data basis for multimodal learning. The dataset is partitioned into training (70%), validation (15%), and test (15%) subsets, with proportions adjusted relative to the official MAESTRO v2.0.0 configuration (approximately 75.4%/10.7%/13.9% by performance count) to provide more balanced evaluation and test set sizes. Although the exact numerical proportions deviate from the official configuration, the fundamental composition-level exclusivity principle established in the original MAESTRO split design is strictly preserved: complete compositions are assigned exclusively to individual partitions such that no work performed by multiple contestants appears across multiple subsets, regardless of performer identity. This design prevents information leakage that could otherwise arise if exposure to one pianist’s recording of a given composition were to influence model predictions on a different pianist’s recording of the same work. The split strategy assigns complete compositions to distinct partitions such that no individual recording appears in multiple splits. Error injection through controllable perturbations is applied exclusively after data partitioning, where each base recording from train, validation, and test sets undergoes independent stochastic perturbation using identical protocols detailed in the subsequent paragraphs, ensuring consistent error distribution characteristics across all three splits while maintaining zero overlap in perturbed instances between partitions. This protocol guarantees exclusive separation of training, validation, and test perturbations, eliminating data leakage during model development.

Since the MAESTRO is comprised of only correct performances, the controllable perturbation-based approach is proposed for the generation of error samples. Pitch error is introduced by the process of random note substitution, and the perturbation range is varied between ± 1 and 3 semitones, modeling the process of note misreading. Timing error is introduced by the process of temporal axis shifting, and the onset times of the notes are varied between ± 50 and 500 milliseconds, covering situations ranging from subtle delays to obvious mistakes. The dynamics error is introduced by varying the MIDI velocity by ± 20 to 40, which reflects a lack of control in dynamics. Touch error is introduced by varying note duration with perturbation ratio ranging from ± 10% to 50% of actual duration, thus mimicking errors in staccato and legato touch. The pedal errors are stochastic variations of the timing of onset and release of the sustain pedal, causing acoustic problems resulting from incorrect operation of the sustain pedal. The ratio of injecting errors is dynamically set between 5% and 20%, and the annotation data include types of errors, timestamps, and durations of errors. Across the test set, 52.3% of temporal windows contain no active perturbation category, 35.8% contain exactly one simultaneously active category, and 11.9% contain two or more co-occurring perturbation types. Per-class observed prevalence rates are: pitch 11.2%, timing 13.4%, dynamics 12.8%, touch 11.9%, and pedal 12.1%, consistent with the 5%–20% injection probability range. It is important to note that the term “error” throughout this study serves as a proxy label for stochastically injected perturbation events rather than mistakes produced by human learners; the detection task is accordingly characterized as controlled perturbation event detection, and all reported performance metrics reflect model capability on synthetic perturbation instances under laboratory conditions rather than on naturally occurring student errors.

The purpose of multimodal feature extraction is to preprocess the raw data. Audio data is converted into Mel spectrograms through the application of the short-time Fourier transform. The frequency dimension is set to 128 Mel bins, the frame size is 25 milliseconds, and the hop size is 10 milliseconds. The MIDI sequences are represented as four-dimensional vectors based on the parameters of pitch, onset time, duration, and velocity. Due to the absence of video recordings in the MAESTRO dataset, kinematic features representing hand posture are synthetically derived from MIDI parameters rather than captured through independent visual sensors. The derivation employs a deterministic mapping function that transforms MIDI pitch values to approximate key positions on the 88-key keyboard (ranging from A0 to C8), converts MIDI velocity values (0–127) to estimated finger force and planar hand elevation parameters, and infers lateral hand trajectories from onset timing intervals and pitch intervals between consecutive notes. The resulting feature vectors adopt the 21-keypoint skeletal topology defined in the MediaPipe Hands schema while retaining only the planar (x, y) coordinates per keypoint, yielding 21 × 2= 42 dimensions in total. Since the kinematic modality is synthetically derived from MIDI parameters rather than captured through physical sensors, reliable estimation of z-axis depth information is not achievable through deterministic transformation alone, making the planar representation the appropriate scope for this proxy under the controlled experimental conditions of the present study. This MIDI-derived kinematic proxy serves as a structured intermediate representation that facilitates multimodal fusion architecture validation, though it inherently lacks the independence of genuinely captured visual signals.

The deterministic derivation of the kinematic modality from MIDI introduces methodological limitations that warrant explicit acknowledgment, given that the resulting representations lack the observational independence of genuine visual signals. Consequently, performance gains attributed to “multimodal fusion” should be interpreted as benefits from structured spatial encoding and adaptive modality weighting within the attention mechanism, rather than definitive evidence of advantage from independent visual observation. This controlled experimental design enables systematic evaluation of cross-modal attention architectures while avoiding the confounding factors present in unaligned or low-quality video data, though validation with authentic video-captured hand posture remains necessary for establishing real-world applicability.

Threats to validity of synthetic perturbations

The controllable perturbation approach enables systematic benchmarking yet introduces validity threats regarding generalizability to authentic educational scenarios. The perturbation distributions were designed based on preliminary observations of intermediate-level student performances combined with experimental control requirements. Pitch perturbations of ± 1 to 3 semitones approximate common misreading errors where note substitutions occur within adjacent scale degrees. Timing shifts of ± 50 to 500 milliseconds capture the spectrum from subtle hesitations to pronounced coordination failures. Velocity variations of ± 20 to 40 MIDI units represent typical dynamic control fluctuations between piano and mezzo-forte levels. Duration scaling of ± 10% to 50% models articulation difficulties in staccato versus legato execution. Pedal timing jitter approximates coordination errors in sustain pedal operation.

These synthetic perturbations approximate several categories of human learner behaviors. Random pitch substitution within limited semitone ranges captures notation misreading phenomena. Uniform temporal shifts model tempo inconsistency and hand coordination delays. Velocity perturbations reflect graduated control difficulties in developing dynamic precision. Duration variations approximate articulation uncertainties during legato transitions and staccato passages. Pedal timing jitter represents coordination challenges when integrating pedal operation with melodic execution.

However, the synthetic error model fails to capture essential characteristics of authentic learner mistakes. Real student errors exhibit strong contextual dependencies absent in uniform random injection: timing mistakes cluster at phrase boundaries, technically demanding passages, and ornament executions rather than distributing uniformly across all notes. Touch and articulation errors depend critically on pedaling context and harmonic progression interactions. Pedal errors manifest as systematic over-pedaling during harmonic transitions or delayed release at cadential points, exhibiting musically meaningful patterns rather than stochastic noise. The perturbation distributions also neglect cognitive and biomechanical grounding: student misreading errors show systematic bias toward visually similar positions, timing errors produce characteristic acceleration or deceleration curves, and dynamic control errors accumulate progressively through fatigue rather than appearing as independent discrete adjustments.

Addressing this validity gap requires a combination of complementary strategies. Collection and annotation of student performance datasets with teacher-identified errors would provide ground truth for direct validation and enable investigation of error co-occurrence patterns. Context-aware injection heuristics represent an intermediate step, where error probability is weighted according to local musical complexity: increased likelihood in passages with rapid tempo, dense chords, large leaps, or complex rhythms. Phrase-boundary-sensitive perturbations could better approximate structural clustering of timing and pedal errors. Collaboration with piano instructors would identify error patterns most diagnostically valuable for educational feedback, ensuring detection capabilities align with authentic teaching priorities.

Self-supervised multimodal transformer architecture

This study proposes a self-supervised multimodal Transformer architecture that integrates audio, MIDI, and a MIDI-derived spatial/kinematic proxy using cross-modal attention mechanisms; the proxy functions as a structured spatial encoding derived deterministically from MIDI parameters to improve the separability of perturbations involving duration and articulation control as well as pedaling temporal patterns, rather than constituting a genuinely new independent sensor modality. For the end-to-end training of the model from raw multimodal data to the fine-grained detection of errors, the proposed architecture is divided into three sequential components: the unimodal encoder modules, the cross-modal Transformer fusion module, and the fine-grained error classification head, where each module collaborates to complete the entire process of feature extraction, modal alignment, and error recognition, as shown in Fig. 1.

Fig. 1.

Overall method framework diagram.

Figure 1 demonstrates the complete process for information processing. The three parallel encoders encode the audio spectrogram, MIDI sequences, and hand keypoints into a common representation space. The fusion module subsequently learns inter-modal correspondences using multi-layer attention mechanisms. Finally, the classification head predicts the temporal locations and intervals for which five different types of errors occur, thus allowing for hierarchical modeling from low-level signals to semantic understanding.

The unimodal encoder is responsible for transforming the raw features associated with each modality into a unified feature space. The audio encoder employs two-dimensional convolutional neural networks to process Mel spectrogram inputs, applying kernel size 3 × 3 across both time and frequency dimensions while progressively increasing channel dimensions from 64 to 512. The Squeeze-and- Excitation mechanism³¹ is employed for adaptive channel attention. The architectural hyperparameters are selected based on established design principles validated through preliminary empirical evaluation. The 3 × 3 convolutional kernel size balances receptive field coverage for local time-frequency pattern extraction with computational efficiency, a common choice in spectrogram processing tasks. The feed-forward network expansion ratio of 4× follows the standard Transformer architecture design, providing sufficient capacity for non-linear feature transformation. The depth of the cross-modal fusion module is determined through systematic ablation experiments evaluating configurations ranging from 4 to 12 layers, with results reported in Sect.  3.3. The MIDI encoder is responsible for encoding the discrete symbolic sequences into continuous vectors through the embedding layer, where the coordinate attention mechanism³² is used to extract the interaction information between the pitch and temporal dimensions to improve the feature representation of rhythmic patterns. The kinematic encoder adopts spatial-temporal graph convolutional networks, and it incorporates the self-attention mechanism of the Audio Spectrogram Transformer³³ in order to handle the long-term dependencies and the correlations of the hand posture variation across the measures.

The cross-modal Transformer fusion module constitutes the core innovation of the model and supports deep interaction and alignment of features from the three encoders. Given the substantial differences across modalities in temporal resolution, feature dimensionality, and semantic granularity, simple feature concatenation or weighted averaging proves insufficient for effective cross-modal integration. As a result, this study adopts an adaptive fusion strategy based on attention mechanisms and supports cyclic information interaction and refinement between modalities via eight stacked transformer modules, as shown in Fig. 2.

Fig. 2.

Cross-modal transformer fusion module detailed architecture.

Figure 2 details the internal structure of a single fusion module, where the multi-head self-attention layers are used for capturing the temporal dependencies in the different modalities, the cross-modal cross-attention layers help in the bidirectional querying process of the different modalities, and the feed-forward networks are used for performing the non-linear transformations in the different layers. The use of residual connections and layer normalization helps in ensuring the training stability of deep networks, where the recursive stacking of 8 layers enables bottom-level local features to progressively fuse into top-level global semantic representations.

Multi-head self-attention layers model temporal dependencies within each modality using 8 attention heads with dimension 64 per head. The self-attention operation for modality at layer l computes:

1

where each attention head i is defined as:

2

3

with representing the input sequence of length and dimension for modality m, projection matrices with , and output projection .

Cross-modal cross-attention layers enable bidirectional information exchange between modality pairs. For audio attending to MIDI (denoted as ), the cross-attention operation uses audio features as queries and MIDI features as keys and values:

4

5

This allows the model to associate acoustic spectral energy patterns in with symbolic note duration and onset timing encoded in . Symmetrically, MIDI attending to kinematic features () correlates velocity values with estimated finger force and hand trajectory patterns:

6

where represents the kinematic feature sequence. To align temporal resolutions across modalities with different frame rates (audio: 100 fps, MIDI: event-based, kinematic: 30 fps), sinusoidal positional encodings are applied independently to each modality before cross-attention:

7

8

where pos denotes the temporal position index within each modality’s native frame rate, and i indexes the embedding dimension. The positional encodings preserve relative timing information while allowing the attention mechanism to learn alignment between modalities with different temporal granularities. The feed-forward networks consist of two fully connected layers with the dimension of the intermediate layer increased to 4 times the dimension of the input layer and the activation function is GELU. Residual connections and layer normalization are used in the design of the deep networks for stable training.

Fine-grained error classification head performs error detection and error type identification at the temporal level. The hybrid attention module³⁴ combines channel attention and spatial attention. It adjusts the weight distribution across feature dimensions to improve overall feature sensitivity. The pyramid convolution structure³⁵ captures error patterns with respect to the duration of the error. Multi-scale receptive fields with kernel sizes of 1, 3, and 5 address instantaneous, short-term, and long-term perturbation patterns respectively. The dynamic sparse attention mechanism³⁶ adaptively selects salient time steps for computation, reducing computational complexity without compromising detection precision. The five binary classifiers work independently to determine the presence of different errors in the multi-label classification framework. The temporal localization head employs a 1-second sliding window boundary regression module to generate continuous onset and offset probability estimates for each error category during training and inference, with the complete evaluation-stage post-processing pipeline specified in Sect.  2.4.

Model training and optimization strategy

The training process of the model is based on a two-step approach, which is tailored to take full advantage of the unlabeled data and adapt to the error detection task. During the self-supervised pre-training step, general musical representations are obtained from the full MAESTRO dataset. The aim of contrastive learning is to maximize the similarity of representations among the various modalities of the same execution and to minimize the similarity among the various executions, so as to induce the model to learn musical semantics invariant to modalities. The InfoNCE loss function constructs positive pairs from synchronized audio and MIDI representations extracted from the same performance, while treating representations from all other performances in the batch as negative samples. For a batch containing N performances, the contrastive loss is defined as:

9

where and represent -normalized embeddings from audio and MIDI encoders for the anchor performance, denotes the MIDI embedding from the j-th performance in the batch including both the positive match (when j corresponds to the anchor) and negative samples, N is the batch size, computes cosine similarity between -normalized vectors, and τ is the temperature parameter controlling the sharpness of the similarity distribution, with the exact value provided in Sect.  2.4. Masked prediction randomly masks 15% of frames in the modality input and requires the model to reconstruct the missing content, while temporal reconstruction requires predicting future frames from preceding context. The total loss function for the pretraining stage is expressed as:

10

with weight coefficients (0.5 for masked prediction, 0.3 for temporal reconstruction) determined through a small-scale validation sweep on 10% of training data, selecting the combination that achieved the fastest convergence while maintaining balanced loss magnitudes across all three objectives.

The error detection fine-tuning stage freezes the first two convolutional layers (layers 1–2) of the audio encoder, the embedding layer of the MIDI encoder, and the first graph convolutional layer of the kinematic encoder, while optimizing all parameters in the 8-layer cross-modal Transformer fusion module and the fine-grained error classification head. This selective freezing strategy preserves low-level feature extractors learned during pretraining while allowing task-specific adaptation of high- level semantic representations and decision boundaries. Focal Loss is used to handle class imbalance in the five error types, and it reduces the weights of easily classified instances using modulation factors to focus the model on the harder instances of errors. The multimodal fusion-based music signal classification model³⁷ demonstrates that fusion strategies have a great impact on performance, and in this study, attention-weighted late fusion was adopted, where the model learns dynamically how much each modality can contribute to identifying different types of errors. Temporal localization loss optimizes prediction of error start and end times through boundary regression, with the total loss function for the fine-tuning stage given in:

11

The optimizer employs AdamW with weight decay coefficient of 0.01, with the complete hyperparameter specification provided in Sect.  2.4. The pretraining stage uses a batch size of 32 and trains for 100 epochs. The fine-tuning stage uses a batch size of 16 and a total of 50 epochs. The learning rate schedule uses a cosine annealing with warm restarts, where the learning rate is restarted to half of its initial value after each 10 epochs. Data augmentation strategies include time stretching (speed factor 0.9–1.1), pitch shifting (± 2 semitones), and Gaussian noise injection (signal-to-noise ratio 30–50 dB). Prior work on high-quality video-to-audio synthesis demonstrates that training strategy design is critical to the success of multimodal learning frameworks³⁸. Gradient conflicts during multi-task joint optimization are mitigated through the two-stage decoupled training strategy, which allows the modality encoders to focus on modality-specific representation learning during pretraining and on task adaptation for controlled perturbation detection during fine-tuning.

Implementation details for reproducibility

To facilitate reproducibility, the complete implementation specifications are documented here. Regarding input alignment across modalities, audio Mel spectrogram frames are extracted at 100 frames per second with a 10 ms hop size and serve as the reference temporal grid; MIDI event sequences are quantized onto this 10 ms grid by assigning each note’s pitch, velocity, duration, and onset attributes to the corresponding frame index, with inter-event frames populated using zero-velocity silence tokens to maintain sequence continuity; the 42-dimensional kinematic proxy vectors are generated at 30 frames per second and subsequently linearly interpolated to the 100-frame-per-second audio grid, producing temporally aligned sequences of identical length across all three modalities prior to sinusoidal positional encoding.

Regarding ground truth segment construction, each injected perturbation event is converted to a binary frame-level label sequence by marking all frames within the perturbation’s annotated onset-to-offset interval as positive for the corresponding error category, with the five binary label sequences maintained independently to support multi-label classification. Error injection operates independently across the five perturbation categories, with each type sampled according to its own stochastic injection probability drawn uniformly from the 5% to 20% range per recording, such that simultaneous co-occurrence of multiple perturbation types within a given temporal window is possible under this independent sampling scheme; annotation records all co-occurring perturbation types, timestamps, and durations for each frame, enabling the multi-label classification head to handle windows containing zero, one, or multiple simultaneously active perturbation categories.

Regarding post-processing consistency, the temporal localization head employs a 1-second sliding window boundary regression module during training and inference to generate continuous onset and offset probability estimates; during evaluation, frame-level classification probabilities are thresholded at 0.5 and grouped into discrete temporal segments via connected component analysis, constituting the evaluation-stage pipeline for IoU-based segment matching as described in the Experimental Setup and Evaluation Metrics subsection, with the two components forming sequential stages of a unified detection pipeline. The complete set of training hyperparameters is as follows: AdamW optimizer with learning rate 1 × 10^− 4 and weight decay 0.01; contrastive learning temperature ; masked prediction frame masking ratio 15%; pretraining loss weights 0.5 for masked prediction and 0.3 for temporal reconstruction as specified in the pretraining loss formulation; fine-tuning localization loss weight 0.2 as specified in the fine-tuning loss formulation; cosine annealing warm restart period of 10 epochs with restart learning rate 5 × 10^− 5; gradient accumulation steps of 4 with mixed precision training on NVIDIA Tesla V100.

For consolidated reference and to facilitate independent reproducibility, Table 1 provides a complete summary of all hyperparameter configurations across both training stages, consolidating the values specified throughout Sect.  2.3 and 2.4 into a single reference.

Table 1.

Complete hyperparameter configuration for reproducibility.

Category	Hyperparameter	Value
Optimizer	Algorithm	AdamW
	Initial learning rate	1 × 10− 4
	Weight decay	0.01
	Gradient accumulation steps	4
Pretraining	Batch size	32
	Training epochs	100
	Frame masking ratio	15%
	Loss weight	0.5
	Loss weight	0.3
	Contrastive temperature	0.07
Fine-tuning	Batch size	16
	Training epochs	50
	Localization loss weight	0.2
LR Schedule	Warm restart period	10 epochs
	Restart learning rate	5 × 10− 5
Hardware	GPU	NVIDIA Tesla V100 (32 GB)
	Training precision	Mixed (FP16/FP32)

Results

Experimental setup and evaluation metrics

All experiments are conducted on a server equipped with NVIDIA Tesla V100 GPU (32GB memory), and the software stack was PyTorch 2.0 and Python 3.9. Mixed precision training with gradient accumulation steps of 4 was used. Detection performance is quantified through multiple complementary metrics that capture distinct aspects of model capability. Macro-averaged F1 score serves as the primary criterion for overall model capability, computed by averaging class-specific F1 scores across the five error types to ensure balanced evaluation despite class imbalance. Mean Average Precision (mAP) evaluates detection accuracy at the segment level: predicted error segments are generated by applying a confidence threshold of 0.5 to the classification head outputs, followed by connected component analysis to group consecutive positive frames into temporal segments. This evaluation-stage segment extraction procedure is consistent with the training-stage 1-second sliding window boundary regression described in Sect.  2.2, with the two components constituting sequential stages of a unified post-processing pipeline as detailed in Sect.  2.4. Ground truth and predicted segments are matched using greedy Intersection over Union (IoU) matching, where a predicted segment is considered a true positive if its IoU with a ground truth segment of the same error type exceeds 0.5. Average Precision for each error type aggregates precision-recall pairs across confidence thresholds from 0.1 to 0.9 with 0.1 increments. Temporal localization precision is quantified through mean IoU, calculated as the average IoU score across all matched segment pairs, reflecting the model’s ability to accurately delineate error boundaries. For segments with multiple matches, the highest IoU pairing is selected. These metrics collectively assess classification accuracy (Macro-F1), detection reliability across varying confidence levels (mAP), and temporal precision (IoU). It bears emphasis that all evaluation is conducted on injected perturbation instances constructed per the protocol in Sect.  2.1.1, such that the reported Macro-F1 of 87.3%, mean mAP of 0.850, and mean IoU of 0.784 characterize controlled perturbation detection performance and should not be interpreted as direct indicators of educational error detection capability without further validation on authentic student performance data. Per-class prevalence and window-level co-occurrence distributions across the test set are reported in Sect.  2.1.1. The selection of baseline methods covers classical and recent studies, including DTW-HMM, CNN-LSTM, Attention-RNN, Music Transformer, MMT, CLMR and MT3, to ensure the comprehensiveness and fairness of comparison.

To ensure fair comparison under identical task conditions, all baseline methods were adapted to the five-category error detection task with temporal localization. DTW-HMM, CNN-LSTM, and Attention-RNN employ audio-only input without pretraining and were re-implemented with task-specific architectures for multi-class classification and temporal boundary prediction. Music Transformer processes MIDI-only symbolic sequences without pretraining, re-implemented with modifications to output error classifications and temporal segments. CLMR utilizes audio-only input with self-supervised contrastive pretraining followed by supervised fine-tuning, implemented using publicly available code adapted to the error detection task. MMT integrates audio and MIDI modalities without pretraining, re-implemented for multi-label temporal localization. MT3 employs audio-only input with publicly available pretrained weights from music transcription tasks, fine-tuned through task-specific classification heads. All baselines were trained on identical perturbation-injected datasets using consistent train/validation/test splits and unified evaluation metrics; each method underwent an independent hyperparameter grid search spanning learning rate values of {10^− 4, 3 × 10^− 4, 10^− 3, 10^− 2}, batch sizes of {16, 32, 64}, and task-appropriate regularization strength, with each search constrained to an equivalent computational budget of 20 GPU-hours on the same NVIDIA Tesla V100 hardware to ensure comparability across methods. All reported mean ± standard deviation values in Table 2, covering both the baseline methods and the proposed model, are derived from three fully independent retraining runs initialized under different random seeds, with no hyperparameter settings shared across runs.

Table 2.

Overall performance comparison with baseline methods on MAESTRO Test Set.

Method	Pitch F1	Timing F1	Dynamics F1	Touch F1	Pedal F1	Macro-F1	Pitch mAP	Timing mAP	Dynamics mAP	Touch mAP	Pedal mAP	Mean mAP	Mean IoU
DTW-HMM	82.4	78.3	68.2	65.7	78.5	74.6 ± 1.8	0.793	0.751	0.658	0.621	0.742	0.713 ± 0.016	0.624 ± 0.018
CNN-LSTM	85.7	80.1	73.6	71.2	80.9	78.3 ± 1.4	0.831	0.774	0.709	0.683	0.779	0.755 ± 0.013	0.672 ± 0.015
Attention-RNN	87.2	82.5	75.1	72.8	81.4	79.8 ± 1.3	0.847	0.798	0.726	0.701	0.786	0.772 ± 0.012	0.691 ± 0.013
Music Transformer	86.2	83.7	74.3	72.6	80.7	79.5 ± 1.2	0.839	0.811	0.718	0.698	0.781	0.769 ± 0.011	0.684 ± 0.012
CLMR	88.1	84.2	76.8	74.1	79.3	80.5 ± 1.1	0.856	0.816	0.743	0.714	0.767	0.779 ± 0.010	0.703 ± 0.011
MMT	88.9	85.3	77.2	76.1	79.0	81.3 ± 1.0	0.864	0.827	0.748	0.736	0.764	0.788 ± 0.009	0.718 ± 0.010
MT3	89.7	86.4	78.9	77.3	78.2	82.1 ± 0.8	0.873	0.839	0.764	0.748	0.757	0.796 ± 0.008	0.731 ± 0.009
Proposed	91.2	89.5	84.7	82.1	88.9	87.3 ± 0.7	0.889	0.871	0.824	0.798	0.867	0.850 ± 0.007	0.784 ± 0.008

Macro-F1, Mean mAP, and Mean IoU are reported as mean ± standard deviation across three independent runs with different random seeds. Per-class F1 and mAP values are reported from the median run. Baseline results are obtained under the standardized re-implementation protocol described in the Experimental Setup and Evaluation Metrics subsection.

Overall detection performance and baseline comparison

In order to evaluate the effectiveness of the proposed framework on the five controlled perturbation detection tasks, a series of experiments have been conducted on the MAESTRO test set, and the proposed framework has been compared to the seven baselines. The goal of the experimental design is to validate the benefit of multi-modal fusion over the pretraining strategies and the supervised single-modal frameworks, and to investigate the detection balance of the model across different perturbation categories. The test set includes 190 pieces that are completely new to the networks, ranging from baroque to more modern pieces with varying degrees of difficulty. Perturbation samples are generated according to the strategy described in Sect.  2.1, and the injection ratio is controlled at 8%–15% to approximate plausible characteristics of real learning scenarios under controlled laboratory conditions; accordingly, all reported detection metrics characterize synthetic perturbation event recognition performance rather than authentic learner-error identification capability. The evaluation process calculates performance metrics independently for each error type, which reflect the overall detection capability of the model by macro averaging, as shown in Table 2.

Table 2 shows the proposed method achieves a Macro-F1 of 87.3%, mean mAP of 0.850, and mean IoU of 0.784, surpassing the best baseline MT3 by 5.2% points in Macro-F1, by an absolute margin of 0.054 in mean mAP, and by an absolute margin of 0.053 in mean IoU. Pedal error exhibits the largest gains across both classification and detection metrics, with F1 improving from 78.2% to 88.9% and mAP increasing from 0.757 to 0.867 compared to MT3. Per-class mAP analysis reveals consistent improvements across all error types, with pitch mAP of 0.889, timing mAP of 0.871, dynamics mAP of 0.824, touch mAP of 0.798, and pedal mAP of 0.867, demonstrating superior detection reliability across varying confidence thresholds. Integration of MIDI-derived kinematic proxy elevates dynamics and touch detection to 84.7% F1 and 0.824 mAP for dynamics, 82.1% F1 and 0.798 mAP for touch. Mean IoU of 0.784 indicates precise boundary delineation, with the model achieving high temporal localization accuracy across all error categories. Consistent improvements across Macro-F1, per-class mAP, mean mAP, and mean IoU collectively validate the effectiveness of the cross-modal attention mechanism and the temporal localization module.

Among the five perturbation categories, pedal perturbation detection exhibits disproportionate performance gains (10.7% points improvement from MT3’s 78.2% to 88.9%), substantially exceeding improvements in pitch (1.5% points), timing (3.1% points), dynamics (5.8% points), and touch (4.8% points). This discrepancy reflects fundamental differences in perturbation temporal signatures across modalities. Sustain pedal operation produces gradual acoustic envelope decay spanning multiple notes rather than instantaneous changes characteristic of pitch or dynamics errors, creating extended temporal patterns that multimodal fusion can exploit more effectively. MIDI representations encode pedal state as discrete binary events (onset/release timestamps) without capturing acoustic decay profiles, while audio spectrograms capture harmonic resonance patterns but lack explicit pedal timing annotations. The MIDI-derived kinematic proxy encodes hand trajectory patterns exhibiting temporal correlation with pedal operation through biomechanical coupling in piano performance, where hand position transitions often precede or coincide with pedal changes. Cross-modal attention associates harmonic resonance decay in audio with pedal state transitions in MIDI and hand movement timing in kinematic representations, enabling detection of temporal coordination errors where pedal operation misaligns with harmonic progression. Baseline methods relying solely on audio struggle to isolate pedal-induced resonance from natural note sustain without symbolic timing context (MT3 at 78.2% F1), whereas multimodal integration leverages complementary temporal information across modalities to achieve 88.9% F1. Pedal error detection benefits particularly from this approach because temporal coordination patterns span longer time scales than instantaneous acoustic or symbolic events, allowing cross-modal attention to capture sustained relationships across multiple temporal frames.

In order to deeply analyze the error patterns and inter-class confusion in the model, the confusion matrix was developed in order to assess the classification accuracy across the five perturbation categories. The enumeration of the correspondence between the true labels and the predicted labels of the different types of errors in the test data provides information on the circumstances of the misclassifications made by the model. The diagonal elements of the confusion matrix represent the proportion of correct classifications, and the off-diagonal elements show the measure of misclassification among classes. The normalization process makes the sum of elements in each row equal to 1 to facilitate cross-category comparison, as shown in Fig. 3.

Fig. 3.

Confusion matrix for five error types.

Figure 3 shows that the diagonal elements represent the correct classification accuracy, where the highest pitch error is 93.1% and the lowest touch error is 85.3%. Furthermore, there are 8.7% of Dynamics errors that are mistakenly classified as Touch errors, and 10.2% of Touch errors are mistakenly classified as Dynamics errors. This bidirectional confusion arises from acoustic and kinematic coupling between velocity control and articulation execution. Dynamics errors (incorrect MIDI velocity) alter attack transient amplitude, producing acoustic signatures overlapping with touch errors (incorrect note duration) that modulate decay envelope shape. The synthetic kinematic proxy derives estimated finger force from MIDI velocity and hand trajectory patterns from note timing characteristics, introducing systematic correlation where velocity perturbations induce correlated kinematic feature changes resembling timing-based trajectory variations. Audio spectrograms cannot definitively distinguish whether reduced harmonic energy results from softer initial attack or premature key release without independent biomechanical observation. The 8.7–10.2% confusion rate suggests that MIDI-derived spatial encoding inherits limitations from deterministic MIDI-to-kinematic mapping that conflates physically independent performance dimensions, a limitation consistent with the proxy’s deterministic construction as discussed in Sect.  2.1.1.

Ablation study and training efficiency analysis

To systematically investigate the contribution of each architectural component to detection performance, ablation studies were conducted across six experimental groups. These groups evaluate modality contributions, pretraining effectiveness, architectural depth, fusion strategies, kinematic proxy alignment mechanisms, and perturbation distribution generalization respectively. Control variable design compares each condition against the full model baseline to isolate individual factor contributions. The experiment on the effect of the pre-training influence tests the difference in performance between random initialization and pre-training initialization. The experiment on the depth of the architecture tests the expressiveness of the 4-layer, 8-layer, and 12-layer transformers. The experiment on fusion strategies tests the effect of early splicing fusion, late fusion, and attention fusion strategies. All ablation experiments were conducted on the MAESTRO validation set, using the same hyperparameters and training process, and the contribution of each factor was quantified by macro-F1, as shown in Table 3.

Table 3.

Ablation study results.

Ablation group	Configuration	Macro-F1 (%)
Modality Contribution	Audio only	79.3 ± 1.2
	MIDI only	81.7 ± 0.9
	Kinematic proxy only	72.4 ± 1.5
	Audio + MIDI	84.6 ± 0.8
	Audio + Kinematic proxy	82.1 ± 1.1
	MIDI + Kinematic proxy	83.8 ± 1.0
	All modalities (Full)	87.3 ± 0.7
Pretraining Effect	With pretraining	87.3 ± 0.7
	Without pretraining	84.7 ± 1.1
Transformer Depth	4 layers	84.2 ± 0.9
	8 layers	87.3 ± 0.7
	12 layers	86.8 ± 0.8
Fusion Strategy	Early fusion	85.4 ± 0.9
	Late fusion	86.2 ± 0.8
	Attention fusion (Full)	87.3 ± 0.7
Kinematic Proxy Alignment	Audio + MIDI	84.6 ± 0.8
	Audio + MIDI + aligned proxy	87.3 ± 0.7
	Audio + MIDI + temporally shuffled proxy	85.1 ± 1.0
	Audio + MIDI + MIDI-source shuffled proxy	83.9 ± 1.2
Perturbation Distribution Shift (Timing)	Train/Test ± 50-500ms (baseline)	87.3 ± 0.7
	Train ± 50-200ms / Test ± 200-500ms	84.2 ± 1.0
	Train ± 200-500ms / Test ± 50-200ms	83.8 ± 1.1

Table 3 shows that MIDI achieves the highest single-modality performance at 81.7 ± 0.9%, while integration of all three modalities (audio, MIDI, and MIDI-derived kinematic proxy) improves performance to 87.3 ± 0.7%, with standard deviations computed across three independent runs using different random seeds. Self-supervised pretraining improves F1 from 84.7% to 87.3%, representing a 2.6% point gain. The 8-layer Transformer configuration reaches optimal performance at 87.3%, with deeper 12-layer architecture degrading to 86.8% due to overfitting, while attention-based fusion (87.3%) outperforms early fusion (85.4%) and late fusion (86.2%) strategies. Kinematic proxy alignment ablation reveals that the 2.7% point gain from incorporating the temporally aligned structured spatial encoding (84.6% to 87.3%) substantially diminishes when temporal alignment is disrupted, with temporally shuffled proxy achieving 85.1% and MIDI-source shuffled proxy dropping to 83.9%; these results should be interpreted as evidence that the proxy operates as a structured spatial encoding of MIDI information that improves the separability of duration-related and articulation-related perturbations—particularly touch and pedaling temporal patterns—rather than as evidence of complementarity from an independent visual sensor, given that the proxy is deterministically derived from MIDI and may function as a reparameterized MIDI stream through the kinematic pathway. The 0.5% point residual gain from the shuffled proxy over the Audio+MIDI baseline reflects the contribution of static key-position spatial encoding that remains partially informative even without temporal alignment, while the larger 2.2% point degradation from the aligned to the shuffled condition confirms that the primary benefit derives from temporally coherent structured spatial encoding rather than from MIDI redundancy alone. Perturbation distribution shift experiments demonstrate cross-range generalization capability, where models trained on narrow timing perturbation ranges maintain 83.8–84.2% performance when tested on complementary ranges, representing 3.1–3.5% point degradation relative to full-range baseline, indicating that the model learns error semantics characterizing temporal coordination failures rather than memorizing specific perturbation magnitudes. Taken together, these ablation results indicate that the observed performance advantages are more appropriately attributed to temporally coherent structured spatial representations and adaptive cross-modal attention weighting than to any form of observational independence on the part of the kinematic modality, an interpretive constraint consistent with the deterministic MIDI-derived construction of the proxy as established in Sect.  2.1.1.

To verify the effectiveness of the proposed two-stage training approach and understand the corresponding learning process related to pre-training and fine-tuning, the training process of loss functions and performance metrics was observed completely. During the pre-training phrase, the convergence of three self-supervised loss functions (contrastive learning, mask prediction, and time series reconstruction) in 100 epochs was observed to examine the optimization results of inter-modality alignment and time series modeling tasks. The fine-tuning phase observes the progress of Focal Loss and macro-F1 over 50 epochs, and compares the early results and rates of convergence in the presence and absence of the pre-training phase. The purpose of the experiment is to show how self-supervised learning can accelerate the adaptation process for the task by relying on the general features learned in the pre-training phase, as shown in Fig. 4.

Fig. 4.

Training curves for two-stage strategy.

Figure 4a shows that all three losses for self-supervised learning have a decreasing pattern during pre-training, with a fastest convergence for contrastive loss, followed by mask prediction loss, and a slow and steady decrease for temporal reconstruction loss until the end of training. Figure 4(b) shows that in the fine-tuning stage, Focal Loss becomes more stable after an initial sharp drop, while the Macro-F1 values keep rising, peaking in the middle to late stages and then showing very small variations. The behavior of both functions further supports that optimizing the loss function and improving performance are correlated.

To compare the effectiveness of self-supervised pre-training in the presence of a small amount of labeled data, comparative experiments were carried out by varying the amount of training data. The need for carrying out the above-mentioned task arises from the fact that large amounts of manually labeled incorrect data are difficult to come by. The model was trained with and without pre-training for training data splits of 10%, 20%, 50%, and 100%. The number of samples in the validation and test sets is fixed, and the amount of samples for fine-tuning is varied. Each data condition is repeated three times, and the results are averaged to remove the effects of random variation. The aim of the experiment is to determine if self-supervised learning can decrease the reliance on the samples during the fine-tuning process, as shown in Fig. 5.

Fig. 5.

Performance vs. training data size with and without pretraining.

Figure 5 shows that both strategies improve as the number of training data increases, but the pre-trained model always maintains a significant advantage. Although the gap between the two methods is maximum in the small sample setting, the difference becomes less pronounced in the large sample setting, thus proving the effectiveness of the pre-training technique.

While Fig. 5 demonstrates performance advantages of self-supervised pretraining under limited annotation scenarios, practical deployment requires evaluation of computational-annotation cost tradeoffs. Table 4 quantifies GPU computational hours for pretraining and fine-tuning alongside expert annotation hours required across different training data splits, integrating performance metrics from Fig. 5 to assess whether computational investment in pretraining offsets annotation labor requirements.

Table 4.

Computational efficiency and annotation cost analysis.

Training data split	Training strategy	GPU Hours (V100)	Annotation Hours	Macro-F1 (%)	Total GPU Hours
10%	With pretraining	73.0 + 1.2	38	79.8	74.2
	Without pretraining	1.2	38	68.3	1.2
20%	With pretraining	73.0 + 2.1	77	84.6	75.1
	Without pretraining	2.3	77	73.5	2.3
50%	With pretraining	73.0 + 5.1	192	86.1	78.1
	Without pretraining	5.4	192	77.2	5.4
100%	With pretraining	73.0 + 8.7	385	87.3	81.7
	Without pretraining	9.1	385	81.9	9.1

Pretraining requires 73.0 GPU-hours on a single NVIDIA Tesla V100 for 100 epochs on full MAESTRO dataset (batch size 32). Fine-tuning GPU-hours scale with training data size. Annotation hours calculated based on 18 min per recording for expert temporal localization across five error categories. Macro-F1 values correspond to performance metrics in Fig. 5.

Table 4 reveals that self-supervised pretraining requires 73.0 GPU-hours upfront computational investment, while fine-tuning costs scale proportionally with data splits (1.2 to 8.7 GPU-hours), constituting 1.6–11.9% of pretraining expense. Manual annotation demands substantially higher resource investment, ranging from 38 h for 10% splits to 385 h for complete dataset annotation. The critical comparison emerges at equivalent performance thresholds: achieving approximately 80% F1 with pretraining requires 74.2 total GPU-hours plus 38 annotation-hours (10% data), whereas reaching comparable performance without pretraining demands approximately 78.1 GPU-hours plus 192 annotation-hours (50% data interpolated from Fig. 5 curve). The 73 GPU-hour pretraining investment enables 5-fold reduction in annotation requirements (38 versus 192 h), yielding net resource savings when GPU infrastructure exists and expert annotation constitutes the primary bottleneck. Conversely, scenarios with limited computational budgets may accept higher annotation overhead (192 versus 38 h) to reduce total GPU consumption from 78.1 to 5.4 h, representing a 14-fold computational savings at the cost of 5-fold annotation increase. These computational analyses pertain exclusively to the controlled experimental setting described throughout this study; the resource estimates are not intended to characterize requirements for production deployment, which would additionally necessitate validation on authentic student performance data, privacy-preserving data management infrastructure, and alignment of detection outputs with pedagogical feedback standards established by expert instructors.

Fine-grained error detection capability verification

To evaluate the generalization capacity and stability of the proposed framework in controlled perturbation event detection across varying repertoire complexity levels, the test dataset is partitioned into three difficulty tiers classified according to the technical demands of the corresponding MAESTRO compositions. The aim of this test is to evaluate whether it is possible to maintain a balanced level of error detection between simple repetitive mistakes and more complex technical mistakes. The primary works (about 60 pieces) contain simple melodies and basic harmonies, the intermediate works (about 70 pieces) introduce more complex rhythm changes and harmonic progression, and the advanced works (about 60 pieces) involve dense chords, fast scale runs and complex pedal techniques. Macro-F1 is measured separately for each series. The robustness benefit of multimodal fusion for complex scenes is compared to the two best baselines (MT3 and Music Transformer), as shown in Fig. 6.

Fig. 6.

Performance comparison across different difficulty levels.

Figure 6 shows the performance comparison of the three methods at the three difficulty levels, and the proposed method maintains the highest F1 value at all difficulty levels. With the increase of difficulty, the performance of all approaches becomes harder to achieve, but the performance degradation of the proposed method is less and it always has a significant lead over the state-of-the-art method. The more pronounced performance degradation of Music Transformer on advanced-difficulty compositions further confirms the robustness advantage conferred by multimodal fusion under complex perturbation conditions.

The observed performance decline from beginner to advanced repertoire (89.1% to 83.2% F1, a 5.9% point reduction) reflects complexity factors inherent to technically demanding works. Advanced pieces exhibit elevated note density, reducing temporal separation between error events and increasing probability of overlapping acoustic signatures where multiple simultaneous errors compound detection difficulty. Pedal technique complexity escalates through frequent partial pedal applications and rapid changes synchronized with dense harmonic progressions, contrasting with sustained pedal patterns spanning extended phrases characteristic of beginner-level repertoire. Polyphonic texture density in advanced works obscures individual note-level errors within dense harmonic clusters, challenging pitch and dynamics discrimination. The synthetic kinematic proxy becomes less reliable at high performance velocities, as deterministic MIDI-to-trajectory mapping inadequately models biomechanical constraints during rapid hand position transitions and complex fingering sequences. Despite these challenges, the proposed method maintains 4.6 and 9.7% point advantages over MT3 and Music Transformer respectively on advanced pieces, demonstrating that multimodal fusion provides robustness when individual modality reliability degrades under complexity.

Visualization analysis and typical cases

In order to explain the mechanism of the proposed model and evaluate the effectiveness of the strategy of cross-modal fusion, attention weights corresponding to different types of errors are analyzed. The goal of this analysis is to explain how the proposed model adjusts the weight of information from each modality according to the perturbation category, and to confirm the superiority of the proposed cross-modal attention mechanism compared to fixed-weight fusion. Thirty samples of each of the three representative perturbation categories—pitch, timing, and touch—are randomly selected from the test set to ensure coverage of qualitatively distinct error signatures. The attention weight matrix from the eighth layer of the cross-modal Transformer fusion module is extracted, and the mean attention weight assigned by each modality to the remaining modalities is computed across the selected samples. The pattern of information flow between the modalities is depicted in Fig. 7.

Fig. 7.

Cross-modal attention weights visualization for different error types.

Recalling that the kinematic modality constitutes a deterministically derived MIDI-based proxy rather than an independently captured visual stream, the attention weight patterns reported here reflect the model’s adaptive exploitation of heterogeneous structured representations rather than genuine audio-visual complementarity arising from independent sensors. Figure 7(a) shows that attention weights between audio and MIDI are highest during pitch perturbation detection, with kinematic proxy weights remaining comparatively low. Figure 7(b) demonstrates a relatively balanced attention distribution across the three modalities for timing perturbation detection. Figure 7(c) reveals significantly enhanced weights directed toward the kinematic proxy modality during touch perturbation detection, consistent with the proxy’s structured encoding of articulation-related temporal patterns and confirming that cross-modal attention mechanisms adaptively reallocate modality contributions according to the representational structure most discriminative for each perturbation category.

To demonstrate the complete detection process performed by the model on actual performance samples, a case study is performed by analyzing the Chopin Etude Op.10 No.3 test sample. This sample lasts for roughly 8 s and consists of two intentionally inserted perturbation events—a pitch perturbation occurring at 2.3 s and a timing perturbation at 5.1 s—representing controlled synthetic instances consistent with the injection protocol described in Sect.  2.1.1. The visualization for the case demonstrates the end-to-end process from the input to the output for fault localization, which includes the Mel spectrum, the detection probability curves for five types of errors, and the temporal evolution of the weights for cross-modal attention. This visualization demonstrates the interpretability and detection accuracy of the model under realistic perturbation conditions, as shown in Fig. 8.

Fig. 8.

Typical error detection case study (Chopin Etude Op.10 No.3 Fragment).

Figure 8a presents the input spectrogram with annotated pitch and timing error intervals. Figure 8(b) shows the detection probability curves, which show high peaks above the threshold at the two error points, and the probabilities for other types of errors remain low. Figure 8(c) displays attention dynamics, where audio modality weights increase significantly during pitch errors and kinematic proxy modality weights enhance during timing errors, thus validating the adaptive adjustment capability of cross-modal attention mechanisms.

Discussion

Under controlled experimental conditions on the MAESTRO test set, the proposed framework achieved a macro-F1 score of 87.3%, representing a 5.2% point improvement over the best baseline MT3 (82.1%). The core contribution demonstrated by these results is the fusion across audio spectral features, symbolic MIDI representations, and a MIDI-derived spatial/kinematic proxy, and the demonstration of cross-modal attention’s ability to exploit such heterogeneous representations under controlled conditions; these findings do not constitute evidence that adding video-based hand pose observation would necessarily yield similar gains, which remains a direction for future validation with video-equipped datasets. The cross-modal attention mechanism dynamically weights the contribution of each modality based on the perturbation category. Lin et al. demonstrated parameter efficiency advantages of multimodal joint learning through compact architectural designs³⁹. The current work prioritizes feature interaction flexibility through cross-modal attention mechanisms over parameter reduction, enabling adaptive modality weighting for different error types. This architectural choice accepts increased computational complexity in exchange for improved detection performance across diverse error characteristics, with parameter efficiency optimization through model compression or knowledge distillation remaining directions for future work targeting resource-constrained application scenarios pending validation on authentic student performance data. Wang et al. identified constraints of audio-only frameworks in performance evaluation under single-modal conditions⁴⁰; the present work extends beyond this boundary by incorporating a MIDI-derived kinematic proxy that, despite its synthetic nature, demonstrably improves touch perturbation detection to 82.1% F1 through structured spatial encoding of articulation-related features under controlled experimental conditions.

Performance disparities among baseline methods reflect fundamental methodological limitations for fine-grained error detection. DTW-HMM achieves substantially lower performance on touch error detection compared to pitch, timing, and pedal categories, exposing limitations of template matching approaches when error characteristics manifest as subtle temporal and spectral variations rather than discrete event mismatches. DTW alignment optimizes global sequence distance without explicitly modelling local perturbation signatures. As a result, touch perturbations that produce gradual envelope changes through duration modifications tend to be conflated with natural expressive timing variations present in correct performances. CNN-LSTM baselines improve touch detection through learned feature representations capturing local acoustic patterns, yet remain constrained by unimodal audio input lacking symbolic timing annotations and spatial articulation cues. Transformer-based methods leverage self-attention to model long-range temporal dependencies, demonstrating stronger touch perturbation detection capabilities than CNN-LSTM and DTW-HMM approaches under the controlled experimental conditions of this study. However, single-modality architectures cannot exploit complementary information across acoustic, symbolic, and kinematic domains. The progressive performance improvements from template-based methods to learned unimodal representations to multimodal fusion demonstrate that cross-modal attention mechanisms address fundamental limitations of prior approaches for distinguishing subtle articulation variations.

The self-supervised pre-training method is most helpful where there is a scarcity of annotations, and the pre-trained model outperformed the randomly initialized model by 2.6%. While the music score-guided network proposed by Huang and Hung relies on a vast amount of data on symbol-audio alignments⁴¹, the present study uses contrastive learning to learn general representations from unannotated data. Park et al. constructed a multi-level perceptual feature dataset for evaluation⁴², and the two-stage training paradigm used provides a feasible solution for addressing the problem of lack of annotated samples. Ramoneda and Jeong demonstrated the effect of repertoire complexity on evaluation accuracy⁴³; consistent with this finding, controlled perturbation detection performance decreases across all approaches as repertoire complexity increases, while multimodal fusion constrains this degradation to 5.9% points under the present experimental conditions. Kato and Nakamura’s computational analysis method provides a theoretical basis for piano performance assessment⁴⁴, and the present study utilizes pyramid convolution and dynamic sparse attention to maintain a controlled perturbation detection F1 of 83.2% on advanced-difficulty compositions. Worrall et al. pointed out that expressive performance evaluation in outdoor environments faces challenges⁴⁵, and this study achieves state-of-the-art performance in controlled environments to provide a foundation for future transfer. Jeon and Li emphasized that key segments play an essential part in the evaluation process⁴⁶, and the temporal localization module of the present study achieves fine-grained perturbation boundary detection under the controlled experimental conditions established in Sect.  2.1.1.

This research method has several limitations. While controllable perturbations allow systematic benchmarking, they may not reproduce the contextual and pedagogical structure of real learner mistakes; thus, the reported scores should be interpreted as performance on injected perturbation events—where “error” functions as a proxy label for stochastically generated perturbations—rather than as indicators of detection capability on naturally occurring student mistakes, and any application to music education settings remains conditional on validation against teacher-annotated student performance datasets. The demonstrated capabilities represent a scalable framework for learning robust representations and detecting controlled error events under limited annotations. Specifically, there exist differences in the characteristics of error distributions between synthetic perturbations and authentic human learner behaviors, as detailed in Sect.  2.1.2. Future studies should involve teaching data to improve generalization capacity. For extreme cases, slight pitch variation in close chords and rhythm mistakes in very fast tempos are still difficult to model accurately. There is a trade-off between the computational complexity of the model and real-time constraints, and lightweight approaches and hardware solutions have been pinpointed as the key areas to focus on in future work. The cross-instrument generalization ability has not been validated yet; when applying the model to different instruments, it may be required to change the modal configuration. While the controlled experimental validation on synthetic error distributions demonstrates the technical feasibility of the proposed architecture, deployment in authentic educational settings requires validation on teacher-annotated student performance data to confirm generalization to naturally occurring learner mistakes. Subject to validation on teacher-annotated student performance data confirming generalization to naturally occurring learner mistakes, this scalable framework constitutes a technical foundation for potential future integration into music education support tools, with the scope of any such deployment appropriately bounded by the pedagogical and privacy considerations inherent to automated assessment contexts.

Conclusion

This study proposes a self-supervised multimodal Transformer architecture that integrates audio, MIDI, and MIDI-derived kinematic proxy representations to achieve fine-grained detection of controlled perturbation events in piano performance under controlled experimental conditions. The proposed two-stage training approach enables the learning of generic music representations during pre-training and error detection optimization during the fine-tuning phase, thus greatly diminishing the need for labeled data. Experiments conducted on the MAESTRO test set show that the proposed system achieves a macro-F1 of 87.3% across the five controlled perturbation detection tasks, representing a 5.2% point improvement over the state-of-the-art baseline MT3. Controlled perturbation detection accuracies for touch and pedal categories reach 82.1% and 88.9% respectively, and the model maintains an F1 of 83.2% on advanced-difficulty compositions under the synthetic perturbation evaluation protocol.

Subject to external validation on teacher-annotated student performance data, this method constitutes a scalable framework whose demonstrated contribution is the fusion across audio, symbolic MIDI, and a MIDI-derived spatial/kinematic proxy, and the ability of cross-modal attention to exploit such heterogeneous representations under controlled conditions—findings that do not imply that video-based hand pose observation would necessarily yield similar gains, which remains a direction for future work. The cross-modal attention mechanism makes it possible to adaptively allocate the weights and control the influence of different modalities based on the perturbation category being detected. The capability of precise temporal localization is the basis for the feasibility of the application of automatic assessment. Future research directions involve the usage of real learner data to minimize the distribution gap between the errors of the synthesized data and the real errors, improving the architecture of the model to make it more efficient in real-time processing, as well as the expansion of the proposed method to other instruments to validate the generalization capability on different instruments, thereby establishing a new technical paradigm for fine-grained music performance analysis under controlled conditions.

Author contributions

K.Y. and J.C. wrote the main manuscript text and K.Y. prepared figures. All authors reviewed the manuscript.

Funding

This work was supported by the High-Level Talent Research Projects of Guangxi Minzu Normal University for Nationalities (Grant Nos. 2025SBNGCC026 and 2025SBNGCC016).

Data availability

The original contributions presented in the study are included in the article Material, further inquiries can be directed to the corresponding authors.

Declarations

Competing interests

The authors declare no competing interests.