1. Introduction

Emotion is a fundamental part of a human’s mental state that contributes to the richness and adaptive quality of human mental life, such as interactions between people, decision making, and education^[1,2]. Human emotion analysis and recognition have been active and challenging research topics within the research community for a long time due to their wide range of applications. Rapid advances and model improvements in deep learning and machine learning techniques have enabled researchers to expand the application of emotion recognition to different areas. The primary focus in the early stages of emotion recognition research was on unimodal approaches that relied on behavioural data or physiological signals, treating each modality separately. Although unimodal approaches provide valuable information and have demonstrated promising performance across studies, they often struggle to accurately capture emotional states when relying on a single data source. A single modality may not sufficiently represent individual differences, be sensitive to noise, or be affected by data variability, which may reduce the generalization capability of emotion recognition systems. This makes it challenging to describe an emotional state comprehensively using a single modality^[3,4].

As a result, there has been growing interest in treating emotion recognition as a multimodal problem that integrates information across multiple data sources. Multimodal emotion recognition (MER) refers to the process of identifying human emotions using a combination of physiological and behavioural signals, such as facial features and physiological measurements. Emotion recognition systems that leverage multiple modalities offer several notable benefits. By integrating complementary information from multiple sources, multimodal approaches can capture richer representations of emotional states and often achieve higher accuracy than unimodal models. They also tend to be more robust and reliable in diverse real-world scenarios, as different modalities can compensate for one another’s limitations. Furthermore, multimodal systems reduce ambiguity by jointly analyzing multiple signals, which is particularly beneficial when emotional expressions are subtle, complex, or mixed^[5,6]. In addition, previous studies suggest that multimodal signals capture richer emotional information than unimodal inputs and more closely reflect natural patterns of human expressive behaviour^[7].

However, in real-world scenarios, it is not always assured that data from all modalities will be consistently available due to many factors such as sensor malfunctions, hardware limitations, privacy constraints, environmental interference, and data transmission issues^[8]. In addition, models must understand the complex intramodal and cross-modal interactions to predict emotion. This challenge highlights the importance of developing methods that can effectively handle incomplete or missing modalities while preserving the advantages of multimodal integration. To address the missing modality problem, several early studies in the research community employed multimodal data fusion techniques, including early, late, and intermediate fusion, combined with various imputation techniques during model inference. However, these approaches often degraded the overall performance of the model. Recent multimodal methods have increasingly focused on model based approaches that learn a joint or coordinated representation across modalities^[9-11]. Despite promising results when modalities are partially missing, these methods usually lack robustness under more challenging conditions where an entire modality or more than one modality is missing. In addition, in practical settings, the pattern of missing modalities is often uncertain, where one or more modalities are partially or entirely absent in a random form. To address these challenges, a model should be capable of handling situations where multiple modalities are missing (whether partially or entirely) while still learning robust joint representations under such uncertainty. This work introduces a regenerative network into the variational autoencoder (VAE) to effectively handle missing modalities. To further capture temporal dynamics and classify emotions from sequential data, we incorporate an attention-enhanced long short-term memory (LSTM).

In summary, the main contributions of this work are as follows:

• We propose a cross-modal latent regeneration (CMLR) framework for robust MER under incomplete modality conditions. The proposed framework integrates latent regeneration, attention-based multimodal fusion, and temporal modelling to improve robustness when one or more modalities are partially or completely unavailable during inference.

• We introduce a cross-modal attention-LSTM to model sequential dependencies, enabling adaptive interaction between modalities and improving the quality of multimodal feature integration.

• We demonstrate that the proposed method maintains robust performance and achieves 5.33%-17.48% improvement compared to existing baseline models.

• To further evaluate the generalization capability of the proposed framework, we additionally include experiments using different combinations of modalities beyond electroencephalogram (EEG) and facial data. This allows us to assess whether the proposed latent regeneration and attention-based framework can generalize across heterogeneous multimodal settings, demonstrating its flexibility beyond specific modality pairs.

The remainder of this paper is organized as follows. Section 2 reviews existing studies on MER and discusses challenges associated with incomplete modalities. Section 3 presents the proposed framework for emotion recognition under incomplete data conditions. Section 4 outlines the training procedure. Section 5 describes the datasets, presents the experimental results, and evaluates the performance of the proposed approach. Finally, Section 6 concludes the paper and discusses potential directions for future research.

2. Related Work

Research in psychology has shown that human emotions encompass complex combinations of subjective feelings, physiological reactions, and behavioural responses triggered by internal or external stimuli^[12]. Early research in emotion recognition tasks was primarily focused on uni-modal approaches where the different modalities are treated independently. Researchers have analyzed a wide range of behavioural or physiological signals, such as facial expressions^[13-17], speech^[18-20], gestures^[21], electrocardiogram (ECG)^[22,23], and EEG^[24-26], where each of these approaches has very distinct statistical properties. Among these modalities, EEG signals and facial expressions are widely used for human emotion analysis. As discussed in Section 1, although unimodal approaches have demonstrated promising performance across various studies, they are inherently limited in capturing the full complexity of human emotional states when relying on a single modality. In addition, behavioural data and physiological signals are the primary sources for identifying human emotions. However, behavioural data often have reliability issues and can be ineffective since humans may involuntarily or consciously conceal their real emotions (so-called social masking)^[27]. Moreover, collecting physiological signals relies on specialized equipment during sample acquisition, which is often invasive, expensive, and needs technical guidance from professionals^[12]. In addition, the complexity of emotions and individual differences in physiological responses may lead to declining prediction performance from a classification perspective when using a single physiological signal^[28]. These limitations have motivated researchers to consider emotion recognition as a multimodal task.

The rest of this section presents an overview of the prior work on MER and a brief overview of related deep learning architectures used in emotion recognition.

2.1 MER

MER aims to infer human emotional states by integrating behavioural indicators and physiological responses obtained from multiple data sources. Compared with unimodal approaches, multimodal frameworks generally provide more reliable predictions because complementary modalities can compensate for limitations present in individual signals. Emotional responses are inherently complex and vary across individuals, which can introduce ambiguity when relying on a single modality. By jointly considering multiple sources of information, multimodal systems can reduce uncertainty and provide a more comprehensive representation of affective states^[29,30]. Several studies have explored MER using diverse combinations of sensory inputs and machine learning techniques. For instance, the studies^[31,32] developed frameworks that integrate facial expression and body gesture information to improve emotion prediction performance. In physiological-based emotion analysis, Dessai and Virani^[33] investigated early fusion of ECG and galvanic skin response (GSR) features and evaluated multiple classifiers, showing improved classification performance when combining modalities. A hypercomplex multimodal neural network was proposed in the work^[34], where parameterized hypercomplex multiplications were used to model interactions between EEG, ECG, GSR, and eye-tracking signals, demonstrating improved modelling of cross-modal relationships. Similarly, Pan et al.^[35] introduced a deep learning architecture combining facial expression, speech, and EEG modalities using specialized subnetworks designed for each signal type. Transformer-based approaches have also been investigated, such as the Token-disentangling Mutual Transformer proposed in the work^[36], which separates modality-specific and shared emotional characteristics through token-level representation learning. Additional multimodal architectures integrate physiological and visual signals for emotion classification. For example, Ali and Hughes^[37] proposed a unified transformer-based framework combining facial images with ECG or PPG signals, demonstrating improved performance compared to unimodal configurations. In the study^[38], a dual-branch neural architecture was introduced to jointly process EEG and eye movement signals, where multi-scale fusion and attention mechanisms were applied to improve feature selection. The Transformer Autoencoder presented in the study^[7] addresses missing modality challenges by learning intra- and inter-modal representations using transformer encoders, followed by reconstruction and classification components. Other studies have explored self-supervised representation learning strategies for multimodal emotion analysis^[39,40], leveraging transformer architectures to capture relationships across heterogeneous data sources. A number of works have specifically focused on emotion recognition using EEG signals and facial videos, which are also considered in this study. Several approaches^[41-45] have investigated joint modelling of these modalities using deep learning techniques. Among these, Chen et al.^[44] proposed a convolutional network (CNN)-based framework that extracts spatial EEG features using convolutional kernels of different receptive fields and incorporates attention mechanisms to enhance facial feature representations. The work^[45] applied CNN-based facial feature extraction together with bidirectional LSTM modelling of EEG signals, where classification results from each modality were combined using evidence theory-based fusion. Another related study^[46], introduced a decision-level fusion strategy combining EEG signals, peripheral physiological signals, and facial expression data within the valence-arousal framework.

Recent studies have also explored alternative modality combinations such as EEG with eye movement signals. For example, Song et al.^[47] proposed a cosine-similarity-based fusion framework that integrates EEG and eye movement representations using gated attention mechanisms, demonstrating strong performance on the SEED-IV dataset. Cross-modal interaction mechanisms have also been investigated to exploit complementary modality information better. In the study^[48], a cross-modal guidance network was introduced where EEG representations are used to refine eye movement features through adaptive weighting and feature recalibration, improving emotion recognition accuracy.

Multimodal learning frameworks offer advantages over unimodal approaches by providing richer and more complementary information, which can improve the reliability and clarity of emotion recognition systems. Accurately predicting emotional states requires the model to capture both relationships within each modality and interactions across modalities. In practical scenarios, multimodal systems must also remain stable when some inputs are noisy, partially corrupted, or unavailable during inference. A key challenge in multimodal learning lies in effectively integrating heterogeneous sources of information into a unified representation^[49]. Multimodal fusion strategies aim to combine features extracted from different data sources, such as visual, physiological, or textual signals, to improve predictive performance. Early research explored several categories of fusion strategies, including early fusion, where modalities are combined at the input level, late fusion, where predictions from individual modalities are aggregated, and intermediate fusion, where interactions are learned at the feature level^[50,51]. Several recent studies have explored robust MER using advanced fusion and cross-modal learning strategies, including GCNet^[52], SDR-GNN^[53], CIF-MMIN^[54], and MoMKE^[55]. These methods utilize mechanisms such as graph neural networks, modality interaction learning, and incomplete modality handling to improve emotion recognition performance under challenging conditions. Moreover, some recent approaches increasingly rely on representation learning techniques that enable the model to discover shared structures across modalities automatically. Among these approaches, multimodal variational autoencoders (MMVAEs) have emerged as a prominent framework, extending the standard VAE formulation to handle multiple modalities jointly^[56]. MMVAEs are designed to learn both shared and modality-specific latent representations, facilitating cross-modal understanding and improving the effectiveness of multimodal integration. In recent years, a growing body of work has explored MMVAE-based architectures for multimodal representation learning and fusion^[57-61]. In this study, we proposed a MER framework that combines latent regeneration, cross-modal attention, and temporal sequence modelling to improve robustness under incomplete modality scenarios.

3. Proposed Method

The proposed framework for robust MER is composed of three main components: 1) a VAE for modality-specific representation learning, 2) a regenerative joint VAE encoder for cross-modal latent projection and regeneration, and 3) a cross-modal Attention LSTM for sequential emotion classification.

3.1 The VAE

VAE^[62] is a probabilistic generative model that encodes input data into a latent representation and reconstructs it from latent samples. Unlike traditional autoencoders, the VAE constrains latent variables to follow a standard Gaussian prior N(0,I). This constraint ensures that the latent space captures meaningful variations in the data while preserving consistency with the prior distribution. The model is trained by maximizing the evidence lower bound:

(1)$L(x;\theta ,\varphi )=-E_{q_{\varphi }(z\mid x)}[\log p_{\theta }(x\mid z\left)\right]+D_{\mathrm{KL}}\left(q_{\varphi }\right(z\mid x)\Vert p(z\left)\right)$

where the reconstruction term ensures the accuracy between the input and output, and the Kullback-Leibler (KL) divergence regularizes the approximate posterior. Therefore, the VAE modules provide probabilistic embeddings for each modality. First, we trained two separate VAEs to capture modality-specific representations of EEG signals and facial images. Each VAE consists of an encoder-decoder pair, generating latent variables z_eeg and z_face as follows:

(2)$z_{\mathrm{eeg}}=q_{\varphi }^e\left(x^{\mathrm{eeg}}\right),z_{\mathrm{face}}=q_{\varphi }^f\left(x^{\mathrm{face}}\right)$

where $q_{\varphi }^e$ and $q_{\varphi }^f$ represent the encoder distributions for the EEG and face modalities, respectively. We adopted a ResNet 101-based VAE pre-trained on diverse external face datasets^[63] for the facial modality, to address limited training data in DEAP and MAHNOB-HCI while enabling effective feature extraction. For the EEG modality, we employed a CNN based encoder, adapted to dataset-specific input representations, consisting of convolutional and max-pooling layers to capture temporal patterns, followed by fully connected layers that estimate the parameters (mean and log-variance) of the latent distribution. This architecture allows the encoder to capture essential temporal dynamics from raw EEG while maintaining a compact and informative latent representation. Figure 1 illustrates the overall architecture of the VAE. Each encoder was trained for 10 epochs with a batch size of 32, using the Adam optimizer with an initial learning rate of 0.001.

Display Full Size

Download

Figure 1. The architecture of VAE. VAE: variational autoencoder.

3.2 CMLR

To address the missing modality problem in MER, we extend unimodal VAEs into a regenerative multimodal framework. Inspired by the study^[64], we introduce cross-modal projections to the pre-trained encoders to regenerate the latent representation of one modality from another. This ensures that informative features can still be exploited even in the absence of other modalities. Given a paired input sample (x^eeg, x^face), where x^face, we obtain latent codes z_eeg and z_face from the frozen encoders. Two residual multilayer perceptrons (MLPs) are introduced as projection networks:

(3)${\hat{z}}_{\mathrm{eeg}}=G_{f\to e}\left(z_{\mathrm{face}}\right),{\hat{z}}_{\mathrm{face}}=G_{e\to f}\left(z_{\mathrm{eeg}}\right)$

where G_f→e maps from the face latent space to the EEG latent space, and G_e→f performs the reverse mapping. Both networks share the architecture 128→256→128, ensuring dimensional consistency while enabling non-linear transformations.

3.3 Regeneration loss

To enforce alignment between the original and regenerated representations, we introduce a latent space consistency loss as follows.

(4)$L_{\mathrm{reg}}=\frac{1}{2}({\Vert {\hat{z}}_{\mathrm{eeg}}-z_{\mathrm{eeg}}\Vert }_2^2+{\Vert {\hat{z}}_{\mathrm{face}}-z_{\mathrm{face}}\Vert }_2^2)$

This regenerative objective ensures that the projected latent vectors remain faithful to their true modality-specific counterparts. By design, this mechanism enables robust inference of missing modalities, since the regenerated latent $\tilde{z}$ can substitute for the unavailable z during downstream tasks.

We considered cosine similarity and KL divergence (using the pre-trained decoder) as alternative objectives. Cosine similarity was less effective, as it only enforces directional alignment, while L2 penalizes scale and directional discrepancies. While KL divergence is standard in VAE training, it is not suitable here, since the regeneration networks produce deterministic outputs rather than distributions. Moreover, enforcing Gaussian priors on regenerated latents distorted the modality-specific geometry and degraded downstream classification. For these reasons, we adopt L2−Loss as the regeneration objective.

3.4 Sequential classification with LSTM

After training the cross-modal latent regeneration framework, we used it to extract modality-specific latent representations z_face and z_eeg from facial and EEG modalities, respectively. Inspired by the work^[65], we implemented a cross-modal temporal attention block to model cross-modal interactions at the sequence level. Formally, the attention mechanism is defined as follows.

(5)$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d}}\right)V$

The mechanism operates with queries Q = Z_face, keys K = Z_eeg, and values V = Z_eeg. This operation yields an attended sequence $\tilde{Z}$, where $\tilde{Z}\in R^{60\times d}$ for the DEAP dataset and $\tilde{Z}\in R^{30\times d}$ for the MAHNOB-HCI dataset, with d denoting the latent dimensionality, where d is the latent dimensionality. In cases where a modality is missing, the regenerated latent representation $\tilde{z}$ from the cross-modal projection is used instead of the unavailable z.

The attended sequence $\tilde{Z}$ was then passed into a two-layer LSTM with 128 hidden units to capture temporal dependencies in the multimodal data. The LSTM selectively retains relevant temporal information through its gating mechanisms and produces a sequence of hidden states $\left\{h_t\right\}$. The final hidden state h_T is mapped to emotion predictions via a fully connected layer followed by softmax activation.

(6)$\hat{y}=\mathrm{softmax}(W\cdot h_T+b)$

The LSTM classifier was optimized using cross-entropy loss with a dropout rate of 0.3 to reduce overfitting. During training, we considered four combinations of latent inputs: $(z_{\mathrm{face}},z_{\mathrm{eeg}}),(z_{\mathrm{face}},{\hat{z}}_{\mathrm{eeg}}),({\hat{z}}_{\mathrm{face}},z_{\mathrm{eeg}})$, and $({\hat{z}}_{\mathrm{face}},{\hat{z}}_{\mathrm{eeg}})$. Each epoch was divided into four passes, one for each pairing, all of which were processed through the attention block and LSTM to update the model with shared weights. The experiments were conducted using 5-fold cross-validation to ensure the reliability and robustness of the results. This architecture enables the model to integrate modality-specific representations from the regenerative framework, capture cross-modal dependencies through attention, and exploit temporal dynamics for robust emotion recognition.

4. Training Procedure

The overall training procedure is composed of three sequential stages: (1) independent pre-training of two VAEs for reconstructing facial images and EEG signals, where the facial branch utilizes a ResNet-101 backbone, (2) optimization of the CMLR module to learn mappings between modality-specific latent spaces, and (3) extraction of latent representations using the trained CMLR framework for subsequent LSTM-based emotion classification. To ensure consistency across stages, the same hyperparameter configuration was adopted throughout the training pipeline. These settings were determined based on commonly used configurations reported in previous studies, along with preliminary empirical evaluation. A batch size of 32 was selected to balance computational efficiency and stable convergence on modern GPU hardware. The learning rate was fixed at 0.001, a standard choice that demonstrated reliable convergence behaviour when combined with the Adam optimizer. The dataset was divided into 80% for training and 20% for validation. The models were trained for 10 epochs, as extending the training duration resulted in only minor performance gains while increasing computational requirements. This configuration allowed the networks to learn stable reconstructions for both EEG and facial modalities without introducing noticeable overfitting or instability. Since the objective of this stage is to obtain informative latent representations rather than enforce strong disentanglement constraints, the weighting parameter was set to β = 1, corresponding to the standard VAE formulation. This choice maintains an appropriate balance between reconstruction quality and latent space regularization. Although higher β values (β > 1) can promote more disentangled latent factors, they typically impose a stronger penalty on reconstruction accuracy, which may negatively affect representation quality for downstream tasks^[66]. The pre-trained VAE components were subsequently incorporated into the multimodal framework’s facial encoder and decoder. This initialization improved reconstruction performance compared to training solely on the DEAP and MAHNOB-HCI datasets, resulting in more stable and transferable facial feature representations.

4.1 VAE training

Image encoder: Both the DEAP and MAHNOB-HCI datasets contain a relatively small number of participants, which limits the ability to train a robust VAE directly on these datasets while maintaining strong generalization in facial representation learning. To overcome this limitation, we pre-trained a VAE with a ResNet-101 backbone (Figure 2a) using a larger and more diverse collection of facial image datasets aggregated from multiple publicly available sources^[67-71]. The ResNet-101 architecture was selected due to its proven effectiveness in extracting hierarchical visual features, enabling the VAE to learn discriminative facial representations that capture both structural characteristics and expression related information. This strategy improves the ability of the model to generalize across subjects and experimental conditions.

Display Full Size

Download

Figure 2. Overview of the proposed model. (a) Two pre-trained encoders are used to obtain actual and regenerated latent representation with the loss computed using regeneration consistency; (b) When one modality is missing (e.g., EEG), the model uses the regenerated latent ${\hat{z}}_{\mathrm{eeg}}$ in place of the unavailable z_eeg before passing it through the attention and LSTM modules. EEG: electroencephalogram; LSTM: long short-term memory.

The Chicago Face Database^[67] provides high-quality facial images accompanied by extensive annotations and is widely used in studies related to facial perception and affective computing. The FERET dataset^[68], originally developed to support advances in face recognition research, contains facial images captured under varying illumination conditions, poses, and expressions, making it suitable for learning robust identity-invariant representations. The FEEDTUM dataset^[69] includes video recordings of natural and spontaneous facial expressions, offering valuable temporal variation in emotional displays commonly used for affect recognition research. The MUCT Landmarked Face Database^[70] contains facial images with manually annotated landmark points, facilitating reliable facial alignment and supporting the learning of spatially consistent facial features across individuals. The Person Face Dataset^[71] consists of synthetically generated facial images created using generative adversarial networks, providing additional diversity and enhancing robustness by introducing photorealistic samples. By combining these datasets during pre-training, the facial encoder-decoder benefits from increased variability in identity, pose, and expression, leading to more transferable and stable latent representations. For image-based datasets such as DEAP and MAHNOB-HCI, facial regions were first detected and aligned using multi-task cascaded convolutional networks (MTCNN) to ensure spatial consistency across samples. The extracted facial images were then converted to red-green-blue format, resized to 224 × 224 pixels, and normalized using dataset-specific statistics before being provided as input to the VAE.

EEG encoder: In addition to the facial branch, we trained a dedicated VAE for the EEG modality to learn compact and informative latent representations prior to multimodal integration. The EEG encoder was implemented using a convolutional architecture. Specifically, the encoder consists of two convolutional layers with 32 and 64 filters, respectively, each followed by a ReLU activation function, and an adaptive average pooling layer. The resulting representation is then flattened and projected through a fully connected layer to produce a 128-dimensional latent vector. This design enables the model to extract stable and discriminative EEG features suitable for downstream emotion recognition.

For the DEAP and MAHNOB-HCI datasets, the EEG inputs were represented as one-second segments with dimensions corresponding to 32 channels and 128 time samples. In contrast, the SEED-IV dataset provides pre-extracted differential entropy (DE) features computed from 62 EEG channels across five standard frequency bands. To ensure temporal alignment between modalities, each trial was segmented into fixed-length chunks consisting of 10 consecutive windows. As a result, each EEG sample was represented as a tensor of size 62 × 10 × 5, where 62 denotes the number of channels, 10 represents the temporal windows within each segment, and 5 corresponds to the frequency bands. This structured representation preserves both temporal and spectral characteristics of the EEG signals.

Similar to the facial branch, the EEG VAE was trained for 10 epochs using a batch size of 32 and the Adam optimizer with a learning rate of 0.001. Training the EEG VAE separately allowed the model to learn a stable and continuous latent space directly from electrophysiological data, providing a robust foundation for multimodal fusion and subsequent temporal modelling.

4.2 CMLR training

For training the CMLR module, EEG and visual modalities were used, with the visual modality corresponding to facial expression information in the DEAP and MAHNOB-HCI datasets. Since the objective is to learn relationships between latent spaces rather than reconstruct the original inputs, the encoders obtained from the previously pre-trained VAEs are utilized to extract modality-specific latent representations.

The EEG modality is processed using a pre-trained CNN-based encoder to capture relevant EEG signals. For both datasets, each EEG segment has dimensions 32 × 128, corresponding to one-second windows sampled at 128 Hz. The encoder transforms these signals into compact latent vectors that retain discriminative information related to emotional states.

For the visual modality, a ResNet-101-based encoder pre-trained in the previous stage is employed to obtain facial latent representations. For video-based datasets such as DEAP and MAHNOB-HCI, one representative frame per second is selected to preserve temporal structure while reducing redundancy. Facial regions are first detected and aligned using MTCNN, then cropped and normalized. The processed images are then passed through the encoder to obtain latent vectors that capture identity-invariant structural and expression-related characteristics. After obtaining latent representations from both modalities, denoted as z_eeg and z_face, cross-modal relationships are learned directly in the latent space through a regeneration mechanism.

Then, instead of enforcing a shared latent space, two MLPs are introduced to map between modality-specific latent spaces, where G_e→f(·) and G_f→e(·) denote the cross-modal projection networks.

While the encoder-decoder networks are pretrained to learn meaningful latent representations, the CMLR training stage focuses on aligning latent spaces. During this stage, the encoder parameters are kept fixed and only the projection networks are optimized using a latent regression objective.

(7)$L_{\mathrm{CMLR}}={\Vert {\hat{z}}_{\mathrm{face}}-z_{\mathrm{face}}\Vert }_2^2+{\Vert {\hat{z}}_{\mathrm{eeg}}-z_{\mathrm{eeg}}\Vert }_2^2$

To evaluate the effectiveness of the selected latent regression loss, we compare the L2 reconstruction loss and cosine similarity. While the L2 loss measures the numerical difference between the projected and original latent representations, cosine similarity evaluates their directional agreement in the latent space. The results are summarized in Table 1.

As shown in Table 1, the proposed projection network achieved a low L2 reconstruction loss of 0.063 and a high cosine similarity of 0.84. These results indicate that the projected latent representations closely match the original modality specific embeddings in both magnitude and direction. Based on the findings presented in Table 1, the L2 loss was preliminarily selected as the optimization objective for the CMLR module. Its effectiveness is further evaluated in Section 5.4.

The model was trained using a batch size of 32, a learning rate of 0.001, and 10 training epochs. After training the projection networks, the encoders are used to extract temporal latent sequences for each modality. The learned projection networks can generate approximate latent representations for missing modalities, which are then used for emotion classification, enabling robust performance under both complete and missing modality scenarios, as illustrated in Figure 2b.

4.3 CMLR-ALSTM-LSTM-based emotion classification

After training the CMLR framework, we extracted latent representations from the EEG and visual modalities for sequential emotion classification. For the DEAP and MAHNOB-HCI datasets, when both modalities were completely/partially available, the original latent features z_face and z_eeg were used. In cases where a modality was missing, its corresponding latent representation was replaced by the regenerated latent feature produced by the cross-modal mapping network. For the SEED dataset, the original latent representations were used without applying cross-modal regeneration. These latent features were then passed to a cross-modal attention block, where the facial latent representation attends to the EEG latent representation in order to improve multimodal feature integration. This operation produces an attended latent sequence $\tilde{Z}$, which captures cross-modal interactions while preserving temporal structure.

Emotional responses are inherently dynamic and evolve through temporal variations, which are reflected in both EEG activity and facial expressions. Consequently, modelling sequential dependencies is important for obtaining reliable emotion predictions. The attended latent sequence, denoted as $\tilde{Z}$, was provided as input to a two-layer LSTM network with 128 hidden units to capture temporal relationships within the multimodal representations. The LSTM’s gating structure enables the network to preserve informative temporal patterns while reducing the influence of transient noise or inconsistencies arising from partial or degraded observations. Model training was performed using the cross-entropy loss function for five epochs. A dropout rate of 0.3 was applied to improve generalization and mitigate overfitting. The last hidden state, h_t, was then fed into a fully connected layer, followed by a softmax activation function to obtain the final emotion prediction. This classification framework enables the model to combine robust latent representations from the VAE encoders, capture cross-modal dependencies through attention, and exploit temporal dynamics through the LSTM, resulting in a more reliable MER system under both complete and incomplete data conditions.

5. Results and Discussion

In this section, we provide a detailed description of the accuracy of emotion classification. Since emotion recognition has a significant impact on many areas, to better understand the classification output, we present emotion classification results in addition to classification robustness.

5.1 Dataset

To evaluate the performance of the proposed MER framework, we conducted experiments on three widely used benchmark datasets: DEAP^[72], MAHNOB-HCI^[73], and SEED IV^[74]. These multimodal human emotion datasets are designated exclusively for academic research, where each participant signed a consent form stating that their data can be used for research and imagery can be published. The multimodal emotion datasets summarized in Table 2 are publicly available for research purposes, and all participants provided informed consent allowing the use of their data for scientific analysis, including the publication of visual materials. The DEAP dataset includes EEG and peripheral physiological recordings from 32 participants, along with frontal facial videos available for 22 of these participants. Each participant viewed 40 one-minute video clips designed to elicit emotional responses. The MAHNOB-HCI dataset provides multimodal recordings from 30 participants, including EEG, facial video, audio, gaze information, and additional physiological signals, with stimulus durations ranging approximately from 35 to 117 seconds.

To improve data reliability and comparability across subjects, the EEG recordings in both datasets were preprocessed by the dataset providers using standardized procedures. These steps include downsampling the signals to 128 Hz, removal of physiological and environmental artifacts, band-pass filtering within the 4-45 Hz frequency range, re-referencing using a common average reference, and normalization. Emotional annotations in both datasets are based on self-assessment ratings defined on a 1-9 scale according to Russell’s valence-arousal model^[75]. For the classification task, ratings between 1 and 5 were considered as low levels, while ratings between 5 and 9 were treated as high levels, resulting in binary labels for both valence and arousal dimensions. In the case of the DEAP dataset, only participants 1-22 were included in the experiments, as facial video recordings are available only for this subset.

We excluded nine trials where objects/hands covered the participant’s face during part of the trial, and the final data segments consisted of 60 seconds after removing the 3-second pre-trial baseline. For MAHNOB-HCI, 526 valid trials were used, where the central 30 seconds were selected for each trial. The EEG was aligned with facial data by dividing the trials into 1-second segments and extracting the corresponding middle video frame for each EEG segment.

The SEED-IV dataset consists of synchronized EEG recordings and eye-movement signals collected from 15 participants across three experimental sessions, with each session comprising 24 emotion-elicitation trials. In each trial, participants watched a short clip while their EEG and eye-movement signals were recorded. The clips were selected to induce one of four emotional states, including neutral, happy, sad, and fear. EEG data were recorded using a 62-channel acquisition system, downsampled to 200 Hz, and filtered into five conventional frequency bands (delta: 1-4 Hz, theta: 4-8 Hz, alpha: 8-14 Hz, beta: 14-31 Hz, gamma: 31-50 Hz). The dataset providers extracted DE features for EEG and statistical descriptors for eye-movement signals, resulting in 1,080 paired EEG samples with a feature dimension of 62 × 10 and 1,080 corresponding eye-movement samples with a feature dimension of 32 × 10. In this study, we included only trials for which both EEG and eye-movement modalities were fully available, ensuring consistent multimodal input and excluding incomplete or partially missing modality pairs from downstream analysis. Unlike DEAP and MAHNOB-HCI, which include facial video data, the SEED dataset provides eye movement features as the second modality, which are used in place of facial information in our framework.

5.2 Experimental setup

The experiments were performed on a workstation equipped with an NVIDIA GeForce RTX 2080 Ti GPU with 11 GB of VRAM, an Intel Core i9-10920X CPU running at 3.50 GHz, and 64 GB of RAM. The models were implemented in PyTorch 2.1 and trained using CUDA 12.2. All experiments were conducted using an 80%-20% train-validation split, and the reported results correspond to the average performance over 5-fold cross-validation. For all datasets, a subject dependent evaluation strategy was employed. Specifically, all trials from all subjects were first combined and shuffled before splitting. As a result, the same subject could appear in both the training and testing sets, but different trials from that subject were used in each set. Importantly, temporal segments extracted from the same trial were always kept together and were never distributed across both training and testing sets, preventing segment-level data leakage caused by temporally correlated samples. The proposed CMLR-ALSTM framework contains a total of 89,211,844 trainable parameters, including the pretrained encoders, latent projection (regenerative) networks, cross-modal attention module, and LSTM classifier.

For comparison with recent robust MER approaches, we reproduced and adapted the publicly available implementation of the MoMKE^[55] framework for our experimental settings. The adapted implementation was evaluated under fully available modality conditions and complete missing modality scenarios where one modality was entirely unavailable during inference. The original implementation was modified to support the multimodal configurations used in this work, including EEG and face on the DEAP dataset.

5.3 Comparison with baselines and previous work

In this section, the performance of the proposed CMLR-ALSTM framework is evaluated against several baseline approaches and previously reported state-of-the-art methods on the DEAP, MAHNOB-HCI, and SEED-IV datasets. For the DEAP and MAHNOB-HCI datasets, both EEG and facial feature modalities are incorporated, and the corresponding results are shown in Table 3. As shown in Table 3, the proposed method achieves performance comparable to existing approaches on the DEAP dataset while maintaining competitive effectiveness relative to prior studies. Specifically, the proposed method attains 96.93% accuracy for valence and 97.08% for arousal, which is comparable to the best-performing prior models, including the pre-trained disentangled multimodal variational autoencoder (DMMVAE)-LSTM framework. The reproduced MoMKE model achieved performance comparable to our previous DMMVAE-LSTM framework because both approaches rely on learning a joint multimodal latent representation. In particular, both methods combine modality-specific latent embeddings into a shared representation space using a Product-of-Experts-based fusion strategy, enabling the model to capture complementary emotional information across EEG and facial modalities before classification. This indicates that the proposed latent regeneration mechanism can preserve discriminative multimodal features without relying on explicit shared latent-space modelling.

For the MAHNOB-HCI dataset, the proposed model consistently outperforms all baseline and prior approaches. In particular, it achieves 97.28% accuracy for valence and 97.05% for arousal, surpassing both the CNN-based and DMMVAE-based methods. This improvement suggests that the cross-modal latent projection strategy effectively captures complementary information between modalities, resulting in more robust and informative feature representations.

To further evaluate the generalization performance of the proposed approach under different combinations of modalities, we also included experiments on the SEED-IV data set, which contains EEG and eye movement signals instead of facial data. The objective of this evaluation is to assess the effectiveness of the proposed latent regeneration and temporal modelling strategy under different multimodal settings. Unlike DEAP and MAHNOB-HCI, the SEED-IV dataset is labeled as a multi-class classification task. Therefore, performance is reported as overall classification accuracy. As shown in Table 4, the proposed CMLR-ALSTM model achieves an accuracy of 93.84%, outperforming all compared methods, including the Transformer Autoencoder (92.69%) and DMMVAE-LSTM (92.77%). The improvement over these approaches demonstrates the effectiveness of the proposed latent regeneration mechanism for learning meaningful cross-modal relationships, even when the modalities differ (facial features vs. eye movement signals).

In addition, we conducted an ablation study using the DEAP dataset to analyze the contribution of the major components of the proposed CMLR-ALSTM framework. Specifically, we evaluated the impact of the attention mechanism and the proposed VAE-based latent representation strategy on emotion recognition performance. We implemented an AE-ALSTM baseline, in which the EEG modality was encoded using a standard autoencoder and facial features were extracted using a convolutional autoencoder. The same attention-based classifier was employed to ensure that the comparison focused on the representation-learning component. To evaluate the contribution of the cross-attention mechanism, we further replaced the cross-attention fusion module in the proposed framework with an intermediate fusion strategy that concatenates the latent representations from both modalities before passing them to the LSTM classifier. This variant is referred to as CMLR-LSTM. The results presented in Table 3 further demonstrate the contribution of the proposed VAE-based latent representation strategy and cross-attention mechanism. Compared with AE-ALSTM, the proposed CMLR-ALSTM framework achieves an average accuracy improvement of 1.34%, highlighting the effectiveness of the VAE-based latent representation in learning a more structured latent space. This structured representation facilitates latent regeneration and enables more informative multimodal embeddings than conventional autoencoders. Furthermore, the proposed CMLR-ALSTM framework achieves an average accuracy improvement of 0.80% over CMLR-LSTM, with the cross-attention module replaced by an intermediate fusion strategy. This result demonstrates the importance of the cross-attention mechanism in capturing complementary information between EEG and facial latent representations. By enabling interactions between modalities before temporal modelling, the attention module produces more discriminative fused representations, thereby improving emotion recognition performance.

The experimental findings across all datasets indicate that the proposed framework consistently outperforms baseline models and previously reported methods. The integration of CMLR with LSTM-based temporal modelling allows the framework to effectively capture both inter-modal relationships and temporal dependencies, contributing to enhanced performance in MER tasks.

5.4 Robustness analysis with missing modality

We further evaluate the robustness of the proposed framework under missing modality conditions, focusing on the extreme case of complete absence (100%) of one modality, as shown in Table 5. To assess this setting, we used our previous models^[63] as baselines and compared their performance under full missing modality scenarios.

For all baseline methods, the missing modality was approximated by replacing the absent input with the average computed across the dataset. In contrast, the proposed CMLR-ALSTM framework handles missing modalities at the latent level through cross-modal regeneration. Specifically, during inference, when one modality is unavailable, its latent representation (z_eeg or z_face) is replaced by its regenerated counterpart $\left({\hat{z}}_{\mathrm{eeg}}\text{or}{\hat{z}}_{\mathrm{face}}\right)$, obtained via cross-modal mapping from the available modality. This allows the model to operate effectively under incomplete multimodal inputs.

To further evaluate the contribution of the proposed CMLR mechanism, we included additional experiments in which the regeneration network was not used under the fully missing modality setup. In this setting, when a modality was completely missing, its latent representation was replaced by the average latent representation computed from the training data, as in the baseline methods. In addition, we investigated the effect of the latent regression objective by replacing the proposed L2 reconstruction loss with cosine similarity loss during the training of the cross-modal projection network. This experiment was conducted to assess whether preserving directional alignment alone is sufficient for latent regeneration, compared with directly minimizing the reconstruction error between projected and target latent representations. As shown in Table 5, the CMLR-ALSTM framework without cross-modal regeneration achieves lower performance under fully missing modality conditions than the complete CMLR-ALSTM framework. When the proposed regeneration network is employed, performance consistently improves across all missing-modality scenarios. This observation highlights the contribution of the CMLR mechanism, indicating that the ability to estimate the latent representation of a missing modality from the available modality plays a key role in maintaining robust emotion recognition performance under complete modality absence. Furthermore, the variant trained using cosine similarity loss also achieves lower performance than the proposed L2-based formulation. This result suggests that directly minimizing the reconstruction error between projected and target latent representations is more effective for CMLR than enforcing directional alignment alone.

Additionally, the results show that CMLR-ALSTM maintains substantially higher performance than the baseline model across all missing modality scenarios for both the DEAP and MAHNOB-HCI datasets. For example, under missing face conditions, the proposed model achieves over 84% accuracy in both valence and arousal on DEAP, compared to approximately 78% for DMMVAE-LSTM and significantly lower performance for the LSTM baseline. A similar trend is observed when the EEG is missing, where the proposed framework consistently outperforms the baselines by a notable margin.

To provide a fair assessment under missing-modality conditions, we also evaluated single-modality LSTM baselines using EEG-only and face-only inputs on the DEAP and MAHNOB-HCI datasets. As shown in the Table 6, EEG-only models consistently outperform face-only models across both datasets, indicating that the EEG modality contributes more strongly to the overall multimodal performance under incomplete modality conditions.

These results demonstrate the effectiveness of the proposed latent regeneration mechanism in preserving discriminative multimodal information even when one modality is completely absent. While joint representation models achieve competitive results when both modalities are available, their performance degrades considerably under full missing modality conditions due to their inability to recover missing information. In contrast, the proposed framework remains significantly more stable by leveraging CMLR, highlighting its practical suitability for real-world multimodal emotion recognition applications where sensor failure or data loss may occur.

5.5 Robustness analysis with data dropout

To evaluate the robustness of the proposed framework under partial data degradation, we conducted additional experiments designed to simulate real-world scenarios, including sensor noise, transmission errors, and temporary signal loss. In this setting, instead of removing an entire modality, we randomly drop a percentage of the input data from both modalities. Specifically, for a given missing rate (e.g., 10%), the dropout is applied equally across modalities (e.g., 5% to EEG and 5% to facial data), ensuring a balanced degradation of multimodal inputs. For this analysis, we used both the DEAP and SEED-IV datasets. These datasets provide synchronized multimodal signals along with temporal sequences, making them suitable for LSTM-based modelling. The temporal structure enables the model to capture sequential dependencies and mitigate the impact of missing or corrupted inputs through temporal aggregation.

The results for the DEAP dataset (Table 7) show that the proposed CMLR-ALSTM framework consistently outperforms DMMVAE-LSTM and the Transformer Autoencoder across all dropout rates. At 10% dropout, the proposed model achieves 93.55% and 93.92% accuracy for valence and arousal, respectively, demonstrating only a minor performance reduction compared to the full-data setting. As the dropout rate increases, degradation remains gradual, indicating that the model effectively utilizes the remaining informative features. Even at 70% dropout, CMLR-ALSTM maintains stable performance (78.37% and 78.94%), outperforming both baseline methods by a clear margin.

We observed a similar behaviour for the SEED-IV dataset (Table 8), where the proposed framework consistently achieves higher accuracy than competing approaches across all dropout levels. In particular, at higher dropout rates such as 50% and 70%, the performance difference between CMLR-ALSTM and the baseline methods becomes more noticeable, indicating the robustness of the proposed model under more challenging data conditions. This robustness stems from the model’s ability to learn complementary representations across modalities, enabling it to leverage the available information when parts of the input are missing. In addition, the LSTM component plays a critical role by integrating temporal information across sequences, smoothing inconsistencies caused by data dropout, and improving classification stability.

Overall, these findings demonstrate that the proposed framework is resilient to partial data corruption and maintains stable performance under varying levels of dropout. This makes it well-suited for real-world MER applications, where data quality and availability may fluctuate over time.

5.6 Robustness analysis with modality dropout

An additional experiment was conducted on the DEAP dataset to further examine the robustness of the proposed framework under conditions of partial data loss, reflecting practical challenges such as sensor noise, communication disturbances, or temporary signal degradation. In this setting, instead of fully removing one modality, a certain proportion of the input samples from each modality was randomly discarded during training. This experimental design enables evaluation of the model’s capability to operate under incomplete inputs without explicitly reconstructing the missing observations. As presented in Table 9, the CMLR-ALSTM framework maintains stable predictive performance even when a portion of the input data is randomly removed. In comparison with DMMVAE-LSTM, the proposed approach consistently achieves higher classification accuracy across all dropout levels. For example, when 10% of the input data is omitted, only a limited reduction in accuracy is observed, indicating that the model effectively utilizes the remaining informative features. With increasing dropout rates, performance declines gradually rather than suddenly, suggesting that the proposed framework is resilient to partial information loss. This behaviour can be attributed to the combined effect of cross-modal feature interaction and temporal modelling. The interaction between modalities enables the extraction of complementary information, while the LSTM component aggregates temporal context across sequences, reducing inconsistencies introduced by incomplete or noisy inputs. These results indicate that the proposed framework is robust not only in scenarios involving the complete absence of a modality but also when the data are partially corrupted. Maintaining reliable performance under different levels of data degradation highlights the potential applicability of the framework in real-world MER settings, where data quality and availability may fluctuate over time.

6. Conclusion

In this study, we proposed a novel MER framework that combines CMLR and attention-enhanced LSTM-based temporal modelling. The proposed CMLR-ALSTM framework learns latent representations of the modality-specific from EEG and visual input. The cross-modal projection networks regenerate missing latent features when one modality is unavailable. By combining latent-space regeneration with sequential modelling, the framework captures both cross-modal relationships and temporal dependencies inherent to MER.

The experimental results demonstrate that the proposed model achieves competitive, and in several cases superior, performance compared to state-of-the-art unimodal and multimodal approaches across different datasets and experimental settings. On the DEAP and MAHNOB-HCI datasets, the framework achieved strong performance in valence and arousal classification. In contrast, on the SEED-IV dataset, it outperformed previous baseline methods in multi-class emotion recognition using EEG and eye-movement signals. These results confirm the effectiveness of the proposed approach across different modality combinations and benchmark datasets.

Through two robustness-oriented experimental setups, we further validated the stability of the proposed framework. First, under complete missing modality scenarios, the model maintained substantially higher performance than baseline approaches by replacing the unavailable latent representation with its cross-modally regenerated counterpart. Second, under partial data dropout, the proposed approach maintained strong classification performance even at high dropout rates, demonstrating its resilience to gradual data corruption. These findings indicate that the combination of CMLR and temporal learning provides a practical advantage in settings where multimodal inputs may be incomplete or unreliable.

In addition, the findings emphasize the benefits of MER, illustrating how complementary modalities can mitigate limitations of individual data sources and enhance robustness and generalization. The proposed framework’s ability to sustain strong performance under both fully available and partially degraded input conditions indicates its suitability for practical applications, where data acquisition may be affected by noise, signal corruption, or limitations in sensing hardware.

Despite the promising results, this study has several limitations. The proposed framework was evaluated on publicly available benchmark datasets using a subject-dependent evaluation protocol. Although this protocol is widely adopted in MER research, evaluating the framework under subject-independent settings and on larger, more diverse datasets would provide a more comprehensive assessment of its generalization to unseen subjects and recording conditions. In addition, the current study focused on EEG, facial images, and eye-movement signals. While the proposed framework demonstrated strong performance across these modality combinations, its applicability to other physiological and behavioural modalities remains to be explored. Future work will include evaluating the proposed framework under subject-independent protocols, investigating more realistic missing-modality scenarios, and validating its performance on larger and more diverse multimodal datasets. In addition, future studies will explore the integration of additional physiological and behavioural modalities, such as ECG and speech signals, to further enhance multimodal representation learning.

Authors contribution

Mahaseni B: Methodology, investigation, formal analysis, writing-original draft, writing-review & editing.

Khan NM: Conceptualization, methodology, formal analysis, writing-original draft, writing-review & editing.

Conflicts of interest

The authors declare no conflicts of interest.

Ethical approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and materials

Data supporting the findings of this study are available from the corresponding author upon reasonable request.

Funding

This research was funded by Natural Sciences and Engineering Research Council of Canada (Grant Nos. RGPIN-2020-05471 and Alliance 577563-2022).

Copyright

© The Author(s) 2026.