Aims: Multimodal emotion recognition (MER) can outperform unimodal approaches by integrating complementary information from multiple sources. However, real-world applications often involve incomplete or missing modalities, reducing the reliability of existing MER models. This study proposes a framework that remains robust under missing-modality conditions while preserving the advantages of multimodal integration.

Methods: To address this challenge, we propose a cross-modal latent regeneration and attention long short-term memory (CMLR-ALSTM) framework. The framework combines pretrained variational autoencoder encoders with residual projection networks, optimized with L2 loss, to achieve stable latent-space alignment across modalities. Regenerated and available latent representations are integrated using a cross-modal attention mechanism and processed by an LSTM to model temporal dependencies and improve multimodal fusion under incomplete data.

Results: The proposed framework was evaluated on three benchmark datasets under complete, partially missing, and completely missing modality scenarios. Experimental results demonstrate that CMLR-ALSTM achieves up to 17.22% improvement under missing-modality conditions while maintaining competitive performance under complete-modality settings. The results also show that the proposed latent regeneration strategy effectively preserves cross-modal relationships and robust latent representations.

Conclusion: The experimental results confirm the effectiveness of the proposed framework, particularly in realistic environments where data availability is inconsistent. By leveraging CMLR to reconstruct missing representations and modelling temporal dependencies through LSTM, the proposed approach provides a more robust and reliable MER framework for practical deployment. The results across different modality combinations demonstrate its ability to generalize across heterogeneous multimodal settings.

Aims: Mental illnesses affect millions worldwide. Despite greater public awareness, significant stigma remains. Anti-stigmatization interventions can help reduce prejudice by promoting education and meaningful contact with people with mental illness. Technology-based interventions may further support these efforts by simulating such contact through approaches such as storytelling. Social robots may offer additional advantages for storytelling-based interventions, especially through the integration of music and sound effects.

Methods: In multimodal robotic storytelling, non-speech sounds remain largely overlooked, despite their importance in related media. To address this gap, we compared a robotic storyteller narrating a story about a person experiencing a panic attack using only voice and bodily expression with versions integrating sound effects, background music, or both.

Results: While the addition of sounds did not affect recipients’ prejudice differently than storytelling without them, adding both sound effects and music increased transportation and improved associative empathy. In contrast, adding only music decreased associative empathy. Although mediation was not indicated, the results revealed transportation as a predictor for empathy and prejudice, while being only minimally influenced by sound integration itself.

Conclusion: The effects of sound integration in a robotic storytelling intervention on recipients’ prejudice were mixed, recommending either the combination of both sound types or complete omission. Transportation was indicated as a key lever for increasing empathy and decreasing prejudice that warrants further investigation. Future work is needed to gain deeper insights into robotic storytelling as an intervention tool for reducing prejudice, including work on transportation as a modifiable factor and the integration of pre-post-measurements.

Self++ is a conceptual design framework for human–Artificial Intelligence (AI) symbiosis in extended reality (XR) that preserves human authorship while still benefiting from increasingly capable AI agents. Because XR can shape both perceptual evidence and action, apparently ‘helpful’ assistance can drift into over-reliance, covert persuasion, and blurred responsibility. Self++ grounds interaction in two complementary theories: Self-Determination Theory (autonomy, competence, relatedness) and the Free Energy Principle (predictive stability under uncertainty). It operationalises these foundations through co-determination, treating the human and the AI as a coupled system that must keep intent and limits legible, tune support over time, and preserve the user’s right to endorse, contest, and override. These requirements are summarised as the co-determination principles (T.A.N.): Transparency, Adaptivity, and Negotiability. Self++ organises augmentation into three concurrently activatable overlays spanning sensorimotor competence support (Self: competence overlay), deliberative autonomy support (Self+: autonomy overlay), and social and long-horizon relatedness and purpose support (Self++: relatedness and purpose overlay). Across the overlays, it specifies nine role patterns (Tutor, Skill Builder, Coach; Choice Architect, Advisor, Agentic Worker; Contextual Interpreter, Social Facilitator, Purpose Amplifier) that can be implemented as interaction patterns, not personas. The contribution is a role-based conceptual map that generates testable design propositions for XR-AI systems that grow capability without replacing judgement, enabling symbiotic agency in work, learning, and social life and resilient human development.

Aims: The right to play is a basic human right. However, sport participation is often limited for children with complex motor disabilities. We developed a brain-computer interface (BCI)-enabled Boccia system that allows children with severe motor disabilities and communication difficulties to play independently. The purpose of this study was to partner with persons with lived experience (PWLE) to improve the BCI-Boccia system.

Methods: Following the Strategy for Patient-Oriented Research framework, we engaged seven PWLE. In the first session, we gathered comments from the PWLE, which were translated into a list of required features. The software was developed using an Agile approach. The second session involved a demonstration to collect additional feedback. In the third session, two PWLE tested the system in person. Engagement was evaluated using the Public and Patient Engagement Evaluation Tool (PPEET).

Results: Comments from the PWLE focused on improving the software controller and the mechanical stability of the ramp. New software controllers for coarse and fine movements were designed, and a new base was developed to enhance stability while allowing faster assembly and disassembly. The PPEET confirmed that PWLE felt their suggestions were considered and that sufficient resources were provided to support their participation.

Conclusion: We demonstrate that a patient engagement strategy can inform and facilitate improvements to a BCI-enabled Boccia system. Involving diverse PWLE throughout the design cycle may improve accessibility and user adoption in disability sports. Inclusive participation likely helps ensure that improvement efforts directly address the needs of end users.

Affective Computing aims to build systems that can sense, interpret, and influence human emotions. In parallel, virtual reality (VR) has matured into a powerful solution for immersion, presence, and interaction. When combined, these two technologies enable a new generation of emotionally adaptive systems that can recognize users’ inner states and react in real time. In this article, we explore the opportunities and challenges of emotion recognition and emotion elicitation in VR, focusing on the Virtual Emotion, Elicitation and Recognition Loop, a closed-loop framework where virtual environments adapt based on recognized and targeted emotions. We discuss applications in manufacturing, product design, and education, and highlight key ethical and privacy concerns, especially in light of recent regulations such as the European Commission released the proposed Regulation on Artificial Intelligence (EU AI Act). Finally, we outline open research directions toward responsible, privacy-aware, and scalable affective VR systems, suggesting that the convergence between VR and Affective Computing can become a crucial foundation for the next generation of human-centric interactive technologies.

This perspective reframes human–robot interaction (HRI) through extended reality (XR), arguing that virtual robots powered by large foundation models (FMs) can serve as cognitively grounded, empathic agents. Unlike physical robots, XR-native agents are unbound by hardware constraints and can be instantiated, adapted, and scaled on demand, while still affording embodiment and co-presence. We synthesize work across XR, HRI, and cognitive AI to show how such agents can support safety-critical scenarios, socially and cognitively empathic interaction across domains, and extend physical capabilities with XR and AI integration. We then discuss how multimodal large FMs (e.g., large language models, large vision models, and vision-language models) enable context-aware reasoning, affect-sensitive situations, and long-term adaptation, positioning virtual robots as cognitive and empathic mediators rather than mere simulation assets. At the same time, we highlight challenges and potential risks, including overtrust, cultural and representational bias, privacy concerns around biometric sensing, and data governance and transparency. The paper concludes by outlining a research agenda for human-centered, ethically grounded XR agents, emphasizing multi-layered evaluation frameworks, multi-user ecosystems, mixed virtual–physical embodiment, and societal and ethical design practices to envision XR-based virtual agents powered by FMs as reshaping future HRI into a more efficient and adaptive paradigm.

Aims: Users’ empathy towards artificial agents can be influenced by the agent’s expression of emotion. To date, most studies have used a Wizard of Oz design or manually programmed agents’ expressions. This study investigated whether autonomously animated emotional expression and neural voices could increase user empathy towards a Virtual Human.

Methods: 158 adults participated in an online experiment, where they watched videos of six emotional stories generated by ChatGPT. For each story, participants were randomly assigned to a virtual human (VH) called Carina, telling the story with either (1) autonomous expressive or non-expressive animation, and (2) a neural or standard text-to-speech (TTS) voice. After each story, participants rated how well the animation and voice matched the story, and their cognitive, affective, and subjective empathy towards Carina were evaluated. Qualitative data were collected on how well participants thought Carina expressed emotion.

Results: Autonomous emotional expression enhanced the alignment between the animation and voice, and improved subjective, cognitive, and affective empathy. The standard voice was rated as matching the fear and sad stories better, the sad animation was better, creating greater subjective and cognitive empathy for the sad story. Trait empathy and ratings of how well the animation and voice matched the story predicted subjective empathy. Qualitative analysis revealed that the animation conveyed emotions more effectively than the voice, and emotional expression was associated with increased empathy.

Conclusion: Autonomous emotional animation of VHs can improve empathy towards AI-generated stories. Further research is needed on voices that can dynamically change to express different emotions.

Aims: Sharing a robot’s intentions is crucial for building human confidence and ensuring safety in robot co-located environments. Communicating planned motion or internal state of a robot in a clear and timely manner is challenging, especially when users are occupied with other tasks. Augmented reality (AR) offers an effective medium for delivering visual cues that convey such information. This study evaluates a smartphone-based AR interface designed to communicate a robot’s navigation intentions and enhance users’ situational awareness (SA) in shared human–robot settings.

Methods: We developed a mobile AR application using Unity3D and Google ARCore to display goal locations, planned trajectories, and the real-time motion of a Robot Operating System enabled mobile robot. The system provides three visualization modes: Goal Visible, Path Visible, and Robot Visible, with spatial alignment achieved via a reference image marker. Fifty-eight participants, with varying levels of experience in robotics and AR, took part in an online user study. SA was assessed using a query-based evaluation adapted from the Situational Awareness Global Assessment Technique (SAGAT), examining perception, comprehension, and projection levels. A post-assessment survey captured user opinions on usability and perceived benefits.

Results: Participants achieved an average SAGAT score of 86.5%, indicating improved awareness of the robot mission, spatial positioning, and safe zones. AR visualization was particularly effective for identifying obstacles and predicting unobstructed areas. In the post-assessment survey, over 96.6% of participants agreed that the interface enhanced their confidence and understanding of robot motion intentions.

Conclusion: A mobile AR interface can significantly enhance SA in shared human–robot environments by making robot intentions more transparent and comprehensible. Future work will include in-situ evaluations with physical robots, integration of richer robot-state information such as velocity and sensor data, and the exploration of additional visualization strategies that further strengthen safety, predictability, and trust in human–robot collaborative environments.

Aims: This study aims to investigate the relationship between eye movement behaviors and social competence, particularly focusing on listening patterns in multi-person conversations. The goal is to identify objective, quantifiable behavioral indicators of social competence to inform the development of assessment tools and training interventions.

Methods: A three-person setting was designed with two conversational partners and one primary listener, and immersive eye-tracking data were collected using Microsoft HoloLens 2 during conversations on naturalistic topics. Social competence was rated by a clinical psychiatrist using standardized behavioral criteria, while analysis targeted three pre-selected indicators: nodding frequency, selective attention allocation to speaker’s regions of interest (head versus shoulders), and vertical gaze stability during listening.

Results: The results revealed that nodding frequency showed a strong positive correlation with social competence scores, indicating its potential as a robust nonverbal biomarker. Participants with higher social competence demonstrated greater attention to the speaker’s head region while minimizing focus on less informative areas such as the shoulders. Furthermore, individuals with higher scores exhibited significantly lower vertical gaze dispersion, reflecting more focused and stable attentional control during social listening.

Conclusion: This study establishes reliable eye movement-based indicators of social competence, highlighting their potential for assessing and enhancing social skills in real-world interactions. By integrating multimodal behavioral analysis, the findings provide a theoretical and technical foundation for developing personalized, real-time feedback systems for social competence training.

Aims: This study introduces and evaluates a virtual reality (VR) prototype designed for the Loving-Kindness Meditation (LKM) to support mental health rehabilitation and relaxation in clinical contexts. The aims include the co-creation of a VR-based mindfulness experience with clinical experts and the evaluation of its usability, user experience, and short-term effects on relaxation, affect, and self-compassion.

Methods: Following a design thinking and co-creation approach, the VR-based LKM experience was developed iteratively with input from clinicians and computer scientists. The final prototype was implemented for the Meta Quest 3 and included five immersive scenes representing phases of the LKM and transition scenes guided by a professionally narrated audio track. Eleven participants (M = 36.5 years, SD = 14.6) experienced the 12-minute session. Pre- and post-session measures included relaxation, positive and negative affect schedule, and self-compassion, complemented in the end by the Igroup Presence Questionnaire, usability measures and a semi-structured qualitative interview.

Results: Participants reported significant decreases in negative affect (t(10) = -2.512, p = .0307, d = -1.037) and stress (t(10) = -3.318, p = .007, d = -1.328), as well as increases in relaxation (t(10) = 5.487, p < .0001, d = 2.471) and self-compassion (t(10) = 2.231, p = .0497, d = 0.283). Usability was rated as excellent (M = 92.5), and presence as good (M = 4.0, SD = 0.43). Qualitative feedback described the experience as calming, aesthetically pleasing, and easy to engage with, highlighting the falling leaves and pulsating orb as effective design elements.

Conclusion: The co-designed VR-LKM prototype was perceived as highly usable and beneficial for inducing relaxation and self-compassion, suggesting its potential as a supportive tool for clinical mindfulness interventions. The results indicate that immersive VR can effectively facilitate engagement and emotional regulation, providing a foundation for future clinical trials and broader implementation in therapeutic and wellness settings.

Aims: We present an early investigation of how people interact with human-like Artificial Intelligence (AI) agents in virtual reality when discussing ideologically sensitive topics. Specifically, it examines how users respond to AI agents that express either congruent or incongruent opinions on a controversial issue, and how alignment, modality, and agent behaviors shape perceived conversation quality, psychological comfort, and agent credibility.

Methods: We conducted a 2 (agent opinion: congruent vs. incongruent) × 2 (input modality: text vs. voice) between-subjects experiment with 36 participants who engaged in five-minute virtual reality (VR)-based conversations with a GPT-4-powered AI agent about U.S. gun laws. Participants completed pre- and post-study measures of opinion and emotional states, evaluated the agent, and reflected on the interaction. In addition, dialogue transcripts were analyzed using the Issue-based Information System (IBIS) framework to characterize argument structure and engagement patterns.

Results: Participants engaged willingly with the AI agent regardless of its stance, and qualitative responses suggest that the interactions were generally respectful and characterized by low emotional intensity. Quantitative results show that opinion alignment influenced perceived bias and conversational impact, but did not affect the agent’s competence or likability. While voice input yielded richer dialogue, it also heightened perceived bias. Qualitative findings further highlight participants’ sensitivity to the agent’s ideological stance and their preference for AI agents whose views aligned with their own.

Conclusion: Our study suggests that AI agents embodied in VR can support ideologically challenging conversations without inducing defensiveness or discomfort when designed for neutrality and emotional safety. These findings point to early design directions for conversational agents that scaffold reflection and perspective-taking in politically or ethically sensitive domains.

Aims: This study examined the speed of approach-avoidance actions in virtual reality (VR) as an indicator of psychological “readiness” to interact with social avatars.

Aims: This study examined the speed of approach-avoidance actions in virtual reality (VR) as an indicator of psychological “readiness” to interact with social avatars.

Methods: Given that fast response is a key psychological factor reflecting a user’s interest, motivation, and willingness to engage, we analyzed the response time of pulling or pushing inputs, typical actions showing approach-avoidance tendency, via bare-hand interaction in VR. We specifically investigated how the response time varied according to participants’ social resources, particularly the richness of their social lives characterized by broader networks of friends, social groups, and frequent interactions.

Results: Results showed that participants with richer social lives exhibited faster pulling (vs. pushing) actions toward both same- and opposite-sex avatars. These effects remained significant regardless of participants’ gender, age, and prior VR experience. Notably, the observed effects were specific to social stimuli (i.e., avatars) and were not revealed with non-social stimuli (i.e., a flag). Additionally, the effects did not occur with other indirect interactions (i.e., a mouse wheel or a virtual joystick).

Conclusion: The findings suggest that social resources may facilitate approach-oriented bodily affordances in VR environments.

Aims: This scoping review systematically maps the existing literature at the intersection of virtual reality (VR), synthetic speech, and affective computing. As immersive and voice-based technologies gain traction in education, mental health, and entertainment, it is critical to understand how synthetic speech shapes emotional experiences in VR environments. The review clarifies how these concepts are defined, how they contribute to empathic computing, and identifies common applications, methodological approaches, research gaps, and ethical considerations.

Methods: A comprehensive search across multiple databases (e.g., ACM Digital Library, IEEE Xplore, ScienceDirect) was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) framework. Eligible studies investigated synthetic or computer-generated speech in VR or comparable immersive 3D settings and assessed emotional responses or related outcomes. Data were extracted on study characteristics, applied technologies, affect-related measures, and reported effects.

Results: The findings reveal a growing interdisciplinary body of research at the convergence of synthetic speech technologies, embodied virtual agents, and affective data processing in immersive environments. Interest in this area has accelerated with the development of advanced text-to-speech models, suggesting increasing relevance and expansion of this topic.

Conclusion: This review underscores a rapidly expanding yet fragmented research landscape. It highlights conceptual and methodological gaps, stressing the need for clearer definitions, standardized evaluation measures, and ethically informed design of synthetic speech in VR. The results provide a foundation for advancing research and applications in emotionally responsive virtual environments.

Aims: Storytelling has evolved alongside human culture, giving rise to new media such as social robots. While these robots employ modalities similar to those used by humans, they can also utilize non-biomimetic modalities, such as color, which are commonly associated with emotions. As research on the use of colored light in robotic storytelling remains limited, this study investigates its integration through three empirical studies.

Methods: We conducted three studies to explore the impact of colored light in robotic storytelling. The first study examined the effect of emotion-inducing colored lighting in romantic storytelling. The second study employed an online survey to determine appropriate light colors for specific emotions, based on images of the robot’s emotional expressions. The third study compared four lighting conditions in storytelling: emotion-driven colored lights, context-based colored lights, constant white light, and no additional lighting.

Results: The first study found that while colored lighting did not significantly influence storytelling experience or perception of the robot, it made recipients felt more serene. The second study showed improved recognition of amazement, rage, and neutral emotional states when colored light accompanied body language. The third study revealed no significant differences across lighting conditions in terms of storytelling experience, emotions, or robot perception; however, participants generally appreciated the use of colored lights. Emotion-driven lighting received slightly more favorable subjective evaluations.

Conclusion: Colored lighting can enhance the emotional expressiveness of robots. Both emotion- driven and context-based lighting strategies are appropriate for robotic storytelling. Through this series of studies, we contribute to the understanding of how colored lights are perceived in robotic communication, particularly within storytelling contexts.

Aims: This study aims to enhance the design of augmented reality (AR) technologies for remote collaboration by examining the complex relationships among individual factors (user characteristics), psychological and physiological states during AR-mediated remote collaboration, and outcomes within an Input-Mediator-Output (IMO) model. The goal is to evaluate how individual characteristics influence psychological and physiological experiences, as well as task performance, in AR-mediated collaboration.

Methods: We hypothesize and evaluate an IMO model and use correlation analyses to examine the relationships among person-related input variables (e.g., predispositions, traits, attitudes, states, and contextual factors), psychological and physiological emergent states, and performance-related output variables.

Results: Our results demonstrate that individual characteristics significantly influence subjective experiences, physiological responses, and task performance, emphasizing the critical role of individual differences, alongside task- and technology-related factors, in shaping collaboration experiences and performance. These findings highlight the importance of considering individual characteristics in the design of AR tools to optimize user well-being and performance outcomes.

Conclusion: Our study provides a foundational framework for understanding the interplay between individuals, tasks, and technology, underscoring the need for AR tools that align with user characteristics. It also lays the groundwork for future IMO research in AR-mediated remote collaboration, contributing to the development of more effective and health-promoting AR technologies.

This paper explores two virtual child simulators, BabyX and the VR Baby Training Tool, which provide immersive, interactive platforms for child-focused research and training. These technologies address key ethical and practical constraints, enabling the systematic study of caregiver-infant interactions and the development of professional relational skills with young children. We examine their design, features, and applications, as well as their future trajectories, highlighting their potential to advance research and improve training methodologies.

Aims: To give a comprehensive understanding of current research on immersive empathic computing, this paper aims to present a systematic review of the use of Virtual Reality (VR), Mixed Reality (MR), and Augmented Reality (AR) technologies in empathic computing, to identify key research trends, gaps, and future directions.

Methods: The PRISMA methodology was applied using keyword-based searches, publishing venue selection, and citation thresholds to identify 77 papers for detailed review. We analyze these papers to categorize the key areas of empathic computing research, including emotion elicitation, emotion recognition, fostering empathy, and cross-disciplinary applications such as healthcare, learning, entertainment and collaboration.

Results: Our findings reveal that VR has been the dominant platform for empathic computing research over the past two decades, while AR and MR remain underexplored. Dimensional emotional models have influenced this domain more than discrete emotional models for eliciting, recognizing emotions and fostering empathy. Additionally, we identify perception and cognition as pivotal factors influencing user engagement and emotional regulation.

Conclusion: Future research should expand the exploration of AR and MR for empathic computing, refine emotion models by integrating hybrid frameworks, and examine the relationship between lower body postures and emotions in immersive environments as an emerging research opportunity.

Aims: Recent world events have resulted in companies using remote meeting tools more often in design processes. The shift to remote meeting tools has had a notable impact on collaborative design activities, such as design reviews (DRs). When DRs depend on computer-aided design (CAD) software, the lack of direct support for CAD functionalities in videoconferencing applications introduces novel communication challenges, i.e. friction. This study investigates friction encountered in real world remote DRs when using a combination of standard CAD and videoconferencing applications. The objective was to understand the main sources of friction when carrying out DRs using a combination of CAD and videoconferencing applications.

Methods: At a single Swedish automobile manufacturer, 15 DRs of a fixture component were passively observed. These observations were subjected to a qualitative thematic analysis to identify categories and sources of friction during these DRs. The DRs were carried out using a combination of CATIA CAD software and Microsoft Teams for videoconferencing.

Results: The analysis of the 15 remote DRs identified four recurring friction categories: requesting specific viewpoints, indicating specific elements, expressing design change ideas, and evaluating ergonomics. Each category highlights specific challenges that were observed during the DRs and emerged due to constraints imposed by existing methods and technologies for remote meetings.

Conclusion: This study provides a framework for understanding the current sources of friction in remote DRs using videoconferencing tools. These insights can support the future development of DR software tools, guiding the integration of features that address these friction points. Additionally, the results serve as a guideline for organizations to implement methods that reduce friction in remote DRs and improve DR quality and efficacy.