Dae Hyeon Kim RESEARCH 한국어

Multimodal emotion recognition

Built an older-adult speech dataset and evaluated graph-based fusion of speech and text for emotion recognition.

Follow-up research: emotionheart
Role
Technical work-package lead
Contribution
Designed IRB and emotion-elicitation protocols. Co-developed multimodal models and supported technology transfer.
Period & affiliation
2023.04 ~ 2025.12 IITP

01 / RESEARCH DETAILOlder-adult speech: acquisition and transcription effects

Challenge
Older-adult speech and transcription errors can alter text-based emotion classification.
Approach
Compared speech, original text and transcribed text using the same subject-independent split of fixed-sentence utterances.
Contribution
Designed IRB and emotion-elicitation protocols, co-developed models and supported data construction and technology transfer.
Model architecture from the KOSOMBE 2024 presentation
Model architecture from the KOSOMBE 2024 presentation
View full-size figure
Weighted F1 before and after transcription (%) Original text: Text only 92.33, Audio + text 92.33; Transcribed text: Text only 81.64, Audio + text 81.38 0 25 50 75 100 92.33 81.64 Text only 92.33 81.38 Audio + text Original text Transcribed text
Weighted F1 before and after transcription (%)
Older-adult speech acquisition and evaluation
Acquisition / evaluation Experimental conditions
Older-adult experiment 24 participants, ages 66–81
Utterances 7 emotions, 1,512 utterances
Evaluation / outputs Subject-independent 4-fold / Best Poster Award
Evaluation conditions and scopeThe experiment involved 24 participants aged 66 to 81, seven emotions and 1,512 utterances. Transcription reduced weighted F1; adding speech did not recover this loss under the tested condition.

02 / RESEARCH DETAILEmotionHeart: dialogue graphs and missing modalities

Challenge
The model must represent speaker relationships and modality differences while retaining context when an input is missing.
Approach
Encoded within-speaker, between-speaker and cross-modal relations, then aligned modalities with an augmentation-free contrastive objective.
Contribution
Designed the heterogeneous graph and alignment loss and evaluated missing modalities. First author, ICASSP 2026.
Heterogeneous Graph Transformer and cross-modal alignment
Heterogeneous Graph Transformer and cross-modal alignment
View full-size figure
Embeddings before training, without GCL and with EmotionHeart
Embeddings before training, without GCL and with EmotionHeart
View full-size figure
Weighted F1 (%) with full and missing inputs
Evaluation setting Baseline Proposed model
IEMOCAP, all inputs 68.99 73.13
MELD, all inputs 66.75 68.99
IEMOCAP, audio only 54.09 60.26
Evaluation conditions and scopeWeighted F1, averaged over 10 random splits. The full-input baseline is Transformer; the audio-only baseline is GRACE. IEMOCAP: 73.13 with all inputs and 60.26 with audio only.