An Agentic Approach with Evidence-Grounded Verification for Multimodal Emotion Reasoning in Conversation
Paper · https://arxiv.org/abs/2610.31685
arXiv:2610.31685v1 Announce Type: new Abstract: LLM-based multimodal emotion reasoning systems perform well on standard benchmarks, but improving them through instruction tuning can be computationally expensive. Agentic pipelines built from frozen models avoid this cost, but introduce a different challenge: when a prediction is wrong, the system must identify which agent or stage caused the error. We address this problem with an agentic framework for traceable multimodal emotion reasoning and prompt-space adaptation. Modality-specific perception agents analyze visual, acoustic, and lexical inputs and post evidence-cited observations to a shared board. An integration agent combines these observations with retrieved multimodal examples to produce an emotion description, label set, and supporting citations, while an MLLM critic can request one revision before the output is finalized. For incorrect predictions, a separate evidence-grounded verification stage applies deterministic checks to verify that cited evidence and references are valid, and an entailment check to determine whether the final claims are supported by that evidence. Detected failures are then grouped by the responsible component and failure type, converted into targeted prompt patches, and accepted only after validation on a held-out set. This adapts the agent prompts without updating model weights. On the MER2025 dataset, targeted prompt adaptation outperforms the frozen static pipeline on all four MLLMs, ranging from 2B to 38B parameters, and matches or outperforms undifferentiated reflection baselines on each of them: VideoLLaMA3-2B, Phi-3.5-vision-instruct, Qwen2.5-VL-7B-Instruct, and InternVL2.5-38B. It improves F-score by 0.6--2.2 points depending on the backbone, including an increase from 52.3 to 54.5 on Qwen2.5-VL-7B.
First seen by Oriel on .
AI summary
The paper presents a traceable multimodal emotion-reasoning pipeline that improves frozen MLLMs through evidence-grounded error analysis and targeted prompt adaptation, without updating model weights.
- Visual, acoustic, and lexical agents post evidence-cited observations to a shared board; an integration agent combines them with retrieved examples to produce emotion descriptions, labels, and citations.
- An MLLM critic can request one revision, while deterministic checks verify citation validity and whether final claims are supported by the cited evidence.
- Failures are attributed to components and types, converted into targeted prompt patches, and accepted only after held-out-set validation.
- On MER2025, adaptation improved F-score by 0.6–2.2 points across four MLLMs from 2B to 38B parameters and matched or outperformed undifferentiated reflection baselines.
- Qwen2.5-VL-7B-Instruct improved from 52.3 to 54.5 F-score.
#multimodal ai #emotion recognition #llm agents #prompt adaptation
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2610.31685v1 | https://arxiv.org/abs/2610.31685 Read discussion |