Oriel — AI & technology worth noticing, every day.

Multimodal Cross-Condition Learning for Temporal Sentence Grounding in Videos

Paper · https://arxiv.org/abs/2611.00033

arXiv:2611.00033v1 Announce Type: new Abstract: Temporal sentence grounding requires a video representation that preserves query-relevant evidence and a sentence representation that reflects the visual context in which it is interpreted. Prior end-to-end grounding addresses the first requirement through sentence-conditioned visual adaptation, but leaves the language representation independent of the video. This work proposes Multimodal Cross-Condition Learning (MCCL). Its Video Prefix Adapter (VPA) converts pooled visual context into a sequence of continuous tokens that condition a frozen DistilBERT encoder. Combined with the Sentence Conditioned Adapter (SCADA), VPA supports an ordered exchange from vision to language and back to vision. A comprehensive study examines two video backbones, LSTM and Mamba temporal modules, and four conditioning modes on two datasets. Bidirectional conditioning exceeds the stronger one-way mode by 1.6--2.3 mIoU points in all four Charades-STA settings. ActivityNet Captions exhibits a different pattern: bidirectionality helps I3D--Mamba, whereas video-to-text conditioning is stronger with C3D. The study extends one-way visual adaptation into a configurable multimodal framework and reports descriptive gains over the corresponding baselines.

First seen by Oriel on .

No saved summary available.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.00033v1https://arxiv.org/abs/2611.00033 Read discussion