Integrated AI framework Sa2VA: Achieving in-depth understanding of images and videos

Driven by multimodal large language models (MLLMs), revolutionary progress has been made in image and video-related tasks, including visual question answering, narrative generation, and interactive editing. However, achieving fine-grained video content understanding still faces significant challenges. This challenge involves tasks such as pixel-level segmentation, tracking with language description, and visual question answering on specific video cues.

Although current state-of-the-art video perception models perform well on segmentation and tracking tasks, they still fall short in open language understanding and conversational capabilities. In addition, video MLLMs perform well on video understanding and question answering tasks, but still fall short in handling perceptual tasks and visual cues.

There are two main existing solutions: multimodal large language models (MLLMs) and reference segmentation systems. MLLMs initially focused on improving multi-modal fusion methods and feature extractors, and gradually developed into frameworks for instruction tuning on LLMs, such as LLaVA. Recently, researchers have tried to unify image, video and multi-image analysis into a single framework, such as LLaVA-OneVision. At the same time, the citation segmentation system has also undergone changes from basic fusion modules to integrated segmentation and tracking. However, these solutions still fall short in the comprehensive integration of perception and language understanding abilities.

Researchers from UC Merced, Bytedance Seed Team, Wuhan University, and Peking University propose Sa2VA, a groundbreaking unified model designed to enable dense basic understanding of images and videos. The model overcomes the limitations of existing multi-modal large language models by minimizing one-time instruction tuning and supporting a wide range of image and video tasks.

Sa2VA innovatively integrates SAM-2 with LLaVA, unifying text, images and videos into a shared LLM token space. In addition, the researchers launched an extensive automatically annotated dataset called Ref-SAV, which contains more than 72K object expressions in complex video scenes, as well as 2K human-verified video objects to ensure robust baseline capabilities.

The architecture of Sa2VA mainly consists of two parts: an LLaVA-like model and SAM-2, which adopts a novel decoupling design. LLaVA-like components include a visual encoder for processing images and videos, a visual projection layer, and an LLM for text token prediction. The system uses a unique decoupling approach that allows SAM-2 to operate alongside a pretrained LLaVA model without direct token exchange, thus maintaining computational efficiency and allowing pluggable functionality with a variety of pretrained MLLMs. connect.

Research results show that Sa2VA achieves state-of-the-art results in the reference segmentation task. Its Sa2VA-8B model achieves cIoU scores of 81.6, 76.2 and 78.9 on RefCOCO, RefCOCO+ and RefCOCOg respectively, surpassing previous systems such as GLaMM-7B. . In terms of dialogue capabilities, Sa2VA achieved excellent results of 2128, 81.6 and 75.1 on MME, MMbench and SEED-Bench respectively.

In addition, Sa2VA also significantly outperforms the previous state-of-the-art VISA-13B on video benchmarks, demonstrating its efficiency and effectiveness in image and video understanding tasks.

Paper: https://arxiv.org/abs/2501.04001

Model: https://huggingface.co/collections/ByteDance/sa2va-model-zoo-677e3084d71b5f108d00e093