AI Just Got Better at Watching Videos

Table of Links

• Abstract and 1 Introduction
• 2. Related Works
• 3. PG-Video-LLaVA
• 3.1. Overview
• 3.2. Architecture
• 4. Experiments
• 4.1. Implementation Details
• 4.2. Stronger Baseline
• 4.3. Spatial Grounding in Videos
• 4.4. Zero-Shot Visual Question Answering
• 5. Conclusion and References

Supplementary Material

• A. Audio Modality Integration
• B. Visual Grounding: Quantitative Evaluation
• C. Qualitative Results for Visual Grounding
• D. Quantitative Evaluations of Video-based Conversation Performance

4.2. Stronger Baseline

This section provides an overview of the quantitative evaluations conducted to determine the effects of the strengthened baseline on PG-Video-LLaVA. To evaluate the impact of the enhanced baseline on PG-Video-LLaVA, we apply the benchmarking framework from Video-ChatGPT[22]. This framework measures performance on several axes critical for video-based conversational agents, including correctness of information, detail orientation, contextual understanding, temporal understanding, and consistency.

In order to facilitate a reliable and reproducible evaluation, we have updated our assessment pipeline by replacing GPT-3.5-Turbo with Vicuna-13b-v1.5. This adjustment addresses the limitations in reproducibility inherent to the closed-source nature of GPT-3.5-Turbo. Subsequently, we have re-assessed both PG-Video-LLaVA and other recent models to ensure a fair and consistent comparison. The results shown in Table 1 demonstrate that PG-Video-LLaVA outperforms the foundational Video-ChatGPT model and exhibits superior performance when compared to other recent contributions in the domain.

Following the quantitative assessment, the qualitative results in Figure 3 indicate the enhanced baseline’s impact on PG-Video-LLaVA’s performance. The PG-Video-LLaVA (13B) model exhibits improved accuracy in the information presented, a deeper level of descriptive detail, and a stronger alignment with the context and temporal progression of the videos. This advancement is particularly noticeable in the precise depiction of the child’s engagement with their surroundings and the giraffe’s behaviour, indicating a refined interpretation of both the activities and their settings. These qualitative insights are consistent with the quantitative results, highlighting the augmented baseline’s role in advancing PG-Video-LLaVA’s capacity in video understanding.