Open Internet by MindsNet

Evaluating Video VLMs: The Wrong Unit of Comparison

The current method of comparing video VLMs by model swap is misleading. The output depends on the whole setup, including segmentation, frame sampling, resolution, prompt, reasoning budget, and post-processing. Changing these factors has a greater impact on results than changing the model.

Computing & Technology, Computer Science, Artificial Intelligence