This paper was accepted on the Studying from Evaluating the Evolving LLM Lifecycle workshop at NeurIPS 2025.
Security analysis of multimodal basis fashions typically treats imaginative and prescient and language inputs individually, lacking dangers from joint interpretation the place benign content material turns into dangerous together. Present approaches additionally fail to tell apart clearly unsafe content material from borderline instances, resulting in problematic over-blocking or under-refusal of genuinely dangerous content material. We current Imaginative and prescient Language Security Understanding (VLSU), a complete framework to systematically consider multimodal security by fine-grained severity classification and combinatorial evaluation throughout 17 distinct security patterns. Utilizing a multi-stage pipeline with real-world pictures and human annotation, we assemble a large-scale benchmark of 8,187 samples spanning 15 hurt classes. Our analysis of 11 state-of-the-art fashions reveals systematic joint understanding failures: whereas fashions obtain 90%-plus accuracy on clear unimodal security indicators, efficiency degrades considerably to 20-55% when joint image-text reasoning is required to find out the security label. Most critically, 34% of errors in joint image-text security classification happen regardless of right classification of the person modalities, additional demonstrating absent compositional reasoning capabilities. Moreover, we discover that fashions battle to stability refusing unsafe content material whereas nonetheless responding to borderline instances that deserve engagement. For instance, we discover that instruction framing can scale back the over-blocking price on borderline content material from 62.4% to 10.4% in Gemini-1.5, however solely at the price of under-refusing on unsafe content material with refusal price dropping from 90.8% to 53.9%. General, our framework exposes weaknesses in joint image-text understanding and alignment gaps in present fashions, and gives a crucial check mattress to allow the subsequent milestones in analysis on sturdy vision-language security.
