Reasoning has develop into a central paradigm for giant language fashions (LLMs), constantly boosting accuracy throughout numerous benchmarks. But its suitability for precision-sensitive duties stays unclear. We current the primary systematic research of reasoning for classification duties below strict low false optimistic price (FPR) regimes. Our evaluation covers two duties—security detection and hallucination detection—evaluated in each fine-tuned and zero-shot settings, utilizing commonplace LLMs and Giant Reasoning Fashions (LRMs). Our outcomes reveal a transparent trade-off: Assume On (reasoning-augmented) era improves general accuracy however underperforms on the low-FPR thresholds important for sensible use. In distinction, Assume Off (no reasoning throughout inference) dominates in these precision-sensitive regimes, with Assume On surpassing solely when larger FPRs are acceptable. As well as, we discover token-based scoring considerably outperforms self-verbalized confidence for precision-sensitive deployments. Lastly, a easy ensemble of the 2 modes recovers the strengths of every. Taken collectively, our findings place reasoning as a double-edged device: useful for common accuracy, however typically ill-suited for purposes requiring strict precision.
- ‡ Equal contribution
- † College of Maryland, Faculty Park
- ** Work completed whereas at Apple
