U.S. FEDResearch
arXiv study: legal LLMs name authorities far more often than verdicts depend on them
An October 8, 2026 arXiv paper by Sadhu, Arora and Seth finds that when seven open-weight models must name the governing legal authority in chain-of-thought, they name the correct one in 66.7% to 100% of generations, but verdicts change far less often when that authority is swapped, across ECHR, SCOTUS, CaseHOLD and ContractNLI benchmarks. In a red-team test of five models, compliance with hidden adversarial instructions exceeded swap sensitivity by a wide margin.
Key points
- Authors test seven open-weight models from 8B to 70B with an authority-swap counterfactual and hidden-state verdict decoding.arXiv
- Models name the correct authority in 66.7% to 100% of generations when required to justify a verdict by naming it.arXiv
- Verdict-swap sensitivity is 0.0% to 21.7% on CaseHOLD, 30.0% to 76.7% on ECHR and SCOTUS, and 43.3% to 50.0% on ContractNLI.arXiv
- Neither scale nor a purpose-built legal-reasoning model, reproduced via LoRA, closes the naming-versus-dependence gap.arXiv
- Hidden adversarial instructions in case facts drew 73.3% to 96.4% compliance across five red-teamed models.arXiv