U.S. FEDResearch
arXiv: LePhilBench tests whether AI models grasp the legal philosophy behind Supreme Court opinions
In an arXiv preprint posted October 14, 2026, researchers led by Shuyuan Zheng of the University of Osaka and Tao Huang of City University of Hong Kong introduce LePhilBench, a benchmark measuring whether large language models understand judges' legal philosophy and apply it in legal reasoning. Its test set covers 220 U.S. Supreme Court opinions annotated by law-trained reviewers.
Key points
- A law professor designed 14 annotation questions on legal philosophy for each opinion.arXiv
- Three annotators with formal jurisprudence training answered every question, producing 9,240 labels.arXiv
- One task tests understanding of legal philosophy; a second tests applying it in legal reasoning.arXiv
- Adding a judge's jurisprudential profile raised accuracy on dissents more than it lowered accuracy on court opinions for every model tested.arXiv
- Models tested included GPT-5.6, Claude and Qwen variants.arXiv