CHINAResearch
arXiv: LexPilot finds LLMs struggle when laypeople leave key facts out of civil-case questions
A new arXiv benchmark, LexPilot, tests whether legal AI models ask the right follow-up questions when non-lawyers describe disputes incompletely. Built from 432 real Chinese civil judgments across 36 causes of action, it pairs complete and underspecified consultations with the courts' actual verdicts. Across eight leading models, verdict prediction from underspecified consultations trailed the fully specified setting by 23.84% on average, even when models could ask clarifying questions.
Key points
- The benchmark holds 864 consultations, one complete and one underspecified version for each of the 432 first-instance judgments.arXiv
- In a pilot with 16 undergraduates, those without legal training conveyed 68.1% of expert-identified key details, versus 86.6% for law majors.arXiv
- The best model at eliciting missing facts, Claude-Opus-5, still asked no valid clarifying question in 25.83% of consultations.arXiv
- The authors warn that models often fall back on generic advice that ignores the missing facts, risking misleading laypeople.arXiv
- Two legal experts with master's degrees in law reviewed the data, and personal details were masked or pseudonymized.arXiv