GERMANYResearch
arXiv: TUM researcher maps how to build trustworthy AI graders for law exams and legal AI benchmarks
In an arXiv preprint posted October 14, 2026, Sebastian Nagl of the Technical University of Munich systematizes how open-text legal answers, such as exam case solutions and memos, can be graded by humans and large language models. He compares eight evaluation paradigms against eight desiderata and proposes a "trustworthy rubric judge" with human examiner oversight.
Key points
- The desiderata, drawn from educational measurement and examination law, include validity, reliability, reasons, robustness to gaming and human oversight.arXiv
- A rubric written for the specific task is the strongest lever for agreement between an LLM judge and human graders.arXiv
- Reported LLM-grader agreement figures rarely come with the human-to-human agreement needed to interpret them.arXiv
- The main validation gap involves defensible alternative solutions, follow-on errors and examiner correction of model judgments.arXiv
- The proposed safeguards include code-checked evidence, repeated runs, calibration against independent human grades, logging and an examiner reading every script.arXiv