EUResearch
SSRN: Study finds stale EU AI Act corpora return right article numbers but wrong current wording
In an October 12, 2026 SSRN preprint, Aghasalim Mustafazada and Murad Eyvazov measure how legislative amendments break legal question-answering systems, showing that hybrid retrieval over a pre-amendment EU AI Act corpus scored 63.6% correct article numbers but 0.0% current wording on questions answerable only after Regulation (EU) 2026/1744, versus 81.8% and 63.6% on the consolidated July 27, 2026 text.
Key points
- Mustafazada and Eyvazov posted the preprint on SSRN on October 12, 2026.SSRN
- They versioned the EU AI Act using the July 2024 Official Journal text and the July 2026 consolidated text.SSRN
- On 11 amendment-only questions, stale-corpus retrieval hit the right article 63.6% but current wording 0.0%.SSRN
- On the current corpus those figures rose to 81.8% article hits and 63.6% current-wording hits.SSRN
- Paragraph-level chunking raised full recall on 33 original questions from 69.7% to 75.8%.SSRN
- They released code and results; generation was not rerun on the amended text.SSRN