TL;DR
A California prosecutors’ office has admitted using AI in at least one court filing that contained “hallucinated” legal citations. The case has prompted a petition to the California Supreme Court and raised serious questions about AI reliability in criminal justice proceedings.
The Admission
Nevada County District Attorney Jesse Wilson confirmed to the Sacramento Bee that a prosecutor “recently used artificial intelligence in preparing a filing, which resulted in an inaccurate citation.” The filing was immediately withdrawn once the error was discovered.
Defence and civil rights attorneys argue the prosecutors’ office used AI in multiple criminal court filings beyond the one acknowledged. Lawyers for defendant Kyle Kjoller have filed a petition with the California Supreme Court identifying three cases they say contain errors typical of generative AI, including nonexistent quotations and misinterpreted court rulings.
Due Process Concerns
The defence team, which includes the non-profit Civil Rights Corps, has argued that “prosecutors’ reliance on inaccurate legal authority can violate ethical rules, and represents an existential threat to the due process rights of criminal defendants and the legitimacy of the courts.”
Twenty-two scholars, lawyers, and criminal justice advocates have filed a brief supporting Kjoller’s case. The California Supreme Court has not yet decided whether to take up the matter.
Response and Policy Changes
District Attorney Wilson attributed other mistakes to human error rather than AI, noting that prosecutors work “diligently and in good faith under heavy caseloads and time constraints.” He added: “It cannot now be assumed that every citation error stems from the use of artificial intelligence.”
Following the incident, the office has conducted new training and implemented an AI policy. Staff have been reminded to “verify all legal citations independently and not rely on AI-generated material without confirmation from reliable sources.”
Looking Forward
This case appears to be the first instance of a US prosecutors’ office using generative AI in a court filing. Whilst lawyers across multiple countries have been fined for AI errors, prosecution offices have largely avoided such scrutiny until now. The outcome of the California Supreme Court petition could establish important precedents for AI use in criminal proceedings—a development UK legal professionals and regulators will be watching closely.
Source: The Guardian