California State Bar used AI to develop bar exam questions
California's State Bar admitted that 23 of 171 scored multiple-choice questions on its February 2025 bar exam were developed with AI assistance by ACS Ventures, its independent psychometrician. The disclosure sparked criticism that non-lawyers used AI to draft questions and that the same company assessed the validity of its own questions, creating a conflict of interest.
What happened
The State Bar of California disclosed on April 23, 2025 that 23 of 171 scored multiple-choice questions on its February 2025 bar exam were developed with artificial intelligence by ACS Ventures, the State Bar's independent psychometrician. The remaining questions came from Kaplan (100 questions) and a first-year law student exam (48 questions).
The State Bar said it will ask the California Supreme Court to adjust test scores for February exam takers. The disclosure came nearly two months after prospective lawyers complained of technical problems and irregularities on the exam.
Reported impact
Law professors and experts criticized the admission as a conflict of interest because non-lawyers used AI to draft questions and the same company was tasked with assessing the validity and reliability of the questions it authored. Mary Basick, assistant dean at UC Irvine Law School, said having non-lawyers use AI to draft questions was "unbelievable," and test-takers had complained after the exam that some questions felt as if they were written by AI.
Katie Moran, associate professor at the University of San Francisco School of Law, called the disclosure "a staggering admission" and said the State Bar employed a company to have non-lawyers use AI to draft questions, then paid that same company to assess and approve those questions.
Index analysis
Andrew Perlman, dean of Suffolk University Law School and advisory council member of the American Bar Association Task Force on the Law and Artificial Intelligence, said he was not aware of AI being used to develop bar exam questions or standards governing such use. He stated that while AI can be useful for developing assessment questions, critical guardrails include careful vetting by subject matter experts.
Governance controls
The State Bar of California is an administrative arm of the California Supreme Court. The disclosure prompted discussion about governance gaps, as Perlman noted he had not heard of standards or guardrails in place for using AI to develop bar exam questions in the legal profession.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.