Pakistani judges clear 1,848 extra cases yearly with AI training, $38.50 return per dollar

A multinational research team from ETH Zurich, the New Economic School, and Imperial College London conducted a large-scale field experiment in Pakistan's judiciary, enrolling 1,559 judges across 118 courts—representing roughly half of all Pakistani trial court judges. The study tested JudgeGPT, an AI assistant built on OpenAI's GPT-4 that uses retrieval augmented generation to search a database of 129,235 documents: 128,292 court rulings and 943 Pakistani laws. When a judge enters a query, the tool retrieves the ten most relevant passages and generates a cited answer.

Researchers divided judges into three groups. The first received JudgeGPT access plus targeted training: six 90-minute lectures delivered over three weeks by ETH Professor Elliott Ash, held after court hours. The curriculum taught judges which tasks suited the tool, where it fell short, and how to verify its output. A second group received the same AI access but only a general seminar on technology and law. The control group attended the technology seminar with no JudgeGPT access. After 40 weeks, the contrast was stark: trained judges averaged nearly 60 logins and over 200 prompts, while the comparison group (with AI access but only the general seminar) averaged about 20 logins and fewer than 50 prompts. AI access alone produced minimal usage.

The results on case resolution were substantial. Districts with more trained judges resolved significantly more cases. At moderate exposure levels, trained judges' districts cleared about 1,848 extra cases per year per district, a 6.3 percent increase. Even districts in the bottom quartile—the least exposed to trained judges—still resolved about 616 more cases annually. Importantly, these gains came without compromising quality or working conditions. The appeal rate per 1,000 resolved cases fell slightly, judges worked the same hours, and work-life balance remained unchanged. An analysis of roughly 4,000 court judgments found that while AI-flagged text appeared more frequently (as expected), readability, length, and the number of legal arguments held steady. An LLM-based quality check validated by two Pakistani lawyers showed a slight improvement: rulings from trained judges were rated better in 59 percent of pairwise comparisons, compared to 42 percent in the control group. Crucially, the study found no evidence that AI use increased gender or religious bias in judicial language.

Analysis of anonymized chat logs from about 1,500 judges revealed that legal research, text editing, and text generation were the most common tasks, with around 60 percent of queries seeking information about laws, procedures, or legal concepts. Training shifted behavior significantly: trained judges used JudgeGPT more for editing and summarizing text—tasks where language models are reliable—and asked fewer broad legal questions, where hallucination risk is higher. Only about a fifth of requests involved what researchers call 'substantive AI delegation,' where judges asked the tool to evaluate decisions or draft reasoning independently. Training made judges more likely to decide cases themselves and use AI only to write up their reasoning. Economically, researchers estimate a return of $38.50 per dollar invested, based on the cost to hire additional judges to produce the same output; even conservative estimates place the return at 'at least' $10 per dollar. The researchers stress that their findings do not support replacing judges with AI, but instead demonstrate that AI can boost public sector productivity when paired with training that directs users toward appropriate tasks. They note that JudgeGPT ran on GPT-4, a pre-reasoning model that was the best available for the project at the time, and that today's reasoning models 'write better, hallucinate less, and handle complex tasks more reliably,' suggesting the productivity gains in this study may be a floor rather than a ceiling.

The trial reveals a nuanced picture of AI adoption in the public sector. Raw access to an AI tool—in this case JudgeGPT, built on GPT-4 and searching a database of 129,235 documents including 128,292 court rulings and 943 Pakistani laws—produced minimal gains. Judges without training averaged only about 20 logins and fewer than 50 prompts over 40 weeks. The decisive factor was the targeted training program, which shifted both usage volume (trained judges averaged nearly 60 logins and over 200 prompts) and usage pattern (toward reliable tasks like text editing, away from risky broad legal questions). This behavioral steering proved crucial: trained judges resolved significantly more cases—1,848 extra per year per district on average, with even bottom-quartile districts clearing about 616 more—while maintaining or improving ruling quality and avoiding bias drift.

The economic case is striking. A return of $38.50 per dollar invested (or conservatively at least $10 per dollar) reflects the cost savings of avoiding new judge hires to achieve the same output. Notably, the gains came without degrading the judicial process: appeal rates per 1,000 resolved cases fell slightly, judges worked the same hours, and work-life balance did not change. The researchers emphasize that these findings do not support replacing judges with AI, but rather demonstrate that thoughtfully deployed AI with proper training can amplify human productivity in constrained public systems. The implication—that training steers users toward appropriate, lower-hallucination tasks while preserving human decision-making—may apply beyond the judiciary to other sectors deploying language models in high-stakes settings.

No comments yet. Be the first to share your thoughts!