Judges caution against Artificial Intelligence in courts, flag ‘hallucinated’ citations | Bengaluru News

Bengaluru: Judges at South Zone Regional Judicial Conference Saturday cautioned against excessive reliance on Artificial Intelligence (AI) in courtrooms, referencing cases involving “hallucinated” citations and fabricated judgments. While acknowledging AI’s value as a research aid, they stressed it cannot replace human reasoning or constitutional principles in judicial decision-making.During a session titled ‘Bridging the digital divide: The role of e-services’ at the conference, Justice M Sundar, Chief Justice of Manipur high court, highlighted that the judiciary’s digital divide encompasses more than just economic factors or computer literacy. He identified three key categories: digital natives and immigrants; the digital rich and poor with varying access to devices and connectivity; and individuals who are digitally skilled versus unskilled based on their technological competence. A fourth emerging digital divide is now centred on AI, cutting across all existing categories between those who see AI as a useful aid to judges and those who fear it could undermine independent judicial thinking.He warned that AI outputs should be “considered, not relied upon,” citing instances of fabricated case citations (referred to as “hallucinations”) and inconsistent AI responses during court proceedings. “You can consider AI, but you cannot depend on it entirely to make a judgment. AI tunes itself as per the prompt you give; it doesn’t have emotions like us to analyse the situation completely in legal cases,” he said.Regarding judicial decision-making, he raised concerns about the use of AI, mentioning cases where lawyers presented AI-generated material to courts, including an incident where a fabricated Supreme Court judgment with fake citations was produced — an event described as “hallucination” that led to the dismissal of the document and the initiation of administrative action.He emphasised that AI doesn’t possess true cognition; it merely detects patterns through data and algorithms. He underscored the need to address the digital divide through practical measures, such as expanding e-service centres in remote areas to enhance access to justice.“We are not looking at robot judges but cyborg judges — part human, part machine — using AI’s computational strength while applying independent human reasoning. AI can assist but cannot replace judicial decision-making. To bridge the digital divide, technology must reach litigants directly,” he concluded.