Supreme Court Judge Raises Concerns Over AI Biases

In a recent hearing related to the Krishna River water dispute, Justice Surya Kant of the Supreme Court of India highlighted some critical concerns regarding the potential biases and drawbacks of artificial intelligence (AI). His remarks came after a discussion with Senior Advocate Jaideep Gupta about the use of AI in legal proceedings and the growing dependency on technology for retrieving information. Justice Kant expressed his apprehension about blindly relying on AI tools like ChatGPT or Gemini for critical information, emphasizing the potential risks and challenges posed by AI's limitations and biases.

AI Technology in Legal Proceedings: A Double-Edged Sword

The conversation initially revolved around the possibility of utilizing AI-powered apps to simplify legal research. Senior Advocate Jaideep Gupta suggested that, with the help of AI technologies, the Court could simply ask questions about the Krishna River—for instance, asking what the tributaries of the river are. Gupta pointed out that AI tools can swiftly retrieve such information without the need for time-consuming manual research. While this may seem like an efficient solution, Justice Kant raised concerns about the consequences of over-relying on AI tools for fundamental legal queries.

Justice Kant’s Caution Against Over-Reliance on AI

Justice Kant expressed caution, stating that, while AI can undoubtedly offer convenience, it is essential to avoid over-dependence on technology. "At least I don’t do that. I am very afraid of doing that. It’s human nature that when we start depending upon something, we are bound to become used to it," he remarked. The Supreme Court judge explained that relying too heavily on AI might lead to individuals losing their ability to independently think and research essential information. This could potentially affect their judgment and ability to critically assess the reliability of the data provided by AI systems. According to Justice Kant, it’s important to preserve human intuition and reasoning, even as technology advances.

Loss of Source Verification and Ethical Concerns with AI

Senior Advocate Gupta further emphasized the risks of becoming overly reliant on AI tools, particularly in the context of source verification. He raised an important point about the loss of the ability to authenticate the information that AI presents. He cautioned that if humans lose the ability to trace back to the original sources of information, it could undermine the accuracy and trustworthiness of the data AI produces. Gupta pointed out that, while AI may give the appearance of having an answer, it is essential to ensure that the answers are rooted in verified and credible sources.

Adding to the concerns, Gupta shared an ethical example, where an AI system might suggest an inappropriate or harmful response. He mentioned a hypothetical scenario where an AI could say, "remove the person," when such a suggestion is not only unethical but potentially illegal. In such cases, he argued, AI should follow an ethical guideline that prevents harmful or unlawful recommendations, underscoring the importance of ethical AI use in legal and other professional spheres.

AI’s Inherent Biases and the Need for Ethical Guidelines

Justice Kant wholeheartedly agreed with Gupta’s concerns about AI's inherent biases. He stated, "Absolutely, very serious biases," acknowledging that AI systems are not free from flaws. These biases often stem from the data used to train AI models, which can inadvertently reflect societal prejudices or historical injustices. Justice Kant further discussed the necessity for developing ethical guidelines for AI systems, especially in the context of legal applications. While AI can be a powerful tool, it must be governed by a framework that ensures it provides answers ethically, objectively, and without harm.

Gupta supported this view, stressing that AI should be subject to strict ethical rules to ensure that the system functions in a responsible manner. He pointed out that one of the ongoing efforts in the development of AI is the creation of a set of ethical standards that AI systems must follow when answering queries. Such guidelines, he suggested, would help ensure that AI tools do not propagate harmful or biased behavior.

Case Title: THE STATE OF ANDHRA PRADESH Versus UNION OF INDIA AND ANR., 2025