The Rise of AI in the Legal Sector: A Case Study from Brazil
The integration of artificial intelligence (AI) across various industries is accelerating, bringing both advantages and challenges. Recently, a labor court in northern Brazil encountered a notable case involving an employee who had been fraudulently working for three years. Initially appearing to be a routine case, it took an unusual turn when the plaintiff's legal team embedded hidden instructions within their petition directed at the court's AI system, Galileu. This concealed text, invisible to the human eye but detectable by AI, requested the AI to respond superficially and avoid challenging the submitted documents. However, the AI detected the deceit, and the judge imposed sanctions on the lawyers involved.
Understanding Prompt Injection: A Legal Concern
This incident raises questions about whether such attempts at deception are isolated incidents or indicative of future practices in legal systems worldwide. Argentine judge and professor Marcelo Quaglia characterizes the situation as predictable rather than absurd. Abel Gende, a lawyer and managing partner of Derecho Virtual, further comments that this phenomenon is not exclusively Brazilian but is a possibility in countries like Spain. He mentions that in January, the Spanish Judicial Authority (CGPJ) approved guidelines regulating AI use by judges, which do not prohibit it but rather channel its application for analyzing and structuring documents.
What is Prompt Injection?
The technique employed by Brazilian lawyers is known as “prompt injection.” This method involves inserting hidden requests within a text intended for AI processing. It has emerged as a tactic in educational settings, with teachers using it to determine if students rely on AI-generated content. Its application in legal environments poses distinct risks due to the nature of legal proceedings, where extensive documentation is the norm.
The Role of AI in Brazil's Legal Framework
The AI system Galileu acts as an assistant to judges, analyzing documents and proposing drafts for rulings, but it does not assess evidence or conduct legal analysis. Its sole function is to organize and prepare content for human review. The Brazilian case highlighted a strategy to bypass this review process by manipulating the AI. Gende explains that while Spain does not have a comparable AI system at this scale, it is moving in a similar direction.
Emerging Cases of AI Manipulation
Following the initial case, other courts in Brazil have reported similar instances. In one case in Sao Paulo, a lawyer instructed the AI to approve urgent requests for legal assistance based on specific criteria designed to manipulate the outcome. This raises concerns about the broader implications of AI in making judicial determinations.
The Subtle Risks of AI in Legal Decision-Making
The potential danger of AI in the legal sphere lies not in its autonomous judgment but in how it influences the decision-making process. Quaglia identifies the critical issue as AI's role in shaping how information is organized and presented to judges rather than directly delegating decisions. This manipulation can fundamentally alter the context in which judges and lawyers operate, leading to what he terms “invisible fraud.”
Challenges for Professional Ethics
AI's influence extends beyond judges to interactions among law firms, where hidden instructions could be easily embedded in exchanged documents. Gende emphasizes that this scenario will challenge legal ethics significantly, as lawyers might feel tempted to employ legitimate means of defense to “outsmart the machine.” Thus, establishing clear boundaries is imperative to differentiate between vigorous defense and attempts to deceive the court.
