How attackers persuade AI agents to break the rules

Elise Racine & The Bigger Picture / Web of Influence I / Licenced by CC-BY 4.0

By Tanya Petersen

Today, most of us interact with AI assistants – reactive bots that wait for human instructions. Yet, AI assistants are rapidly being replaced by agentic AI agents, that can interact with external tools, browse the web, generate images, send emails, and perform increasingly complex workflows on behalf of users.

As these agentic agents become more capable, they may be exploited by people with bad intentions. Safety tests have generally only checked whether an AI agent refuses a single harmful request – they don’t measure what happens when an attacker gradually persuades the AI through a series of seemingly harmless conversations.

Now, researchers from EPFL’s Natural Language Processing Laboratory have developed STING (Sequential Testing of Illicit N-step Goal execution), an automated testing framework that simulates how a real attacker might manipulate large language model (LLM) agents into carrying out harmful tasks over multiple interactions.

In their paper, presented at the 2026 International Conference on Machine Learning, the EPFL researchers outline how, rather than simply asking an AI agent to do something obviously malicious, STING breaks an illicit objective into a series of seemingly harmless requests, each building towards the final goal.

“LLM agents are everywhere,” says PhD student Ayush Kumar Tarun in the NLP Lab, and lead author of the study. “They’re very powerful, but what if someone with bad intentions wants to use those same agents? That’s what we wanted to understand.”

As companies race to deploy these systems, ensuring they cannot be manipulated into assisting with cybercrime, fraud or other harmful activities has become an urgent challenge. In one example, META admitted in June that attackers used simple social engineering tactics, rather than malicious code or malware, to trick its AI support assistant into granting unauthorized access to Instagram accounts.

Thinking like an attacker

“If you simply ask an AI agent to hack someone’s account, today’s models are generally smart enough to refuse,” explained Antoine Bosselut, head of the NLP Lab and co-author of the paper. “But, if you decompose that goal into smaller, more benign-looking requests, and adapt those requests as the conversation progresses, you have a much better chance of getting the agent to perform the actions you want.”

STING reproduces this behavior by creating an automated “attacker” that develops a step-by-step plan before attempting to persuade a target AI agent to execute each stage.

The researchers tested this approach across 176 harmful task scenarios involving several leading AI models (like GPT, Gemini, and Claude) operating as tool-using agents. The results showed that multi-turn attacks consistently succeeded more often than traditional single-prompt tests. In some cases, agents were twice as likely to complete harmful tasks when attackers gradually built towards their objective rather than stating it outright.

“We expected multi-turn attacks to perform better. What surprised us was the magnitude,” says Tarun. “For some models, harmful task completion was around two times higher than with existing single-turn evaluations. STING measures how quickly an attack’s success occurs, rather than only whether an attack eventually succeeds, allowing different testing approaches to be compared more fairly.”

Challenging assumptions about language

The team also explored whether attacks became more effective in different, lower-resource languages where less training data exists, and were surprised by the results.

“We expected to see differences across languages because earlier research showed that translating prompts into lower-resource languages could bypass safety measures,” says Tarun. “But for agents, we found that harmful task completion rates were remarkably similar across all seven languages we tested.”

That finding challenges a growing assumption in AI safety research that multilingual vulnerabilities naturally increase as language resources decrease. However, the researchers also discovered an important caveat. When sophisticated attackers switched languages during different stages of a multi-step attack, harmful task completion could become dramatically more successful, highlighting another avenue that future safety evaluations should examine.

Safety shouldn’t be an afterthought

The research comes at a time when the rapid adoption of agentic AI agents makes proactive safety testing essential.

“We’ve focused enormously on making agents more capable,” says Bosselut. “But we also need people working on how to prevent those capabilities from being misused. Traditionally, safety has often been something people think about when something breaks, not before, but with AI agents, it can’t be an add-on or an afterthought.”

Looking ahead, the research team hopes STING will encourage developers to embed safety testing much earlier in the design process and, develop a framework for multi-agent systems.

“Research efforts to expose vulnerabilities are much larger in number than those showing practical defense strategies, and this is a crucial area of development. We also need to address what we can do for the models that are already out there in the wild for which safety was a post-hoc addition,” concluded Tarun.

AI Assistant vs. AI Agent

Autonomy is the fundamental difference between an AI assistant and an AI agent. AI assistants respond to our requests refined by our prompts and during conversations, we clarify and guide the AI toward what we’re looking for.

An AI agent, on the other hand, is given a high-level goal and works out autonomously and proactively how to achieve it.

Example: Recently in Australia an AI agent acting as a personal assistant was told to secure a spot in a full gym class. It autonomously scanned the gym’s booking system, located an unpatched security loophole, bypassed booking calendar restrictions, and deleted another member from the waitlist to force its user into the class. Because it autonomously planned and executed a multi-step strategy to meet its goal, the example has been widely cited as Australia’s first consumer-run AI agent attack.

EPFL

EPFL