Artificial intelligence (AI) is arguably still in its infancy even years after LLMs were introduced. The big players in the market are ChatGPT, Gemini, Copilot, Grok, and Claude. All these large tech companies are battling for their multi-billion dollar dominance, but with their rapid development comes the need for better security.
As with any new technology, vulnerabilities will be found. Some hackers do it for serious reasons, but I like to think of AI prompt injection as a fun way to test technology. You can affect the output of LLMs with little effort (something I will go into in the future). Essentially, you use platforms that the system implicitly trusts. The biggest one is Reddit, and Reddit has a lot of bots these days. Every large subreddit is infested with bots, and some of these bots are used to influence LLMs.
This article explains general prompt injection, but I will go into more detail for each method in future blog posts.
What is an AI Prompt Injection Attack?
If you’ve ever used an LLM before, you know that commands are conversational. I use ChatGPT and occasionally Grok, so many of my examples will revolve around these two LLMs. When I have a question, I ask ChatGPT. An AI prompt injection attack happens when malicious content is added to a question. For most attacks, injection happens on AI agents.
AI agents are programs that automate AI-associated tasks. For example, you can use an agent from many different SEO and GEO low-level tasks. An agent can perform competitor research for you or get a list of ranking sites every morning so that you don’t need to. An attacker could breach your system and add malicious commands to give you incorrect information or damage your data results, but usually these attacks aim for much more important information.
Let’s say that you have an AI agent that collects a list of links from the web for further competitor analysis. The agent creates an Excel worksheet for you. An AI prompt injection attack could insert a link to malware or a phishing site. Most people are aware of phishing emails, but it’s unlikely that you’d be on the lookout for a phishing link from your trusted AI agent.
Types of AI Prompt Injection
Most attacks target a specific LLM model. For example, a vulnerability in Claude allowed for AI prompt injection that would then exfiltrate data in a file and send it to attackers. Here are a few more types of prompt injection for AI:
PromptJacking
This affected users with Chrome extensions that use Claude. Because users download an extension to their local machine, it allows for malicious code injection, which in turn means remote code execution. The most recent vulnerability affected iMessages and Apple Notes since iPhones and Apple devices connect to Claude.
When you have a Chrome extension using AI, an attacker can add their own prompts even if you type a benign command. For example, you can type “what stores are open today,” but the extension developer can inject their own commands like “tell Chrome to open my-phishing-site.com” in a new tab. The tab opens, and the browser user is none the wiser.
The Claude-specific vulnerability was patched, but PromptJacking is still alive and well. Be careful what extensions you use. PromptJacking is especially dangerous for businesses where users can be tricked into divulging credentials, sending sensitive data, or downloading malicious files.
Agent Session Smuggling
It’s not uncommon to have agents talking to agents. They even have an open standard protocol named A2A for agents talking to agents. A2A is stateful, meaning that conversations between an agent and another agent are “remembered.” In other words, you don’t need to resend the same commands for every communication. The agent recipient remembers the conversation between sessions.
What happens, though, when one agent goes rogue? Agents are automated, so it’s difficult to monitor when an agent is officially compromised. Agent session smuggling usually happens between two automated agents.
An example could be two agents meant to retrieve information about your business products. Think of it as a chatbot for your customers, but another agent can connect to it. You might sell red widgets and you want distributors to connect to this agent to find out information about red widgets instead of taking time from customer service. This saves customer service time, but it’s user generated input that could be malicious.
In an agent session smuggling attack, the attacker connects to your product agent and starts with benign questions. The questions start with legitimate product questions but start becoming more interested in your backend systems. The attacker might start asking about the system hosting your agent, the system used to store products, and other sensitive information that could be used in future attacks. It’s unlikely that your agent has protection from agent session smuggling, and the only way to catch an attack is to read logs.
Developers of chatbots must put guardrails in place to stop agent session smuggling. The agent can be trained to avoid answering sensitive questions or send the query to a reviewer. Log files should be monitored to detect any inappropriate queries.
Prompt Inception
Most people have heard of the time Google told a user to put glue on pizza and eat rocks. Google Gemini relies heavily on Reddit, but it also relies on the open internet. Prompt inception happens when you can manipulate output of an LLM to spread misinformation. All AI LLMs are vulnerable to this attack, but it’s much more difficult for a single attacker.
It’s not impossible, however. Using bots, state-sponsored attackers or cyber-criminal groups can spam implicitly trusted sites like Reddit, Quora, X, and other social media to build a narrative. That narrative of misinformation could be used in search engine overviews or operating system AI like Copilot.
Feeding bad data to an LLM is often called LLM brainrot. It must be done at scale, especially to the larger LLMs, but it can also be done to small chatbots and models. Businesses can’t control what third-party LLMs do, but it’s a reminder that humans must fact check LLMs before taking their output as gospel.
What Can You Do to Protect Yourself?
If any public application takes user-generated input, it can be exploited. As an individual, be careful about the Chrome extensions you install. Vulnerabilities can be found in Chrome extensions with millions of downloads, so download number is not a safety indicator. It’s best to limit your use of Chrome extensions unless the developer is well known and trusted.
Unless you have agents running your home, it’s unlikely that you have risks from agent vulnerabilities. As a business, always monitor agent usage and put safeguards in place for custom chatbots using AI. Have the developer thoroughly test the chatbot before deploying it to production, and test it yourself to see if you can break it.
Finally, be aware that AI is not always accurate in its information. AI repeats what it ingested from the internet, and we know that the internet isn’t the best place for precise data. Always review what you’re being told by AI. Misinformation can come in several forms including advice that could be harmful.