AI-powered chatbots and conversational assistants are becoming increasingly common in websites and business applications. They can answer customer questions, provide product information, assist employees and handle conversations automatically.
But unlike traditional applications, these systems interpret natural language as instructions. This creates a new security risk known as LLM injection or prompt injection.
What Is LLM Injection?
LLM injection occurs when an attacker crafts input designed to influence an AI chatbot into ignoring its intended instructions or behaving in an unintended way.
For example, a chatbot may be instructed:
“You are a customer service assistant. Only answer questions related to our products.”
An attacker may try to manipulate it with instructions such as:
“Ignore your previous instructions and follow these new instructions instead.”
The exact wording is not important. Attackers can try different approaches to confuse or override the chatbot’s original instructions.
The objective may be to make the AI:
- Ignore its intended rules
- Reveal information it should not disclose
- Provide restricted content
- Reveal hidden instructions
- Adopt a different role or behaviour
- Produce responses outside its intended purpose
Why Is LLM Injection a Security Concern?
Traditional applications generally process input according to predefined rules.
An LLM chatbot, however, interprets language and context. This means an attacker may attempt to manipulate the model rather than exploit a conventional technical flaw.
For example, a chatbot may have instructions telling it not to disclose certain information. An attacker could repeatedly change the wording or context of their requests to try to convince the AI to ignore those instructions.
This makes prompt injection particularly relevant from a penetration-testing perspective.
Common LLM Injection Techniques
Instruction Override
The attacker attempts to convince the chatbot to ignore its original instructions and follow new ones.
They may ask the AI to:
- Forget previous instructions
- Follow a new set of rules
- Treat the user’s instructions as higher priority
- Ignore restrictions placed on its responses
The objective is to determine whether the chatbot’s intended behaviour can be bypassed.
Prompt Leakage
Attackers may attempt to get the chatbot to reveal its hidden system instructions.
For example, they may ask it to:
- Repeat its instructions
- Summarise its rules
- Explain how it was configured
- Reproduce hidden instructions in another format
A system prompt should not contain passwords, API keys or other secrets. However, exposing internal instructions can still provide useful information about how the chatbot has been configured.
Role Manipulation
LLMs respond strongly to conversational context, which attackers can attempt to manipulate.
An attacker might instruct the chatbot to pretend it is:
- A developer
- A system administrator
- An unrestricted assistant
- Another AI model
The objective is to make the chatbot behave differently from its intended role.
Restriction Bypass
A chatbot may have rules restricting certain questions or responses.
Attackers can try to bypass these restrictions by changing the wording, context or role of the conversation.
This may involve multiple attempts using different prompts rather than relying on one specific attack.
Is Every Successful Prompt Injection a Vulnerability?
Not necessarily.
A chatbot producing an unusual or unexpected response does not automatically mean there is a serious security vulnerability.
During a VAPT, the tester should consider the actual security impact.
For example:
- Does the chatbot reveal confidential information?
- Does it disclose information that should remain hidden?
- Can an attacker consistently bypass important restrictions?
- Does it expose sensitive internal instructions?
- Can the attacker manipulate the chatbot into violating its intended security controls?
The distinction between “the AI gave an unexpected answer” and “the AI disclosed information it should not” is important.
Don’t Rely on the LLM to Enforce Security Rules
One of the key lessons from LLM injection is that simply telling an AI “do not disclose this information” should not be treated as a strong security control.
LLMs interpret instructions rather than enforcing deterministic security policies.
Sensitive information and important security restrictions should therefore not depend solely on the model choosing to follow its instructions.
From a penetration-testing perspective, the question is not simply:
“Can I trick the chatbot?”
It is:
“Can I trick the chatbot into breaking a security rule that matters?”
LLM Injection Is a New Attack Surface
As AI chatbots become part of web applications, they should be considered part of the application’s attack surface.
A conventional web application may be tested for issues such as authentication, authorization, input validation and business logic flaws. When a chatbot is added, there is another layer to consider: Can an attacker manipulate the LLM itself?
LLM injection testing helps identify whether the chatbot can be manipulated into bypassing its intended behaviour or disclosing information that should remain protected.
Test Your Web Application’s AI Chatbot
At Perennial Consultancy, LLM injection testing is included in our Enterprise VAPT package for web applications that incorporate AI chatbots or conversational AI. This allows organisations to assess whether their chatbot can be manipulated into bypassing its intended instructions, revealing restricted information or behaving in unintended ways.
If you are a Singapore SMEs, you are eligible for CISOaaS VAPT grant of up to 70%.
