Anatomy of a Cyberattack: AI Prompt Injection and Agent Authorization

A support ticket is data. A refund is an action. Between them sits a question that no amount of fluent model output can answer on its own: who authorized this operation, against this resource, with these arguments?

Anatomy of a Cyberattack series: this is the AI installment. Automated web abuse has a different threat model, covered in the separate bots article.

That is the engineering thread explored here through my Anatomy of a Cyberattack repository. This walkthrough focuses on direct prompt injection, poisoned support tickets and excessive agent authority: how attacker-controlled content can influence a model, and why that influence must not become permission to act.

This walkthrough follows the implementation, not just the threat labels. We will examine the request path, the enforcement points and the evidence behind the result displayed on screen. The aim is to build a useful mental model for designing defenses, not to turn a deliberately small teaching application into a production security product.

Scope: the source links below pin the review to commit 48a70ef. Secrets are fictional teaching markers. Agent tools are simulated. Production recommendations are identified separately from controls implemented in the labs.

Start with the runtime, not the attack name

The combined launcher loads three exercise modules into a shared Python HTTP server. Each module supplies its scenario, attack catalog and an ask(message, attack_id, defended) method. The server exposes module metadata and dispatches requests through /api/module/<id>/ask.

The loader also addresses a practical Python problem: all three exercise directories contain similarly named files such as attacks.py and defenses.py. It temporarily adjusts the import path and removes newly loaded exercise-local modules from the import cache between loads. Otherwise, one exercise could accidentally reuse another exercise’s implementation.

Exercises one and two use the shared Ollama client to call localhost:11434/api/chat. The default model is qwen2.5:1.5b, with temperature zero, a 120-second timeout and non-streaming responses. The client returns text alongside input and output token counts when supplied by Ollama.

Exercise three does not ask an LLM to choose tools. A deterministic keyword router makes the authorization lesson reproducible.

This separation matters when interpreting a result. A successful model call, a detected marker and a policy decision are three different observations. None should silently stand in for the others.

Direct injection: message roles are not a secret store

The first target is a fictional customer-care assistant. Its system prompt contains a deliberately protected discount code, SECRET-2025, together with instructions not to disclose it. The user controls the other message.

The API request really does distinguish roles. In abbreviated form, its structure is:

{
  "model": "qwen2.5:1.5b",
  "messages": [
    {"role": "system", "content": "<customer-care rules and fictional code>"},
    {"role": "user", "content": "<untrusted customer message>"}
  ],
  "stream": false,
  "options": {"temperature": 0.0}
}

Calling this a literal flat string would misdescribe the application. The important limitation is subtler: a role label is not an independently enforced information-flow boundary. The model receives the protected value, and the application is relying on its response behavior to keep that value out of the answer.

The catalog explores instruction override, invented personas, forged delimiters, claimed authority and reframing. A forged “system message” inside user text does not become an actual API system message. It attempts to influence how the model interprets the content. Whether it succeeds depends on the model and execution conditions, not a universal rule that the latest instruction always wins.

The defended path adds two application-side gates. A case-insensitive input scanner can reject recognized patterns before inference. An output scanner can replace a response that contains the protected marker before returning it to the caller.

The leak judge uppercases text and keeps only alphanumeric characters. That makes SECRET-2025 and S E C R E T - 2 0 2 5 comparable. The shared judge expresses the same normalization operation:

def normalize(text: str) -> str:
    return "".join(ch for ch in text.upper() if ch.isalnum())

This is a useful deterministic detector, not a general proof of confidentiality. It recognizes certain textual representations of a known value. It does not establish coverage for arbitrary encodings, partial disclosure or all semantically equivalent responses. Broad signatures can also reject legitimate customer questions.

The stronger architectural decision is to avoid placing actual secrets in model context when the task does not require them. If a privileged operation needs a secret, a separately authorized service should use it without returning the credential to the model.

Indirect injection: the operator did not write the attack

The second exercise changes who controls the input. A trusted staff member asks SupportSummarizer to summarize a customer ticket. The customer, not the staff member, controls the ticket body.

That difference is operationally important. Authenticating the employee does not make every document the employee processes trustworthy. The employee’s authority and the document’s provenance must remain separate.

The ticket catalog demonstrates three cases: an instruction embedded in the ticket body, an instruction inside an HTML comment, and a request to append an attacker-selected reference marker to the summary.

The HTML example illustrates a representation problem. A comment may be invisible in rendered HTML while remaining present in the text passed to a model. In this lab, the ticket is supplied as text; there is no crawler or complete RAG ingestion pipeline. In a production retriever, the exposure depends on what the parser preserves and what reaches model context.

The third case targets integrity rather than the fictional escalation secret. If a customer can dictate additional content in an internal summary, the downstream reader may mistake that content for the summarizer’s trusted assessment.

Spotlighting helps express intent; code enforces the gates

The module implementation confirms the actual sequence: scan the ticket; if allowed, call the model with SPOTLIGHT_SYSTEM_PROMPT and spotlight_task(ticket); then scan the response.

Spotlighting wraps the ticket in explicit BEGIN UNTRUSTED TICKET and END UNTRUSTED TICKET markers and instructs the model to treat their contents as data. This clarifies the intended boundary. It does not create a cryptographic or authorization barrier around that text.

The output detector checks the fictional escalation code and any markers associated with the selected catalog entry. That last detail is essential. The lab knows the expected marker for its demonstration; it does not discover every attacker-chosen output goal. A custom ticket without a matching catalog entry does not acquire an arbitrary new marker detector.

The resulting compromised flag must therefore be read within that detection model. A green result is not a certification that a summary is semantically correct, free from manipulation or safe to feed into a privileged downstream workflow.

For production, retain provenance as application metadata, restrict retrieval to resources the caller may access, and evaluate whether outputs remain faithful to the authorized task. If summaries trigger actions, authorize those actions independently. Summarization must not launder customer text into employee authority.

Excessive agency: move the decision outside the model

The third exercise shifts the failure mode from what a model says to what an application permits.

Its router returns a ToolCall containing a name and arguments. Four mock capabilities are described: order lookup, refund, email and record deletion. The router uses keywords and simple extraction rules; it is not evidence about an LLM’s tool-selection accuracy.

Without the defense layer, the module labels the selected operation executed. With the defense layer, it passes the proposal to apply_policy. Nothing actually deletes customer records, sends email or transfers money.

The important design pattern is the separation between proposal and permission. A planner can propose an action. A different component decides whether that action is allowed.

The policy checks something a prompt cannot authorize

The policy implementation applies an allowlist, rejects wildcard targets, restricts email recipients to the configured internal domain, and checks refund amounts against a limit of 500.

Refunds above that limit are blocked. Refunds within it are held, rather than automatically approved. Order lookup and internal email can pass the implemented checks. Deletion is excluded from the allowlist.

For example, the policy distinguishes these proposals:

Proposed operation	Defended outcome	Reason
Look up order 4471	executed	Allowed mock read operation
Refund order 4471 for 25	held	Sensitive operation requires approval
Refund order 4471 for 9999	blocked	Exceeds the configured cap
Email an external recipient	blocked	Recipient fails the domain rule
Delete customer records	blocked	Tool is outside the allowlist

These results demonstrate a control boundary, not complete business authorization. The example policy does not establish resource ownership, tenant membership or a durable approval workflow. Its simple parsing and numeric handling are teaching code, not a reusable financial validation library.

Even read-only access needs authorization in a real system. An order lookup that exposes another customer’s information is not safe merely because it performs no write.

What production enforcement needs to bind

A production tool gateway should validate a tuple such as caller, tenant, action, resource and arguments against trusted context. The model must not supply the identity evidence that grants itself authority.

For refunds, validate the amount’s type, finiteness, sign, currency and relationship to the original payment. Use an appropriate money representation rather than treating a floating-point comparison as a complete business rule.

An approval should bind the exact operation and arguments, have an expiry, and be consumed in a replay-safe way. Changing the amount or recipient after approval should invalidate it. Recheck relevant authorization at execution time, and use idempotency controls to prevent duplicate side effects during retries.

This is consistent with OWASP’s Excessive Agency guidance: minimize functionality, permissions and autonomy, and enforce authorization in downstream systems. Better instructions may improve behavior; they should not become the authority that permits a payment.

Evaluate boundaries, not just green badges

The repository has separate tests for detectors, model-call wiring, policies, HTTP endpoints and optional live integrations. Those layers support different claims.

For the LLM labs covered here, 69 deterministic tests passed and four live-model tests were deliberately skipped. Ollama requests were disabled while local HTTP calls remained available for mocked endpoint tests. This did not exercise a live model.

Evidence	What it establishes	What it does not establish
Mocked model requests	Expected roles, payload fields and response handling	A model’s resistance to unseen injection
Detector tests	Matching and blocking for supplied examples	Complete leakage or poisoning detection
Policy tests	Expected decisions for known mock proposals	Production authorization or real side effects

An input-block test is especially valuable when it asserts that the model was not called. An output-block test should verify that disallowed content does not reach the caller. For a real tool gateway, instrument the execution boundary and assert that denied or held operations produce no side effect.

Extend evaluation beyond attack success rates. Measure false refusals on legitimate tasks, malformed arguments, dependency failures, cross-tenant access attempts, approval replay and duplicate submissions. Keep the model version, prompt version, policy version and test corpus identifiable so results can be compared meaningfully.

Record enough telemetry to explain the decision: request identity, selected tool, redacted arguments, enforcement stage and outcome. Do not put raw secrets or complete customer tickets into logs merely to make troubleshooting convenient.

Reproduce the right part of the lab

The authorization exercise is a useful starting point because it needs neither a model nor a cloud service:

cd Anatomy-of-Cyberattack/exercises/LLM/03-excessive-agency
python3 -m unittest -v

For the combined UI, start from the actual directory in the reviewed repository:

cd Anatomy-of-Cyberattack/exercises/LLM
python3 serve.py

The UI listens on http://localhost:8000. Model-backed exercises require Ollama and the configured model to be available separately. Installing models or exercising a live model was not part of the offline verification described above.

The repository also includes deepfake demonstration media and reference documents. These support discussion, but they are not an implemented deepfake detector or provenance-verification service. Treat convincing media as evidence to verify, not permission to bypass an established approval process.

The architectural takeaway

Model instructions describe desired behavior. Retrieved content supplies evidence. Tool policies authorize operations. These responsibilities should reinforce one another without being collapsed into a single “trusted” flag.

The practical question is not whether an assistant can be persuaded to produce a dangerous-looking sentence. It is whether that sentence can cross into a system that treats it as permission.

That is the perspective I bring to Anatomy of a Cyberattack: follow the input, locate the authority, and make the enforcement point observable. The course overview explains the broader course coverage. The course details above provide the current session and registration information.

The permanent course archive keeps the session history in one place.

Implementation references are pinned to the reviewed commit. The short Python excerpt is adapted from the repository’s MIT-licensed code, copyright Razi Rais. Diagrams and explanatory prose were created for this article; third-party report pages and diagrams are not reproduced.