# Agent traces are security telemetry

By Derik Callahan · 2026-09-30 · https://derikcallahan.com/writing/agent-traces-are-security-telemetry

> If you can't replay what an agent saw, decided, and did, you can't investigate it. OpenTelemetry already gives us the format. The SOC needs to treat it like EDR data.

When an endpoint does something strange, my team can rebuild exactly what happened: the process tree, command lines, network connections, and file writes. EDR gave us that, and it changed incident response.

When an AI agent does something strange, most organizations can rebuild almost nothing. Maybe there's a chat transcript. Maybe there's an API bill. The prompt that steered it, the documents it retrieved, the tools it called and the arguments it passed are gone, or they sit in a developer's debug log that nobody in security can query.

That's the gap I spend the most time on. **An agent's trace is security telemetry, and it should be collected, retained, and watched like any other.**

## What to capture

OpenTelemetry already has a model that fits agents: a trace made of nested spans. One user request becomes a root span. Each model call, retrieval, and tool call becomes a child span. The useful attributes for security are:

- **Identity.** Which human started this, and which non-human identity the agent acted as.
- **Inputs.** The system prompt version, the user prompt, and *where retrieved content came from*. Indirect prompt injection arrives in retrieved content.
- **Tool calls.** The tool name, its arguments, and the result, including denials from a gateway or guardrail.
- **Decisions.** Whether a human approved, rejected, or never saw an action that needed approval.

None of this is exotic. The hard part is organizational: getting every team that builds agents to emit traces in the same shape and send them somewhere the SOC can see.

## What to detect

Once traces land in the SIEM, the detections start to resemble ones we already know:

- **Tool use the task doesn't explain.** A summarization agent that calls a send-email tool. That's the agent version of Word spawning PowerShell.
- **Injection markers in retrieved content**, especially when a tool call happens right after them.
- **Data leaving through an unexpected route.** Large tool arguments going to external destinations, or sensitive labels showing up in outbound content.
- **Guardrail denials that cluster.** One blocked call is noise. Twenty from the same session is someone probing.
- **Drift.** An agent identity suddenly using tools or data it has never touched before.

## Where gateways fit

An AI or MCP gateway gives you a choke point. It can enforce which tools an agent may call, inspect arguments, and require approval for destructive actions. It's also the natural place to emit consistent telemetry. But a gateway without trace retention is a firewall without logs: it will block things, and you'll never know what it saw.

## The test

Here's the question I ask about any agent going to production: **if this agent does something harmful at 2 a.m., can the on-call analyst rebuild what happened by 2:30?** If the answer is "we'd have to ask the developer," it isn't ready yet.