Skip to main content

OpenAI Agents SDK Observability

Observability is the ability to understand exactly what your agent is doing, at every step, in real time. In the OpenAI Agents SDK, observability is built in—every agent run, tool call, handoff, and guardrail check is automatically captured as a trace. This article is your practical guide to enabling, interpreting, and acting on those traces, logs, and metrics to debug, optimise, and operate production‑grade agent systems.

What Is Observability in the OpenAI Agents SDK​

Observability means you can answer questions like:

  • What path did this user request take through the agents?
  • Which tool was called, with what arguments, and what did it return?
  • Why did the agent hand off to a different specialist?
  • Where is the latency coming from—LLM, tools, or guardrails?
  • How much did this conversation cost?

The SDK provides three pillars of observability:

  • Tracing – a structured, end‑to‑end record of an agent run, including every sub‑step.
  • Logging – you augment traces with custom log messages for your own business logic.
  • Metrics – aggregated numerical data (latency, error rates, token counts) derived from traces and logs.

Everything is centred around the OpenAI trace dashboard, but you can also export data to your own tools via the API.

A simple example: seeing a trace.

from agents import Agent, Runner, set_tracing_export_api_key
import os

# Enable tracing
os.environ["OPENAI_API_KEY"] = "sk-..."
set_tracing_export_api_key(os.environ["OPENAI_API_KEY"])

agent = Agent(name="Assistant", instructions="Be helpful.", model="gpt-4o")
result = await Runner.run(agent, "Hello!")
# A trace of this run is now visible in the OpenAI platform.

Open the dashboard and you’ll see the entire conversation flow: the user input, the LLM call, and the agent’s output. As you add tools and handoffs, the trace becomes richer.

Why Observability Matters​

Without observability, production agents are a black box. You’re blind to:

  • Debugging agent behaviour – why the agent gave a certain answer, why it chose a specific tool.
  • Understanding tool usage – which tools are called most, which fail, which consume the most time.
  • Tracking handoffs – how often routing changes, and whether the correct specialist is reached.
  • Measuring latency – finding the slowest step in a multi‑turn conversation.
  • Monitoring costs – token consumption and tool API costs per user or workflow.
  • Improving reliability – detecting error patterns and proactively fixing them.

Observability transforms post‑mortem guesswork into precise, data‑driven analysis.

Core Observability Concepts​

ConceptPurpose
TraceThe complete, end‑to‑end record of a single agent run.
SpanA unit of work within a trace (e.g., one LLM call, one tool execution, one handoff).
LogA developer‑emitted message containing context about a specific event.
MetricA numeric measurement aggregated over time (latency, success rate, cost).
AlertA notification triggered when a metric crosses a predefined threshold.

In the SDK, a trace corresponds to one Runner.run() invocation. Each internal operation (LLM reasoning, tool call, guardrail check) becomes a span nested inside that trace.

How Tracing Works​

When you enable tracing, the SDK automatically instruments the entire execution lifecycle:

Every span contains:

  • Trace ID – unique to the entire run.
  • Span ID – unique to this step.
  • Parent span ID – connects the span to its parent operation.
  • Timing – start and end timestamps.
  • Metadata – input/output, tool name, arguments, guardrail results, error messages.

You view these in the OpenAI dashboard under Traces. The UI shows a tree of spans, with detailed payloads for each.

Trace Structure​

A typical multi‑agent trace might look like:

Trace: user-123-session-abc
├── span: runner.start (input)
├── span: agent.triage.reasoning
│   ├── span: llm.call (model: gpt-4o, tokens: 800)
│   └── span: handoff.request (target: Billing Agent)
├── span: agent.billing.reasoning
│   ├── span: llm.call (model: gpt-4o, tokens: 600)
│   ├── span: tool.execute (name: lookup_invoice, args: {…}, duration: 1.2s)
│   └── span: llm.call (model: gpt-4o, tokens: 400)
├── span: agent.billing.final_output
└── span: runner.complete

Each indented span is a child of the span above it. This tree makes it trivial to see how a request flowed through the system and where time was spent.

Logging Best Practices​

Traces capture the SDK’s internal steps; logs add your business context. Use Python’s logging module and structure your messages as JSON.

What to log​

  • Agent execution logs – log the start and end of each custom node (if you’re wrapping the runner). Include thread_id and agent_name.
  • Tool call logs – inside your tool function, log the tool name, key arguments (sanitised), and a summary of the result.
  • Handoff logs – log the source agent, target agent, and reason if captured.
  • Guardrail logs – log every time a guardrail blocks a request or rewrites an output.
  • Error logs – log full tracebacks for unexpected exceptions, with the conversation context.

Sample structured log​

{
  "timestamp": "2026-06-11T14:22:01Z",
  "level": "INFO",
  "thread_id": "sess-001",
  "agent": "Billing Agent",
  "event": "tool_call",
  "tool_name": "lookup_invoice",
  "arguments": {"invoice_id": "INV-2026"},
  "duration_ms": 1200,
  "result_summary": "Found invoice, status: paid"
}

Ship these logs to a central system (CloudWatch, ELK, Datadog) and include the trace ID so you can correlate logs with traces.

Metrics for Agent Systems​

Metrics are derived from traces and logs and give you a high‑level view of system health.

CategoryKey MetricsWhy
LatencyEnd‑to‑end duration, LLM call duration, tool execution duration, handoff durationIdentify slow steps and bottlenecks.
ReliabilitySuccess rate (per agent, per tool), error rate, retry rateTrack stability and detect regressions.
UsageToken consumption per run, tool call frequency per run, handoff frequencyUnderstand system load and plan capacity.
CostEstimated cost per request, cost per workflow, tool API costControl budgets and optimise expensive paths.

Instrument these metrics using the trace spans’ timing and metadata. For example, the trace’s root span duration is the end‑to‑end latency. Each LLM span contains token counts. Tool spans contain the tool name and duration.

Set up dashboards in your observability platform (Grafana, Datadog, Azure Monitor) to track these metrics over time.

Monitoring Agent Workflows​

Beyond individual requests, you need to monitor the overall health of your agent service.

  • Workflow completion tracking – what percentage of conversations reach a successful end? Track terminated vs. failed runs.
  • Failure tracking – monitor error rates by agent, tool, or guardrail. A spike in output guardrail blocks may indicate a prompt drift.
  • Bottleneck identification – use P95 latency breakdowns to find the slowest step across all requests.
  • Performance trends – compare metrics before and after a model update or prompt change.

A production dashboard might show:

  • Request rate (RPM)
  • P50 / P95 latency
  • Error rate (overall and by type)
  • Token usage per minute
  • Cost per minute
  • Tool failure rate

Observability for Tool Calling​

Tool calling is often the most critical and failure‑prone part. Observability here focuses on:

  • Tool selection tracking – which tools are called, and is the agent choosing the expected one?
  • Argument logging – are the arguments well‑formed? Inspect spans for tool.execute to see the exact payload.
  • Execution timing – how long does each tool take? A slow API might need caching.
  • Failure analysis – see error messages returned by tools. If many Error: strings appear, the tool’s resilience may need improvement.

Pattern: add a custom span or log inside your tool function to capture business‑specific details (e.g., “invoice found”, “API returned 404”).

Observability for Handoffs​

Handoffs can silently misroute users. Track:

  • Handoff frequency – too many handoffs might indicate unclear instructions.
  • Routing decisions – inspect the handoff request span to see the target agent and the reason (if the model outputs one).
  • Context transfer tracking – ensure that the conversation history after the handoff is complete; traces show the message list size before and after.
  • Handoff latency – the time between the source agent’s last output and the target agent’s first response.

A spike in a specific handoff path (e.g., “General → Fallback”) may signal that the routing prompt needs tuning.

Observability for Guardrails​

Guardrails are safety nets; monitor them to ensure they aren’t too aggressive or too permissive.

  • Validation failures – count how often input and output guardrails block. High input blocking could mean you’re receiving junk traffic (or your guardrail is too strict).
  • Policy violations – categorise blocks by rule. This feeds into policy tuning.
  • Blocked actions – if a runtime tool check blocks a dangerous call, log it prominently and alert.
  • Runtime checks – instrument your custom runtime checks with logs and metrics.

Guardrail spans appear in the trace, showing the rule that fired and the action taken.

Debugging Agent Behavior​

When an agent produces an unexpected result, follow this practical workflow:

  1. Find the trace – locate the trace for that user’s thread_id (session ID) in the dashboard.
  2. Identify the failing span – look for error spans (red icons) or unexpected outputs.
  3. Inspect logs – your custom logs (correlated by trace ID) provide context that the SDK spans might not.
  4. Analyse tool calls – did the wrong tool get called? Were the arguments correct? Was the tool output truncated?
  5. Verify outputs – check the final answer span. Did the model misinterpret tool results?
  6. Apply fixes – adjust instructions, tool descriptions, or guardrails, then re‑test.

The trace tree makes it easy to zoom in on the exact moment something went wrong.

Production Monitoring​

Set up proactive monitoring to catch issues before users report them.

Dashboards​

Create a real‑time dashboard with:

  • Key latency percentiles
  • Error rate by agent and tool
  • Token usage and cost
  • Guardrail block rate

Alerts​

  • High‑urgency: error rate > 5% for 5 min, latency p95 > 30s for 5 min, guardrail output blocking rate > 10%.
  • Warning: token usage approaching daily budget, tool timeout rate > 2%.

SLA Monitoring​

Define internal SLAs (e.g., 99% of requests complete within 30s) and track compliance.

Anomaly Detection​

Use your monitoring tool’s anomaly detection to flag unusual patterns (e.g., sudden drop in token usage may indicate the model is returning empty responses).

Capacity Planning​

Use request rate and average latency to plan scaling of your agent service.

Common Observability Patterns​

PatternDescription
Request TraceFull end‑to‑end trace for every user request; store for N days for debugging.
Workflow TraceFor multi‑turn conversations, a single trace or linked traces per turn, correlated via session ID.
Tool MonitoringDedicated dashboard tracking per‑tool latency and error rate.
Cost MonitoringDaily/weekly cost reports broken down by agent, user, and workflow.

These patterns are straightforward to implement using the SDK’s built‑in tracing plus your logging/metrics stack.

Common Beginner Mistakes​

  1. No tracing enabled – flying blind in production. Always enable tracing.
  2. Logging sensitive data – capturing full user messages or PII in logs without masking.
  3. Missing error logs – not logging tool exceptions or guardrail failures.
  4. Tracking too many metrics – creating metrics for everything without a clear purpose; focus on what matters.
  5. Ignoring cost metrics – token costs can spiral without monitoring.
  6. Not setting alerts – relying on manual checking of dashboards.

Best Practices​

  • Enable tracing from day one – even in development. It pays for itself in debugging time.
  • Use structured logs – JSON format, with standard fields (timestamp, level, thread_id, span_id, message).
  • Track business metrics – not just technical metrics; track completion rate, user satisfaction (if feedback available).
  • Monitor costs continuously – set budget alerts and review weekly.
  • Correlate traces and logs – always include the trace ID in your logs.
  • Set actionable alerts – alerts should indicate a problem that needs human attention, with a clear runbook.
  • Test observability – verify that traces appear, logs are shipped, and metrics are correct in staging.

Practical Example: Customer Support Agent with Observability​

We’ll instrument a support agent that searches knowledge, calls a billing tool, and may hand off to a specialist. The goal is to have full visibility.

1. Enable tracing and set up structured logging​

import os, logging, json
from agents import Agent, Runner, set_tracing_export_api_key, trace

os.environ["OPENAI_API_KEY"] = "sk-..."
set_tracing_export_api_key(os.environ["OPENAI_API_KEY"])

# Structured logger
logger = logging.getLogger("support_agent")
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter('%(message)s'))  # output raw JSON
logger.addHandler(handler)

def log_event(event_type, **kwargs):
    payload = {"timestamp": datetime.utcnow().isoformat(), "type": event_type, **kwargs}
    logger.info(json.dumps(payload))

2. Create tools that emit logs​

from agents import function_tool

@function_tool
def search_knowledge(query: str) -> str:
    start = time.time()
    # Simulated search
    result = "Knowledge base article about returns..."
    duration = (time.time() - start) * 1000
    log_event("tool_call", tool="search_knowledge", query=query, duration_ms=duration)
    return result

@function_tool
def get_invoice(invoice_id: str) -> str:
    # Runtime validation + logging
    if not invoice_id.startswith("INV"):
        log_event("tool_error", tool="get_invoice", error="Invalid invoice ID format")
        return "Error: Invalid invoice ID format."
    # ... actual API call
    log_event("tool_call", tool="get_invoice", invoice_id=invoice_id)
    return f"Invoice {invoice_id}: paid"

3. Create agents with handoffs​

billing = Agent(name="Billing Agent", instructions="...", tools=[get_invoice])
support = Agent(name="Support Agent", instructions="...", tools=[search_knowledge], handoffs=[billing])

4. Run with a custom trace​

async def main():
    config = {"max_turns": 5}
    with trace("Customer Support Workflow", group_id="support-bot-v1"):
        result = await Runner.run(support, "Tell me about invoice INV-123", run_config=config)
    print(result.final_output)

Trace structure produced:

Trace: Customer Support Workflow
├── span: runner.start
├── span: Support Agent.reasoning
│   ├── span: llm.call (tokens: 350)
│   ├── span: tool.execute (search_knowledge, args: {…})
│   └── span: handoff.request (target: Billing Agent)
├── span: Billing Agent.reasoning
│   ├── span: llm.call (tokens: 400)
│   ├── span: tool.execute (get_invoice, args: {invoice_id: "INV-123"})
│   └── span: llm.call (tokens: 200)
└── span: runner.complete

Logs emitted: structured JSON entries for each tool call, including timing and success/failure.

Metrics to track in production:

  • Agent turn count (Support Agent: 1 turn; Billing Agent: 1 turn).
  • Tool latency: search_knowledge 80ms, get_invoice 1.2s.
  • Token usage: 950 total.
  • Estimated cost: $0.02.
  • Handoff success: 1 (Support → Billing).
  • Error rate: 0.

Alerts configured: notify if tool error rate exceeds 2%, or if end‑to‑end latency exceeds 20s.

With this foundation, you can debug any anomaly, optimise performance, and confidently scale your support system.

Observability in Production Systems​

In production, observability becomes your operational backbone.

  • Incident investigation – traces and logs give you the evidence to quickly pinpoint failures.
  • Performance optimization – latency metrics guide where to cache, upgrade models, or add parallelism.
  • Capacity planning – usage trends (RPM, token consumption) inform infrastructure scaling.
  • Reliability improvement – error metrics highlight flaky tools or guardrails that need tuning.
  • Cost management – continuous cost tracking prevents budget overruns.

Observability and Other SDK Concepts​

Observability is the lens through which you view all other SDK capabilities.

  • Core Concepts – every agent, instruction, and session interaction is traced.
  • Tool Calling – tool spans show arguments, results, and errors.
  • Handoffs – handoff spans reveal routing decisions and context transfer.
  • Guardrails – guardrail spans log blocks and rewrites.
  • MCP – MCP tools appear as tool spans; you can wrap them to add custom logs.

For cross‑framework observability comparisons, see the LangGraph Observability and CrewAI Observability articles. The framework comparison also highlights observability capabilities.

FAQ​

1. What is observability in the OpenAI Agents SDK?​

It’s the built‑in ability to capture traces (full execution records), logs, and metrics for every agent run, giving you deep insight into behaviour and performance.

2. What is tracing?​

Tracing captures the entire lifecycle of a Runner.run() call as a tree of spans, including LLM calls, tool executions, handoffs, and guardrails.

3. What metrics should I track?​

At minimum: end‑to‑end latency, token usage, cost per request, tool success rate, and error rate. Add business‑specific metrics like task completion rate.

4. How do I debug agent failures?​

Find the trace for that session, identify the span where the error occurred, inspect its inputs/outputs, and check correlated logs for context.

5. How do I monitor tool calls?​

Each tool invocation appears as a span in the trace. Aggregate these spans to get per‑tool latency and error rates. Add custom logs inside tools for business details.

6. How do I track costs?​

Traces include token counts. Multiply by model pricing to estimate LLM cost. For tool API costs, log estimated cost inside the tool and aggregate as a metric.

7. Is observability required for production?​

Yes. Without it, you cannot reliably operate, debug, or improve your agent system. It’s a production necessity, not a luxury.

8. Can I export traces to my own observability platform?​

The SDK currently sends traces to the OpenAI dashboard. You can use the API to export or mirror them, or implement a custom exporter using the trace events.

9. How do I correlate logs with traces?​

Include the trace_id and span_id in your log messages. The SDK provides these through the RunContext.

10. What is a span?​

A span is a single unit of work within a trace, like one LLM call or one tool execution. It has a start time, end time, and metadata.

11. How long are traces stored?​

In the OpenAI dashboard, traces are stored for a configurable period (typically 7–30 days depending on your plan). For longer retention, export them to your own storage.

12. Can I create custom spans?​

Yes, using the trace() context manager or the lower‑level TraceProvider to add custom spans for business logic.

13. How do I set up alerts?​

Use your monitoring stack (e.g., Datadog, Grafana) to create alerts based on metrics derived from traces/logs. For example, alert if error rate > threshold for 5 minutes.

14. Does observability add overhead?​

Minimal. Traces are sampled and sent asynchronously. The performance impact is negligible for production use.

15. Can I disable tracing for specific runs?​

Yes, you can control tracing via environment variables or by not setting the export API key for local development.

Conclusion​

Observability is not an afterthought in the OpenAI Agents SDK—it’s a core feature. By enabling tracing, adding structured logs, and tracking metrics, you gain complete visibility into your agent’s decision‑making, performance, and cost. This empowers you to debug with confidence, optimise relentlessly, and operate production systems reliably.

Key takeaways:

  • Tracing gives you an automatic, detailed map of every agent run.
  • Logging adds your own business context to the traces.
  • Metrics turn raw data into actionable insights.
  • Monitoring and alerting keep you ahead of problems.
  • Observability is critical for production and should be set up from day one.

Now, continue mastering the SDK:

For cross‑framework observability patterns, explore LangGraph Production, CrewAI Production, and the full framework comparison. To integrate standardised tools, visit the MCP Guide.

Now, turn on those traces, watch your agents in action, and never guess again.