Trace

What it does

The trace package records every agent run as a tree of spans: the run itself, each model call, each tool call, handoffs, memory access, retrieval, and LLM middleware such as retries. Each span carries a typed payload with the full input and output of the operation, so backends can show the exact prompt, response, tool arguments and results.

Spans are parented through the context.Context. Concurrent runs never collide, a sub-agent used as a tool nests under that tool call, and a run started under an existing span (yours, or an OpenTelemetry one) becomes its child.

Quick start

import (
    "github.com/henomis/phero/v2/trace"
    "github.com/henomis/phero/v2/trace/text"
)

p := trace.NewProvider(trace.WithProcessor(text.New(os.Stderr)))
a.SetTracing(p)

result, err := a.Run(ctx, llm.Text("What is 2+2?"))
fmt.Println(result.Summary.StopReason, result.Summary.Usage.CostUSD)

An agent without its own provider uses the one in the context (trace.WithProvider(ctx, p)), so sub-agents inherit their parent's provider with no extra wiring. Without any provider, spans are still recorded to build Result.Summary, but nothing is exported.

Span kinds

Durations are End - Start. A run's stop reason is one of final, max_iterations, handoff, stopped (a tool called agent.StopRun; the span is not marked as failed), cancelled, guardrail or error.

Processors

• text.New(w, opts...): prints the span tree with colours; text.WithVerbose() also prints start lines, text.WithMaxLen(n) truncates content
• jsonfile.New(path): writes one NDJSON line per span with the full payload and totals; closed by Provider.Shutdown
• otel.New(tp, opts...): exports OpenTelemetry spans following the GenAI semantic conventions
• tracetest.NewRecorder(): in-memory recorder for tests

Pass several trace.WithProcessor options to fan out to several backends. Wrap a processor to change what it sees without affecting the others:

p := trace.NewProvider(
    trace.WithProcessor(text.New(os.Stderr)),
    trace.WithProcessor(trace.Batch(
        trace.Content(trace.ContentPolicy{Capture: true, MaxBytes: 4096}, myExporter),
    )),
    trace.WithProcessor(trace.Redact(func(d *trace.SpanData) { /* mask PII on a copy */ }, other)),
)
defer p.Shutdown(ctx)

Processors are called synchronously; wrap exporters that do I/O in trace.Batch. ContentPolicy clears content, truncates it, or replaces inline images with their type, size and sha256.

OpenTelemetry

import pherootel "github.com/henomis/phero/v2/trace/otel"

exp := pherootel.New(tp) // tp: your OpenTelemetry TracerProvider (SDK + OTLP exporter)
p := trace.NewProvider(
    trace.WithIDSource(exp.IDSource()), // phero and OTel share trace/span IDs
    trace.WithProcessor(exp),
)
a.SetTracing(p)

• Span names and attributes follow semconv v1.41.0: invoke_agent {agent}, chat {model}, execute_tool {name}, gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.*, gen_ai.response.finish_reasons, gen_ai.conversation.id.
• Content is off by default. pherootel.WithContent(true) adds gen_ai.system_instructions, gen_ai.input.messages (the full prompt), gen_ai.output.messages, tool definitions, arguments and results; pherootel.WithMaxAttrBytes(n) sets the size limit (64 KiB by default).
• Run totals are exported as phero.run.* attributes on the run span.

Run summary

Every Result carries a *trace.RunSummary computed from the run span, including the work of sub-agents run as tools:

• StopReason, Iterations, MaxIterations, TraceID, SpanID, SessionID
• LLMCalls, ToolCalls, ToolErrors, MemoryRetrieved, MemorySaved
• Usage: input, output, cache read/write and reasoning tokens, cost in USD
• Latency: total, LLM, tool, memory and time to first token (streaming)
• Tools (calls, errors, duration per tool), ByModel, ByAgent, HandoffAgents, Error

Context: sessions, attributes, propagation

ctx = trace.WithSession(ctx, "conversation-42")              // copied to every span
ctx = trace.WithAttributes(ctx, trace.String("tenant", "acme")) // added to every span

// Across processes (W3C trace context):
header := trace.Inject(ctx)                // traceparent of the current span
ctx = trace.Extract(ctx, incomingHeader)   // later spans become its children

// Correlate logs with traces:
slog.InfoContext(ctx, "step done", slog.Any("trace", trace.LogAttrs(ctx)))

A handoff records a span referenced by Handoff.Span. When you run the target agent outside the same span, link it back with trace.WithLinks(ctx, h.Span).

Trace raw LLM calls

Outside an agent, wrap a client with trace.InstrumentLLM to record a chat span for each call. Inside an agent the wrapper adds nothing, since the agent already records the call.

client := trace.InstrumentLLM(openai.New(apiKey))
res, err := client.Execute(trace.WithProvider(ctx, p), messages)

Try it

# from repo root
go run ./examples/tracing        # text span tree
go run ./examples/tracing-otel   # OpenTelemetry spans with GenAI attributes

Related packages

• agent: records run, model call, tool, handoff and memory spans
• llm: finish reasons, reasoning tokens and the Describer interface feed the spans; middleware records retry, limiter, cache and guardrail activity
• rag: records retrieval and embedding spans