LLM

A minimal chat interface + first-class function tools.

What it is

The llm package is the thin waist of Phero: a minimal interface for chat models plus a small tool system. Higher-level packages (like agent) build on this to orchestrate multi-turn loops and tool execution.

All message and content types are defined in this package — no dependency on any specific provider SDK is required.

Using an LLM backend

Any backend that implements llm.LLM can power agents and tools. Phero includes an OpenAI-compatible client at llm/openai and an Anthropic Messages API client at llm/anthropic and a Gemini API client at llm/google.

Call options: tools, tool choice and structured output

Execute(ctx, messages, opts...) takes the conversation plus any number of llm.CallOptions. Everything that belongs to a single call travels this way, so several agents can share one client with different tools and output contracts: 

res, err := client.Execute(ctx, msgs,
    llm.WithTools(search, fetch),
    llm.WithToolChoice(llm.ToolChoiceRequired),
)

Build a response format from a Go type with llm.NewResponseFormat[T]. The schema is generated the same way as tool schemas and normalized to the strict form OpenAI and Anthropic both require. Use llm.NewRawResponseFormat when you already have the schema as a map. The answer's text is a JSON document you decode into T: 

type CV struct {
    Name   string   `json:"name"`
    Skills []string `json:"skills"`
}

cvFormat, err := llm.NewResponseFormat[CV]("cv", "a parsed curriculum vitae")
if err != nil {
    panic(err)
}

res, err := extractor.Execute(ctx, msgs, llm.WithResponseFormat(cvFormat))
if err != nil {
    panic(err)
}

var cv CV
if err := json.Unmarshal([]byte(res.Message.TextContent()), &cv); err != nil {
    panic(err)
}

Backends check the options before sending anything. An inconsistent combination, such as ToolChoiceRequired with no tools or a forced tool that is not offered, fails locally with llm.ErrInvalidToolChoice instead of an opaque provider error. Code that implements or wraps llm.LLM reads the options with llm.NewCallConfig(opts...).

Upgrading from v1

In v1 tools were a positional argument: Execute(ctx, msgs, tools). Callers now write Execute(ctx, msgs, llm.WithTools(tools...)), or simply Execute(ctx, msgs) instead of passing nil. A custom backend still written against the v1 signature can be wrapped with llm.FromLegacy(old). It receives the tools, and a call that also sets a tool choice or a response format fails with llm.ErrUnsupportedCallOption rather than silently dropping it.

Anthropic backend

llm/anthropic implements llm.LLM using Anthropic's Messages API. The same Phero message/tool types are used at the boundary, so the rest of the framework (agents, tools, memory) doesn't need to change. 

import (
    "os"

    "github.com/henomis/phero/v2/llm/anthropic"
)

llmClient := anthropic.New(
    os.Getenv("ANTHROPIC_API_KEY"),
    anthropic.WithModel("claude-sonnet-4-6"),
    anthropic.WithMaxTokens(2048),
)

If you pass an empty API key, the underlying Anthropic SDK will fall back to its environment variable configuration.

Available options:

llmClient := anthropic.New(
    os.Getenv("ANTHROPIC_API_KEY"),
    anthropic.WithModel("claude-sonnet-4-6"),
    anthropic.WithPromptCaching(),
    anthropic.WithThinking(2048),
)

result, _ := llmClient.Execute(ctx, messages, llm.WithTools(tools...))
fmt.Println("reasoning:", result.Message.ReasoningContent())
fmt.Println("answer:", result.Message.TextContent())

On models where the fixed thinking budget has been removed, adaptive thinking plus an effort level is the equivalent configuration: 

llmClient := anthropic.New(
    os.Getenv("ANTHROPIC_API_KEY"),
    anthropic.WithModel("claude-opus-4-5"),
    anthropic.WithPromptCaching(),
    anthropic.WithAdaptiveThinking(),
    anthropic.WithEffort(anthropic.EffortXHigh),
)

Google backend

llm/google implements llm.LLM and llm.StreamingLLM using the Gemini API, through Google's official Go SDK (google.golang.org/genai). Tools, tool choice, structured output and streaming work as with the other backends. 

import (
    "os"

    "github.com/henomis/phero/v2/llm/google"
)

llmClient := google.New(
    os.Getenv("GEMINI_API_KEY"),
    google.WithModel("gemini-3.5-flash"),
)

If you pass an empty API key, the SDK falls back to the GOOGLE_API_KEY or GEMINI_API_KEY environment variable.

Gemini 3 models attach a thought signature to function calls and reject a conversation that does not send it back on the same call. The client keeps each signature in a reasoning part of the assistant message (with empty text, so it never shows in ReasoningContent() or TextContent()) and puts it back on the next request, so agents keep working across tool turns without any extra code.

Available options:

Token usage maps cached prompt tokens to CacheReadTokens and counts thinking tokens as output. API errors are wrapped in *llm.ProviderError, with RetryAfter read from Gemini's RetryInfo. Images must be inline (llm.ImageBase64 / llm.ImageFile): the Gemini API does not fetch image URLs.

OpenAI backend

llm/openai implements llm.LLM using the OpenAI Chat Completions API. It also implements llm.Transcriber and llm.SpeechSynthesizer for audio (see below). 

import (
    "os"
    "github.com/henomis/phero/v2/llm/openai"
)

llmClient := openai.New(
    os.Getenv("OPENAI_API_KEY"),
    openai.WithModel("gpt-4o"),
    openai.WithTemperature(0.7),
)

Available options:

Messages and content parts

Every message is an llm.Message containing a Role and a slice of llm.ContentPart values. A content part is either text or an image (URL or base64-encoded bytes). 

// Plain text
llm.Text("Hello, world!")

// Image by URL
llm.ImageURL("https://example.com/photo.png")

// Image from a local file (MIME type is detected automatically)
part, err := llm.ImageFile("/path/to/photo.jpg")

// Image as raw base64 bytes
llm.ImageBase64("image/png", base64EncodedData)

Role constants and message constructors:

// Role constants
llm.RoleSystem    // "system"
llm.RoleUser      // "user"
llm.RoleAssistant // "assistant"
llm.RoleTool      // "tool"

// Constructors
llm.SystemMessage("You are a helpful assistant.")
llm.UserMessage(llm.Text("Hello!"))
llm.UserMessage(llm.Text("Describe this image:"), llm.ImageURL("https://..."))
llm.AssistantMessage([]llm.ContentPart{llm.Text("Hi!")})
llm.ToolResultMessage(toolCallID, llm.Text("42"))

When a tool fails, set ToolError on its result message so the model is told the call errored rather than succeeded. The agent loop does this automatically — a handler returning an error yields a tool-result message with ToolError: true carrying the error text — but you can set the field yourself when constructing messages by hand. 

msg := llm.ToolResultMessage(toolCallID, llm.Text("division by zero"))
msg.ToolError = true

To extract the plain text from a message or from loose parts, use TextContent: 

// From a message
text := msg.TextContent()

// From loose content parts
text := llm.TextContent(parts...)

Multimodal input

Pass image parts alongside text to send multimodal messages to vision-capable models. The agent's Run method accepts variadic ContentPart values: 

imagePart, err := llm.ImageFile("screenshot.png")
if err != nil {
    panic(err)
}

result, err := a.Run(ctx,
    llm.Text("What does this image show?"),
    imagePart,
)
if err != nil {
    panic(err)
}
fmt.Println(result.TextContent())

See examples/multimodal for a complete working example.

LLM middleware

Just as tools support middleware, the llm package provides a Middleware type and a llm.Use function to compose decorators around any llm.LLM. This is the right place to add caching, rate limiting, logging, or automatic retries without modifying individual backends. 

// A simple logging middleware
func loggingMiddleware(next llm.LLM) llm.LLM {
    return llm.Func(func(ctx context.Context, msgs []llm.Message, opts ...llm.CallOption) (*llm.Result, error) {
        fmt.Printf("calling LLM with %d messages\n", len(msgs))
        result, err := next.Execute(ctx, msgs, opts...)
        fmt.Printf("LLM responded; err=%v\n", err)
        return result, err
    })
}

// Wrap a base client with one or more middlewares
base := openai.New(os.Getenv("OPENAI_API_KEY"))
wrapped := llm.Use(base, loggingMiddleware)

Middlewares are applied in declaration order: llm.Use(base, m1, m2) means m1 is outermost and runs first. A middleware should pass opts... through unchanged; one that needs to look at the tools or the response format reads them with llm.NewCallConfig(opts...). See examples/llm-middleware for a full example.

The llm/middleware package ships ready-made middlewares: NewRetry (exponential back-off), NewRateLimit, NewGuardrails, and NewSemanticCache.

Provider errors

When a provider's API answers with an HTTP error status, the OpenAI and Anthropic backends return an *llm.ProviderError. It carries the Provider, the StatusCode and, when the provider says how long to wait, RetryAfter (Anthropic only: the OpenAI SDK does not expose response headers). The SDK's own error stays reachable underneath, and the error text is unchanged. Code that may not import a provider SDK, such as a retry policy or the NATS server choosing an error code, can still tell a rate limit from a bad API key: 

if pe, ok := errors.AsType[*llm.ProviderError](err); ok && pe.StatusCode == 429 {
    time.Sleep(pe.RetryAfter)
}

Errors that never reached the API, such as a network failure or an invalid request built locally, are returned as they are.

Semantic response caching

middleware.NewSemanticCache wraps any backend with a cache keyed by the semantic similarity of the conversation rather than an exact string match. Each request is embedded with an embedding.Embedder and looked up in a vectorstore.Store; when the nearest neighbour's cosine similarity meets the configured threshold, the stored response is returned without calling the model. Cache failures degrade gracefully to a normal, uncached call. 

cacheMW, err := middleware.NewSemanticCache(
    embedder, // embedding.Embedder
    store,    // vectorstore.Store
    middleware.WithSimilarityThreshold(0.97),
)
if err != nil {
    panic(err)
}

client := llm.Use(base, cacheMW)

By default a cache hit reports zero token usage (no model call was made), keeping cost accounting truthful, and requests that carry tools bypass the cache so cached tool calls are never silently replayed. Both behaviors are configurable via WithReportCachedUsage and WithSkipToolCalls.

Similarity only decides whether two conversations are the same. The shape of the answer must match exactly: each entry records the call's response format and tool choice, and a lookup only considers entries stored under the same ones. A free-text answer is never served to a call that asked for a schema, however close the conversations are.

Audio: transcription and speech

The llm/openai client also implements llm.Transcriber and llm.SpeechSynthesizer, so the same client used for chat can also transcribe audio and synthesize speech. 

// Speech-to-text (Transcriber)
result, err := llmClient.Transcribe(ctx, llm.TranscriptionRequest{
    Input: llm.AudioFile("recording.mp3"),
})
fmt.Println(result.Text)

// Text-to-speech (SpeechSynthesizer)
speech, err := llmClient.SynthesizeSpeech(ctx, llm.SpeechRequest{
    Input:  "Hello from Phero!",
    Format: llm.SpeechResponseFormatMP3,
})
// speech.Data holds the raw MP3 bytes; speech.MIMEType is "audio/mpeg"

See examples/audio for a runnable example.

Function tools

The main way you integrate capabilities is via function tools. In examples/conversational-agent, a get_current_time tool is exposed to the agent. 

type TimeInput struct{}

type TimeOutput struct {
    CurrentTime string `json:"current_time" jsonschema:"description=The current local time in RFC3339 format"`
}

func getCurrentTime(_ context.Context, _ *TimeInput) (*TimeOutput, error) {
    return &TimeOutput{CurrentTime: time.Now().Format(time.RFC3339)}, nil
}

tool, err := llm.NewTool(
    "get_current_time",
    "Get the current local time",
    getCurrentTime,
)
if err != nil {
    panic(err)
}

Tools are added to an agent with AddTool, and the agent will run them when the model requests a tool call.

Tool middleware

Tools support middleware via tool.Use(...). This is the place to add validation, permission checks, logging, or other cross-cutting behavior without baking it into each tool handler. 

timeTool, err := llm.NewTool(
    "get_current_time",
    "Get the current local time",
    getCurrentTime,
)
if err != nil {
    panic(err)
}

timeTool.Use(func(tool *llm.Tool, next llm.ToolHandler) llm.ToolHandler {
    return func(ctx context.Context, arguments string) (any, error) {
        fmt.Printf("running %s with args %s\n", tool.Name(), arguments)
        return next(ctx, arguments)
    }
})

Middleware order is preserved: if you call tool.Use(m1, m2), then m1 runs before m2. This replaces older per-tool validation helpers and keeps approval logic at wiring time.

Tracing raw LLM calls

The trace package can wrap any llm.LLM with trace.InstrumentLLM to record a chat span for each direct Execute call made without an agent. 

import (
    "github.com/henomis/phero/v2/trace"
    "github.com/henomis/phero/v2/trace/text"
)

p := trace.NewProvider(trace.WithProcessor(text.New(os.Stderr)))
traced := trace.InstrumentLLM(llmClient)

result, err := traced.Execute(trace.WithProvider(ctx, p), messages, llm.WithTools(tools...))

Inside an agent the wrapper adds nothing: the agent already records each call as a span.

Token usage and cost

Every Result carries Usage{InputTokens, OutputTokens, CacheReadTokens, CacheWriteTokens, ReasoningTokens}, the resolved Model name and a normalised FinishReason (stop, length, tool_calls, content_filter, error; the provider's value is in RawFinishReason). ReasoningTokens is a subset of OutputTokens, and cached input is reported only in CacheReadTokens, so it is billed once. Call usage.Cost(model) for a best-effort US-dollar estimate using a built-in per-model pricing table; unknown models return 0. Override or add rates with llm.RegisterPricing. Agents aggregate this into RunSummary.Usage.CostUSD. 

// Override or add pricing (USD per 1M tokens).
llm.RegisterPricing("my-local-model", llm.Pricing{
    InputPer1M:  0.50,
    OutputPer1M: 1.50,
})

cost := result.Usage.Cost(result.Model)
fmt.Printf("this call cost $%.4f\n", cost)

Streaming

Backends that support incremental responses implement llm.StreamingLLM (both the OpenAI and Anthropic clients do). Use llm.StreamOrBuffer to consume any backend uniformly: it streams when the client supports it and otherwise falls back to a single buffered chunk. Each StreamChunk carries a TextDelta (or ReasoningDelta); the terminal chunk (Done) holds the complete Message, Usage, and Model. 

for chunk, err := range llm.StreamOrBuffer(ctx, llmClient, messages, llm.WithTools(tools...)) {
    if err != nil {
        panic(err)
    }
    fmt.Print(chunk.TextDelta) // print tokens as they arrive
    if chunk.Done {
        fmt.Printf("\n[tokens in=%d out=%d]\n", chunk.Usage.InputTokens, chunk.Usage.OutputTokens)
    }
}

At the agent level, use Agent.RunStream (see the agent docs) for streamed text and tool events.

Putting it together

The minimal loop is: create an LLM client, register one or more tools, then run an agent. This is the core pattern used throughout examples/. 

# from repo root
go run ./examples/simple-agent

go run ./examples/conversational-agent

Related packages