Skip to main content

Tracing

Goal: report every ticket to Langfuse or an OpenTelemetry collector.

Prerequisites​

  • A reachable Langfuse 4 instance, or an OpenTelemetry collector that accepts OTLP/HTTP JSON.
  • The host and any keys the tracer needs, in environment variables.

Steps​

  1. Name a tracer in the Runtime manifest, so every ticket is reported to an observability backend. type names a plugin's tracers port.

  2. To send OTLP/HTTP to a Langfuse 4 instance, use the afe-langfuse plugin. The three variables name the host and the project keys, never their values:

    spec:
      tracer:
        type: langfuse
        hostEnv: LANGFUSE_HOST
        publicKeyEnv: LANGFUSE_PUBLIC_KEY
        secretKeyEnv: LANGFUSE_SECRET_KEY

  3. To send the same spans to any OpenTelemetry collector over OTLP/HTTP JSON, at <endpoint>/v1/traces, use the afe-otlp plugin. Tempo, Jaeger and most vendor agents accept it.

    spec:
      tracer:
        type: otlp                                  # reads the standard exporter variables
        endpointEnv: OTEL_EXPORTER_OTLP_ENDPOINT    # the default, e.g. http://otel-collector:4318
        headersEnv: OTEL_EXPORTER_OTLP_HEADERS      # the default, optional: key=value,key=value

A missing variable or an unreachable host stops the process before any run. The keys never appear in an error or a log.

Notes​

  • A tracer takes only the fields of its type. Header values go only into the request, never into a span, an error or a log. afe serve opens the tracer too.
  • A ticket is one trace, whichever processes ran it. The ticket span is the root. Each round is a span under it, each node a span under its round, and each model call and tool call a span under its node. Every span carries tokens, cost and duration. The engine computes the span ids from the ticket, so a ticket resumed on another worker continues the same tree. In Langfuse the ticket is also the session.
  • afe serve records a span for each ticket.start, ticket.resume and ticket.cancel. The start span is the parent of the ticket span. Resume and cancel sit under the ticket. When the caller passes a W3C traceparent to ticket.start, the whole ticket joins the caller's trace. An invalid one is ignored, and a new trace starts instead.
  • Every text — prompts, replies, tool arguments and results — passes through the engine's redaction before it leaves the process. Without a tracer, nothing is sent. A failing tracer is logged and never fails a ticket.

Troubleshooting​

runtime: set LANGFUSE_HOST (or LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, OTEL_EXPORTER_OTLP_ENDPOINT) One of the tracer's required variables is not set in the process environment. Export it before afe run or afe serve starts. The engine checks every required variable before opening the tracer.

runtime: cannot open the tracer at <host> (ConnectError) The host resolved, but the process could not reach it: the wrong host, a closed port, or a network policy. The message never includes a key, only the exception class and the host.

No spans appear in Langfuse or the collector A startup failure would have stopped the process, so the tracer opened. The send is failing silently instead. A tracer failure never fails a ticket. Look at the worker's log for a warning from afe.langfuse or afe.otlp.

See also​