feat(alloy): forward Claude Code traces to Langfuse #208
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "land/alloy-langfuse-traces"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Rebased off current master and re-verified. Original commit b637232 sat unshipped on a local branch since 2026-07-26 (66 commits behind).
Verified:
kubectl kustomize apps/alloybuilds;langfuse-web.langfuse.svc:3000exists in-cluster (46d);sops -d apps/alloy/secret.sops.yamldecrypts.The OTLP receiver has had no `traces` output since it was added, so every span Claude Code sent was accepted and silently dropped. The metrics could tell us a session cost $X and touched Y lines; nothing could answer what it actually did. Langfuse has meanwhile been running for over a week with no Claude Code data in it at all. Claude Code emits spans once CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 and OTEL_TRACES_EXPORTER=otlp are set: claude_code.interaction as the per-prompt root, with .llm_request / .tool / .tool.execution / .hook beneath it. This adds the missing traces pipeline to carry them. Langfuse rather than Tempo because it is already the LLM-observability surface here - apps/litellm forwards gateway traces to the same project via its langfuse_otel callback - and it scores and evaluates traces, which a generic tracing backend does not. Claude Code talks to Anthropic directly rather than through the gateway, so it was the one agent surface Langfuse could not see. Alloy is the junction rather than exporting straight from the workstation: the receiver already terminates OTLP on a hostPort on every node, so this adds no client-side fan-out and keeps the Langfuse credential in the cluster instead of in a shell rc file. Notes for the next person: - otlphttp, not otlp. Langfuse has no gRPC ingestion endpoint; it speaks OTLP over HTTP only. Endpoint is the base /api/public/otel and the exporter appends /v1/traces itself. - Auth is HTTP Basic over base64("pk:sk") plus x-langfuse-ingestion-version. Reuses the existing project keys from apps/litellm - both write to the one langfuse.kblab.me project, so rotate them together. - Alloy resolves sys.env() at config load, so rotating the secret needs a pod restart, not just a secret update. - sending_queue + retry_on_failure are there so a Langfuse outage degrades to dropped traces rather than backpressuring the metrics and logs pipelines that share this receiver. Verified before commit: - Ingestion proven end-to-end: a hand-built span POSTed to /api/public/otel/v1/traces returned HTTP 200 with authCheck.validKey=true and was queued to Langfuse's otel-ingestion-queue. - `kubectl kustomize apps/alloy` builds; the Secret and ConfigMap render. - Config loaded in a real grafana/alloy:v1.5.1 container: both otelcol.processor.batch.langfuse and otelcol.exporter.otlphttp.langfuse evaluate cleanly, so the component arguments are valid on this version. (The only load errors are the expected out-of-cluster ones - kubernetes discovery and /var/log/journal.)