What is TruLens and what does it do for AI agents?

TruLens is an open-source, OpenTelemetry-native tool for evaluating and tracing AI agents. It records every step an agent takes (latency, inputs, outputs, tokens, and cost per span), scores each step with benchmarked LLM judges that explain their scores, and helps you decide which agent version to ship. It fits teams building or maintaining agents who need to find where an agent fails and where cost can be cut without losing quality — not teams looking for a general-purpose APM or a hosted-only platform.

The problem it targets

Agents fail differently than normal software: a bad answer can come from a retrieval step, a tool call, a prompt, or a model choice, and without a per-step record you can't tell which. TruLens is built around that gap — traces to locate the cause, evals to quantify it.

Tracing: every span, understood

TruLens instruments your agent with OpenTelemetry and records, per step:

• latency
• inputs and outputs
• tokens
• cost

The trace view presents this as a tree/timeline with raw attributes available, so a bad answer has a traceable cause rather than a vague "the agent was wrong." Because it is OpenTelemetry-native, the instrumentation follows an open standard rather than a proprietary format.

Evaluation: scores on every step, with explanations

Each step is scored by LLM judges, and the judge explains the score — so a low score points to exactly what went wrong rather than just flagging that something did. Judges are benchmarked against human annotations out of the box, and can then be tuned to your domain: add a rubric, add examples, or adjust the score range, with the plumbing handled for you.

Benchmarks TruLens reports

These are vendor-published figures; treat them as a starting point for your own comparison rather than a substitute for testing on your data.

Customizing judges to your domain

Out-of-the-box judges are the starting point. To align them with your domain, you supply your private data as a rubric and/or examples and adjust the score range; the surrounding plumbing is handled for you. The site illustrates the entry point with a Metric and Selector import from trulens.core, e.g. defining a relevance metric with a provider implementation. The practical sequence is: run the default judges, inspect where their scores disagree with your own judgment, then add rubric/examples until they align.

When TruLens fits — and when it may not

Choose TruLens if:

• You are building or iterating on an agent and need per-step traces plus scored evals in one place.
• You want open-source tooling and OpenTelemetry-based instrumentation you can integrate with existing observability.
• You need judges you can tune to a specific domain rather than a fixed scoring scheme.
• You want evidence for a ship/no-ship decision between agent versions, including cost trade-offs.

Look elsewhere or verify first if:

• You need a fully managed, hosted evaluation service — the site describes the project as open source but does not state hosting or pricing terms, so confirm deployment and cost details directly.
• You need evaluation for non-agent systems only; the focus here is agent traces and step-level scoring.
• You require benchmark results independent of the vendor; the published numbers above come from TruLens and its cited third-party benchmarks.

Getting started

The site's entry point is "Get started" plus a GitHub star link, and it references a helpdesk RAG case study as a worked example. A reasonable first pass: instrument one agent with the OpenTelemetry integration, run a handful of representative tasks, review the trace tree to confirm spans capture the steps you care about, then check whether default judge scores match your own assessment before tuning. Common sticking points to plan for are ensuring every meaningful step (retrieval, tool calls, generation) is captured as a span, and validating judge alignment on your domain before trusting scores for release decisions.