trulens.org

Start with TruLens's out-of-the-box judges, then align them to your domain by supplying your own private data as rubrics and examples, and adjust the score range as needed — TruLens handles the plumbing. The goal is to move from generic benchmarked judges to metrics that reflect your domain's definition of quality, without rebuilding the scoring infrastructure yourself.

What "customizing a judge" actually means

A TruLens judge is a metric implementation that scores a span (a step in your agent's execution). Out of the box, TruLens ships benchmarked judges — for example, groundedness and context relevance — that are graded against human annotations. Customization means keeping that scoring machinery but changing what the judge looks for and how it grades.

The site describes three levers you can pull:

• Rubrics — written criteria that define what a good or bad output looks like in your domain.
• Examples — sample inputs and outputs that anchor the judge's scoring.
• Score range — the scale the judge returns, adjusted with the plumbing already handled.

Step-by-step: aligning a judge to your domain

1. Confirm the baseline judge works first

Before tuning, run the out-of-the-box judge on a sample of your agent's traces. The site's framing is "start with state of the art judges, align judges to your domain with your private data" — so the baseline is the starting point, not something to skip. If the generic judge already scores your domain well, you may only need light tuning.

Expected result: a set of scores on real traces you can compare against your own judgment of which outputs are good and bad.

2. Write a rubric for your domain

A rubric states, in plain terms, what the judge should reward and penalize. For a support agent, that might mean "answers must cite the policy section they rely on" or "do not confirm a refund the customer is not eligible for." The rubric is where domain knowledge enters the judge.

3. Add examples

Pair the rubric with examples: a few inputs with outputs you consider correct and incorrect. Examples resolve ambiguity a rubric alone can't — edge cases, tone, and the specific failure modes your agent exhibits.

4. Adjust the score range

If your team works on a 1–5 scale rather than a probability, or needs a binary pass/fail, set the range. The site notes the plumbing for this is done for you, so you are configuring the judge rather than writing scoring code from scratch.

5. Define the metric in code

The site's example shows the pattern: import Metric and Selector from trulens.core, then construct a metric with an implementation and a selector that determines which part of the trace it scores.

from trulens.core import Metric, Selector

f_relevance = Metric(
    implementation=provider,   # the judge implementation
    # selector determines what the metric is applied to
)

The Selector is what ties a metric to a specific span or field — for example, the retrieved context versus the final answer. This is how you keep a customized judge scoped to the step you care about.

6. Re-score and compare

Run the tuned judge over the same traces you used for the baseline. Compare where scores changed and whether the changes match your own assessment. The site's customization section is framed as tuning "to your domain" — the check is whether the judge now flags the failures you actually see.

Why tuning matters: the reliability argument

TruLens's case for its judges rests on benchmark performance, which is also the reason to tune rather than assume:

These numbers describe the generic judges. They tell you the starting point is strong; they don't tell you the judge understands your domain's rules. That gap is exactly what rubrics, examples, and score-range tuning close.

Common sticking points

• Tuning before baselining. If you customize immediately, you lose the comparison that tells you whether tuning helped. Score with the stock judge first.
• Rubrics that are too vague. "Be helpful" won't change scores. Criteria tied to observable behavior — citations, eligibility checks, required fields — will.
• Examples that contradict the rubric. If your examples reward something the rubric penalizes, the judge gets mixed signals. Keep them consistent.
• Forgetting the selector. A well-tuned judge applied to the wrong span (say, scoring the retrieved context when you meant the final answer) produces misleading results. Check what each metric is scoped to.

When customization is worth it

Customize when your domain has quality rules the generic judges can't infer — regulated answers, house style, or failure modes specific to your agent. If your agent's outputs are already well-scored by the stock judges and your traces show no domain-specific failures, the baseline may be sufficient. The site's own progression — out of the box, then tuned to your domain — suggests treating customization as a second step driven by evidence from your traces, not a default.

An open-source UI element library is a collection of individual, ready-made interface pieces—buttons, cards, inputs, toggles, loaders—that you copy into your own project and adapt. A UI framework, by contrast, is a structured system of components, conventions, and often a theming layer that governs how your whole interface is built. The practical difference: an element library gives you a snippet; a framework gives you a way of working. If you need a polished button in ten minutes, reach for the element library. If you're building a 40-screen product with a team, you probably want the framework.

What "open-source UI element library" actually means

The term gets used loosely, so it helps to separate the parts:

• Open-source: the code is publicly available, and the license tells you what you may do with it—copy, modify, redistribute, or use commercially.
• UI element: a single, self-contained piece of interface, usually small enough to read in one sitting. A button with hover states, a pricing card, a search field.
• Library: a browsable, searchable collection of those elements, typically contributed by many different people.

On a site like Uiverse, elements are shared by a community and written in plain CSS or Tailwind. You find one you like, copy the markup and styles, paste them into your project, and adjust colors, spacing, and text to fit. There's no package to install and no build step required—which is exactly the appeal, and also the source of most of the confusion.

Element library vs. UI framework: the core differences

The table isn't a verdict. It's a map of trade-offs. Element libraries win on speed and freedom; frameworks win on consistency and maintenance.

Licensing and attribution: what to check before you paste

This is where people get into trouble, and it's worth slowing down for.

1. Find the license. Every element or collection should state one. Common open-source licenses include MIT, Apache-2.0, and BSD. Some projects use copyleft licenses like GPL, which can impose obligations if you redistribute your code.
2. Understand what the license permits. MIT and Apache-2.0 are permissive: you can typically use the code in commercial and closed-source projects. Copyleft licenses may require you to release derivative source under the same terms.
3. Check attribution requirements. Permissive licenses usually require you to keep the copyright notice and license text somewhere in your project. That's a real obligation, not a formality.
4. Look for per-element terms. On community sites, the site's overall terms and the individual contributor's stated wishes may differ. If a contributor asks for credit, honor it.
5. When in doubt, ask or avoid. If a snippet has no license at all, you don't have clear permission to reuse it. Treat "no license" as "not open source," even if the code is publicly visible.

This article is general information, not legal advice. For commercial products with real exposure, have someone qualified review the licenses you're relying on.

How to use a community element in your project: a practical workflow

Here's a repeatable process that avoids most of the usual mess.

1. Start from a real need, not a browsing session

Decide what you need first—"a compact primary button with a loading state"—then search. Browsing aimlessly produces a pile of pretty snippets that don't fit together.

2. Copy the smallest version that works

Take the markup and the styles. Strip anything you don't need: demo wrappers, extra animations, decorative layers. Less code means fewer surprises.

3. Convert it to your conventions

If your project uses design tokens or CSS variables, replace hard-coded values:

/* Before: hard-coded */
.button { background: #4f46e5; border-radius: 8px; }

/* After: token-based */
.button { background: var(--color-primary); border-radius: var(--radius-md); }

This one step is what keeps a copied element from looking like a foreign object in your UI.

4. Check accessibility before you ship

Community elements vary widely here. Verify at minimum:

• Keyboard focus is visible and the element is reachable by Tab.
• Color contrast meets WCAG AA (4.5:1 for normal text).
• Interactive elements use semantic HTML (<button>, not a clickable <div>).
• Form inputs have associated labels.
• Motion respects prefers-reduced-motion.

5. Test in context

Paste it into a real page with real content. Long labels, small screens, and dark mode break more copied elements than anything else.

6. Note where it came from

Keep a short comment or an internal credits file: source, license, date. Future you—and your legal reviewer—will be grateful.

Where element libraries genuinely shine

• Prototypes and demos: you need something clickable today, not a design system.
• Landing pages and marketing sites: a handful of distinctive elements, each custom.
• Filling gaps: your framework lacks one specific component, and you don't want to build it from scratch.
• Learning: reading well-made CSS is one of the fastest ways to improve.
• Small projects: a personal site doesn't need a theming architecture.

Where they fall short

• Consistency at scale: ten elements from ten contributors rarely look like one product.
• Maintenance: you own every copy. When your design changes, you edit each one.
• Accessibility debt: you inherit whatever the contributor did or didn't do.
• No upstream fixes: a bug fixed in the original won't reach your copy.
• Integration friction: different naming conventions, different units, different assumptions about resets.

When to choose which

Choose an element library when the scope is small, the timeline is short, or you need a few distinctive pieces rather than a whole system.

Choose a framework or design system when multiple people build multiple screens over months, when consistency is a product requirement, or when accessibility and theming need to be guaranteed rather than checked.

A hybrid works well for many teams: adopt a framework for the structural components—forms, navigation, layout—and borrow individual elements for the places where you want personality. Just route every borrowed element through the same token and accessibility checks, so it lands as part of your system rather than beside it.

The short version: open-source UI element libraries are a fast, flexible way to get good-looking interface pieces into a project. They are not a substitute for a design system, and the license and accessibility details are the part worth reading carefully.

TruLens scores every step of an AI agent—not just the final answer—by attaching LLM-judge feedback to each span in an OpenTelemetry trace, and it reports that its judges are benchmarked against human annotations. Reliability is best judged by those published benchmarks: the vendor reports 95% of agent errors caught with Agent GPA on TRAIL/GAIA (267 of 281 human-annotated errors, versus 55% for a baseline trace judge), a 0.81 groundedness F1 on an 11,000-example LLM-AggreFact holdout, and 0.93 context relevance NDCG@5 in an AIMultiple comparison. Treat these as vendor-reported figures and validate them on your own domain before relying on them for release decisions.

How scoring works: judges attached to every span

TruLens instruments your agent with OpenTelemetry and records latency, inputs, outputs, tokens, and cost per step. Each step (span) can then carry a score from an LLM judge, so a bad final answer has a traceable cause rather than an unexplained outcome.

Two design choices matter for interpreting a score:

• Scores are per step, not per conversation. A low score on a retrieval span points to retrieval; a low score on a synthesis span points to generation.
• Judges explain the score. The feedback includes the judge's reasoning, so a low score tells you what went wrong, not only that something did.

The site's example shows a support-agent trace (v1-passthrough, record c17f0a94-…) with tree, timeline, and raw-attribute views alongside feedback results per metric.

What the benchmarks say about reliability

TruLens publishes several comparisons. The table below summarizes them with their stated conditions.

How to read these numbers:

• They are benchmark-specific. TRAIL/GAIA, LLM-AggreFact, and the AIMultiple harness are not your traffic. High scores there indicate the judges are well-calibrated on those distributions, not that they will catch your agent's failure modes.
• The adversarial result is the most decision-relevant for RAG. A 4.2:1 win-loss on near-copies with one swapped fact suggests the judge is sensitive to small factual differences—the failure mode that matters most in retrieval-grounded answers.
• The baseline comparison is the useful signal. 95% vs. 55% on the same traces says more about judge quality than the absolute 95%.

Tuning judges to your domain

Out-of-the-box judges are the starting point; the site describes aligning them to your data by adding a rubric, examples, and adjusting the score range, with the plumbing handled for you. The excerpt shows the entry point as a Metric object built from a provider implementation and a selector, e.g.:

from trulens.core import Metric, Selector

f_relevance = Metric(
    implementation=provider,  # judge implementation
    # selector + rubric/examples configure what is scored
)

The practical implication: if your domain has conventions the default judge misreads (internal jargon, domain-specific correctness), supply examples and a rubric rather than accepting raw scores. Re-check the benchmark-style behavior on a small labeled set of your own traces after tuning.

Deciding whether to trust the scores

Use this sequence before letting judge scores gate a release:

1. Label a sample yourself. Take 30–50 traces, mark which steps are actually wrong, and compare against judge output. This is the same method behind the 267-of-281 figure, applied to your data.
2. Check the explanation, not just the number. If the judge's stated reason for a low score doesn't match the span, the score is noise for that case.
3. Test near-miss cases. Include answers that are almost right but contain one wrong fact; the adversarial benchmark suggests this is where judge quality separates.
4. Tune, then re-measure. Add rubric and examples for the failure patterns you found, and confirm the labeled-set agreement improved.
5. Use scores for triage, not verdicts. Per-step scores are strongest at pointing to where to look; the final ship decision should combine them with your labeled evidence.

If your agent's failures are concentrated in retrieval grounding, the published groundedness and context-relevance results are the most relevant evidence. If failures are in multi-step tool use, the TRAIL/GAIA agent-error result is closer—but still validate on your own traces, since that benchmark's error distribution may not match yours.

TruLens is an open-source, OpenTelemetry-native library for evaluating and tracing AI agents. It instruments your agent so that every step is recorded as a trace span, scores those steps with benchmarked LLM judges, and helps you decide which version to ship. You can use it if you work with LLM agents or RAG pipelines and want per-step visibility into failures, latency, token use, and cost. The project is distributed via GitHub, and the site's "Get started" and "Star on GitHub" links point there. The page does not list pricing, so treat it as an open-source project rather than assuming a paid tier or a free hosted service.

What TruLens actually does

TruLens addresses a specific problem: agents fail differently than normal software, so you need traces and evals to find out why. It provides two connected capabilities.

Tracing: every span understood

TruLens records latency, inputs, outputs, tokens, and cost per step. A bad answer therefore has a traceable cause rather than an unexplained output. The site shows a trace view with tree, timeline, details, and raw attributes for a support agent run, which is the practical shape of the output you get: a per-step record you can inspect.

Evaluation: scores on every step

Each step receives a score, and the judges explain the score so a low result points to what went wrong. TruLens ships benchmarked judges and lets you tune them to your domain with private data by adding a rubric, examples, and an adjusted score range.

Integration model: OpenTelemetry-native

The key integration fact is that TruLens instruments agents with OpenTelemetry. That means tracing follows an open standard rather than a proprietary format, which matters if your stack already collects telemetry. The page describes TruLens as "Open source, OpenTelemetry-native" and frames it for AI agents and RAG pipelines generally; it does not enumerate specific framework connectors, so check the GitHub repository for the current list of supported integrations before committing.

How reliable are the judges?

The site makes several benchmark claims, all attributed to external evaluations:

These are vendor-published claims with cited sources, so they are checkable rather than asserted without support. They are still benchmark results on specific datasets; your domain may behave differently, which is exactly why the customization path exists.

Customizing judges for your domain

The documented approach is to start from the built-in judges and align them to your data. The page shows a code snippet beginning with from trulens.core import Metric, Selector and defining a metric such as f_relevance = Metric(...), indicating that custom metrics are defined in code with a selector choosing what to score. The page excerpt cuts off before the full implementation, so consult the documentation for the complete signature and provider setup.

When TruLens is a good fit

• You need to debug agent failures, not just log outputs. Per-step traces with inputs, outputs, tokens, and cost give you a cause, not just a symptom.
• You want evaluation tied to traces. Scores attached to each span let a low score point to a specific step.
• You prefer open standards. OpenTelemetry-native instrumentation avoids locking tracing into a proprietary format.
• You need domain-specific judges. Rubrics, examples, and score-range adjustments are supported.

If you only need aggregate quality scores without step-level tracing, or if you require a managed hosted service with published pricing, the page does not confirm that TruLens provides it—verify on GitHub and in the docs first.

TruLens traces every step of an AI agent—each span records latency, inputs, outputs, tokens, and cost—so a bad final answer can be traced back to the specific step that caused it. It is built for teams debugging agents that fail in ways normal software doesn't, and it is open source and OpenTelemetry-native. Use it when you need per-step visibility rather than only an end-output check; the tracing value depends on your agent being instrumented so its actions are captured as spans.

Why agent failures need step-level traces

TruLens frames the core problem directly: agents fail differently than normal software. A wrong answer at the end may originate in retrieval, a tool call, a prompt, or a model response several steps earlier. Checking only the final output tells you that something went wrong, not where. Tracing closes that gap by recording what happened at each step, so a bad answer has a traceable cause.

What each trace captures

Per the site, every span is recorded with:

• Latency — how long each step took
• Inputs and outputs — what went into and came out of each step
• Tokens and cost — consumption recorded per step

Because these are attached per step, you can compare a failing run against a healthy one and see which span diverged—and whether the fix is a quality problem or a cost problem.

Reading a trace: tree and timeline views

TruLens presents traces in two complementary views:

The trace detail view also exposes raw attributes, so you can inspect the underlying recorded data for a span rather than relying only on the summary.

Connecting scores to steps

Tracing alone shows what happened; scores show how good it was. TruLens scores every step, and its judges explain the score—so a low score points to exactly what went wrong rather than just flagging a bad output. The site reports that its judges are benchmarked against human annotations out of the box, including:

• 95% of agent errors caught with Agent GPA on TRAIL/GAIA (267 of 281 human-annotated errors, versus 55% for the baseline trace judge), per arXiv:2510.08847
• 0.81 Groundedness F1 on an LLM-AggreFact holdout
• 0.93 Context relevance NDCG@5 in an AIMultiple benchmark (23 March 2026)

Treat these as the vendor's cited benchmarks, not guarantees for your workload—your agent's domain and data determine how well the judges transfer.

Tuning judges to your domain

Out-of-the-box judges are the starting point. TruLens lets you align them to your domain with your private data: add a rubric, provide examples, and adjust the score range, with the plumbing handled for you. The site's example begins with importing Metric and Selector from trulens.core to define a custom metric. This matters because a generic judge may not recognize what "correct" means in your specific agent—tuning is how you make the failure signal trustworthy.

A practical way to use it

1. Instrument the agent so its actions are emitted as OpenTelemetry spans.
2. Reproduce a failure and open its trace.
3. Use the tree view to locate the branch where behavior diverged, and the timeline to check for slow or out-of-order steps.
4. Read the per-step scores and judge explanations to identify the specific step and reason for the failure.
5. Tune the judge with your own rubric and examples if the score doesn't match your notion of quality.
6. Re-run and compare traces to confirm the fix and check whether cost can be cut without losing quality.

Where it fits and what to check

TruLens is aimed at teams that need both tracing and evals for agents and want an open-source, OpenTelemetry-native option. Before adopting, confirm that your stack can emit OpenTelemetry spans and that the benchmarked judges perform well on samples from your own domain—the published numbers are evidence of capability, not a substitute for validating on your data. Pricing and access details beyond "open source" are not specified on the page, so verify those directly if they affect your decision.