How to Customize TruLens Judges for Your Domain

Start with TruLens's out-of-the-box judges, then align them to your domain by supplying your own private data as rubrics and examples, and adjust the score range as needed — TruLens handles the plumbing. The goal is to move from generic benchmarked judges to metrics that reflect your domain's definition of quality, without rebuilding the scoring infrastructure yourself.

What "customizing a judge" actually means

A TruLens judge is a metric implementation that scores a span (a step in your agent's execution). Out of the box, TruLens ships benchmarked judges — for example, groundedness and context relevance — that are graded against human annotations. Customization means keeping that scoring machinery but changing what the judge looks for and how it grades.

The site describes three levers you can pull:

• Rubrics — written criteria that define what a good or bad output looks like in your domain.
• Examples — sample inputs and outputs that anchor the judge's scoring.
• Score range — the scale the judge returns, adjusted with the plumbing already handled.

Step-by-step: aligning a judge to your domain

1. Confirm the baseline judge works first

Before tuning, run the out-of-the-box judge on a sample of your agent's traces. The site's framing is "start with state of the art judges, align judges to your domain with your private data" — so the baseline is the starting point, not something to skip. If the generic judge already scores your domain well, you may only need light tuning.

Expected result: a set of scores on real traces you can compare against your own judgment of which outputs are good and bad.

2. Write a rubric for your domain

A rubric states, in plain terms, what the judge should reward and penalize. For a support agent, that might mean "answers must cite the policy section they rely on" or "do not confirm a refund the customer is not eligible for." The rubric is where domain knowledge enters the judge.

3. Add examples

Pair the rubric with examples: a few inputs with outputs you consider correct and incorrect. Examples resolve ambiguity a rubric alone can't — edge cases, tone, and the specific failure modes your agent exhibits.

4. Adjust the score range

If your team works on a 1–5 scale rather than a probability, or needs a binary pass/fail, set the range. The site notes the plumbing for this is done for you, so you are configuring the judge rather than writing scoring code from scratch.

5. Define the metric in code

The site's example shows the pattern: import Metric and Selector from trulens.core, then construct a metric with an implementation and a selector that determines which part of the trace it scores.

from trulens.core import Metric, Selector

f_relevance = Metric(
    implementation=provider,   # the judge implementation
    # selector determines what the metric is applied to
)

The Selector is what ties a metric to a specific span or field — for example, the retrieved context versus the final answer. This is how you keep a customized judge scoped to the step you care about.

6. Re-score and compare

Run the tuned judge over the same traces you used for the baseline. Compare where scores changed and whether the changes match your own assessment. The site's customization section is framed as tuning "to your domain" — the check is whether the judge now flags the failures you actually see.

Why tuning matters: the reliability argument

TruLens's case for its judges rests on benchmark performance, which is also the reason to tune rather than assume:

These numbers describe the generic judges. They tell you the starting point is strong; they don't tell you the judge understands your domain's rules. That gap is exactly what rubrics, examples, and score-range tuning close.

Common sticking points

• Tuning before baselining. If you customize immediately, you lose the comparison that tells you whether tuning helped. Score with the stock judge first.
• Rubrics that are too vague. "Be helpful" won't change scores. Criteria tied to observable behavior — citations, eligibility checks, required fields — will.
• Examples that contradict the rubric. If your examples reward something the rubric penalizes, the judge gets mixed signals. Keep them consistent.
• Forgetting the selector. A well-tuned judge applied to the wrong span (say, scoring the retrieved context when you meant the final answer) produces misleading results. Check what each metric is scoped to.

When customization is worth it

Customize when your domain has quality rules the generic judges can't infer — regulated answers, house style, or failure modes specific to your agent. If your agent's outputs are already well-scored by the stock judges and your traces show no domain-specific failures, the baseline may be sufficient. The site's own progression — out of the box, then tuned to your domain — suggests treating customization as a second step driven by evidence from your traces, not a default.