What Evaluation Metrics Does Ragas Provide?

Updated

Ragas provides four families of evaluation metrics: RAG metrics (context precision, context recall, faithfulness, response relevancy, and others), agent/tool-use metrics (tool call accuracy, tool call F1, agent goal accuracy, topic adherence), general-purpose LLM-judged metrics (aspect critic and rubric-based scoring), and traditional non-LLM metrics (BLEU, ROUGE, CHRF, exact match, string similarity). Which set you use depends on what you are evaluating: a retrieval pipeline, an agent's tool behavior, open-ended output quality, or similarity to a reference answer.

RAG metrics

These are the core of Ragas and target retrieval-augmented generation systems, where you need to judge both the retrieved context and the generated answer.

Metric What it measures
Context Precision Whether relevant chunks in the retrieved context are ranked near the top
Context Recall Whether the retrieved context covers the information needed to answer
Context Entities Recall Whether key entities from the reference are present in the retrieved context
Noise Sensitivity How much incorrect or irrelevant content in the context affects the answer
Response Relevancy Whether the answer actually addresses the question asked
Faithfulness Whether claims in the answer are supported by the retrieved context

A practical split: context precision, context recall, and context entities recall evaluate the retriever; faithfulness and response relevancy evaluate the generator. Noise sensitivity sits between them, showing how robust the pipeline is to imperfect retrieval. If you are diagnosing a RAG app, running retrieval metrics and generation metrics separately tells you which half to fix.

Agent and tool-use metrics

For agents that call tools or take multi-step actions, Ragas offers metrics aimed at the trajectory rather than a single answer:

  • Tool Call Accuracy — whether the agent called the right tools.
  • Tool Call F1 — a precision/recall-style balance over tool calls.
  • Agent Goal Accuracy — whether the agent actually achieved the user's goal.
  • Topic Adherence — whether the agent stayed within its intended scope.

For example, if you are evaluating a support agent that looks up orders and issues refunds, tool call accuracy catches wrong-tool selection, while agent goal accuracy catches cases where the right tools were called but the task still failed.

General-purpose metrics

When your task is not RAG or agent-specific, Ragas provides LLM-judged metrics you can point at almost any output:

  • Aspect Critic — score an output against a specific criterion you define (for example, harmlessness or conciseness).
  • Simple Criteria Scoring — a lightweight pass/fail or scored judgment against a stated criterion.
  • Rubric-based scoring — score against a detailed rubric, either shared across all samples or defined per instance (instance-specific rubrics).

These are the right choice when you have a quality bar that is easier to describe in words than to compute. The trade-off is that they depend on an LLM judge, so results reflect the judge model's behavior.

Traditional non-LLM metrics

Ragas also includes deterministic metrics that need no LLM call:

  • BLEU, ROUGE, CHRF — n-gram and character-based overlap with a reference answer.
  • Non-LLM string similarity — direct string comparison.
  • Exact match — strict equality with the expected answer.
  • String presence — whether a given string appears in the output.

These are cheap, fast, and reproducible, which makes them useful for regression tests and CI. Their limitation is that they reward surface similarity, not correctness — a paraphrased but correct answer can score poorly on BLEU or exact match.

Specialized metric groups

Beyond the four main families, the documentation lists additional groups for narrower tasks:

  • NVIDIA metrics — answer accuracy, context relevance, and response groundedness.
  • Natural language comparison — factual correctness and semantic similarity.
  • SQL — SQL execution-based scoring, a Datacompy-based score, and SQL query equivalence.
  • Summarization — metrics for evaluating summaries.

How to choose

Start from the artifact you are testing. Evaluating retrieval and grounded answers: use the RAG metrics. Evaluating an agent's decisions: use the tool-use and goal metrics. Evaluating open-ended quality against your own standard: use aspect critic or rubrics. Needing fast, deterministic checks in a pipeline: use the traditional metrics. Most real evaluations combine several — for instance, context recall plus faithfulness plus exact match on a structured field.

Metric definitions and usage examples are documented per metric in the Ragas docs under Core Concepts → Metrics.