docs.ragas.io

Ragas includes a dedicated set of agent and tool-use evaluation capabilities: agent-specific metrics (Topic Adherence, Tool Call Accuracy, Tool Call F1, Agent Goal Accuracy), a guide for evaluating a new LLM agent, and testset generation tailored to agent and tool-use scenarios. These are relevant if your application routes requests to tools or pursues multi-step goals, rather than only answering from retrieved context.

What Ragas Covers for Agents

The documentation groups agent evaluation under "Agents or Tool Use Cases," alongside the RAG and general-purpose metric families. The listed agent metrics are:

These sit next to the RAG metrics (Context Precision, Context Recall, Faithfulness, Response Relevancy, Noise Sensitivity, Context Entities Recall) and general metrics (Aspect Critic, rubric-based scoring), so an agent that also retrieves documents can be assessed on both retrieval quality and tool behavior.

When This Fits Your Case

Use the agent metrics when your system's output includes actions, not just text — for example, an assistant that must call a search tool, a database query tool, or an internal API before answering. If your application only produces answers from retrieved passages, the RAG metrics are the better starting point; the agent metrics add little there.

Ragas also documents an "Evaluate a Text-to-SQL Agent" guide and an "Align an LLM as a Judge" guide, which are useful when tool calls produce structured output that needs its own correctness check.

Testset Generation for Agents and Tool Use

Beyond metrics, Ragas lists "Testset Generation for Agents or Tool use cases" under Test Data Generation. This matters because agent evaluation needs inputs that exercise tool selection and multi-step behavior, not just question–answer pairs. The same section includes RAG testset generation, KG building, and scenario generation, so you can build evaluation data for a system that mixes retrieval and tool calls.

Getting Started

The documented path is: install Ragas, work through the Quick Start, then follow the tutorial "Evaluate an AI Agent." From there, the how-to guides cover customizing models, adapting metrics to languages, and training/aligning metrics. The docs do not state pricing or login requirements on this page, so check the main site before assuming access terms.

A practical sequence for a new agent:

1. Define the goal and the tools the agent may call.
2. Generate a testset for agent/tool-use scenarios.
3. Score runs with Tool Call Accuracy, Tool Call F1, Agent Goal Accuracy, and Topic Adherence.
4. Use the "Evaluate a New LLM Agent" guide to structure the comparison.

If you need a quick decision: choose Ragas' agent metrics when tool-call correctness and goal completion are the failure modes you care about; stay with RAG metrics when answer grounding is the primary concern.

To get started with Ragas, install the package, run the Quick Start, then follow the tutorial that matches what you are building: a prompt, a RAG system, an AI workflow, or an AI agent. Ragas is an evaluation framework for AI applications, so the fastest path is to evaluate one small, real system you already have rather than reading the whole documentation first. The steps below follow the structure of the official docs at docs.ragas.io.

Step 1: Install Ragas

Ragas is a Python package. Install it in the environment where your AI application runs, then confirm the import works before writing any evaluation code.

pip install ragas

import ragas
print(ragas.__version__)

Expected result: the version prints without errors. If the import fails, check that you installed into the same Python environment your application uses (a common issue when a virtual environment is active in one terminal but not another).

Step 2: Run the Quick Start

The docs place Get Started → Installation → Quick Start at the front of the navigation. Work through the Quick Start before the tutorials: it shows the smallest complete evaluation loop — supply a dataset, choose a metric, run the evaluation, read the scores. Once that loop runs end to end, every later page is a variation on it.

Step 3: Pick the tutorial that matches your application

The Tutorials section is organized by application type. Choose based on what you are actually evaluating:

Start with the one closest to your current project. Running one tutorial against your own data teaches more than reading all four.

Step 4: Learn the core concepts

The Core Concepts section explains the three things every Ragas evaluation combines:

• Datasets — the inputs, contexts, responses, and reference answers your evaluation runs on.
• Metrics — the scoring functions. The docs group them into families: RAG metrics (context precision, context recall, noise sensitivity, response relevancy, faithfulness), agent and tool-use metrics (topic adherence, tool call accuracy, tool call F1, agent goal accuracy), natural language comparison (factual correctness, semantic similarity), traditional non-LLM metrics (BLEU, ROUGE, CHRF, exact match, string similarity), SQL metrics, and general-purpose metrics (aspect critic, rubric-based scoring).
• Experimentation — running evaluations repeatedly to compare versions of your application.

Read the Metrics Overview and Available Metrics pages with your own use case in mind. For a RAG system, faithfulness and response relevancy are the natural starting points; for an agent, tool call accuracy and agent goal accuracy.

Step 5: Use the how-to guides when you hit a specific need

The How-to Guides are task-oriented and are best consulted on demand rather than read front to back. Common entry points:

• Customise models and LLM Adapters — point Ragas at the model provider you already use.
• Run Config and Caching — control how evaluations execute and avoid repeating expensive calls.
• Modify Prompts and Adapt Metrics to Languages — adjust metric behavior, including for non-English applications.
• Testset Generation — generate evaluation data, including single-hop and multi-hop queries and non-English testsets.
• Prompt Evaluation and Optimization — iterate on prompts systematically rather than by trial and error.

Practical tips

• Evaluate a small sample first. A handful of examples is enough to confirm your dataset schema and metric choices are right before running a full evaluation.
• Match metrics to your architecture. Applying RAG retrieval metrics to a system with no retrieval step produces scores that do not mean anything.
• Treat scores as comparisons, not absolutes. The Experimentation concept exists because the useful signal is usually the change between two versions of your application.

The docs also include benchmarking guides (evaluating a new LLM, aligning an LLM as a judge, evaluating a text-to-SQL agent) and a RAG evaluation guide for evaluating and improving a RAG app — useful once the basics are working.

Yes. Ragas (docs.ragas.io) documents test data generation as a first-class capability, covering both RAG applications and agents/tool-use cases. According to the site's navigation, the Test Data Generation section includes RAG testset generation, KG building, scenario generation, and testset generation for agents or tool use cases. The How-to Guides add customization options: non-English testset generation, persona generation, custom single-hop queries, custom multi-hop queries, and using pre-chunked data.

The condition to keep in mind: these pages describe the framework's features and workflows, not a hosted data service. You generate testsets by running Ragas in your own environment, so you need a working Python setup and the LLM or model access your generation config expects.

What Ragas can generate

How generation fits the Ragas workflow

Ragas is described as an evaluation framework for AI applications. Testset generation is one stage of that loop: you produce a dataset, then score your application against it using the available metrics. The site organizes metrics by application type, which tells you what a generated testset is meant to feed:

• RAG metrics — context precision, context recall, context entities recall, noise sensitivity, response relevancy, faithfulness.
• Agent or tool-use metrics — topic adherence, tool call accuracy, tool call F1, agent goal accuracy.
• General-purpose and comparison metrics — factual correctness, semantic similarity, aspect critic, rubric-based scoring, and traditional non-LLM metrics such as BLEU, ROUGE, CHRF, and exact match.

The practical implication: decide which metrics you will report before generating data. A RAG testset aimed at faithfulness and context recall looks different from an agent testset aimed at tool call accuracy, and Ragas separates the generation paths accordingly.

Getting started

The site's Get Started path lists Installation, Quick Start, and Tutorials. The tutorials include evaluating a prompt, a simple RAG system, an AI workflow, and an AI agent — a sensible order if you are new to the framework. For generation specifically, start from the Test Data Generation section, then consult the How-to Guides when you need customization.

A reasonable first pass, for example when you have a document collection and want a RAG testset:

1. Install Ragas and confirm the Quick Start runs in your environment.
2. Build the knowledge graph from your source documents (KG building).
3. Generate a RAG testset from that graph.
4. Score your pipeline with RAG metrics such as context precision and faithfulness.
5. If coverage is thin, add custom single-hop or multi-hop queries, or generate personas to diversify the questions.

For an agent, the equivalent path runs through scenario generation and agent/tool-use testset generation, then the agent metrics above.

Practical considerations

• Customization is documented, not automatic. Non-English testsets, personas, and custom query types each have their own guide; plan to configure them rather than expect them by default.
• Pre-chunked data is supported. If your documents are already split, the "Using Pre-chunked Data" guide covers that case, which matters when your chunking strategy is fixed by your retrieval stack.
• Pair generation with optimization. The site also documents prompt evaluation and systematic prompt optimization, so generated testsets can feed improvement work, not just scoring.
• Pricing and access are not stated on the pages reviewed. Treat Ragas as software you run; check installation requirements and any model provider costs on your side.

If your goal is simply to know whether Ragas covers testset generation for RAG and agents, the answer is yes, with dedicated sections for both and documented customization for query type, persona, language, and pre-chunked input.

Ragas provides four families of evaluation metrics: RAG metrics (context precision, context recall, faithfulness, response relevancy, and others), agent/tool-use metrics (tool call accuracy, tool call F1, agent goal accuracy, topic adherence), general-purpose LLM-judged metrics (aspect critic and rubric-based scoring), and traditional non-LLM metrics (BLEU, ROUGE, CHRF, exact match, string similarity). Which set you use depends on what you are evaluating: a retrieval pipeline, an agent's tool behavior, open-ended output quality, or similarity to a reference answer.

RAG metrics

These are the core of Ragas and target retrieval-augmented generation systems, where you need to judge both the retrieved context and the generated answer.

A practical split: context precision, context recall, and context entities recall evaluate the retriever; faithfulness and response relevancy evaluate the generator. Noise sensitivity sits between them, showing how robust the pipeline is to imperfect retrieval. If you are diagnosing a RAG app, running retrieval metrics and generation metrics separately tells you which half to fix.

Agent and tool-use metrics

For agents that call tools or take multi-step actions, Ragas offers metrics aimed at the trajectory rather than a single answer:

• Tool Call Accuracy — whether the agent called the right tools.
• Tool Call F1 — a precision/recall-style balance over tool calls.
• Agent Goal Accuracy — whether the agent actually achieved the user's goal.
• Topic Adherence — whether the agent stayed within its intended scope.

For example, if you are evaluating a support agent that looks up orders and issues refunds, tool call accuracy catches wrong-tool selection, while agent goal accuracy catches cases where the right tools were called but the task still failed.

General-purpose metrics

When your task is not RAG or agent-specific, Ragas provides LLM-judged metrics you can point at almost any output:

• Aspect Critic — score an output against a specific criterion you define (for example, harmlessness or conciseness).
• Simple Criteria Scoring — a lightweight pass/fail or scored judgment against a stated criterion.
• Rubric-based scoring — score against a detailed rubric, either shared across all samples or defined per instance (instance-specific rubrics).

These are the right choice when you have a quality bar that is easier to describe in words than to compute. The trade-off is that they depend on an LLM judge, so results reflect the judge model's behavior.

Traditional non-LLM metrics

Ragas also includes deterministic metrics that need no LLM call:

• BLEU, ROUGE, CHRF — n-gram and character-based overlap with a reference answer.
• Non-LLM string similarity — direct string comparison.
• Exact match — strict equality with the expected answer.
• String presence — whether a given string appears in the output.

These are cheap, fast, and reproducible, which makes them useful for regression tests and CI. Their limitation is that they reward surface similarity, not correctness — a paraphrased but correct answer can score poorly on BLEU or exact match.

Specialized metric groups

Beyond the four main families, the documentation lists additional groups for narrower tasks:

• NVIDIA metrics — answer accuracy, context relevance, and response groundedness.
• Natural language comparison — factual correctness and semantic similarity.
• SQL — SQL execution-based scoring, a Datacompy-based score, and SQL query equivalence.
• Summarization — metrics for evaluating summaries.

How to choose

Start from the artifact you are testing. Evaluating retrieval and grounded answers: use the RAG metrics. Evaluating an agent's decisions: use the tool-use and goal metrics. Evaluating open-ended quality against your own standard: use aspect critic or rubrics. Needing fast, deterministic checks in a pipeline: use the traditional metrics. Most real evaluations combine several — for instance, context recall plus faithfulness plus exact match on a structured field.

Metric definitions and usage examples are documented per metric in the Ragas docs under Core Concepts → Metrics.

Ragas is an open-source evaluation framework for AI applications, documented at docs.ragas.io. You use it to measure the quality of AI systems—especially retrieval-augmented generation (RAG) pipelines, agents, and prompts—by running them against datasets and scoring the outputs with defined metrics. It fits teams that need repeatable, metric-based checks on an AI application rather than ad-hoc manual review.

What Ragas actually does

Ragas sits alongside your AI application and evaluates its outputs. The documentation organizes this around three building blocks:

• Datasets — structured collections of inputs, contexts, and responses to evaluate against.
• Metrics — scoring functions that quantify quality dimensions (for example, whether a response is grounded in retrieved context).
• Experimentation — running evaluations and comparing results across versions or configurations.

The framework is described as an "evaluation framework for your AI Application," and its documentation is structured to support both first-time setup and deeper customization.

What you can evaluate

The documentation's tutorials and use-case sections map to distinct application types:

The metric categories

Metrics are the core of Ragas, and the docs group them by what they measure:

RAG metrics

• Context Precision and Context Recall — how well retrieved context matches what's needed
• Context Entities Recall and Noise Sensitivity — entity coverage and robustness to irrelevant context
• Response Relevancy and Faithfulness — whether the answer addresses the question and stays grounded in context

Agent and tool-use metrics

• Topic Adherence, Tool Call Accuracy, Tool Call F1, and Agent Goal Accuracy

Natural language comparison

• Factual Correctness and Semantic Similarity

Traditional non-LLM metrics

• Non-LLM string similarity, BLEU, CHRF, ROUGE, string presence, and exact match

SQL metrics

• SQL execution-based scoring, Datacompy score, and SQL query equivalence

General-purpose metrics

• Aspect critic, simple criteria scoring, and rubrics-based scoring (including instance-specific rubrics)

There are also Nvidia-specific metrics (answer accuracy, context relevance, response groundedness) and a summarization task category.

When Ragas is a good fit

Choose Ragas if you want to:

• Quantify RAG quality with metrics that separate retrieval problems from generation problems.
• Evaluate agents on tool-call correctness and goal completion, not just final answer text.
• Compare versions systematically — the experimentation and benchmarking sections support evaluating a new LLM or comparing configurations.
• Generate test data — the docs include testset generation for RAG, knowledge-graph building, scenario generation, and agent/tool-use testsets.
• Customize evaluation — you can modify prompts, adapt metrics to other languages, train and align metrics, and use LLM adapters, caching, and run configs.

If you only need a one-off manual read of a few outputs, the framework's setup overhead may not be worth it.

How to get started

The documentation's path is: Installation → Quick Start → Tutorials → Core Concepts → How-to Guides.

1. Install the package (see the Installation page for the current command).
2. Work through the Quick Start to run a first evaluation.
3. Pick the tutorial matching your application type (prompt, RAG, workflow, or agent).
4. Read Core Concepts on datasets, metrics, and experimentation before scaling up.
5. Use How-to Guides for customization, optimizers, and benchmarking.

A practical first task: evaluate one simple RAG system with Context Precision, Context Recall, Response Relevancy, and Faithfulness, then inspect which metric is weakest before changing your pipeline.

Common sticking points

• Metric choice matters more than tooling. Faithfulness and Response Relevancy answer different questions; pick metrics that map to the failure you're investigating.
• LLM-based metrics need a judge model. Several metrics rely on an LLM as evaluator; the docs cover aligning an LLM as a judge and customizing models.
• Non-English applications need adaptation. The docs include adapting metrics to languages and non-English testset generation, so plan for this if your app isn't English-only.
• Test data quality caps evaluation quality. Use the testset generation guides rather than hand-writing a handful of cases if you want results you can trust.

The documentation does not state pricing or login requirements on the pages reviewed, so check the site directly for current access terms.