Can Ragas Generate Test Datasets for RAG and Agent Applications?

Updated

Yes. Ragas (docs.ragas.io) documents test data generation as a first-class capability, covering both RAG applications and agents/tool-use cases. According to the site's navigation, the Test Data Generation section includes RAG testset generation, KG building, scenario generation, and testset generation for agents or tool use cases. The How-to Guides add customization options: non-English testset generation, persona generation, custom single-hop queries, custom multi-hop queries, and using pre-chunked data.

The condition to keep in mind: these pages describe the framework's features and workflows, not a hosted data service. You generate testsets by running Ragas in your own environment, so you need a working Python setup and the LLM or model access your generation config expects.

What Ragas can generate

Generation target What it produces Where it fits
RAG testsets Question/answer-style evaluation samples built from your knowledge base Evaluating retrieval and generation quality of a RAG pipeline
KG building A knowledge graph representation used as the basis for scenario and query generation Grounding generated tests in your source documents
Scenario generation Scenarios that seed agent/tool-use tests Testing agents against realistic situations rather than one-off prompts
Agent/tool-use testsets Test cases for agentic or tool-calling behavior Checking tool selection and task completion
Custom single-hop queries Simpler queries you define or shape yourself Targeted checks on straightforward retrieval
Custom multi-hop queries Queries requiring multiple reasoning or retrieval steps Stress-testing complex RAG behavior
Persona generation Synthetic user personas that drive query diversity Broadening coverage beyond your own assumptions
Non-English testsets Test data in languages other than English Multilingual applications

How generation fits the Ragas workflow

Ragas is described as an evaluation framework for AI applications. Testset generation is one stage of that loop: you produce a dataset, then score your application against it using the available metrics. The site organizes metrics by application type, which tells you what a generated testset is meant to feed:

  • RAG metrics — context precision, context recall, context entities recall, noise sensitivity, response relevancy, faithfulness.
  • Agent or tool-use metrics — topic adherence, tool call accuracy, tool call F1, agent goal accuracy.
  • General-purpose and comparison metrics — factual correctness, semantic similarity, aspect critic, rubric-based scoring, and traditional non-LLM metrics such as BLEU, ROUGE, CHRF, and exact match.

The practical implication: decide which metrics you will report before generating data. A RAG testset aimed at faithfulness and context recall looks different from an agent testset aimed at tool call accuracy, and Ragas separates the generation paths accordingly.

Getting started

The site's Get Started path lists Installation, Quick Start, and Tutorials. The tutorials include evaluating a prompt, a simple RAG system, an AI workflow, and an AI agent — a sensible order if you are new to the framework. For generation specifically, start from the Test Data Generation section, then consult the How-to Guides when you need customization.

A reasonable first pass, for example when you have a document collection and want a RAG testset:

  1. Install Ragas and confirm the Quick Start runs in your environment.
  2. Build the knowledge graph from your source documents (KG building).
  3. Generate a RAG testset from that graph.
  4. Score your pipeline with RAG metrics such as context precision and faithfulness.
  5. If coverage is thin, add custom single-hop or multi-hop queries, or generate personas to diversify the questions.

For an agent, the equivalent path runs through scenario generation and agent/tool-use testset generation, then the agent metrics above.

Practical considerations

  • Customization is documented, not automatic. Non-English testsets, personas, and custom query types each have their own guide; plan to configure them rather than expect them by default.
  • Pre-chunked data is supported. If your documents are already split, the "Using Pre-chunked Data" guide covers that case, which matters when your chunking strategy is fixed by your retrieval stack.
  • Pair generation with optimization. The site also documents prompt evaluation and systematic prompt optimization, so generated testsets can feed improvement work, not just scoring.
  • Pricing and access are not stated on the pages reviewed. Treat Ragas as software you run; check installation requirements and any model provider costs on your side.

If your goal is simply to know whether Ragas covers testset generation for RAG and agents, the answer is yes, with dedicated sections for both and documented customization for query type, persona, language, and pre-chunked input.