DatasetOps Autopilot automatically creates RAG evaluation benchmarks using specialized AI agents. Transition from manual scripting to autonomous test generation.
Autopilot coordinates specialized agents to analyze documents, blueprint evaluation scopes, generate QA datasets, and audit/repair errors with human-in-the-loop control.
Upload raw markdown, PDF, or text files. The Source Understander runs hierarchical scans to build semantic topic categories and extract grounding facts.
The Intake Planner compiles slot allocations. Human Plan Approval is required to validate the category targets and token budgets before generation starts.
The Generator drafts single/multi-hop QAs. The Quality Evaluator audits every sample against Faithfulness, Relevance, and Hallucination Risk metrics.
If audits fail, a bounded Generator ↔ Evaluator Repair Loop automatically retries failing samples. The Exporter packages all verified files.
Instead of linear scripts, DatasetOps Autopilot runs specialized agents that collaborate dynamically to build, audit, and repair.
Performs hierarchical text chunking and extracts topics, category distributions, and semantic relationships to map the source context.
Defines slots covering target topics, splits counts by difficulty (easy, medium, hard), and maps target reasoning pathways.
Synthesizes diverse RAG question-answer pairs (single-hop, multi-hop, unanswerable, edge case) grounded strictly in evidence context.
Evaluates candidate pairs against strict criteria: Faithfulness (>=0.85), Answer Relevance (>=0.75), and Hallucination Risk (<=0.25). Participates in a self-repair loop.
Assembles passed QA pairs, answer keys, quality sheets, and markdown cards into a clean, downloadable production export.
Modern LLM systems depend on precise semantic retrieval. Without high-quality, ground-truth benchmarks, measuring drift, precision, and alignment is impossible.
"...Auto-scaling thresholds are managed by the orchestrator daemon. The system scales up nodes automatically when average CPU utilization remains above 80% for a continuous period of 5 minutes."
Under what utilization threshold and duration will the orchestrator daemon scale up nodes?
Nodes scale up when average CPU utilization stays above 80% for 5 minutes.
{"id": "3dac4f21-77ae-49fb-8d3a-9429cec1e26f", "sample_type": "single_hop", "question": "Under what utilization threshold and duration will the orchestrator daemon scale up nodes?", "source_chunk_ids": ["chunk_658cdfea-1944-40f6-b610-1c6a7e582504"], "evidence": [{"chunk_id": "chunk_658cdfea-1944-40f6-b610-1c6a7e582504", "text": "...Auto-scaling thresholds are managed by the orchestrator daemon. The system scales up nodes automatically when average CPU utilization remains above 80% for a continuous period of 5 minutes."}]}
{"id": "8fa3c11d-2831-4be6-a192-1c6a7e582504", "sample_type": "multi_hop", "question": "How do you configure authentication key rotations for cluster daemons?", "source_chunk_ids": ["chunk_c85598f2-c20c-4816-9119-6d3d5e54c6f4", "chunk_658cdfea-1944-40f6-b610-1c6a7e582504"], "evidence": [{"chunk_id": "chunk_c85598f2-c20c-4816-9119-6d3d5e54c6f4", "text": "Authentication secrets are defined under /etc/cluster/daemon.conf."}, {"chunk_id": "chunk_658cdfea-1944-40f6-b610-1c6a7e582504", "text": "Run daemon-reload with SIGHUP to trigger safe key rotation."}]}
{"id": "b91f1a23-45c1-4822-a9b0-9831a293c12a", "sample_type": "unanswerable", "question": "What is the maximum number of external nodes allowed in a hybrid multi-cloud configuration?", "source_chunk_ids": ["chunk_a7b4582f-e1c2-482d-8812-7d2d3e582504"], "evidence": [{"chunk_id": "chunk_a7b4582f-e1c2-482d-8812-7d2d3e582504", "text": "Hybrid cluster limits are described in cloud-provider manuals, not local configs."}]}Explore the core features designed to automate and guarantee RAG evaluation dataset quality.
Ingests raw markdown, PDF, or text files, running semantic scans to build topic concept maps, identify chunk density, and evaluate category coverage.
Why it matters: Ensures evaluation is grounded in actual document coverage rather than random guesses.
Benefit: Creates a structured profile of your knowledge base, preventing uneven test distribution.
Generates an interactive benchmark blueprint outlining topic coverage, difficulty distributions, and sample counts, requiring human review before running.
Why it matters: Gives developers complete control over the evaluation scope before spinning up expensive model generation calls.
Benefit: Prevents API waste and guarantees the generated test suite aligns with project requirements.
Runs an autonomous generation agent that drafts custom single-hop, multi-hop, unanswerable, and edge-case question-answer pairs.
Why it matters: Replaces slow, error-prone manual scripting of test suites with high-speed autonomous generation.
Benefit: Delivers diverse, production-grade test cases covering all potential query pathways.
Audits every generated sample against strict metrics (Faithfulness, Relevance, Hallucination Risk) and auto-repairs failed entries.
Why it matters: Enforces strict hallucination-risk thresholds without requiring human review for every line.
Benefit: Maintains high benchmark integrity by recursively fixing incorrect references.
Provides a clean visual interface to inspect evaluation logs, agent traces, and individual question metrics.
Why it matters: Allows transparent debugging of agent decision-making and quality auditing steps.
Benefit: Allows deep-diving into individual test cases before committing to production export.
Compiles verified samples into structured test packages including datasets, answer keys, and quality reports.
Why it matters: Saves hours of formatting and wrangling, producing plug-and-play evaluation files.
Benefit: Delivers standard .jsonl and .md formats compatible with any RAG testing framework.