Evaluation Datasets
Evaluation datasets are curated sets of test inputs with known expected outputs. They enable consistent, repeatable measurement of agent quality across changes.
6 min•By Priygop Team•Updated 2026
Building an Evaluation Dataset
- Collect representative inputs: 50-200 examples covering common and edge cases
- Include ground truth: for each input, define what a correct outcome looks like
- Cover failure cases: include inputs that should trigger escalation, rejection, or error handling
- Versioned and immutable: once created, the eval set should not change (use a new version for new requirements)
- Separate from training data: if the agent was trained or fine-tuned, do not use those examples for evaluation
Evaluation Dataset Structure
Evaluation Dataset Structure
# Evaluation dataset for a refund processing agent
EVAL_DATASET = [
# Happy path cases
{
"id": "EVAL-001",
"category": "happy_path",
"input": {"order_id": "ORD-100", "reason": "product damaged", "user": "alice@example.com"},
"expected_outcome": "success",
"expected_path": ["get_order", "check_eligibility", "issue_refund", "send_confirmation", "FINISH"],
"expected_tools": {"get_order", "issue_refund", "send_confirmation"},
"should_escalate": False,
},
# Edge cases
{
"id": "EVAL-002",
"category": "outside_return_window",
"input": {"order_id": "ORD-050", "reason": "changed mind", "user": "bob@example.com"},
"expected_outcome": "rejected",
"expected_path": ["get_order", "check_eligibility", "reject_refund", "FINISH"],
"should_escalate": False,
},
{
"id": "EVAL-003",
"category": "large_refund",
"input": {"order_id": "ORD-999", "reason": "defective", "user": "carol@example.com"},
"expected_outcome": "escalated",
"expected_path": ["get_order", "check_eligibility", "ESCALATE"],
"should_escalate": True,
"escalation_reason_contains": "approval",
},
]
print(f"Eval dataset: {len(EVAL_DATASET)} cases")
categories = {}
for case in EVAL_DATASET:
categories[case["category"]] = categories.get(case["category"], 0) + 1
for cat, count in categories.items():
print(f" {cat}: {count} case(s)")Key Takeaways
- Evaluation datasets are curated sets of test inputs with known expected outputs.
- Collect representative inputs: 50-200 examples covering common and edge cases
- Include ground truth: for each input, define what a correct outcome looks like
- Cover failure cases: include inputs that should trigger escalation, rejection, or error handling