Multi-turn evals from first principles
Single-turn evals check one decision. Multi-turn evals check the whole trajectory. A Python harness with three evaluators, an offline test suite, and the judge prompt that actually works.
#Evaluation
Evaluation in AI and ML systems means measuring whether models and agents actually perform correctly on tasks that matter. These articles cover benchmark design, LLM-as-judge patterns, OpenTelemetry-based evaluation pipelines, and the tooling needed to move from demo accuracy to production-grade reliability.
4 posts tagged with evaluation. ← All posts
OpenTelemetry and Evaluation in Multi-Agent Workflows —...
How to instrument multi-agent systems with OpenTelemetry, propagate trace context across an in-memory bus, and build a layered evaluation pipeline — from real-time policy gates to async LLM-as-judge to SLO-based trust scoring. Everything I learned building Genie.
Moving Diagnostic Accuracy 42.9% → 85.7% by Changing Two...
How a single sprint of specialty-rule work — guided by a benchmark that wasn't afraid to print embarrassing numbers — turned a 'demo respiratory differential' into a five-condition rule-based diagnostic engine.
Evaluation
Scoring Microsoft Agent Framework agent outputs offline with a LocalEvaluator.
Score Microsoft Agent Framework agent outputs with evaluate_agent and a LocalEvaluator: run offline keyword, tool-call, and custom evaluator checks, then gate CI via raise_for_status.
All posts on this site are written by Pratik Dhanave, an Agentic AI Architect with 7+ years building production distributed systems, multi-agent AI platforms, and cloud-native infrastructure. About the author → Each article includes working code, architecture diagrams, and references to the specific frameworks and standards discussed. Browse all posts or explore related topics using the tag cloud above.