OTel Evaluation for Multi-Agent Workflows

Trace context across an in-memory bus into layered real-time and offline evaluation

OTel Evaluation for Multi-Agent Workflows Trace context across an in-memory bus into layered real-time and offline evaluation 01 / Agents 02 / Bus 03 / Real-time Eval 04 / Offline Eval 05 / Backend Agents · emit spans · 01 / Agents · trace ctx Agents emit spans trace ctx Agent Bus · in-memory, ctx propagated · 02 / Bus Agent Bus in-memory, ctx propagated Policy Gates · real-time · 03 / Real-time Eval · block/allow Policy Gates real-time block/allow Safety + Cost · live metrics · 03 / Real-time Eval Safety + Cost live metrics LLM-as-Judge · offline scoring · 04 / Offline Eval · async LLM-as-Judge offline scoring async Observability · OTel backend · 05 / Backend · traces + scores Observability OTel backend traces + scores spans + ctx traced events passed runs scores traced events live signals Legend primary data policy / PII async batch data store

Context Travels With Work

  • • Agents emit OpenTelemetry spans as they run
  • • The in-memory bus propagates trace context across hops
  • • Every downstream stage inherits the same trace

Layered Evaluation

  • • Real-time policy gates and safety/cost metrics run inline
  • • The LLM-as-judge scores runs offline, off the hot path
  • • All layers converge into one observability backend

Data-flow diagram • Built with Archify • Create yours ↗ • Hover to trace • R route • Click to focus • +/− zoom • M radar • T theme • E export