IBO

The problem

An agent that answers client requests is a graph of steps: classify, retrieve, reason, draft, check. When the final answer is wrong, a thumbs-down on the output tells you almost nothing. Business experts need to see each step and judge each step. Engineers need to change one prompt and replay only what changed.

What it does

IBO is a workspace where each agent owns its requests, display blocks, feedback criteria and experts.

• Runs, live. “Run the agent” executes in the background; LangGraph’s task stream reports the start and result of every node, sub-graphs included, pushed to the browser over SSE. A node’s feedback form unlocks as soon as the node finishes.
• Re-run from a node. The new run reuses every node that finished before it (copied, greyed out, not billed again) and resumes from the exact saved state, LangChain messages included. The fastest way to test a prompt change on a late step.
• LLM costs. Tokens counted per node and per model (input, cached, output, reasoning), priced from an admin-managed table.
• Performance. Median / p95 run duration, queueing time, success rate; per node, the share of time spent waiting on the LLM versus RAG, tools and code; per model, latency, time-to-first-token and tokens/s.
• Review queues. Admins assign requests or let IBO spread them across the least-loaded experts, with optional double review. Progress per expert is visible to the team.
• Flexible display. Columns, headers and context panels are configured as JSONPath + display type (text, markdown, table, image, PDF…) with a live preview on a real request.

How it is built

FastAPI + SQLAlchemy/Alembic on the backend, LangGraph and LangChain for agents, optional Langfuse and MLflow (agents can be loaded from MLflow or installed as Python wheels). The frontend is Next.js with the App Router, MUI and React Flow for the graph, fully bilingual (FR/EN). The web app proxies the API so cookies and live events stay on one origin.

Fine-grained roles: admins see everything; expert members of an agent act only on their own queue; per-agent options decide whether experts can launch runs or even see intermediate steps.

Why it matters

Agent evaluation is not a metric, it is a conversation between the people who know the business and the people who build the graph. IBO makes that conversation precise: node by node, with cost and latency on the table.