From chatbot to decision system: when to use multi-agent architecture

Architecture spectrum

Workflow first, agency only where uncertainty demands it#

1. 01

   Deterministic workflow

   Known sequence, typed inputs, explicit branches, predictable failure handling.

2. 02

   Single agent

   One context, several tools, dynamic choice inside a bounded task.

3. 03

   Multi-agent system

   Specialized contexts or tools, independent branches, explicit handoffs and synthesis.

Positive signals

Four reasons specialization may pay for itself#

Specialization helps when one component researches, another computes, another checks policy, and a final component synthesizes evidence. It can also isolate sensitive tools or parallelize independent searches.

The boundary must be contractual: input, output, tool access, timeout, retry and failure owner. “Let the agents talk” is not an architecture.

Topology menu

Choose a coordination pattern before choosing a framework#

1. 01

   Prompt chain

   Use a fixed sequence when each stage transforms and validates the previous result.

2. 02

   Parallel fan-out

   Run independent analyses together, then aggregate only evidence that meets the same contract.

3. 03

   Router and specialists

   Classify the request, send it to one bounded specialist, and keep an explicit fallback.

4. 04

   Evaluator and optimizer

   Let one component draft and another critique against a rubric, with a hard iteration limit.

Sanitized product pattern

A conversational analytics system is more than chat#

In a production conversational analytics product, a user request may require intent classification, data retrieval, statistical computation, comparison and explanation. The interface looks like one conversation, but the responsibilities behind it are different.

The useful pattern is not the number of agents. It is routing cheap or deterministic paths early, preserving continuity for follow-up questions, and requiring the final answer to carry the evidence produced by each specialist.

Failure containment

Design handoffs for partial failure#

Every extra agent creates another place for context loss, timeout and contradictory output. Give the orchestrator a deadline, a retry budget and a degraded response path.

Trace each handoff and tool result with stable identifiers. If a final answer is wrong, the team must distinguish bad routing, bad evidence, bad specialist output and bad synthesis.

Architecture experiment

Make the complex design beat a simpler baseline#

Run the same representative set through the typed workflow, the single-agent version, and the proposed multi-agent design. Compare task completion and critical failures first; then inspect p95 latency, cost per successful task, human intervention, and the fraction of runs that degrade gracefully.

Promote the multi-agent design only when the gain survives repeated runs and can be attributed to specialization or containment. If quality ties while latency, cost, and diagnosis worsen, the simpler design is the result—not a prototype you failed to outgrow.

from dataclasses import dataclass
import json

@dataclass(frozen=True)
class Request:
    intent: str
    question: str

def generalist(request: Request) -> dict:
    return {"path": ["generalist"], "answer": "baseline", "evidence": []}

def data_specialist(request: Request) -> dict:
    if "timeout" in request.question.lower():
        raise TimeoutError("data specialist timed out")
    return {
        "path": ["router", "data_specialist"],
        "answer": "compare two validated segments",
        "evidence": ["segment_a", "segment_b"],
    }

def route(request: Request) -> dict:
    if request.intent != "compare_segments":
        return generalist(request)
    try:
        return data_specialist(request)
    except TimeoutError:
        return {
            "path": ["router", "data_specialist", "safe_fallback"],
            "answer": "comparison unavailable",
            "evidence": [],
            "degraded": True,
        }

cases = [
    Request("small_talk", "hello"),
    Request("compare_segments", "compare the top two"),
    Request("compare_segments", "force timeout"),
]
print(json.dumps([route(case) for case in cases], indent=2))

[
  {
    "path": ["generalist"],
    "answer": "baseline",
    "evidence": []
  },
  {
    "path": ["router", "data_specialist"],
    "answer": "compare two validated segments",
    "evidence": ["segment_a", "segment_b"]
  },
  {
    "path": ["router", "data_specialist", "safe_fallback"],
    "answer": "comparison unavailable",
    "evidence": [],
    "degraded": true
  }
]

Decision rule

Ask what complexity buys#

• ✓Can a typed workflow solve the task?
• ✓Does one agent have enough context and tool isolation?
• ✓Are responsibilities meaningfully different?
• ✓Can branches run independently or in parallel?
• ✓Is each handoff typed and observable?
• ✓Does every specialist have a failure owner?
• ✓Can the system degrade when one branch fails?
• ✓Will evals compare the simpler and more complex design?