Architecture and design decisions¶
This document explains why the swarm is built the way it is. Each decision links the
evidence behind it. The full literature review is in
research/Multi agent swarms state of the art.md.
The problem¶
There are thousands of independent tasks (classification, extraction, Q&A, checks), and each one is small. The goal is to finish all of them with the highest success rate, the fewest tokens per task, the shortest wall time and the fewest agents.
This is the regime where multi-agent parallelism is known to help. Controlled studies find large gains on decomposable work but losses on sequential tasks and when a single agent is already strong (Kim et al., Towards a Science of Scaling Agent Systems, arXiv 2512.08296, 2025–26). The swarm therefore does not use agents that talk to each other. Its agents are independent workers on independent tasks, and all coordination is done by code.
Primary use case: deep research¶
v3: free client-side search, and quote grounding¶
Workers no longer depend on Claude's server-side web search ($10 per 1,000 searches). They call client-side tools backed by pluggable providers: - TinyFish Search and Fetch, free within a daily allowance; - your local documents (BM25); - a disk cache in front of either.
This is a manual tool loop that we own:
- parallel tool calls run concurrently, and their results go back in a single user message;
- the conversation is cached on each turn, and the final turn forbids further tool calls;
- the worker's last message is structured JSON: an answer plus sources with verbatim
quotes.
That gives a property server-side search didn't: every quote is checked by code against
the text the worker actually retrieved (case-, markup- and whitespace-insensitive, with
ellipses allowed). Unmatched quotes are dropped. So a citation can't be invented: it must
point to text that exists on the page. Tool output is wrapped in <untrusted_content> as
part of the injection defences.
swarm/research.py puts a research front end on the engine. It runs plan → parallel
web-research workers (one sub-question each) → citation gate → escalation → synthesis →
reference-integrity check. It was chosen because information-seeking is among the most
common uses of chat assistants, and breadth-first research has the strongest multi-agent
evidence. The research-specific choices:
- Server-side web tools (web_search_20260209, web_fetch_20260209): there is no
client tool loop, and pause_turn is resumed by resending the turn so far.
- No structured output on workers, because citations and JSON-schema output can't be
combined. Workers return cited text, and the engine wraps it as JSON for the validator.
- Batch size 1 and no hedging, because each call is already a multi-search job.
- Unverified findings are passed to synthesis labelled as such, and the prompt requires
a Limitations section. They are not silently dropped.
- Claim check after synthesis. The citation gate only counts sources; the claim check
tests each cited sentence against the excerpts of the sources it cites. Claims are batched
ten per call through the same engine. Its verdicts are themselves model judgements, so the
report shows them as marks and a support rate rather than silently deleting sentences.
- Injection defences in layers: prompt rule, delimiters, pattern detection with
exclusion, output sanitising and least-privilege tools. None of these is complete on its
own.
- Money is the budget unit users see. The engine budgets in input-token equivalents,
converted from dollars at the model's list price. Search fees are added as token
equivalents, so one cap covers both.
Pipeline¶
tasks ─► resume (WAL) ─► dedupe/cache ─► pools[(min_rung, difficulty)]
│ lazy batching at dispatch time
▼
scheduler ── policy picks (effort rung, batch size)
│
┌──────────── agent pool (knee-sized, AIMD-limited) ────────────┐
│ budget reserve → breaker route → call (+hedge if straggling) │
└───────────────────────────────┬───────────────────────────────┘
▼
verify: deterministic checker, or agreement of 2 answers
ok → journal fail → next rung (or retry at top)
Decisions¶
1. Code orchestrates, and agents never coordinate with each other¶
Agent-to-agent conversation is where most measured failures come from. In the MAST taxonomy, inter-agent misalignment accounts for roughly a third of failures across 1,642 traces from 7 frameworks (Cemri et al., arXiv 2503.13657). Debate rarely beats simple voting at equal compute (Choi et al., NeurIPS 2025). Kim et al. measure 4.4× error amplification for centralized coordination against 17.2× for independent agents, with the orchestrator acting as a checkpoint. Practice in 2026 has moved the same way, towards code-driven fan-out (e.g. Claude Code dynamic workflows, Managed Agents multiagent workflows) and away from free-form swarms. Here the "orchestrator" is a deterministic scheduler, not an LLM, so it costs zero tokens.
2. A pull-based shared queue instead of synchronous fan-out¶
Anthropic reports that synchronous subagent waves are a bottleneck, because the lead waits for the slowest worker (How we built our multi-agent research system, Jun 2025). Here, workers pull the next batch as soon as they are free. Batches are formed lazily, so each decision uses the latest learned statistics, and a partial batch goes out only when nothing full is ready.
3. Fewest agents: the makespan knee¶
Coordination overhead grows with agent count (58–515% in Kim et al.), and so do rate-limit
pressure and 429s. plan_agents chooses the smallest pool whose estimated makespan is
within 15% of the fastest possible. The benchmark includes a single-agent baseline with the
same pipeline. It has the same token cost but takes 3–5× longer, which shows that the extra
agents buy wall time and nothing else.
4. Many tasks per call (batch prompting)¶
Batch prompting cuts tokens and calls substantially: up to 5× at 6 tasks per call (Cheng et al. 2023), and 18–31% of the tokens at batch size 32 (BatchPrompt). For reasoning models it has also been reported to cut thinking tokens (arXiv 2511.04108). Accuracy drops as batches grow, and how fast depends on the model and the task. That is why batch size is learned, not fixed (§5).
5. Learn batch size and effort online (Thompson sampling + DP)¶
For each (difficulty, effort rung), a Bayesian logistic model learns P(success | batch size), and an online linear model learns tokens per call. A dynamic programme over the ladder minimises expected cost per solved task, including the cost of escalating failures:
Q(r,b) = c(r,b) + (1 − p(r,b))·V(r+1) V(r) = min(min_b Q(r,b), V(r+1))
Prior art includes cascades and routers (FrugalGPT, RouteLLM, AutoMix) and joint model-and-batch-size selection under a budget (RoBatch, arXiv 2605.28268, offline and greedy). No prior work was found that applies online Thompson sampling to batch size. Honest result: once learned (warm), it matches hand-tuned static rules within noise in the simulator. Its value is not needing hand-tuning on a real endpoint. Its cost is an exploration premium on the first run, which is why the learned state is persisted.
6. Escalate effort instead of debating¶
Upgrading the model or effort beat doubling the token budget in Anthropic's analysis. Most
tasks finish at low; only failures move up, then get two more tries at the top rung.
Rate limits, 5xx errors and stragglers never consume one of a task's attempts. The
ladder stays on one model by default, which keeps one cache namespace and avoids drift
between models.
7. Verify deterministically, otherwise require agreement¶
MAST lists verification failures as a whole failure category. Adding a verification step
lifted ChatDev by 15.6% (Cemri et al.). Checks here are code (expected, regex, callable), so
no "judge" LLM is needed. For tasks with no checker, self-reported confidence is a weak
signal (AUROC ≈ 0.52–0.61 in calibration studies), whereas agreement between samples is
better supported (self-consistency; Adaptive-Consistency needs up to 7.9× fewer samples by
stopping at agreement). By default an unchecked answer is accepted once two independent
attempts agree.
8. Provider-aware reliability¶
| Mechanism | Evidence | Behaviour |
|---|---|---|
| AIMD limiter | Anthropic ramp-limit guidance: "increase by at most one step per minute" | Halve on a plain 429; hold on ramp_limit_exceeded; honour retry-after |
| Spend-cap stop | Anthropic: the enforced_spend_limit_reached 429 "keeps failing until access resumes" |
Stop the run cleanly, with no retry storm |
| Circuit breaker | Standard SRE practice; 529 means overload across all users | Open after N 5xx errors; reroute to another rung's model or wait; single half-open probe |
| Hedged requests | The Tail at Scale (Dean & Barroso 2013): large tail cut for ~2% extra requests | Duplicate a call past 1.5 × p95, at most 8% of calls, using reserved limiter headroom |
| Token budget | Cost-aware evaluation (Kapoor et al. 2024; HAL) | Reserve before each call, settle after; leftover tasks end as budget_exhausted |
| Write-ahead journal | Durable execution (Temporal, DBOS, Restate) | Append-only and fsynced, tolerates a torn last line; resume skips finished tasks |
9. Pay less per token¶
- Batches API: 50% off, for workloads that can wait up to 24 h. Engine calls are coalesced into one batch.
- Prompt caching: shared context sits behind a cache breakpoint; reads cost 0.05–0.1× input depending on the model. Cache reads also don't count toward input rate limits. A short system prompt alone falls below the minimum cacheable prefix, which is why
--shared-contextexists. - Structured output removes parse retries and verbose prose.
- Known caveat: changing top-level
effortbetween requests can cause a separate cache entry per rung.
Evaluation methodology¶
This follows the cost-aware evaluation literature (Kapoor et al. 2024; HAL, ICLR 2026): - Several seeds, reporting means with 95% confidence intervals. - Cost and time reported together (Pareto view), not accuracy alone. - A compute-matched single-agent baseline and the previous version as baselines. - A simulator with adversarial conditions (stragglers, a 5xx incident, rate limits) and three endpoint profiles. It is clearly labelled as simulated: it measures the scheduler's trade-offs, not Claude's accuracy.
Gaps from the research report: status¶
| Gap the report raised | Status |
|---|---|
Branch on 429 error codes (spend cap, ramp limit, retry-after) |
Done (classify_429, SpendCapReached, AdaptiveLimiter.on_ramp) |
| Compute-matched single-worker baseline, seeds with CIs, Pareto plot | Done (v2-1agent, ci95, docs/assets/pareto.svg) |
| Self-reported confidence is poorly calibrated | Done: agreement is the default for unchecked tasks |
| Batch position effects | Done: item order is shuffled within every batch |
| Bandit reward should be cost per verified item | Already the case: the DP minimises cost per solved (verified) task |
| Hedging only on the synchronous path | Already the case: hedging is off for the Batches backend |
| Discount old evidence (non-stationarity) | Open |
| Per-message effort to keep the cache across rungs | Open (each call is a fresh one-shot request today) |
| OpenTelemetry GenAI spans | Open (a JSONL event log exists) |
| Test the verifiers themselves | Partly: unit tests cover the checkers |
Known gaps¶
- No real-endpoint benchmark yet (it costs money; see
swarm run). - Agreement mode doubles the minimum calls for unchecked tasks. Semantic (not exact) agreement would help free-form answers.
- No idempotency key on Messages requests, so a crash mid-call can duplicate spend (the journal prevents duplicated results).
- The policy assumes difficulty labels; unlabeled workloads pool into one class.