Schedulers, not swarms, make multi-agent pay¶
As of late 2026, the state of the art in LLM multi-agent systems for large workloads has narrowed to something less flashy than "agent swarms." Code, not an LLM, schedules many context-isolated workers. Those workers mostly read or handle independent items, a single thread performs writes, and a fresh-context or deterministic verifier checks results before they are accepted. The controlled evidence is sobering. A 260-configuration Google/MIT study found the mean multi-agent gain over a single agent was about −0.3%, with results ranging from +80.8% on decomposable financial analysis to −70% on sequential planning, and returns turning negative once the single-agent baseline exceeds roughly 45% success (Kim et al.). Much of the headline multi-agent advantage is bought with tokens. Anthropic reports multi-agent systems use about 15x the tokens of chat, and that token usage alone explains about 80% of performance variance on BrowseComp (Anthropic). The failure literature (MAST) shows that most breakdowns come from system design and verification gaps, not model weakness (Cemri et al.). This means the engineering that pays off sits around the model: stacked provider discounts (Batch API, prompt caching, effort controls), distributed-systems reliability patterns (verification gates, retry budgets, hedging, circuit breakers, durable journals), and honest cost-aware evaluation. For anyone building a high-throughput "swarm," the lesson is to treat agents as interchangeable, verifiable workers behind an adaptive scheduler. Treating them as a society that negotiates is the approach the 2025–2026 evidence now argues against.
Orchestration converged on "code is the scheduler" with single-threaded writes¶
The canonical vocabulary still comes from Anthropic's December 2024 "Building effective agents." It distinguishes workflows (predefined code paths) from agents (LLMs directing their own process). It names five patterns: prompt chaining, routing, parallelization (sectioning and voting), orchestrator-workers, and evaluator-optimizer. Its advice is to start simple and add complexity only when it earns its keep (Anthropic). The field has spent two years confirming that advice. Anthropic's June 2025 research system is the reference orchestrator-worker design. An Opus 4 lead plans, spawns 3–5 parallel subagents with their own context windows, and synthesizes their results, beating single-agent Opus 4 by 90.2% on an internal eval and cutting research time by up to 90% (Anthropic). That same post names the design's main weakness: synchronous subagent execution is a bottleneck, because the lead waits for the slowest member of each batch. It also recommends that subagents write outputs to external storage and pass back lightweight references, which amounts to a blackboard pattern.
The June 2025 "Don't Build Multi-Agents" vs. multi-agent-research exchange, published one day apart, set up the debate. Cognition argued that "actions carry implicit decisions," so parallel writers produce mismatched artifacts (Cognition). By April 2026 the two positions had largely converged. Cognition now endorses read-only search subagents, clean-context reviewers (Devin Review catches about 2 bugs per PR, about 58% of them severe), "smart friend" escalation to a stronger model, and manager agents doing "map-reduce-and-manage." It still finds parallel-writer swarms and unstructured agent negotiation "mostly a distraction" (Cognition). Anthropic's January 2026 guidance draws the same boundary. Multi-agent is justified for context protection, for parallelization valued for thoroughness, or for specialization (signals include 20+ tools). It typically costs 3–10x more tokens. Decomposition should follow context (what each subtask needs to know) rather than problem roles (planner/implementer/tester), because role-based splits spend more tokens coordinating than working and lose fidelity at handoffs (Claude blog). The same guidance warns that parallel systems "often take longer overall" and that verifier agents suffer an "early victory problem," declaring success after a few tests.
The architectural shift of 2026 is from "the LLM is the scheduler" to "code is the scheduler." Claude Code dynamic workflows went GA on May 28, 2026. In them, Claude writes an orchestration program that runs "tens to hundreds of parallel subagents," with progress saved so interrupted jobs resume. The flagship example is the Bun Zig-to-Rust port: about 750k lines of Rust with 99.8% of tests passing in 11 days, using hundreds of porting agents plus two reviewers per file, at "substantially more tokens" (Claude). Claude Managed Agents documents both modes. Subagent delegation is one level deep, with at most 25 concurrent child threads and a shared sandbox and filesystem. The docs recommend programmatic workflows for "most work that needs more than one agent" (Claude Platform Docs). Every major framework now pairs deterministic graph or workflow control with agentic nodes. Microsoft Agent Framework 1.0 (GA April 3, 2026) ships sequential, concurrent, handoff, group-chat and Magentic-One orchestrations with checkpointing and pause/resume (Microsoft). LangGraph, CrewAI Flows and Google ADK follow the same pattern.
The framework landscape has consolidated sharply. Several research-era frameworks are now lineage rather than competitors.
| Framework / protocol | Late-2026 status | Orchestration model |
|---|---|---|
| LangGraph | 1.2.24, ~45k stars; 1.x is LTS (release policy) | StateGraph plus checkpointers; supervisor library archived, swarm handoff library active |
| Microsoft Agent Framework | 1.0 GA Apr 2026 | Graph workflows; successor to Semantic Kernel plus AutoGen |
| AutoGen | Archived, no fixes or support | Conversable agents (lineage); community fork AG2 at v1.3.3 |
| OpenAI Agents SDK | v0.25.0, still pre-1.0 | Handoffs and agents-as-tools; Temporal integration for durability (Temporal) |
| CrewAI | 1.15.44 | Role-based crews; Flows recommended for production |
| Google ADK | v2.15.0 | Parent/sub-agents plus Sequential/Parallel/Loop workflow agents |
| MetaGPT | Dormant since Jan 2026 | SOP roles with publish-subscribe message pool (paper) |
| A2A / MCP | A2A v1.0.1; MCP spec 2026-07-28 | Agent-to-agent and agent-to-tool interop layers |
One structural observation matters for the design discussed at the end of this report. Fan-out in all of these frameworks is push-based: an orchestrator or graph assigns work. None is built around a pull-based shared queue with batch-executing workers and an adaptive scheduler. The closest analogue sits one layer down, in durable-execution engines such as Temporal, whose task queues and worker pools already provide queue semantics.
Multi-agent wins only on decomposable work with a weak baseline¶
The best controlled evidence comes from Kim et al.'s "Towards a Science of Scaling Agent Systems." It covers 260 configurations, 6 agentic benchmarks, five architectures (single-agent, Independent, Centralized, Decentralized, Hybrid), and models from GPT-5-nano through Claude Sonnet 4.5, with 1–4 agents (arXiv). The per-benchmark spread is the finding. On Finance-Agent, centralized multi-agent gained +80.8%. On BrowseComp-Plus, decentralized gained +9.2%, but independent agents lost about 35%. On PlanCraft, every multi-agent architecture degraded, from −39% to −70%. On SWE-bench Verified, every architecture degraded, by 2–15% (arXiv HTML). Three mechanisms explain the pattern. Capability saturation: returns turn negative once the single agent already succeeds about 45% of the time. Tool-coordination overhead: the penalty is negligible at four or fewer tools and grows with tool count. Thinning: under a fixed budget, turn count grows super-linearly with agents and per-agent reasoning becomes "too thin" beyond 3–4 agents. Their predictive model picks the best architecture for 87% of held-out configurations, but its cross-validated R² is only 0.37–0.41, so task structure still leaves much unexplained (arXiv; Google Research).
Normalize by compute and most of the advantage disappears. Kim et al. measured 67.7 successes per 1K tokens for a single agent versus 13.6 for hybrid multi-agent. That is roughly a 5x efficiency gap, driven by coordination overheads of 58% (independent) to 515% (hybrid) (arXiv HTML). Sampling-and-voting results show the same ceiling from another angle. In "More Agents Is All You Need," 40 majority-voted Llama2-13B samples lifted GSM8K from 0.35 to 0.59, beating a single Llama2-70B. But 40 GPT-3.5 samples never surpassed a single GPT-4 call on any of five benchmarks (GSM8K 0.85 vs. 0.88, HumanEval 0.73 vs. 0.88) (arXiv HTML). Anthropic found the same thing in production: upgrading to Sonnet 4 beat doubling the token budget on Sonnet 3.7 (Anthropic). More samples close part of a capability gap but do not replace a better model.
Multi-agent debate, the most-studied "agents talking to each other" pattern, holds up worst. Wang et al. found a single well-prompted LLM matches the best discussion method except when no demonstrations are given (arXiv). "Stop Overvaluing Multi-Agent Debate" tested 5 debate methods on 9 benchmarks with 4 models. Debate often failed to beat chain-of-thought or self-consistency despite using more compute, and model heterogeneity was the one reliable improvement (arXiv). "Debate or Vote" (NeurIPS 2025 Spotlight) showed majority voting accounts for most of debate's gains. It also proved that debate alone induces a martingale over beliefs, so it "does not improve expected correctness" (arXiv). A 2026 study of ten homogeneous 7–8B debaters found sycophantic conformity up to 85.5% and peers overturning correct answers up to 70% of the time, at 2.1–3.4x the tokens of isolated self-correction (arXiv). The counterpoint deserves a fair hearing. At matched compute, one 2026 paper found debate beating self-consistency by +1.3 points and mixture-of-agents by +2.7 on MMLU-Pro/BBH. That margin is modest, and the best gain over CoT (+7.1) needed about 20x CoT compute (arXiv).
Scaling with agent count follows the same shape. MacNet scaled DAG-organized collaboration past 1,000 agents and reported a "collaborative scaling law" of logistic growth. Logistic curves saturate, though, and the study was not compute-matched against a single agent (arXiv). Anthropic's prompt-embedded effort rules point the same way: 1 agent for simple facts, 2–4 for comparisons, 10+ only for complex research. Its production failure list includes spawning 50 subagents for a simple query (Anthropic). The defensible reading is that agent count is not a quality knob. It is a throughput and coverage knob that pays off only when items are genuinely independent, and quality gains beyond 3–4 cooperating agents on a single task are unsupported. Two qualifications are worth stating plainly. The 45% threshold has only been tested with 1–4 agents and nine models. Anthropic's 90.2% result is neither compute-matched nor externally replicated, and was built on an eval that began with about 20 queries.
Most multi-agent failures are design and verification failures¶
MAST, the Berkeley taxonomy ("Why Do Multi-Agent LLM Systems Fail?", NeurIPS 2025), is the standard reference. It annotated 1,642 traces across 7 frameworks, including ChatDev, MetaGPT, HyperAgent, AppWorld, AG2 and Magentic-One, with human inter-annotator agreement κ = 0.88. It found failure rates of 41% to 86.7% (arXiv). Its 14 modes fall into three categories. Summing the per-mode shares gives roughly 44% system design issues: step repetition 15.7%, unaware of termination conditions 12.4%, disobeying the task spec 11.8%. Roughly 32% are inter-agent misalignment, led by reasoning-action mismatch at 13.2% and task derailment at 7.4%. Roughly 24% are task-verification failures: incorrect verification 9.1%, missing or incomplete verification 8.2%, premature termination 6.2% (arXiv HTML). The interventions are just as telling. On ChatDev, adding a high-level objective verification step raised ProgramDev success by +15.6%, and better role and workflow specifications added about +9.4%. Completion rates nonetheless stayed low, and prompt-level fixes had inconsistent effects. The authors conclude that structural fixes matter more: multi-level verification that checks high-level objectives (since "it compiles" is not enough), explicit termination specs, and standardized communication and state management.
Diagnosing failures after the fact is itself unreliable. On the Who&When dataset (failure logs from 127 multi-agent systems), the best automated method identified the responsible agent 53.5% of the time and the decisive step only 14.2% (arXiv). 2026 follow-ups improved this. TraceElephant shows that with full execution traces rather than output-only logs, attribution rises to 65.9% (agent) and 30.3% (step). It also finds that at least 21% of failures cannot be attributed from outputs alone, a direct argument for full tracing (arXiv). Kim et al. quantified error amplification relative to a single agent at 17.2x for independent agents versus 4.4x for centralized ones, attributing the gap to the orchestrator acting as a checkpoint. Their regression, however, found the amplification terms not statistically significant (arXiv HTML).
Long horizons make all of this worse. METR finds agent success falls from near 100% on tasks humans finish in under 4 minutes to under 10% on tasks over 4 hours. The 50% time horizon has doubled about every 7 months (METR; arXiv). Toby Ord's constant-hazard reading implies the 80%-reliability horizon is about one third of the 50% horizon, and the 99% horizon about 1/70 (arXiv). For illustration, a 20-step chain at 95% per step succeeds only about 36% of the time. Every handoff adds hazard. The working rule is to cut work into chunks well below the model's 50% horizon, verify each chunk deterministically, and checkpoint so a failure costs one chunk rather than the run.
Provider discounts stack, and they reward batch-shaped workloads¶
The largest cost levers in late 2026 are on the provider side, and they compound. Anthropic's Message Batches API gives 50% off input and output in exchange for asynchronous completion. Most batches finish within an hour and expire at 24 hours, and expired requests are not billed. Each batch holds up to 100,000 requests or 256 MB, and the Batches API is not subject to ramp limits (Anthropic batch; rate limits). Prompt caching charges 1.25x base input for a 5-minute write or 2x for a 1-hour write. Reads cost 0.1x on most models and 0.05x on Opus 5.5 and Sonnet 5.5, and these multipliers explicitly "stack with … the Batch API discount" (Anthropic pricing). Caching is also a throughput lever: default input-tokens-per-minute (ITPM) limits count only uncached input, so an 80% hit rate turns a 2M ITPM limit into 10M effective input tokens per minute (Anthropic rate limits). Inside batches, caching is best-effort. Anthropic reports hit rates of 30% to 98%, recommends the 1-hour TTL for shared context, and does not support cache pre-warming inside a batch (Anthropic batch). OpenAI's terms have converged: 50% batch discount, 24-hour window, 50,000 requests per file. For GPT-5.6 and later, a 1.25x cache write at a 30-minute default TTL and reads at 0.1x or 0.05x. That replaces the older "free automatic 50%" caching that much 2024–2025 material still cites (OpenAI caching; OpenAI batch).
| Lever | Magnitude (late 2026) | Key caveat |
|---|---|---|
| Batch API | 50% off input and output (Anthropic, OpenAI) | Up to 24 h latency; no fast mode in batch |
| Prompt cache read | 0.1x base (0.05x Opus/Sonnet 5.5) (pricing) | Batch hit rate 30–98%; 1-h write costs 2x |
| Multi-task batch prompting | Up to 5x fewer tokens at 6 items (Cheng et al.) | Accuracy usually drops as batch size grows |
| Effort level | low → max via output_config.effort (effort docs) |
Changing top-level effort can invalidate the cache |
| Routing / cascades | 35–98% savings in original papers (FrugalGPT) | Many routers fail to beat the best single model (LLMRouterBench) |
Stacked, the theoretical floor for cached Opus 5.5 input in batch is about $0.10 per million tokens (0.05x of the $2 batch rate). That figure is derived from the documented multipliers, and realized savings depend on the hit rate. One subtle interaction deserves design attention. Anthropic's effort parameter (low/medium/high/xhigh/max) is now the primary thinking-cost control. It affects all output tokens, including tool calls, and is "a behavioral signal, not a strict token budget." Changing top-level effort between requests "can invalidate cached prefixes," while per-message effort set via a system message "preserves the prompt cache" (Anthropic effort). The same docs warn that max adds significant cost for small gains and "can lead to overthinking" on structured-output tasks. HAL's independent runs found that higher reasoning effort reduced accuracy in the majority of runs (HAL). Effort escalation is therefore not free accuracy, and it needs an empirical sweep on the target workload.
Multi-task batch prompting (packing several items into one call) is the client-side counterpart. The original Batch Prompting paper cut token and time cost up to 5x with six items per call, with "better or comparable" accuracy on 10 datasets (arXiv). BatchPrompt at batch size 32 used 9–16% of the calls. Its permutation-and-ensembling trick (re-running shuffled batches and voting) held BoolQ at 90.9% on 27.4% of the tokens, but its authors note accuracy falls with batch size, partly because later answers condition on earlier ones (arXiv). Degradation is model-specific. In machine-translation evaluation, GPT-4o-mini stayed stable while GPT-4o collapsed at batch size 16 (arXiv). One 2025–2026 result reverses the usual trade-off for reasoning models: going from batch size 1 to 15 cut DeepSeek-R1 and o1 reasoning tokens by 76% (2,950 to 710) while preserving or improving accuracy, apparently by suppressing hedging and overthinking. Prompt-level caps like "≤100 thinking tokens" failed (arXiv). The closest prior art for jointly choosing configuration and batch size is RoBatch (May 2026). It proves the "Route with Batching" problem NP-hard and uses an offline greedy walk along the cost-utility Pareto frontier (arXiv).
Routing and cascades remain useful but overrated. FrugalGPT claimed up to 98% cost reduction at GPT-4 quality (arXiv). RouteLLM reported 35–85% savings against GPT-4-only at 95% of GPT-4 quality (LMSYS). AutoMix reported over 50% savings, using noisy self-verification fed into a POMDP router (arXiv). Unified 2026 evaluation tempers this. LLMRouterBench (400K+ instances, 33 models) found many routers perform similarly, several commercial routers "fail to reliably outperform a simple baseline," and larger model pools show diminishing returns compared with careful model curation (arXiv). Any routing or cascade claim therefore has to be measured against the best single configuration, not against "always use the biggest model."
Reliability now borrows directly from distributed systems¶
Verification is the highest-leverage reliability technique, and its form matters. Self-consistency (sample and majority-vote) is well established (arXiv), and adaptive variants recover most of its cost. Adaptive-Consistency cuts samples up to 7.9x with under 0.1% accuracy loss across 17 datasets (arXiv). Early-Stopping Self-Consistency stops when a sampling window is unanimous and saves 34–84% of samples (arXiv). Self-reported confidence is a poor escalation signal by comparison. LLMs verbalizing confidence are systematically overconfident, and failure-prediction AUROC sits around 0.52–0.61 (Xiong et al.). Agreement across samples, or better a deterministic check (tests, schemas, exact-match invariants), is the better gate. MAST's +15.6% from objective verification and its 9.1% "incorrect verification" mode together imply the verifier must check the actual objective, not a proxy.
The infrastructure layer is classic distributed systems with LLM-specific semantics. Anthropic's 429s come in three kinds that need different handling. A rate-limit 429 carries retry-after. A ramp-limit 429 (ramp_limit_exceeded) caps growth at "your … highest per-minute usage … plus a step," and Anthropic's guidance is to "increase it by at most one step per minute," which is effectively provider-enforced additive increase. A spend-cap 429 (enforced_spend_limit_reached) has no retry-after and "keeps failing until access resumes." A 529 signals platform-wide overload rather than tenant-specific throttling (Anthropic errors; rate limits). Limits use a token bucket that may be enforced over sub-minute windows (60 RPM can mean 1 request per second), which favors smoothing traffic over bursting it. The SDK retries 429s and 5xx twice by default. Requests over about 10 minutes should stream or go to Batches, and streaming errors can arrive after an HTTP 200. Google SRE's overload chapter supplies the client-side discipline. Use adaptive throttling that rejects locally once requests exceed K=2 times accepts, cap retries at 3 attempts per request and 10% of requests per client, and tag batch work as SHEDDABLE so it yields first (Google SRE). Request hedging comes from Dean and Barroso's "Tail at Scale." There, a backup request sent after 10 ms (approximately the p95) cut BigTable p99.9 latency from 1,800 ms to 74 ms for 2% extra load. Secondary summaries differ on whether the figure is p99 or p99.9 (ClickHouse; the morning paper). For LLMs the economics differ. A duplicate call duplicates output-token cost, and a shared cached prefix makes the hedge cheaper on input but no faster on output. Hedging is justified only for latency-critical, short-output calls, triggered after a p95 delay, with the loser cancelled and the total bounded by a retry budget. No provider publishes tail-latency SLOs, and no measured study of LLM hedging benefit was found.
Durable execution became table stakes in 2025–2026. Temporal runs LLM calls as retried Activities inside deterministic workflows, and its OpenAI Agents SDK integration went GA for Python in March 2026 (Temporal; Temporal docs). Restate journals each step before proceeding, and DBOS checkpoints steps into Postgres as a library (developersdigest). Pydantic AI supports Temporal, DBOS, Prefect and Restate as backends (Pydantic AI). Microsoft Agent Framework, LangGraph and Claude dynamic workflows all checkpoint and resume, following Anthropic's 2025 observation that agents are stateful, errors compound, and resuming beats restarting. One gap is the user's responsibility. The fetched Anthropic error and rate-limit docs document no idempotency-key header for Messages requests, so deduplicating retried or replayed calls falls to a client-side journal.
Benchmarks are saturating and cost-aware evaluation is the honest frontier¶
Headline agent benchmarks are near ceiling on vendor-reported numbers, and every figure below should be read as "reported by an aggregator on a date, under some harness." llm-stats lists SWE-bench Verified at 95.0% in November 2026 (llm-stats). Terminal-Bench 2.0's top scores sit around 82% (llm-stats). BrowseComp's top entry, a four-agent configuration, reaches 92.2%, only 0.7 points above the best single-agent entry (Steel). Harness effects are as large as model gaps. Vals.ai's standardized Terminus 2 harness put the Terminal-Bench leader at 73.2% versus roughly 82% vendor-reported (Vals.ai). Scale's standardized SWE-bench Pro tops out around 61.5% with ±3.1–3.6 point CIs that leave the top entries tied (Scale), and Epoch rates SWE-bench Pro "Flawed," with 30%+ of tasks estimated broken (Epoch). Correcting erroneous tests changed the rank of 24.4% of SWE-bench Verified entries (UTBoost). METR's time horizon still discriminates. Claude Mythos Preview's 50% horizon was about 17.4 hours, with a 95% CI of roughly 8.5–55 hours, falling to about 11.3 hours if detected cheating counts as failure (agitracker).
Multi-agent evaluation is less mature still. MultiAgentBench/MARBLE (ACL 2025) scores collaboration and competition across star, chain, tree and graph topologies with milestone KPIs (arXiv). Who&When and MAST cover attribution and taxonomy. No widely tracked multi-agent SOTA leaderboard exists. The methodological standard comes instead from Princeton's cost-aware work. "AI Agents That Matter" argued that evaluations without cost control reward absurdly expensive agents. It called for accuracy-versus-dollars Pareto frontiers and showed that simple baselines (retry, escalation) match complex agents (arXiv). HAL (ICLR 2026) put this into practice over 21,730 rollouts. Its board shows SWE-Agent + Sonnet 4.5 at 72.0% for $464 versus Opus 4.1 at 68.0% for $1,351, o4-mini matching Claude 3.7 Sonnet on TAU-bench Airline at about a quarter of the cost, and a summary line that "agents can be 100x more expensive while only being 1% better" (HAL). HAL's log audits also caught agents searching HuggingFace for benchmark answers (arXiv). A 2026 analysis shows close leaderboard ranks often cannot support "A > B" claims, and that the choice of cost rule can flip which system wins (arXiv). Practice lags theory. In LangChain's 2026 survey of 1,300+ practitioners, 57% have agents in production and quality is the top barrier (32%), but only 52.4% run offline evals (LangChain). OpenTelemetry's GenAI conventions now define invoke_agent spans with token-usage attributes but remain in Development status (OTel). Gartner predicts over 40% of agentic AI projects will be canceled by end-2027 on cost, value or risk grounds (Gartner).
Practical guidance distilled from the evidence¶
Start from one capable agent and justify every additional one. Add workers when items are independent (throughput), when a subtask would flood the main context, or when tool sets genuinely diverge. Do not add workers to make a single hard task "smarter." Keep cooperating agents on one task at roughly 3–4 or fewer, decompose by context rather than by role, and keep writes single-threaded. When extra compute should buy accuracy, prefer parallel samples feeding a vote or, better, a deterministic verifier over debate. Prefer escalation to a stronger model or higher effort over more samples of a weak one. Let code, not an LLM, own scheduling, retries and termination, and make stop conditions explicit, since "unaware of termination" alone is 12.4% of MAST failures. Shape the workload for the provider's discounts: put shared context in a stable cached prefix, use the 1-hour TTL for batch jobs, move anything latency-tolerant to the Batch API, and escalate effort per message rather than at top level so the cache survives. Treat rate limits as a control loop: start bulk jobs low, add one step per minute, cut multiplicatively on 429, and tune on uncached tokens. Branch on error codes so spend caps halt the job and 529s shed load. Bound retries and hedges with an explicit budget. Journal every step with enough identity to deduplicate replays. Evaluate on a held-out set with 3–5 seeds and CIs. Report dollars per verified success alongside tokens, wall-clock percentiles and agent count on a Pareto plot against a compute-matched single-agent baseline. Audit full traces, labeled with MAST categories, for shortcuts.
How this project's design maps to the literature¶
The project is a Python swarm of a few batch-executing workers pulling from a shared queue. A Thompson-sampled scheduler chooses batch size and effort level, with effort escalation, deterministic verification, a circuit breaker, request hedging, AIMD rate limiting, a token budget, a write-ahead journal and a Batches API backend. It sits squarely in the 2026 consensus on most axes, and is genuinely novel on one.
What it follows from the literature. The pull-based queue with code-owned scheduling is the "code is the scheduler" pattern of dynamic workflows and graph frameworks. Continuous refill directly addresses the synchronous-batch bottleneck Anthropic named in its research system. Temporal-style task queues are the nearest infrastructure precedent. "A few workers" matches Kim et al.'s finding that benefits thin out beyond 3–4 agents. Because workers process independent items rather than cooperating on one task, the design avoids the inter-agent misalignment category (about 32% of MAST failures) almost by construction. Deterministic verification is the most strongly supported choice in the whole design. It targets MAST's verification category (about 24%), mirrors the +15.6% ChatDev result, and sidesteps the poor calibration of self-reported confidence. It also stands in for the "orchestrator as checkpoint" that kept centralized error amplification at 4.4x against 17.2x for independent agents. Multi-task batching follows Batch Prompting and BatchPrompt. Effort escalation is a single-model analogue of FrugalGPT/AutoMix cascades and Cognition's "smart friend." The Batches API backend, caching and token budget exploit stacked 50% batch and 0.05–0.1x cache-read pricing, in line with Kapoor et al.'s cost-control argument. AIMD matches Anthropic's documented ramp-limit behavior (additive, one step per minute). The circuit breaker, hedging and retry discipline follow Google SRE and "The Tail at Scale." The write-ahead journal is the Restate/Temporal/DBOS event-history idea, and it fills the gap left by the absence of a documented idempotency key.
Where it is novel. No paper found applies Thompson sampling to choosing multi-task batch size, and none treats batch size × effort level as a joint bandit arm space. The closest neighbors are RoBatch (offline, greedy, model × batch size) and the Thompson-sampling routing and prompt-selection papers (arXiv 2510.00841; OPTS). No published study measures intra-model effort escalation as a cascade, either. The design can therefore be positioned as an online, bandit-based answer to the Route-with-Batching problem, and as a testbed for whether batching suppresses overthinking on adaptive-thinking Claude models, as reported for R1/o1.
Known gaps and tensions to document. The literature points to several issues the design description does not address:
- Bandit reward. Batching degradation is non-linear and model-specific, so the Thompson reward should be cost per verified item, not raw accuracy or raw cost. A contextual variant conditioned on task type or length would track the routing-bandit literature more closely. Non-stationarity (model updates, cache warmth) argues for discounting old evidence.
- Escalation and caching. Escalating via top-level
effortcan invalidate the cached prefix. Per-message effort preserves it. HAL's finding that higher effort often lowered accuracy means escalation needs an empirical sweep rather than an assumption of monotonic gains. - Batch position effects. Later answers condition on earlier ones within a batch. Randomizing item order, tracking per-position error rates, and using BatchPrompt-style permutation voting on items that fail verification are cheap mitigations.
- Hedging and Batches. Hedging only makes sense on the synchronous path. Against a 24-hour Batches backend it is meaningless. On the synchronous path it doubles output cost, so it should be p95-triggered, cancel the loser, and draw from an SRE-style retry budget of about 10%.
- Rate-limit and breaker semantics. AIMD should be tuned on uncached input tokens. The circuit breaker should branch on
enforced_spend_limit_reached(halt),ramp_limit_exceeded(hold, then increase additively) and 529 (shed to batch). - Journal replay. Replay must deduplicate calls whose results already landed, including batch results retrieved after a crash. The Temporal-style discipline is to keep orchestration logic deterministic and isolate nondeterministic LLM calls as journaled steps.
- Verification coverage. The design's strongest guarantee only holds for verifiable tasks. MAST's "incorrect verification" mode (9.1%) is a reminder to test the verifiers themselves.
- Evaluation. To meet the field's best practice, the project should report a compute-matched single-worker baseline and a "best single fixed configuration" baseline (the LLMRouterBench lesson). It should use 3–5 seeds with CIs, dollars per verified success on a Pareto plot, and OTel GenAI spans per worker, with failures labeled using MAST categories.
Conclusion¶
The field's understanding has inverted since 2024. "Multi-agent" used to mean agents collaborating to be smarter. The evidence now says collaboration rarely beats a strong single agent at equal compute, and that debate mostly works as an expensive vote. What does scale is the operational use of many agents: independent work items, deterministic acceptance tests, and a scheduler that treats tokens, rate limits and latency as resources to manage. In that framing the meaningful frontier is no longer agent topology. It is control: how a system decides how much compute each item deserves (batch size, effort, escalation) and proves the result is right before paying for more.
That shift makes the open questions empirical and testable at small scale. Nobody has published whether an online bandit over batch size and effort beats a well-tuned fixed configuration once cost per verified success is the metric, or how much batching changes thinking-token spend on adaptive-thinking models. Nor has anyone measured how often hedging pays for itself against LLM tail latency. A small, well-instrumented swarm with honest baselines can answer these questions. Leaderboards cannot, because they are saturated and do not report cost.