SWARM SEEDS / EXPERIMENTS 01–05
What actually makes AI agent teams smarter?
Five open experiments tested where extra AI compute should go: into deeper reasoning, more independent attempts, structured review, evolved decision systems, or answers that can verify themselves.
NEW / EXPERIMENT 05 A Luna Light system crossed 90% on a fresh hard-sequence gate. It did not trust a judge. It made proposed answers prove themselves first. ↓ OPEN SOURCE / EXPERIMENTS 01–04 Explore the full Swarm Seeds program on GitHub Skills, protocols, benchmarks, prompts, raw outputs, failed attempts, scorers, amendments, and audits. Experiment 05 follows after its release archive is finalized.THE PROBLEM
More AI activity does not automatically produce a better answer.
A difficult task can be given more tokens, more agents, more review stages, or a smarter final selector. Each option spends extra compute differently. Each can also fail differently.
The goal of this program was to find out which forms of extra work create evidence that survives fresh tests, rather than merely creating more confident output.
The task was exact mathematical sequence continuation. An agent saw only a short prefix of integers, had to infer the hidden generating rule, and then predict the next three or five terms with no partial credit for a plausible explanation. Because any finite prefix can support infinitely many continuations, the later benchmarks used defined rule families and rejected cases where equally valid recognized rules led to different answers.
Can one agent search deeply enough?
A direct solver is fast and cheap, but it may settle on the first plausible pattern and never find a better one.
Do more agents find different ideas?
Independent attempts can widen the search. They can also repeat the same memorized guess and form a confidently wrong majority.
Does collaboration improve the evidence?
Critics, verifiers, and judges add review. They also add latency, tokens, and new opportunities to override a correct answer.
Can the system improve without fooling itself?
Prompt and topology search can discover useful mechanisms, but small reused evaluations can turn random variation into a false winner.
THE CENTRAL QUESTION Under a controlled budget, what should another unit of AI effort actually do?
THE RESEARCH ARC
Each result changed the next question.
The program moved from changing one prompt to changing how groups of agents search, review, and decide.
Evolve one prompt
Twenty generations found a validation winner that lost badly on untouched cases.
Baseline wonCompare teams fairly
Direct solving, independent voting, and a 5→2→2→1 swarm shared fixed budgets.
Reasoning dominatedMake the task harder
RuleWeave-5 replaced recognizable sequences with generated five-term challenges.
Tournament promisingEvolve the topology
Symbolic evolution searched ten-call structures while the model and task stayed fixed.
Selection stayed noisyEvolve the gate
Eight rounds searched how proposed answers should be accepted or overridden.
Mechanism improvedMake answers prove themselves
Visible holdouts let candidates earn influence by reconstructing known public terms.
22 / 24 fresh casesBEFORE THE SWARM
The first “winner” was an illusion.
The earliest attempt tried to evolve a better prompt for one Luna Light solver. It looked encouraging through training and validation. The untouched final test reversed the result.
Across 20 generations, 230 prompt candidates were tested. A compact skeptical-check prompt was frozen before the final split opened. On that split, the ordinary baseline solved twice as many complete continuations.
Exact next-three accuracy across three independent runs per arm. A single lucky validation result was not treated as success.
Prompt search can manufacture apparent progress from stochastic variation. Untouched tests are what keep the story honest.
That negative result changed the direction of the work. Instead of asking one agent to think better through wording alone, the next experiment asked whether several isolated agents could contribute different evidence and then combine it.
REASONING VS ROUTING
One task. Three ways to spend compute.
Each solver received ten integers and had to return the next three exactly. Low and medium reasoning were sealed from one another, and the vote and swarm each received ten calls.
Direct
One isolated solver. Ten separate runs estimated expected one-call performance.
Independent vote
Ten agents solved the task separately. A deterministic plurality rule combined exact answers.
Structured swarm
Five agents proposed rules, two challenged them, two verified the surviving logic, and one judge decided.
At medium reasoning, voting matched the full swarm while using about 3.35× less visible-token proxy and 2.43× less parallel latency.
The clearest result was not “swarms win.” Medium reasoning improved every method. At medium, ten independent attempts captured all observed accuracy benefit of the larger workflow. At low, the swarm’s 25-point lead over voting was interesting, but its wide interval crossed zero.
Open Experiment 01 and every underlying recordHARD SEQUENCE SCALING
The second test removed the easy shortcuts.
Recognizable named sequences could overlap model training data. RuleWeave-5 instead generated fresh integer programs with random coefficients, longer prefixes, and five required predictions.
Medium Tournament20 produced the best point estimate at 44 of 48 exact cases. It finished three cases ahead of Medium Vote20, but the paired interval still crossed zero. Averaged across multi-call methods, moving from Light to Medium added 34.38 points; changing equal-call voting into routing added only 1.04 points.
Open Experiment 02, RuleWeave-5, and its 400-call final runHarder tasks made structured collaboration more plausible, but not yet proven better than independent sampling.
EVOLVING LIGHT SWARMS
Could evolution discover a better team?
The third experiment held Luna Light, the prompts, schemas, ten-call budget, and RuleWeave-5 task fixed. Only a symbolic orchestration genome could change.
The evolved champion finished three cases ahead of the corrected generalist Vote10, but the interval included zero. It tied a diversified independent pool and trailed its own founder by one case. The system searched successfully; the hidden evidence did not establish that evolution improved it.
A release audit also found that the first independent baseline had accidentally mixed specialized lenses. The error was preserved and an amendment added the missing generalist baseline. That correction became part of the result, not something removed from the story.
Open Experiment 03, its genomes, amendment, and complete audit trailEVOLVING THE DECIDER
The fourth experiment narrowed the search.
Instead of evolving an unrestricted team shape, the search focused on six legal decision systems: vote, judge, gated critic, dual judge, verified gate, and deliberative gate.
3 useful0 harmful8 neutral
The verified gate created real mechanism evidence: it accepted three useful overrides and no harmful ones. But the full champion tied one generalist Vote10 run and trailed another fresh execution of the same baseline configuration. Identical prompts produced different outcomes, which made inference variance impossible to ignore.
Open Experiment 04, all eight rounds, and the 2,604-attempt ledgerA component can improve its own plurality without proving that the full system is better than a simpler baseline.
ADAPTIVE ORCHESTRATION SEARCH
The breakthrough was making answers prove themselves.
Experiment 05 let a research director revise the orchestration between sealed fresh gates. The final frozen system solved 22 of 24 new hard sequences, reaching 91.7% exact with Luna Light calls.
The system did not ask a model to declare which answer looked best. Instead, each candidate received a shortened version of the public sequence and had to recover the known terms removed from its end while also predicting the five unknown terms. A continuation earned influence only when it reconstructed the visible holdout exactly.
The architecture began with 15 independent prompt lenses. It spent additional calls only when plurality support or public reconstruction was weak: first deep holdout solvers, then recovery solvers, then structural worksheets. Confidence was ignored; verified public evidence determined the final ranking.
A length-weighted replay reached 87.5% on Gate 02, but it was discovered after that gate opened and was treated only as development. The rule was frozen before Gate 03 was generated.
Verification became useful when it stopped asking which answer sounded right and started asking each answer to recover evidence already visible in the problem.
This is a strong fresh-gate result, not a universal 91.7% reasoning claim. Repeated research and multiple new gates discovered a system specialized to this benchmark. Transfer to unrelated reasoning tasks, or a larger replication, remains the more demanding next test.
WHAT THE PROGRAM SUPPORTS
The useful answer is conditional.
There is no universal winning diagram. The best way to spend compute depends on how capable the individual agents are, how hard the task is, and how much evidence exists before a final answer is changed.
Improve the reasoner first.
Moving from Light to Medium produced the largest and most consistent gain across direct, voting, and collaborative methods.
Independent voting is hard to beat.
It preserved diversity, parallelized cleanly, and repeatedly matched or approached more elaborate systems.
Routing helps when agents are weak.
Structured review helped the low-reasoning team in Experiment 01, but not reliably across every harder run.
Make answers recover known evidence.
Visible holdouts turned self-verification into measurable evidence and corrected six base errors without a harmful Gate 03 override.
Evolution has not earned the complexity.
Evolution found plausible policies, but validation selection and run variance remained larger than the observed final leads.
Use disagreement as a routing signal.
A strong future system may vote first and spend extra review only when independent answers genuinely conflict.
HOW THE METHOD EVOLVED
The experiments became harder to fool.
Later runs did not only increase the number of model calls. They strengthened the evidence system around those calls.
Sealed reasoning arms and equal call budgets.
Fresh programs, five-term targets, paired statistics.
A baseline mismatch was preserved and corrected.
Round-by-round release gates and 2,600 logical identities.
A post-hoc insight had to survive a newly generated gate before becoming a result.
- Freeze protocols, prompts, answers, and hashes before the relevant split opens.
- Compare architectures at equal call budgets and report token or latency differences separately.
- Preserve malformed and stalled attempts instead of silently repairing the record.
- Use untouched final cases, paired outcomes, uncertainty intervals, and fresh executions.
- Publish corrections and amendments when an audit changes the interpretation.
New agent systems, research methods, and live experiments are explored in the weekly 1000x Lab.
See the 1000x Lab