Sakana AI & UC Berkeley MASS: Recursive Self-Improvement from Scratch

👤 1. Who Is Involved & Impacted?

• Creators: Researchers from Sakana AI and UC Berkeley (Hyunin Lee, Jinglue Xu, Jeffrey Seely, Donghyun Lee, Somayeh Sojoudi, Matei Zaharia, and Yujin Tang, arXiv:2610.12176).
• Base Model Under Test: Qwen3.6-27B trained in a strictly homogeneous loop where the optimizee is the evaluator, and the optimizer is the optimizee (via LoRA SFT without distilling from larger external frontier models).
• Community: Discussed on Hacker News (138 points, 82 comments), debating whether distilling multi-agent traces back into the same checkpoint is true Recursive Self-Improvement (RSI) or bounded harness tuning.

📦 2. What Did They Build?

Multi-Agent Self-Supervision (MASS) repeatedly optimizes multi-agent workflows and distills that collective intelligence back into a single base model:

1. Three Roles, One Shared Checkpoint (L^(0) -> L^(1) -> L^(2)): A single Qwen3.6-27B model acts simultaneously as the Workflow Optimizer, Multi-Agent Worker, and Internal Evaluator.
2. Synthetic Task Win Rates (Table 1):
   • 8 Training Tasks: Win rate vs. base (L^(0)) rises from 65.6% (Round 1) to 82.5% (Round 2).
   • 3 Held-Out Test Tasks: Win rate climbs from 53.9% (Round 1) to 69.9% (Round 2).
3. Public Benchmark Transfer (1.2x–1.6x Score/Token):
   • Evaluated without a multi-agent harness at test time, Round 2 (L^(2)) achieves 1.2x–1.6x score per output token on MLR-Bench (1.58 -> 2.40), DSBench, ScienceAgentBench, and AstaBench.
   • Token Efficiency: Multi-agent SFT (M+) reaches a 68.3% win rate using 28.8M tokens (-31% fewer tokens) vs. 64.0% for single-agent SFT (S++, 42M tokens).

📅 3. When & Where?

• When: Published on October 11, 2026.
• Where: Interactive blog at pub.sakana.ai/mass, paper on arXiv:2610.12176, and Hacker News Thread #50048522.

🎯 4. Why Does This Matter (and Why Did SWE-bench Dip)?

• Amortizing Multi-Agent Deliberation: Multi-agent debate beats a single forward pass but burns extra inference tokens. MASS compresses those multi-agent gains back into single-pass weights (1.2x–1.6x score/token).
• Out-of-Domain Coding Regression: Self-supervision on scientific research tasks did not transfer to repository-level coding:
  • SWE-bench Verified (500 x 3): Dips from 0.626 (L^(0)) to 0.618 (L^(1)) and 0.611 (L^(2)).
  • Terminal-Bench 2.0 (89 x 3): Dips from 0.375 (L^(0)) to 0.352 (L^(2)).
• Correlated Blind Spots: Because Optimizer, Workers, and Evaluator share the same Qwen3.6-27B priors, shared hallucinations cannot be self-corrected without external compilers/verifiers, causing gains to saturate after 2 rounds.

⚙️ 5. Benchmark Summary & Community Reaction

• Harness Tuning vs. RSI (@cootsnuck): Saturating after 2 LoRA rounds with a SWE-bench drop (0.626 -> 0.611) shows convergence to a local optimum, not open-ended RSI.
• "Ring of Rocks" (@throwyonion): Homogeneous self-evaluation amplifies shared priors unless anchored by a ground-truth environment or compiler.
• Dense Synthetic Data (@zby): Multi-agent rollouts (28.8M vs. 42M tokens) remain a practical way to generate higher-signal SFT trajectories.

✅ CONCLUSION

MASS is best understood as multi-agent trajectory distillation (1.2x–1.6x score per token and 31% fewer training tokens) rather than unbounded self-improvement—proving that models grading themselves quickly hit a ceiling without external verifiers.

References

• Sakana AI & UC Berkeley Blog: Multi-Agent Self-Supervision (MASS)
• arXiv:2610.12176 — Multi-Agent Self-Supervision
• Hacker News Discussion (138 points, 82 comments)