HomeNews

LLMs Design 1,800x Smaller Models That Beat Transformers

Naman
Naman
•
Nov 18, 2026 9:43 AM
•
0
 min read
Select Emergent as your Preferred news source
LLMs Design 1,800x Smaller Models That Beat Transformers

💡 TL;DR

  • Three frontier LLMs independently chose sparse lookup tables over transformers when designing neural networks without gradient descent.
  • The LLM-designed 3.35M-parameter model achieved 54.4% on ARC, outperforming a 3.33M-parameter transformer at 29.8%.
  • Each prediction traces to an editable fact row, enabling 17x faster GPU inference and precise local corrections.

When three frontier language models were asked to design neural networks from scratch, all three rejected transformers in favor of the same alternative: sparse lookup tables. A November 18, 2026 paper from Pohang University of Science and Technology researchers Hyunjong Ok and Jaeho Lee demonstrates that Claude Opus 5.5, GPT-6.1 Sol, and GPT-6 Astra independently converge on associative-memory architectures when writing weights directly, achieving up to 54.4% accuracy on the ARC benchmark with 1,800 times fewer parameters than comparable LLM-written transformers. Officially released on November 18, 2026, the research challenges assumptions about which architectures emerge when gradient descent is removed from the equation.

Three Frontier Models, One Architecture

The experimental setup eliminated the usual constraints of neural network training. Given ten labeled questions, a code sandbox, and complete freedom to design any architecture, Claude Opus 5.5, GPT-6.1 Sol, and GPT-6 Astra all produced sparse lookup tables rather than transformer blocks. The LLMs compiled plain-text facts into indexed tables queried by fixed procedures, with each model arriving at the same core structure independently.

The best-performing design came from Claude Opus 5.5, which produced a 3.35M-parameter sparse table that reached 54.4% test accuracy on ARC. A same-sized two-layer transformer trained through gradient descent by the same LLM scored 29.8%. The 1,800x parameter reduction refers to comparisons with the most accurate transformer the LLMs wrote when constrained to that architecture family.

Performance and Interpretability Trade-Offs

The LLM-designed models lag behind pretrained baselines. Qwen3.5-0.8B achieves 82.1% on the same ARC benchmark. Performance on MMLU hovers near random chance at approximately 25%, and CIFAR-10 accuracy from raw pixels falls between 21% and 28%. The authors frame the contribution narrowly: demonstrating which structure emerges when writing weights, not competing with state-of-the-art pretrained systems.

Two structural properties distinguish the sparse-table design:

  • Each prediction traces directly to a specific written fact row in the lookup table
  • Removing a single row changes approximately one other answer in 3,530 predictions
  • GPU inference runs up to 17 times faster than a same-sized transformer
  • Local edits modify individual facts without retraining the entire model

Implications for Model Architecture Research

The convergence pattern raises questions about whether transformers dominate because of inherent superiority or because they fit gradient-based training. When frontier models write weights by hand, the usual reasons for choosing attention mechanisms and residual connections no longer apply. The sparse-table design optimizes for direct interpretability and fast lookup rather than differentiable parameter updates.

Ok and Lee published code on GitHub alongside the paper, joining a research landscape where open-source AI projects have logged 386 releases over the past 90 days according to public trackers. The associative-memory model they propose structures knowledge as editable rows rather than distributed representations, enabling what the authors describe as fact-level provenance for every model output.

What This Means

This research opens a narrow but clear line of inquiry: what architectures would we build if training methods changed? The fact that three independent frontier LLMs all converged on the same non-transformer structure when freed from gradient descent suggests that current architectures may reflect optimization constraints more than fundamental reasoning requirements. While the resulting models underperform pretrained baselines on standard benchmarks, their interpretability properties and inference speed advantages point toward specialized applications where traceability matters more than absolute accuracy. The 17x speedup and single-row editability could prove relevant for fact-intensive domains where knowing the source of every answer carries compliance or trust requirements.

About the writer

A growth marketer with varied interests and the proven ability to acquire new skills fast. Currently engrossed in all things AI.

Start Building
on Emergent today
Try Emergent