Latent Feedback Cuts Pretraining Compute by 48%

A team from the Institute of Foundation Models, USC, and CMU has published research showing that latent feedback, an architectural technique that pipes each token's final hidden state back into the next token's input, can save up to 48% of pretraining compute and 47% of unique training data compared with vanilla Transformers under compute-matched conditions. The paper, posted November 11, 2026, by Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, and Hector Liu, introduces a joint scaling law that predicts held-out 778M model losses to within 0.3% and the gap between architectures to within 0.05%. Officially released on November 11, 2026, the work addresses a core challenge as high-quality text becomes scarce: how to extract more value from the same data without simply repeating epochs.
How Latent Feedback Works
The latent feedback mechanism routes the top-layer representation of each token into the input of the next token through a learned mapping. This costs extra forward passes per update but allows each token to perform more computational work during training. The authors argue that when unique data runs low, the standard practice of training for more epochs yields diminishing returns because each repeat of the same data contributes less to loss reduction than the last. By contrast, latent feedback lets models extract more signal from the same unique tokens by deepening the computation per token rather than simply cycling through the data again.
Compute Savings Scale With Data Per Parameter
The efficiency gains depend on how much unique data each parameter sees during training, measured in unique tokens per parameter (UTPP). At two epochs and 12.5 UTPP, latent feedback costs 8% more compute than the Transformer baseline. At 100 UTPP, it saves 15% to 18%, and at 400 UTPP, it saves 44% to 48%. On the data side, at one epoch on 50 UTPP it saves 16% of unique tokens at the 271M parameter scale and 20% at 778M. At three epochs on 100 UTPP, it saves 41% at 271M. The crossover occurs around 33 UTPP at two epochs, a regime the authors say today's 1B-10B dense models have long surpassed. They surveyed 257 base model reports and found the median trains on 1,040 tokens per parameter, implying at least 260 UTPP even over four epochs.
Unified Scaling Law Predicts Held-Out Models
The team fitted a single scaling law in which latent feedback behaves like a Transformer with effectively more parameters. Trained on models from 60M to 271M parameters, the law predicts the validation loss of a held-out 778M model to within 0.3% and the performance gap between the two architectures to within 0.05%. This predictive accuracy suggests the gains are not an artifact of small-scale experiments but a systematic property that extends to larger models. The abstract reports no per-benchmark downstream task numbers, only that downstream results agree with the pretraining trends, and the ladder tops out at 778M parameters, well below frontier training runs.
Code Release and Broader Context
The code is publicly available at github.com/ifm-ai/LoRM. The paper arrives amid a steady stream of training-efficiency research as labs confront the twin constraints of limited high-quality text and rising compute costs. While the work stops short of frontier-scale validation, the unified scaling law and compute-matched comparisons offer a clear framework for evaluating latent feedback in larger runs. The authors position the technique as a data and compute amplifier, one that becomes more valuable as training budgets push further into the multi-epoch, high-UTPP regime where most production models already operate.
What This Means
Latent feedback represents a shift from scaling through more data or more layers to scaling through deeper per-token computation. For labs training 1B-10B parameter models on trillions of tokens, the 48% compute savings at high UTPP could translate directly into faster iteration cycles or lower training costs. The joint scaling law also provides a blueprint for predicting when and where the technique pays off, reducing the risk of adopting it in production. As the industry moves toward more efficient architectures, latent feedback offers a concrete path to doing more with less, especially in regimes where simply adding more epochs no longer moves the needle.
on Emergent today





