Linum Trains 2B Text-to-Image Model for $21K With Open

Two-person research lab Linum has released PJiT, a 2-billion-parameter text-to-image model trained for just $21,000 in compute costs. The model uses a novel pixel-space architecture that bypasses traditional VAE compression, processing 660 million samples across 512² and 1024² resolutions with significantly improved efficiency over previous approaches.

Developed by Sahil and Manu Chopra, the Pyramid-JiT (PJiT) architecture represents a stepping stone toward Linum's ultimate goal of building generative video models from scratch. The team has released open weights for the experimental model, which demonstrates competitive image quality despite its relatively small parameter count and training budget.

Pyramid Architecture Enables 4x Token Reduction

PJiT's core innovation lies in its pyramid architecture, which applies 4x more aggressive token reduction than Linum's previous v2 model. Where standard approaches like FLUX-2 use 16×16 token reduction, PJiT achieves 32×32 reduction, cutting the attention context window by 4x while maintaining image quality. This efficiency gain stems from operating directly in pixel space rather than using a variational autoencoder.

The model processes images by splitting them into 32×32 patches, compressing each patch from 3,072 dimensions (32×32×3 RGB values) down to 1,024 dimensions at a 3:1 compression rate. This architectural choice allows the model to converge in 11x fewer samples than Linum v2, dramatically reducing training costs.

Intermediate Prediction Heads Boost Gradient Flow

To address weak gradient signals in early network blocks, PJiT incorporates intermediate prediction heads along the transformer trunk. The architecture includes:

• Two single-modality refiner blocks each for text and image inputs
• 22 identical DiT blocks in the merged processing stream
• Linear readout heads after blocks 10 and 16 that predict quarter and half-resolution versions of the final image
• A 2-layer PixelREPA readout at block 5 that aligns latent space to DINOv3-L features

This multi-scale prediction approach ensures every block in the network contributes meaningful gradients during training, accelerating convergence. The text encoder uses a frozen Qwen3.5-4B model, reading representations from layers 7, 15, and 27.

Training Pipeline and Resolution Curriculum

Linum trained PJiT through several stages, starting with 512² resolution before transitioning to 1024². The team employed a noise schedule curriculum, concentrating on high noise levels for the first 80 million samples to teach basic structure, then widening the distribution for fine-grained detail learning. When scaling to 1024², they doubled the noise scale from 2 to 4 and used RoPE interpolation to align the higher resolution patches to the 512² coordinate system.

Post-training involved a variant of Finite Difference Flow Optimization with JEV-Image and human judges, though the team notes that diffusion model post-training remains a young discipline compared to large language model techniques. The total compute cost for training and post-training came to approximately $21,000.

Release Date and Availability

Officially launched on November 5, 2026, PJiT is available with open weights on Hugging Face. The model includes automatically generated documentation created by Claude Opus 5.5, which extracted and cleaned code from Linum's internal experiment repository. The team emphasizes this is an experimental release demonstrating their architectural progress toward video generation capabilities.

Known Limitations and Model Constraints

PJiT exhibits several limitations inherent to its size and training approach. The model cannot generate text or create posters, as the training dataset deliberately excluded text-heavy images like infographics, PDFs, and presentation slides. This filtration aligns with Linum's focus on animation tools rather than document generation.

At 2B parameters, the model also struggles with specific scenarios including full-body human poses (particularly leg deformities), scenes with many entities (crowds, multiple vehicles), multi-person interactions, and nuanced world knowledge. The team notes these issues appear occasionally and would likely improve with a larger 7B-parameter model and additional post-training.

What This Means

Linum's $21K training run demonstrates that competitive text-to-image models can be developed on modest budgets through architectural innovation. The pyramid architecture's 4x reduction in attention context windows and 11x faster convergence point toward more efficient paths for building multimodal models. For the broader AI community, PJiT's open weights and detailed technical documentation provide a reference implementation for pixel-space diffusion approaches, particularly valuable for labs pursuing video generation as their end goal rather than static image creation.