The transformer, rebuilt in parts

Between July and October 2026, four large model reports changed the same parts of the transformer without replacing it. In three of them, most attention layers became linear. Moonshot, Alibaba and DeepSeek reworked the residual stream, and Alibaba and DeepSeek moved capacity into large lookup tables. DeepSeek cut its global KV cache to 890 bytes per token. Post-training recipes converged as well. Diffusion language models made their clearest progress on speed: Google’s DiffusionGemma writes about 1,500 tokens per second on one H100, at a measurable cost in reasoning scores.

Scope: architecture and training of large language models, as disclosed in technical reports, model cards and papers. Period: 1 July to 27 October 2026. Benchmark and speed figures are the authors’ own.

22 July to 26 August: three linear layers for every global one

• What happened: Upstage’s Solar Open 2, a 250B-parameter mixture-of-experts model, interleaves one softmax attention layer among every three linear-attention layers. It uses no positional encoding and reaches a 1M-token window (Solar Open 2). Moonshot’s Kimi K3, at 2.8T parameters with 104B active, uses the same 3:1 pattern of Kimi Delta Attention and gated MLA, with no position encoding on the MLA layers (Kimi K3 report). Its weights followed on 27 July, as announced (Moonshot; model card). On 26 August Alibaba’s Qwen3.8-Flash-Next kept Gated DeltaNet in three of every four layers and replaced the fourth with a sparse attention that selects context in small blocks (Qwen report).
• Why it matters: linear layers keep a fixed-size state, so only the global layers hold a cache that grows with context (Kimi K3 report). Qwen keeps one global layer in four because it “retains the direct token-level retrieval that no finite-state memory reproduces exactly” (Qwen report). A paper from 7 October finds that the choice of cheap layer still matters: linear-attention hybrids gain more from long-context continued pretraining, while sliding-window hybrids extrapolate better (Liu et al.).
• Confidence: high. All three layouts are documented in the technical reports.

July to September: the residual stream gets redesigned

• What happened: Kimi K3 uses Attention Residuals, in which “each layer selectively retrieves representations from all preceding layers rather than accumulating them uniformly” (Kimi K3 report). Qwen’s Gated Residual widens the residual stream to four branches and reads it through an element-wise gate (Qwen report). DeepSeek-V4.1-Flash lists “Single-Pass mHC”, a revised residual-stream mixing (DeepSeek model card).
• Why it matters: Moonshot describes the standard residual as compressing all prior information into a single state, “a bottleneck reminiscent of RNNs over time” (Kimi K3 report). Qwen reports that one residual change barely moved the loss but clearly improved benchmarks, an example of why it checks both (Qwen report).
• Confidence: medium. Only Qwen publishes a residual-specific ablation; Moonshot reports a combined gain from several changes, and DeepSeek’s change cuts memory traffic (DeepSeek report). There is no shared comparison.

26 August and 10 September: parameters in lookup tables

• What happened: Qwen3.8-Flash-Next has a 125B-parameter main model with 6B active per token, plus 51B parameters of n-gram embedding tables that are prefetched from host memory (Qwen report). DeepSeek-V4.1-Flash carries 196B parameters of “Engram conditional memory”, accessed sparsely by token-based lookup (DeepSeek model card).
• Why it matters: a lookup adds capacity at almost no compute per token, and the tables need not sit in accelerator memory (Qwen report). Qwen reports that its base model leads its 397B predecessor on eight of fourteen pretraining benchmarks and trails by at most 2.6 points on the rest. It needed about a ninth of its predecessor’s training compute. It also warns that a larger n-gram vocabulary keeps lowering loss while downstream accuracy saturates.
• Confidence: medium. The design is documented; the efficiency claims are Qwen’s.

10 September: DeepSeek shrinks the global cache to 890 bytes per token

• What happened: DeepSeek-V4.1-Flash, a 552B mixture-of-experts model, splits 40 layers into a 20-layer causal encoder and a 20-layer decoder. The decoder’s global cache is projected from the encoder’s final states, so the model activates 8B parameters per token in prefill and 16B in decode. With shared sparse-attention indices and a cache stored in FP4, the global cache comes to 890 bytes per token, about a quarter of V4-Flash (DeepSeek model card).
• Why it matters: DeepSeek’s release note says cache-hit charges “often account for a large share of agent costs”. It puts the new cache at a quarter of the HBM and an eighth of the SSD storage of the previous generation. From 14 September, DeepSeek routed all V4-Pro requests to the new model (DeepSeek).
• Confidence: high for the design. The cost effects are DeepSeek’s own.

July to October: one training recipe, many labs

• What happened: three reports describe the same post-training shape. Kimi K3 runs supervised fine-tuning, trains domain experts with reinforcement learning, then merges them by multi-teacher on-policy distillation (Kimi K3 report). Solar Open 2 consolidates twelve domain specialists the same way (Solar Open 2). DeepSeek follows SFT, RL and on-policy distillation “without algorithmic modifications” (DeepSeek model card). In pretraining, Qwen runs the Muon optimizer on its linear maps and AdamW on embeddings, output head and router, and reports that batch-size warmup became unnecessary (Qwen report). Kimi K3 applies quantization-aware training from the fine-tuning stage onward, with MXFP4 weights and MXFP8 activations (Kimi K3 model card).
• Why it matters: DeepSeek says all its substantive post-training changes lie in the data pipeline, mainly synthesised agent tasks and environments (DeepSeek model card). My read: the algorithm is settling, and the labs now compete on environments and data.
• Confidence: high for the recipes as described.

7 and 31 July: diffusion as a decoding mode

• What happened: NVIDIA’s Nemotron-Labs-Diffusion trains one checkpoint on a joint autoregressive and diffusion objective, so it can decode left to right, by diffusion, or with diffusion drafting and autoregression verifying. The 8B model decodes six times more tokens per forward pass than Qwen3-8B at comparable accuracy (Fu et al.). Google’s DiffusionGemma converts Gemma 4 26B A4B with under 10% of its original training tokens and refines 256-token blocks in parallel. At one request it reaches 1,456 tokens per second on an H100: 7.1 times the autoregressive model, 4.8 times with multi-token prediction (DiffusionGemma report).
• Why it matters: the speed has a price. DiffusionGemma scores 69.1 on AIME 2026 against 88.3 for Gemma 4 with multi-token prediction, and 73.2 against 82.3 on GPQA Diamond. Its throughput lead holds up to about 32 concurrent users; beyond that, autoregression wins (DiffusionGemma report). For now, diffusion pays off in low-concurrency, latency-bound work.
• Confidence: high for the facts. Both teams measured their own models.

4 August: diffusion from scratch, with its own scaling laws

• What happened: the LLaDA MoE v2 paper fits scaling laws for mixture-of-experts diffusion models. Compared with autoregressive trends, the optimal batch size grows faster and the optimal learning rate falls faster with compute. The authors then trained a 30B model with 3B active from scratch on 23.5T tokens, which approaches Qwen3 on several benchmarks with about 65% of Qwen3’s pretraining tokens (Zhu et al.).
• Why it matters: converted models inherit choices made for autoregression. DiffusionGemma’s own limitations list names “architectural, optimization, and data-mixture decisions from the AR baseline that may be suboptimal for the discrete diffusion paradigm” (DiffusionGemma report).
• Confidence: medium. One group, and “approaches” covers several benchmarks, not all.

What to watch next

• Qwen4. Alibaba calls Qwen3.8-Flash-Next an early preview of the architecture of Qwen4 (Qwen repository).
• DeepSeek-V4.1-Pro. V4-Pro requests route to V4.1-Flash “until V4.1-Pro launches” (DeepSeek). DeepSeek says the new architecture is designed for scaling to larger models; V4.1-Pro would be its first test. DeepSeek’s 17 October update to Flash uses reinforcement learning with self-feedback to improve end-to-end project development (DeepSeek).
• Hybrid decoding. DiffusionGemma can still decode left to right, which its authors see as “a path toward hybrid diffusion-AR decoding” (DiffusionGemma report). Speculation: the next diffusion releases ship as a fast mode of an autoregressive model, not as a separate model.