Blog Home
Listen
0:00

The Emulation Frontier

New architectures abound in LLMs, interestingly these days they almost invariably follow a particular form; being a restricted subset of existing architectures facilitating comparable performance to larger and/or more intensive models while using fewer resources namely memory, {memory, inter-gpu, inter-node } bandwidth, FLOPs, parameters. Given that these are restrictions that typically means that these can be undone and that these models can be "natively emulated" by other models that are a superset of them. Sometimes by setting a large number of weights as zero or one but often more sophisticated reductions are required.

The result is surprisingly extreme.

I took the top 10k decoder-LLM checkpoints on Hugging Face — 3,906 decoder LMs once duplicate repos are merged, collapsing to 942 distinct architecture-and-size configurations — and found that they can all be natively emulated by no more than 293 of them.

(Almost) everyone knows this is generally true of model families, Llama-3.1 is a clean case: 8B, 70B and 405B are one architecture at three sizes, and going up the ladder every axis either grows or stays the same: - 32 $\rightarrow$ 80 $\rightarrow$ 126 layers - 4096 $\rightarrow$ 8192 $\rightarrow$ 16384 hidden - 32 $\rightarrow$ 64 $\rightarrow$ 128 query heads - 14336 $\rightarrow$ 28672 $\rightarrow$ 53248 FFN size. - 8 $\rightarrow$ 8 $\rightarrow$ 8 KV heads - 128 $\rightarrow$ 128 $\rightarrow$ 128 head dim So 8B is directly a subset of 405B: same slots, more of lots everything , all you have to do is set some weights to zero. Even within a family you don't always get this Llama-2 broke it, the 7B kept full MHA, 32 heads of 128, a 4096-wide cache; the 70B moved to GQA with 8 heads and cached 1024. So the 70B cannot reproduce the 7B's attention without paying to blow the cache back up, despite having 10x the parameters. The efficiency win killed the subset relation for their own family it's not just size, it's which resources a model actually keeps.

What is allowed and what is cheeky?

I am allowing setting of weights, and of any config value that leaves the tensor shapes unchanged and that the host's own code reads (RoPE thetas and scaling, norm epsilons and norm on/off switches, window sizes, attention scales and caps, MoE routing groups and gate normalisation, and the like), however we like, with two exceptions: the activation function and the number of experts each token is routed to stay whatever the host has, since changing those would make some of the most interesting walls vanish by fiat. This should be equivalent to keeping the code to run it and the tensor shapes, and changing only the values in the tensors and the config. I also treat DeepSeek's sparse attention as the dense attention it computes whenever the context is shorter than the number of tokens it selects (2,048). Likewise I treat Phi-3-small's block-sparse attention as the dense attention it approximates (up to 1,024 tokens), and the query temperatures and NTK scalings some models switch on past a set length as plain attention below that length: nothing is credited beyond it. If I were to allow changing sizes almost every model could natively emulate almost every other. Not allowing adjusting things like RoPE thetas could lead to small incompatibilities ruling out emulatability. I will be allowing some amount of bounded error (as in the case of $ReLU \subseteq SiLU$): an approximate swap only counts if the emulating model still generates the same text with high agreement.

I also allow ignoring the size of a tokenizer vocabulary. To me it is a rather superficial feature and if a model is larger in every way than another except for the vocab it seems pretty silly to rule it out for that reason.

I also let the input begin with one fixed special token, a start-of-sequence marker like the BOS most models already prepend. It is the same for every input, so it says nothing about the text, but it gives the host a landmark at position zero. That turns out to matter: with it, a model with no position encoding can reproduce a learned position table exactly, and a model with attention biases can reproduce learned attention sinks. Without a marker, learned absolute positions are provably out of reach of any position-free host, however deep.

Attention masking is a bit of an iffy one, since you would be applying a hard mask where there was none before. But a causal mask can be treated as a parameter, setting some ones to zeros, and local attention is a subset of global attention the same way (RoPE on a few extra dims can approximate a hard local mask very closely). It does mean a diffusion model (full mask) can play an autoregressive one just by masking — though never the other way round.

The cheekiest one, though, is the mixer, and it's one I end up refusing. A big softmax model can in principle play an SSM / linear / hybrid one, but only with thousands of heads per layer ($26$k for mamba-130m, against $128$ for the widest real host), a norm change between layers, or depth that grows with the context (gated DeltaNet). At a price a real host can actually pay, agreement collapses (one fixed-decay layer: $85\%$ top-1, $4\%$ of continuations identical). So the recurrent models stay on the frontier in their own right. (The other way round is free: a hybrid plays a softmax model exactly by zeroing its SSM layers.) Grant that never-exact gap, and the similar cross-layer-KV carry, as free and the frontier shrinks to $265$; demanding every edge reproduce to $0.0$ (softcap and qk-norm folds included) grows it to $320$.

I am also ignoring precision level since to me it does not really feel too specified by the checkpoint, e.g. people running it at lower quantization, we are merging quantizations anyways, so we are including multiple at different quantization levels. However I did want to make sure that things can be emulated with the host model running on fp32 numerics.

What does emulation cost?

In a previous blogpost I showed how RoPE can natively emulate ALiBi. The constructions for MQA, GQA being subsets of MHA is I believe widely known, involving only duplicated weights. But what else is a subset? How deep does it go?

Almost every architectural optimization saves a resource, and emulating the more efficient model usually means spending more of that very resource: MQA shrinks the KV cache, so to play MQA, MHA keeps the same queries and spends more keys and values.

Reductions are not always direct (as in $\mathrm{ALiBi} \subseteq \mathrm{RoPE}$), where the general model only arrives at the special case in a limit — the rotation angle has to go small before the score goes linear. So there are two things worth pinning down for every row: which resource the construction spends, and in what sense the two models are actually equal.

The cookbook (in a few tables)

Each row has a runnable construction with a test. - exact: an identity to fp noise (errors are fp32 max-abs, so they scale with activation size) - module: the stock transformers class parameterized into the special case, forward for forward - limit / floor / gap: an approximation validated as one — a limit goes to 0 as you spend (the rate is given when the resource is width or depth), a floor stops at a nonzero minimum, a gap is a measured nonzero error - exact in-regime: an identity that holds while the inputs stay inside a stated range

The currencies, cheapest first: - free: nothing is spent; the special case is already reachable. - magnitude: a weight, scale, or frequency driven to an extreme (e.g. a huge slope, a tiny angle as in $\text{ALiBi}\subseteq\text{RoPE}$). - fold: absorbed into a neighboring weight matrix, so nothing grows. - KV: wider or more (heads), keys and values - $\infty$: no finite amount is exact — the error shrinks as you spend width but never reaches 0 - mask: the attention mask changes, no weight does. - width / depth: wider hidden/residual size or more layers; 2x where the construction doubles one. - regime: free, but only while the inputs stay bounded; outside that range it drifts. - capacity: total FFN hidden units, active or not. - active capacity: the units actually computed per token

reduction the identity currency validated code
$\text{ALiBi}\subseteq\text{RoPE}$ freeze one head: $\langle R_iq,R_jk\rangle=-N\sin\theta(i-j)\to-(N\theta)(i-j)$ as $\theta\to0$ — the frozen pair needs constant $q,k$ components, i.e. a bias, or an unrotated pair fed by a start token magnitude exact→limit src
$\text{LayerNorm}\subseteq\text{RMSNorm}$ $\operatorname{LN}(x)=\gamma\odot\operatorname{RMS}(Px)+\beta,\ P=I-\tfrac1d\mathbf1\mathbf1^\top$: fold $P$ into every matrix that writes the residual, $\gamma$ into the next weights, $\beta$ into the next layers' biases (including an lm_head bias); tied embeddings must be untied first fold + biases exact $3{\cdot}10^{-5}$ src
$\text{MQA}\subseteq\text{GQA}\subseteq\text{MHA}$ $W_h^K=W_{g(h)}^K$ — tile KV heads with repeat_interleave KV exact 0.0 src
$\text{MLA}\subseteq\text{MHA}$ materialise the keys and values: $W^K=W_{DKV}W_{UK}$, $W^V=W_{DKV}W_{UV}$ (exact for the norm-free form — real MLA RMS-normalises its latents first, which a plain MHA can't reproduce); the shared RoPE key needs a partial-RoPE host KV exact $7{\cdot}10^{-6}$ src
$\text{softmax}\not\subseteq\text{linear}$ $\exp(q^\top k)\approx\phi(q)^\top\phi(k)$, error $O(m^{-1/2})$, never 0 $\infty$ gap src
$\text{local}\subseteq\text{full}$ band mask $M_{ij}\in\{0,-\infty\}$, no weight change mask exact $2{\cdot}10^{-7}$ src
$\text{sparse}\subseteq\text{full}$ apply the $0/1$ pattern $A$ as the attention mask, no weight change (the reverse — a sparse host covering full attention — needs $L\ge\operatorname{diam}$ layers just for reach, $(I\lor A)^L$) mask exact src
retention $=$ linear attention $+$ RoPE $+$ decay (a structural note, not a reduction) $y_n=\sum_{m\le n}\gamma^{n-m}(R_nq_n)^\top(R_mk_m)\,v_m$ — the same $q,k$ rotation as RoPE plus a $\gamma^{n-m}$ decay; the rotation rides in the recurrent state, so the parallel and recurrent forms agree — parallel $\equiv$ recurrent $6{\cdot}10^{-6}$ src
$\text{sink}\subseteq\text{plain}$ a sink is one extra key with a $q$-independent logit $s_h$ and zero value: give the host one more KV slot, a query bias and an unrotated pair so $q^\top k_{\text{sink}}=s_h$ (the reverse is a limit: $s_h\to-\infty$ recovers plain softmax) KV $+1$ exact $6{\cdot}10^{-8}$ src
$\text{dense}\subseteq\text{MoE}$ for any $d_{\text{ff}}\le k\,e_w$ zero the router's input weights so every token routes the same way — which $k$ experts that is depends on the implementation's tie-break (Mixtral: experts 5 and 6), so probe it — deal the dense hidden units across those $k$ blocks, and divide each $W_2$ by its gate weight ($\times k$ for Mixtral, $\times E$ without renormalisation, $\times k/\text{rsf}$ for DeepSeek-V3); a shared expert adds its $e_s$ to the budget active capacity exact $7{\cdot}10^{-7}$ src
$\text{dense}\supseteq\text{top-1 MoE}$ for any $d_{\text{ff}}\ge E\,e_w$ hold all $E$ experts in width (total, not active); never exact: a top-1 MoE jumps at its routing boundary and a dense model is continuous. In practice it is near-exact: one layer computes the router into $E$ tiny carried channels, and a second layer masks the losing experts through a sign-only threshold. That holds for top-1 routers with a unit gate; top-$k{\ge}2$ is a wall: the gate weight is a third factor the carry can't deliver capacity + depth $\times2$ + $E$ channels limit ($100\%$ identical on Huihui-MoE) src
$\text{SwiGLU}=$ sq-FFN $pb=\tfrac14\big((p+b)^2-(p-b)^2\big),\ p=\operatorname{SiLU}(a)$ width $\times2$ + depth exact $10^{-4}$ src
$\text{ReLU}\subseteq\text{SiLU}$ $\operatorname{ReLU}(x)=\lim_{\beta\to\infty}x\,\sigma(\beta x)$, $\beta$ folds into the neighbor magnitude limit src
$\text{GeLU}\approx\text{SiLU}$ best one-parameter Swish-$\beta$ fit misses by $\approx0.0139$ ($0.012$ with the full affine fit $a\,X(cx{+}d)+ex+f$) floor floor src
$\text{swigluoai}\subseteq\text{SiLU}$ $g\,\sigma(1.702g)=\operatorname{SiLU}(1.702g)/1.702$ — fold $1.702$ into $W_{\text{gate}}$ and the $+1$ into an up bias (the host needs one); the gate $\le7$ / up $\in[-7,7]$ clamps are regime fold + bias exact $5{\cdot}10^{-7}$ src
$\text{mish}\approx\text{SiLU}$ $x\tanh(\operatorname{softplus}x)$ — best Swish-$\beta$ fit misses by $\approx0.10$ ($0.010$ with the full affine fit) floor floor src
$\phi\subseteq\text{ReLU-net}$ (SiLU/GeLU/mish/tanh) one hidden layer of $N$ ReLUs is the piecewise-linear $\phi$ ($c_i=$ 2nd differences at the knots) width limit $\sim\!1/N^2$ ($2.4$–$3.9{\cdot}10^{-4}$ at $N{=}256$) src
$\text{SwiGLU},\text{GeGLU}\subseteq\text{ReLU-net}$ the gate is a product $pb{=}\tfrac14((p{+}b)^2{-}(p{-}b)^2)$; one layer suffices but $x^2$ via a Yarotsky sawtooth is exponentially cheaper in depth width $+$ depth limit $\sim\!4^{-L}$ ($3{\cdot}10^{-3}$ at $L{=}6$) src
$\text{seq}\subseteq\text{parallel}$ at $2\times$ depth: a layer with its MLP zeroed, then one with its attention zeroed, gives $x+A+M(x+A)$ depth $\times2$ exact 0.0 src
$\text{parallel}\subseteq\text{seq}$ $x+A+M(x)$ needs the MLP to read $x$, not $x+A$; carrying $x$ in spare channels breaks under a real pre-norm (the norm rescales them), so it costs a norm change norm change exact only without pre-norm src
qk-norm, softcap $\subseteq$ plain $\operatorname{softcap}_c(z)=z-\tfrac{z^3}{3c^2}+O(z^5)$ — folds while $\lvert z\rvert\ll c$ regime exact in-regime src·src
$\text{looped}\subseteq\text{untied}$ $\theta_{t+b}=\theta_t$; depth $Kb$, params $b$ depth exact 0.0 src
$\text{NoPE}\subseteq\text{RoPE}$ $\operatorname{inv\_freq}\to0$ is the identity rotation; rope_theta can't reach it ($\theta^0=1$), but the RoPE scaling factor drives every rotation rate toward $0$ — config only magnitude limit src
$\text{CLA}\subseteq\text{standard}$ carry the shared $K,V$ in spare channels, read them back — the same pre-norm problem width + norm change gap $6{\cdot}10^{-2}$ (pre-norm) src
$\text{MTP}\subseteq\text{plain}$ drop the aux heads; the main next-token logits are identical free exact src
$N{:}M\leftrightarrow\text{dense}$ each dense pair into 2 of 4 slots, nonzeros conserved width $\times2$ exact 0.0 src

Activations, cross-emulated

Zoom in on just the activation slot: the single-op cost of host $X$ emulating target $Y$ — the best $a\,X(cx{+}d)+ex+f$ fit, floor $=\max|\Delta|$ (all analytic, no training):

$Y\!\downarrow$ / $X\!\rightarrow$ ReLU relu² SiLU GeLU mish swigluoai situ xIELU
ReLU — $0.37$ $\to0$ $\to0$ $\to0$ $\to0$ $0.16$ $0.37$
relu² $1.81$ — $0.21$ $0.30$ $0.24$ $0.21$ $0.22$ $\mathbf{0.00}$
SiLU $0.14$ $0.32$ — $0.01$ $0.01$ $\mathbf{0.00}$ $0.07$ $0.32$
GeLU $0.08$ $0.37$ $0.01$ — $0.02$ $0.01$ $0.13$ $0.37$
mish $0.11$ $0.36$ $0.01$ $0.02$ — $0.01$ $0.11$ $0.36$
swigluoai $0.08$ $0.37$ $\mathbf{0.00}$ $0.01$ $0.01$ — $0.12$ $0.37$
situ $0.32$ $0.32$ $0.29$ $0.19$ $0.29$ $0.21$ — $0.48$
xIELU $1.64$ $0.14$ $0.14$ $0.17$ $0.16$ $0.14$ $0.14$ —

Two families fall out. The swish cluster — SiLU, GeLU, mish, and swigluoai (which is $\operatorname{SiLU}(1.702x)/1.702$ exactly, floor $0.00$) — is mutually interchangeable ($\le0.1$) and emulates ReLU near-exactly (the $\beta\to\infty$ limit). The quadratic cluster — squared-ReLU and xIELU — runs one way: an xIELU host plays squared-ReLU exactly ($0.00$), but squared-ReLU and the swish curves reach xIELU only to $0.14$–$0.17$. That is why xIELU "doesn't fold": it lives in a different, quadratic family, and the swish curves sit at $0.14$–$0.37$ from both its members. situ (Kimi-K3) is a distinct saturating curve in between. ReLU is the bottom — a poor single-op host for any curve; it takes a whole ReLU network ($\sim1/N^2$) to bend one — and every pair still closes to $0$ through such a network.

But a floor isn't a currency you can actually pay. I swapped the activation in every FFN of 16 real checkpoints (GPT-2, Pythia, Gemma-2, OPT, SmolLM2, Qwen2.5, Llama-3.2, Apertus, nanochat, a tiny gpt-oss) for the host's best fit and scored it on the original's own greedy text. The exact folds and the $\beta\to\infty$ limits reproduce it token for token; every floor in the table fails, even GeLU from SiLU at $0.012$ — top-1 $97$–$98\%$ but at most $54\%$ of 32-token continuations identical — and the reverse, SiLU from GeLU, manages only $26\%$ top-1 on Qwen2.5-1.5B, whose pre-activations run far outside the fitted range. No cell of the table falls between $0$ and $0.009$, so its verdict doesn't depend on where you draw the line (even "no worse than running the original in bf16" fails at $0.010$); the one measured floor below that, erf vs tanh GeLU at $0.0003$, passes. So in the graph only exact folds, limits and near-zero floors like that one pay; every other single-op floor is a wall. What the graph will pay is width: a $k$-unit fit at $k\times$ the FFN, where the host has it.

Some swing both ways

Some reductions have bidirectional construction — usually a width $\leftrightarrow$ depth trade, you can pay the currency in either resource.

reduction construction A construction B dual code
$\text{MQA}\!\leftrightarrow\!\text{MHA}$ replicate KV heads (MHA copies the shared head) wide shared KV (width $H\!\cdot\!d_h$, each query reads its slice) — A·B
$\text{dense}\supseteq\text{MoE}$ (top-1, unit gate only) all $E\,e_w$ experts in width + hard select $E\times$ depth: one masked FFN sublayer per expert width↔depth src
$\text{seq}\!\leftrightarrow\!\text{parallel}$ parallel emulates seq at $2\times$ depth seq emulates parallel only by changing the norm (spare channels don't survive pre-norm) depth / norm change src
$\text{ReLU}\subseteq$ act $\beta\!\to\!\infty$ limit of Swish $\beta\!\to\!\infty$ limit of GeLU two families src
$\text{sparse}\to\text{full}$ local+strided $\to$ $2$ hops naive local $\to O(N/W)$ depth pattern's diameter src

What's actually irreducible

Little is genuinely incomparable. Most axes have a payable reverse — untie KV → memory, dense → MoE → params, local → full → depth, most pointwise activations → width — so "incomparable" usually means "expensive or approximate one way", and the few walls with no construction at all are what keep the headline above the pure-capacity count. The frontier is then a function of the currency you will spend, and it shrinks monotonically as the budget grows (reproduce):

currency budget frontier what folds in
exact only $320$ nothing — every inexact edge is rejected
$+$ regimes / limits $293$ softcap, qk-norm, ALiBi / ReLU / squared-ReLU limits, the eps lever, the start token and activation width — the headline ($279$ AR $+$ $14$ non-AR); single-op floors don't pay
$+$ the never-exact carries $265$ $\text{softmax}\leftarrow\text{SSM/linear}$ near-gap and the CLA carry, granted free
$+$ fold the type walls $49$ reverse activation / position / mixer, as if every wall were free
$+$ a bigger model $1$ the elephantine dominates the whole Pareto

The second row is the headline. Pay every currency that has a construction behind it and the frontier settles at $293$: some of those are maximal by size (the widest cache, the deepest stack, the biggest FFN), the rest by a wall nobody has a construction for yet — a qk-norm or post-norm the host lacks, biases the host lacks, a top-k that doesn't divide, a recurrent state that doesn't fit, a RoPE target on a NoPE host, squared-ReLU from a swish curve, or any other activation swap that only reaches a floor and that the host has no spare FFN width to fit (GeLU from SiLU pays at $5\times$ width, SiLU from GeLU at $3\times$, and GeLU's erf and tanh forms swap freely). Wave those walls through as free and it falls to $49$, the pure capacity frontier; only the elephantine, bigger than everything, closes the last step to $1$.

So three things actually resist: - Exactness, not possibility. A linear host playing softmax is a true $\infty$ — no finite feature count reproduces the exponential kernel — and the other way round, a softmax host playing a recurrence, needs thousands of heads per layer or depth that grows with the context, so the headline refuses that too. $\text{GeLU}\approx\text{SiLU}$ bottoms at a $0.0139$ floor in a single op and closes only by spending width on a ReLU network — and a $0.01$ floor is already enough to change what the model writes, so in the graph a single op is a wall, not a discount; only a $k$-unit fit at $k\times$ the FFN width pays. - Capacity, and the walls nobody can pay. The $52$ Pareto records — widest cache, deepest stack, biggest FFN, and the trade-offs between them — are irreducible within the zoo, and so are the type walls with no construction yet (a qk-norm or post-norm the host lacks, biases the host lacks, a top-k that doesn't divide, a recurrent state that doesn't fit, RoPE on a NoPE host, squared-ReLU from a swish curve, any other activation swap that only reaches a floor where the host has no spare FFN width to fit it, softmax on a pure-SSM host). Currency buys most type; it never buys size. - Paradigm. AR $\leftrightarrow$ diffusion is only half a resource: a diffusion model plays an autoregressive one by applying the causal mask, but an autoregressive model's own forward pass is causal, so it can't do bidirectional denoising — so the diffusion and block-causal models stay on the frontier however much you pay. (Most autoregressive code will in fact accept a custom attention mask from the caller. If you count that as part of the input, this wall falls too, and I plan to.)

Most of what the earlier draft filed under "hard" folds at a cost: local vs SSM (a hybrid host does both by zeroing the other half, though neither plays the other alone) and the exotic activations (xIELU quadratic, situ finite-$\beta$ — both close to any precision at width, $\sim1/N^2$). Absolute position falls too once the input starts with a fixed token (two extra blocks); without one it is provably a wall.

How much of the headline is walls and how much is size? Suppose every architectural wall had a construction, at the prices the existing ones charge, and only size and the AR/diffusion split still counted. Then 58 shapes would remain. Waive the paradigm split as well and it is 52. That is the floor no clever construction can get under, because nothing in the zoo is bigger than those models on every axis. Everything between it and the headline is a wall, and most frontier models sit behind more than one: no single kind of wall frees more than 22 of them, and 48 have a feature no other shape in the snapshot has.

Caveats on these numbers - **They count shapes, not repos.** A shape is one architecture at one size; fine-tunes and re-uploads of it share it. The snapshot is 3,906 decoder LMs from the Hub on November 9, 2026, collapsing to 942 shapes. A different snapshot gives different counts. - **Emulation here means the same next-token distribution**, checked in fp32. It says nothing about training, speed or memory. - **The rules are the ones under "What is allowed".** Any shape-preserving config value is free, except the activation and the routed top-k. One fixed start token is allowed. DeepSeek's sparse attention and Phi-3-small's block-sparse attention count as the dense attention they compute, and the query-temperature models are credited up to their threshold context. A score or embedding head may be read as a fixed set of output rows at a fixed position, the way decoding reads logits, but mean pooling and normalisation may not. - **Approximate edges are certified by measurement, not by the theorems.** Lean proves each limit exists and how fast it converges. The setting each construction uses is then measured on real checkpoint pairs: top-1 agreement ≥ 99% and ≥ 90% identical 32-token greedy continuations. The theorems' own worst-case thresholds are far too loose to certify any practical setting. - **Some edges hold only up to a context length**: sliding windows, the sparse-attention threshold, the query-temperature threshold. - **A target is judged by everything its family's config could express, not just the values its checkpoint uses.** This keeps the relation transitive, but it is conservative for positions. A plain-RoPE checkpoint of a family that *could* set per-pair frequencies isn't credited to a host whose RoPE code is fixed, even though theta alone would reproduce it. That affects a handful of frontier models. - **The eps-lever constructions need bf16 or fp32 at run time**; in fp16 the tiny values they rely on underflow. - **Models whose loaders need remote code** were classified from their code read as text and checked against transcriptions, not run. 14 models have no code anywhere and are left unplaced. - **A wall means no construction is known**, and for many of them one is provably impossible (each wall's status is in the proofs). Some may still fall to cleverer constructions. With the bias, norm, qk-norm, output-gate and sink walls waved through, the headline would be 250. That is what's at stake, not a claim. - **The four modes nest as they should**, and the relation is transitive on every real pair in every mode. A Z3 model of the rules agrees with the code on all 887,364 ordered pairs. Transitivity beyond the real data is not yet proved symbolically.

What hides in the code

A config file is a summary, and the summary leaves things out. The architecture a model actually runs is in its modelling code, and almost every bug I found in my own graph came from trusting the summary. Some recurring kinds:

Each fix moved the frontier. Altogether the code-level reading, net of the new constructions it prompted, took it from 106 to 293. Every rule now has a machine-checked proof, a measured agreement test, or at least a verified end-to-end instance behind it: 740 Lean theorems for the constructions and the walls, the high-agreement test for every approximate edge, and, for 8 simple switch-offs still waiting on their one-line theorems, a built and checked example. The code that applies the rules is checked too. A Z3 model of the rules agrees with the code on every ordered pair of real shapes, with no disagreements. 70 randomly sampled headline edges were built end to end and all passed (the approximate ones aren't built, so they rest on their agreement tests), and a reduction catalog builds and checks one concrete instance of every kind of reduction the relation uses.

The models

A classifier reads the config.json, and then assigns $17$ slots (norm, pos, kv, ffn, act, block, window, qk-norm, softcap, mixer, sink, xkv, mtp, gen, loop, head, input). Then I use a size-aware relation $\operatorname{emulates}(B,A)$: $B$ can emulate $A$ only with at least $A$'s KV capacity, query capacity, query-head count, FFN total/active width, effective and parameter depth, and hidden size — after paying each differing slot's currency, plus the forward-pass features no slot carries (biases, extra norms, tied embeddings, output gates, router details).

Over the top $10\text{k}$ models on the Hub there are $\sim\!4.4\text{k}$ deduped: $$4384\text{ configs}\ \to\ 3{,}906\text{ models}\ \to\ 942\text{ shapes},\quad \text{frontier}=293\ (279\text{ AR}+14\text{ non-AR}).$$1 The classifier drops $29$ non-decoder types (T5, BERT) and $372$ that can't be placed as causal LMs, and $77$ speculative drafters, leaving $3{,}906$ decoder LMs ($3{,}330$ softmax, $559$ hybrid, $14$ SSM, $3$ linear), which collapse to $942$ distinct arch+size shapes. A bigger model can adopt most activations, positions and mixers at a currency cost, but not all of them, nor every feature: where no construction exists (a qk-norm or post-norm the host lacks, biases the host lacks, a top-k that doesn't divide, a recurrent state that doesn't fit, RoPE on a NoPE host, squared-ReLU from a swish curve, an activation swap that only reaches a floor the host has no spare FFN width to fit, a softmax target on a pure-SSM host, and the other way, an SSM target on a softmax host) the type wall stands. Counting only the walls you can actually pay leaves $293$ maximal shapes, $279$ autoregressive plus $14$ diffusion or block-causal. Treat every wall as free and it falls to $49$; grant the never-exact mixer and CLA carries and it's $265$; demand every edge exact and it's $320$.

The frontier doesn't count vocab / tokenizer size, so vocab_size is off by default (turn it on and the frontier grows from $293$ to $327$ as wide-vocab hosts split off).

The funnel: 4,384 configs → 3,906 classified decoder LMs → 942 arch+size shapes → 293 frontier emulators (pure emulatability).

Every model and every edge is browsable in the interactive graph: search for a model to see what it emulates, what emulates it, and why it is or isn't on the frontier.

It also works as a plain architecture reference. For about 3,906 models it gives the details read from the modelling code, not just the config, and each model's size, KV cache, FLOPs, memory and bandwidth, each marked with whether it has been validated. In a head-to-head on 30 tricky architectures it was the most accurate source I tested. You can paste in any model to look it up, and a Variants view shows which attention, norm and MoE variants are used by which models.

It also says where each model runs: native support in transformers, vLLM, SGLang, llama.cpp, MLX, TGI, ExecuTorch, ONNX and transformers.js, plus the "backdoor" routes, i.e. running an unsupported model through a supported architecture that emulates it.

The crown froze: the top emulator is still Qwen-72B's shape, and that isn't a recent frontier model; it's a 2023 one. (In the table it goes by TW3-JRGL-v2, a checkpoint with Qwen-72B's exact shape saved in Llama format.) Its full-MHA cache ($\mathrm{kv}8192$) is the exact capacity later architectures spend GQA, MLA, and sliding windows to compress, and it's the most general emulator precisely because it wasted nothing. Kimi-K2.6 and DeepSeek-V2 have enormous total parameters but rank-$576$ MLA caches, so neither can emulate that plain 2023 72B: the optimization that made them cheap to run made them less general.

frontier model shape emulates
TW3-JRGL-v2 (Qwen-72B) RoPE·MHA·SwiGLU, $h8192\ L80\ \mathrm{kv}8192$ $\mathbf{1{,}489}$
Qwen1.5-110B RoPE·GQA·SwiGLU, $h8192\ L80$ $1{,}378$
Qwen3-Coder-480B-A35B-Instruct-FP8 RoPE·GQA·MoE, $h6144\ L62$ $1{,}232$
Seed-OSS-36B-Base RoPE·GQA·SwiGLU, $h5120\ L64$ $1{,}215$
K-EXAONE-2.0-750B-A37B RoPE·GQA·MoE, $h6144\ L78$ $1{,}184$
Ling-1T RoPE·GQA·MoE, $h8192\ L80$ $1{,}182$
PseudoBrain-Qwen3-32B-BF16 RoPE·GQA·SwiGLU·hybrid, $h5120\ L64$ $1{,}173$

Every model as a point — parameter count (rough proxy) vs how many models it can emulate. Qwen-72B's shape emulates ~1,489 while models up to ~38× larger emulate far fewer; the frontier (red) is the maximal set.

If we flip things around and ask "What is the model that is able to be emulated by the most others?" the answer is a toy: tiny Llama-shaped checkpoints like SimpleStories-1.25M and v0-llama2-100k are subsets of ~$2{,}880$ models each. Among the recognisable ones, pythia-14m is a subset of $97$, while the recurrent mamba-130m is a subset of only $12$ and nothing in the zoo emulates rwkv-4-169m; the median model is emulable by $55$ others.

Every model by both degrees: how many it emulates (out) vs how many emulate it (in). The two anticorrelate.

A frontier model usually means a capability frontier, newest, shinyist, most capable, Pareto-best on what it can do. My emulation frontier is the set of models for which no other architecture can be reparameterized simulate its whole forward pass at the currencies above (capacity plus every type wall with a construction behind it). Seeking the capability-frontier usually means cutting compute costs, leaning on MLA, GQA, sliding windows, sparse MoE i.e. step down the emulation order in order to spend those savings making them bigger in other dimensions, total parameters depth, context length trained with. So I'd venture that they mostly coincide as we see with Kimi-K3, Nemotron-3-Ultra, DeepSeek-V4-Pro all being on the emulation frontier. A caveat I can't resolve: this is over open models only. I can't see GPT-5, Claude, or Gemini, so I can't tell you whether they're on the emulation frontier. But anyone at those labs can — run the classifier on your own config.json, check whether any of the $293$ frontier specs dominates it. (I'd guess not for flagships and yes for smaller versions of the families).

To make "mostly coincide" concrete, here are the top-ten open models as of August 2026 by each yardstick — most-used on OpenRouter, highest-rated on Artificial Analysis — and whether each sits on the emulation frontier:

OpenRouter (usage) on frontier? Artificial Analysis (intelligence) on frontier?
Hunyuan-Hy3 ✅ Kimi-K3 ✅
DeepSeek-V4-Flash ❌ → DeepSeek-V4-Pro GLM-5.2 ❌ → Hy4-preview-FP8
DeepSeek-V4-Pro ✅ DeepSeek-V4-Pro ✅
MiMo-V2.5 ❌ → MiMo-V2.5-Pro Nemotron-3-Ultra ✅
Nemotron-3-Ultra ✅ DeepSeek-V4-Flash ❌ → DeepSeek-V4-Pro
GLM-5.2 ❌ → Hy4-preview-FP8 MiMo-V2.5-Pro ✅
Kimi-K2.6 ✅ MiniMax-M3 ✅
MiniMax-M3 ✅ Qwen3-235B ✅
Step-3.7-Flash ✅ gpt-oss-120b ✅
gpt-oss-120b ✅ Mistral-Medium-3.5 ❌ → Hermes-4-405B

$7$ of the top-ten by usage and $7$ by intelligence sit on the frontier (each miss names the frontier model that emulates it with the widest coverage). gpt-oss-120b and MiniMax-M3 are on it because folding their swigluoai gate into SiLU needs an up-projection bias, which the bigger hosts lack. Two misses are mid-weights a sibling already emulates: DeepSeek-V4-Flash and MiMo-V2.5 fall to their own -Pro siblings. GLM-5.2 falls to Hy4-preview, and Mistral-Medium-3.5 — which itself emulates 950 models — is still covered whole by Hermes-4-405B. (Step-3.7-Flash is scored by Step-3.5-Flash, the closest checkpoint on the Hub.)

The maximal (frontier) set vs. distinct configs present over time, each class dated by its earliest Hub upload.

as of unique configs maximal set
2022-04 29 10
2023-01 60 15
2024-01 189 36
2024-07 322 68
2025-01 400 86
2025-07 520 131
2026-01 630 167
2026-11 942 293

The minimal universal emulator: elephantine

There is not one huggingface model that emulates everything (hence the 293 element frontier) But what if there was one? What would it look like? What is the smallest giant model that all of the open models we are running could be considered to be doing a subset of the computation of? It would need to be roughly maximal supreme in every dimension.

Compute it over the zoo and it's one concrete spec:

RMSNorm (LayerNorm by centring) $\cdot$ learned-abs and RoPE $\cdot$ MHA $\cdot$ shared-MoE $\cdot$ SwiGLU (plus GeLU, ReLU, squared-ReLU, mish, SiTU, xIELU, swigluoai and PolyNorm branches) $\cdot$ sequential or parallel block $\cdot$ causal, block-causal or full mask $\cdot$ softmax attention + one general gated-delta recurrence $\cdot$ untied, with $h{=}16384$, $L{=}192$, $\mathrm{kv}{=}14336$, $128$ query heads, and an FFN of $\sim\!2.76\text{M}$ expert units per layer ($530\text{M}$ total).

So ~26.5T total parameters (~4.5T active) — about $4.8\times$ the biggest model anyone has shipped, and roughly $11\times$ the per-token compute of Llama-3.1-405B. Big, but not absurd: it's just what you get by stacking records that already exist. One caveat on that count: it's the spec's attention and FFN weights. It leaves out the recurrent (SSM) branch, which the code lays out densely for generality (carrying even the trained mamba-130m that way takes a ~33B-parameter host), so a literal instance of universal.py at these sizes would be considerably bigger. A leaner SSM layout is an engineering problem, not a representational one.

The host is elephantine/universal.py: one architecture in which every operation some family uses (a norm's centring, a parallel residual, partial RoPE, a full mask, the recurrence, each activation) is a branch with a coefficient, off by default. Loading a model means switching on its branches and copying its weights into them; the same code runs every column below.

One elephantine layer, four settings: the same code, different weights and switches.

"Checked" means against the original's own transformers code: identical in float64 (to ~$10^{-12}$ or better), and within the original's own float32 rounding as shipped (e.g. $2.4{\cdot}10^{-4}$ on SmolLM2's logits, with every greedy token the same). That holds for $3{,}531$ of the $3{,}906$ models, and no test that runs remote code is counted. Another $369$, mostly repos whose remote code I couldn't run, are carried slot by slot by the same tested constructions. The last six aren't carried yet: one is in a gated repo, and five use recurrences I haven't classified. You can try it: hf.co/alexlitz/elephantine has six settings (SmolLM2, an inflated SmolLM2, Pythia, Falcon-H1, a Mamba and a Qwen3 diffusion LM), one per folder:

from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("alexlitz/elephantine", subfolder="pythia-160m", trust_remote_code=True)

From the repo, load("org/model") in elephantine/load.py builds the host for any supported model. Inflate it to any size, or run just the active subset of operations for efficiency; same weights either way. It's a proof, not a server: it computes in float64 with no KV cache, so it's only practical for small models. For running models outside transformers, the zoo has downloadable recipes: tested scripts for PyTorch, JAX, tinygrad, TensorFlow and DyNet, each validated against transformers, for 3,334 of the models.

The minimal universal emulator vs the top real models on each capacity axis — it dominates all of them at once.

Why is it bigger than the biggest model at all, when every axis is only a max over things that already exist? Because the records live in different models. No single checkpoint tops every axis at once — the widest cache is bloom's ($14{,}336$, full MHA), the most depth Ouro-2.6B's, the fattest FFN Kimi-K3.1-Pro's $896$ experts, the most query heads ($128$) DeepSeek-R1's, tied with Llama-405B and a handful of others. The join has to hold all of them at the same time, and it carries that whole FFN budget at its own $16{,}384$ hidden width — wider than most of the models it is copying. Stacking records that no single model pays for together is what pushes the supremum past any one of its parts.

The elephantine (the join of every model on the Hub so far) against the largest single model of the day, month by month. On each axis the join is just that axis's record; the gap in total size comes from no one model holding the records at once.

Almost nothing escapes it. It's one architecture — attention with a causal or full mask, one general gated-delta recurrence, MoE, and a parallel-block branch — and nearly every family in the zoo is a setting of its weights: Mamba-1/2/3, gated-DeltaNet, KDA, RWKV-4..7, Hymba, the LLaDA and DiffusionGemma diffusers (the same weights under the full mask) and the rest. The trick is that the gated delta rule, $S_t = (\mathrm{diag}(a_t) - \beta_t k_t k_t^\top)S_{t-1} + \beta_t v_t k_t^\top$, is what nearly every recurrent mixer is a choice of decay and delta in. It hosts 919 of the 942 shapes. The other 23 each have a part it has no branch for yet, such as memory attention, cross-attention to an encoder (a few text-to-speech models), byte patches, document masks, engram blocks, and a few unusual query norms. On size alone it hosts all 942. The zoo and the repo list them. Read another way, it's a spec for an LLM guaranteed to be able to out-do every model that exists: its hypothesis space is a superset of all of theirs, so whatever weights make Deepseek-v4-pro, Kimi K3, or GLM 5.2 tic, an equally-good (or better) setting lives somewhere inside it (as the models themselves do). Consider the architecture search over (until somebody publishes another frontier model). There's just the one catch, hiding in "able to": you still have to find those weights — and training a $26.5$-trillion-parameter Frankenstein that out-maxes Kimi-K3.1-Pro's FFN and matches Llama's width and bloom's cache simultaneously may cost more than all of them combined.

So what?

Representational supremacy is free; the optimum inside it is not. And to a significant extent the architecture search is already over, architectures have converged, we are mostly simplifying improving efficiency, improving other aspects of training, pushing what can be done on the same hardware and using new hardware.

It doesn't means that these architectures are equivalent in practice.

Most architectural innovations results in giving up a little performance or capacity for a lot less resource usage cost.

But it does suggest that the apparent diversity of decoder architectures is much larger than their underlying structural diversity.

Across 3,906 distinct decoder-LM configurations, a surprisingly small number of more general architectures cover the rest. Much of what we call a new architecture is better understood as imposing a carefully chosen restriction on a more general mechanism in exchange for a resource saving. More yet can be understood as facilitating effective learning of representations than representing things more effectively per-say.

To me this suggests a way of thinking about architectural innovation:

How was it restricted, what does that restriction buy us?

Not all architectural innovations are restrictions/down the hierarchy, notable SiLU cannot be well simulated with a single ReLU and to simulate MoE with Dense we have to pay a hefty price.

AI assistance I used LLMs extensively in developing and writing this article, including drafting and editing technical explanations. I take responsibility for the conclusions and any errors. The code, experiments, and tests are linked in the repository for independent inspection.
How to cite
@misc{litzenberger2026emulation,
  title  = {The Emulation Frontier},
  author = {Litzenberger, Alex},
  year   = {2026},
  url    = {https://alexlitzenberger.com/blog/the_emulation_frontier},
  note   = {Code: https://github.com/alexlitz/the_emulation_frontier}
}

Appendix: the detailed tables

More activation constructions

Optimal small-$K$ constructions. The cookbook's $N$-ReLU floors use uniform knots; placing the $K$ breakpoints by minimax (free knots) is far more efficient — 3 optimal units already match ~16 uniform ones (max$|\Delta|$):

units SiLU GeLU mish tanh
$K{=}1$ $0.140$ $0.086$ $0.110$ $0.605$
$K{=}2$ $0.050$ $0.058$ $0.057$ $0.071$
$K{=}3$ $0.050$ $0.058$ $0.042$ $0.071$

It plateaus at small $K$ (SiLU and GeLU gain nothing from the third unit), then falls at least as fast as the uniform $\sim1/N^2$ rate as the layer widens.

The exotic ones fold too. xIELU and situ aren't walls — they're pointwise curves, so a wider FFN closes them like any other. Uniform $N$-ReLU interpolation, max$|\Delta|$ on $[-6,6]$:

units $N$ $2$ $4$ $8$ $16$ $32$ $64$
situ $0.63$ $0.22$ $0.11$ $0.033$ $0.009$ $0.002$
xIELU $7.2$ $1.8$ $0.45$ $0.11$ $0.028$ $0.0070$

situ closes as $\sim\!9/N^2$, xIELU as exactly $28.8/N^2$ (a pure quadratic's interpolation error) — and an xIELU host plays squared-ReLU exactly ($0.00$), though not the reverse ($0.14$). Both are width-priced near-gap edges, not the categorical wall of absolute position. The headline spends only small $k$ ($k{=}3$–$5$, a $k$-unit fit, not $N$ ReLUs) and only where the host's FFN has the room: an $N$-ReLU stand-in multiplies every FFN unit by $N$, which no host in the zoo has to spare. Squared-ReLU from a swish curve still finds no fit the host's FFN pays, so it stays a wall.

Positional encodings, cross-emulated

The same treatment for the positional slot — each scheme's producible logit field, and whether host $X$'s span contains target $Y$'s (floor $=$ normalized min-max residual; $0=$ exact, $✗=$ no useful fit (floor $\ge0.9$), else the floor):

$Y\!\downarrow$ / $X\!\rightarrow$ NoPE ALiBi RoPE T5-rel learn-abs sinus
NoPE — exact exact exact exact exact
ALiBi ✗ — $\theta^2$* exact exact exact
RoPE ✗ ✗ — exact ✗ ✗
T5-rel $0.73$ $0.73$ $0.07$ — ✗ $0.62$
learn-abs ✗ ✗ ✗ ✗ — $0.43$
sinus $0.30$ $0.30$ $0.19$ exact $0.36$ —

NoPE is the bottom (any scheme zeros its field). T5's arbitrary relative bias exactly hosts ALiBi and RoPE; RoPE⊆T5 is exact but T5⊆RoPE floors at $0.07$ (RoPE is band-limited — it can't make sharp bucket edges). Absolute and relative only half-separate: absolute schemes host ALiBi exactly (and NoPE trivially), and T5's relative bias hosts the sinusoidal field exactly, but nothing relative reaches a learned absolute table, and no absolute scheme reaches RoPE ($✗$). *ALiBi⊆RoPE is the project's seed result: one RoPE pair at angle $\theta$ is ALiBi with error $\sim\theta^2\to0$. (The table is one layer, norm-free, with no start token. Under a real pre-norm the ALiBi ramp doesn't survive, and with a start token a NoPE or RoPE host does reach the learned table.)

The other direction (mostly zeros and ones)

The table above is the load-bearing direction — the general architecture doing the special one's job, at a cost. Read the other way, as a setting: every special case is its general parent with something forced to a $0$, a $1$, a tie, or a limit. Worth stating for completeness. The currency column is the same resource the construction spends.

general $\to$ special the setting currency code
$\text{RoPE}\to\text{ALiBi}$ freeze one head to $q{=}(-N,0),\,k{=}(0,1)$ magnitude src
$\text{RMSNorm}\to\text{LayerNorm}$ fold the centering $P=I-\tfrac1d\mathbf1\mathbf1^{\!\top}$ into every residual writer ($\beta$ into biases) fold + biases src
$\text{MHA}\to\text{GQA/MQA}$ tie KV heads $W_h^K{=}W_{g(h)}^K$ — a $0/1$ grouping; MQA is the single group KV src
$\text{MHA}\to\text{MLA}$ factor $W^K{=}W^{DKV}W^{UK}$ at rank $r$ instead of full rank KV src
$\text{full}\to\text{local}$ set $M_{ij}{=}-\infty$ outside the window mask src
$\text{full}\to\text{sparse}$ apply the $0/1$ pattern mask $A$ mask src
$\text{plain}\to\text{sink}$ append one key with logit $s_h$, value $\mathbf0$ fold src
$\text{MoE}\to\text{dense}$ tie all experts: $\sum_{e\in T(x)}p_e(x){=}1$, so the router drops out and every token sees the same $F(x)$ — no router control needed, but one expert's width $e_w$ is the ceiling ($E{=}1$ is the degenerate case) active capacity src
$\text{SiLU}\to\text{ReLU}$ $\beta\to\infty$ magnitude src
$\text{GLU}\to\text{ungated}$ gate branch $b\equiv\mathbf1$ width src
$\text{parallel}\to\text{sequential}$ split each layer into an attention-only and an MLP-only layer depth $\times2$ src
$\text{plain}\to\text{qk-norm/softcap}$ fold the fixed scale $(r_qr_k)^{-1}$ / keep $\lvert z\rvert\ll c$ so the cap stays linear regime src
$\text{untied}\to\text{looped}$ tie $\theta_{t+b}{=}\theta_t$ depth src
$\text{RoPE}\to\text{NoPE}$ $\text{inv\_freq}\to0$ via the RoPE scaling factor (config only) magnitude (limit) src
$\text{standard}\to\text{CLA}$ reuse (tie) an earlier layer's $K,V$ width + norm change src
$\text{plain}\to\text{MTP}$ append zero-init predict heads free src
$\text{dense}\to N{:}M$ apply the $N{:}M$ zero mask width src

The exchange rates, quantified

Every "currency" above is really a scaling law — resource spent vs error — and each is built and measured analytically (explicit weights, no training):

reduction currency law measured
$\phi\subseteq$ ReLU-net (SiLU/GeLU/mish/tanh) width $N$ $\sim1/N^2$ $2.4$–$3.9{\cdot}10^{-4}$ @ $N{=}256$
SwiGLU/GeGLU gate $\subseteq$ ReLU-net depth $L$ $\sim4^{-L}$ $3{\cdot}10^{-3}$ @ $L{=}6$
ALiBi $\subseteq$ RoPE magnitude ($\theta\!\to\!0$) $\sim\theta^2$ exact→limit
local $\subseteq$ full — RoPE Fourier width $N$ + magnitude Gibbs floor at the edge; $\sim1/N$ interior $0.07\%$ leak @ $N{=}64$
local $\subseteq$ full — stacked window $w$ depth $\lceil(T{-}1)/(w{-}1)\rceil$ exact (mask-composed)
MLA $\subseteq$ MHA KV rank $r$ $\sim r^{-0.7}$ (power-law spectrum) $10^{-3}$ @ $r{=}64$
softmax $\not\subseteq$ linear $\infty$ (features $m$) $\sim1/\sqrt m$ never $0$ ($m{=}65\text{k}\!\to\!10^{-2}$)

Three shapes hide under "approximate": exact-in-the-limit (activations and ALiBi → $0$ as you spend the resource), a genuine $\infty$ (softmax←linear improves as $1/\sqrt m$ but never arrives — fp64-zero would need $m\approx10^{33}$), and a Gibbs/edge floor (a hard mask's discontinuity is unreachable in sup-norm, only softened).

Each row is a runnable module: ReLU-net · RoPE-window · stacked masks · MLA-rank · softmax↝linear; the cross-matrices are activations and positions.

Max context is a demand currency. The window/mask reductions aren't priced in the abstract — they're priced against the context length $T$ you must stay cogent over (attend across). Below the $T$-requirement the receptive field simply doesn't span $T$: a hard floor, not a soft one.

support context $T$ KV memory depth cogency floor
full attention $T$ $1$ none
local window $w$ $w$ $\lceil(T{-}1)/(w{-}1)\rceil$ field $<T$ if too shallow — not cogent
RoPE-window $T$ $1$ frequency dims split resolution × range (Nyquist)
MLA rank $r$ $r$ $1$ SVD tail — worsens as $T$ grows

The trades are conserved: local buys back KV memory but owes depth $\lceil(T{-}1)/(w{-}1)\rceil$; MLA buys memory but owes rank/quality; RoPE-window keeps global compute but owes frequency budget. So emulates gains an opt-in max-context check: a host must support at least the target's context, and a local-window host covering a wider-window target must also pay depth $\lceil(w_t{-}1)/(w_h{-}1)\rceil\times$ the target's; the frequency and rank budgets in the table aren't checked yet. Like the vocab axis it only adds constraints — turn it on and the frontier grows from $293$ to $319$ as short-context hosts stop covering long-context targets.


  1. And not to be pedantic, but it isn't quite a DAG: 29 pairs of shapes emulate each other both ways (11 differ in a window, 7 in GeLU's erf vs tanh form, 4 in the rotary width, 2 in a feature each can switch off, 2 in the MoE routing, 1 only in dims, 1 in the position scheme, and 1 in both GeLU form and rotary width), so each such pair counts once on the frontier. ↩