While this is all still speculation, it would be consistent with Mythos\u2019s jump on GraphWalks BFS 265K\u20131M (80% vs Opus 4.6\u2019s 38.7%) \u2014 exactly the kind of recursive graph-traversal task where iterative latent refinement should shine.<\/p>\n\n\n\n
Whether there’s truth to the rumours or not, the underlying idea of Looped Transformers is both interesting and deceptively simple: take a transformer, loop it over its own representations, and watch it reason.<\/p>\n\n\n\n
The lineage behind this idea runs from Universal Transformers (Dehghani et al., 2019<\/a>) through the recent theoretical analysis of Looped Transformers (Saunshi et al., 2025<\/a>), depth-recurrent architectures (Geiping et al., 2025<\/a>), and the TRM\/HRM family of models that my co-authors and I have been working on. The headline result that sparked much of this recent buzz was a model with only 7 million parameters, looped repeatedly over latent states, achieving 44.6% on ARC-AGI-1 \u2014 dramatically outperforming comparably-sized models and notably surpassing many commercial LLM APIs on this benchmark.<\/p>\n\n\n\n Our recent\u00a0ICLR 2026 workshop paper<\/a>\u00a0also sits squarely in this space \u2014 we replaced the Transformer blocks in TRM with Mamba-2 hybrids and found something interesting:<\/p>\n\n\n\n I will get to those results later in this post, but first, I want to step back and think about reasoning with LLMs at a higher level. Before we can answer questions like why looped transformers have a reasoning edge, whether we should be looping all LLMs, or what the final form of a reasoning model looks like, we need to understand what reasoning actually\u00a0is<\/em>\u00a0in the context of these models. <\/p>\n\n\n\n What is the source of reasoning? Is it the visible chain-of-thought \u2014 the quality of the intermediate steps, the right decomposition? \u2014 or is it the iterative process itself, providing enough forward passes for the model to refine its answer regardless of what the tokens say? Does reasoning come from the trace or from the recursion?<\/p>\n\n\n\n These questions are not academic <\/strong>– <\/strong>they determine whether the path forward is better prompting, better training data for chain-of-thought, or fundamentally different architectures that think in latent space. Getting the framing right matters as we invest in any particular direction.<\/p>\n\n\n\n An autoregressive transformer models a conditional distribution over token sequences. Given an input \\(x\\), it assigns probability to a continuation by predicting one token at a time. This is the standard next-token prediction view, and it is correct but incomplete.<\/p>\n\n\n\n The problem is that reasoning is not well captured at the token level. A final answer is a semantic object that can be expressed by many different token strings. \u201c42\u201d, \u201cthe result is 42\u201d, and \u201cforty-two\u201d are all the same answer. The right abstraction is therefore not the token distribution but the induced distribution over semantic outcomes:<\/p>\n\n\n\n \\(P(A = a | x) = \\sum_{y \\in Y(a)} ^{} P(y|x)\\)<\/p>\n\n\n\n where \\(Y(a)\\) is the set of all token sequences that express answer \\(a\\). Reasoning, from this perspective, is the process of shifting probability mass among semantic outcomes \u2014 not among surface strings.<\/p>\n\n\n\n This reframing matters because a model might assign low probability to a terse answer but high probability to a full explanation that ends in that answer. The model\u2019s reasoning behaviour is better studied as a distribution over possible solution trajectories and final outcomes.<\/p>\n\n\n\n It is tempting to describe autoregressive reasoning as repeated self-refinement: the model keeps processing the same problem until the answer distribution converges. But this is not quite right.<\/p>\n\n\n\n Every generated token becomes part of the next condition. After the model emits a token, the context changes, and the next distribution is conditioned on a new state. Autoregressive reasoning is better described as path-dependent conditional refinement<\/strong> rather than convergence. The model is not \u201cgetting more confident by re-reading the prompt.\u201d It is constructing a new context at each step, and its predictions are shaped by the path it has taken.<\/p>\n\n\n\n This process may sharpen the answer distribution: the entropy of the answer given the reasoning trace is often lower than the entropy given only the prompt. But entropy reduction is not the same as correctness. If the model generates an incorrect intermediate step, later predictions condition on that error and may become confidently wrong.<\/p>\n\n\n\n A transformer can reason at several different levels:<\/p>\n\n\n\n The key distinction is that self-generated reasoning usually adds computation and structure, while externally provided context can add genuinely new information. Both sharpen the answer distribution, but generally through different mechanisms.<\/p>\n\n\n\n We think the most useful abstraction is to treat reasoning as inference over latent solution paths. Let\u00a0\\(x\\) be the problem,\u00a0\\(z\\) a latent reasoning process (a proof, a plan, an algorithm), and\u00a0\\(a\\)\u00a0the final answer. Then:<\/p>\n\n\n\n \\(P(a|x) = \\sum_{z} P(a|x,z)P(z|x)\\)<\/p>\n\n\n\n A direct-answer model tries to marginalise over all possible reasoning paths implicitly \u2014 in a single forward pass. A chain-of-thought model usually<\/em> generates an explicit approximation to one such path and then predicts the answer conditioned on that.<\/p>\n\n\n\n This explains both why chain-of-thought helps and why it can fail. Estimating\u00a0\\(P(a|x,r)\\)\u00a0for a specific reasoning trace \\(r\\)\u00a0is often easier than estimating\u00a0\\(P(a|x)\\)\u00a0directly, because\u00a0\\(r\\) introduces intermediate variables and constraints. But if the sampled path\u00a0\\(r\\)\u00a0is wrong or biased, conditioning on it pushes the model toward an incorrect answer \u2014 it can become more confident because the context is more specific, rather than because it is more true.<\/p>\n\n\n\n As an enhancement, self-consistency methods exploit this view: sample multiple candidate paths and aggregate over the answers. In probabilistic terms, this is a crude approximation to marginalising over multiple possible\u00a0\\(z\\)’s rather than trusting one sampled trajectory.<\/p>\n\n\n\n The most obvious benefit of chain-of-thought is computational. A decoder-only transformer has bounded computation per generated token \u2014 one forward pass through the layer stack. If the model answers immediately, it must map the prompt to the answer in that single pass. But if it generates\u00a0\\(T\\) intermediate tokens first, it gets\u00a0\\(T\\) additional forward passes, each conditioned on an expanded context. Chain-of-thought thus buys serial depth at inference time.<\/p>\n\n\n\n Merrill and Sabharwal (2024)<\/a>\u00a0made this precise in \u201cThe Expressive Power of Transformers with Chain of Thought.\u201d They proved that allowing a transformer to generate intermediate tokens fundamentally extends its computational power, with the expressiveness scaling with the number of intermediate steps. A polynomial number of CoT steps enables transformers to solve exactly the class of polynomial-time problems \u2014 something a bounded-depth transformer without intermediate tokens cannot do. The implication is clear: even if the intermediate tokens contained no semantic information at all, the extra forward passes alone could expand what the model can compute.<\/p>\n\n\n\n If the computational depth is the key benefit, a natural question follows: does the content<\/em> of the reasoning trace play any role, or is it just a vehicle for additional forward passes?<\/p>\n\n\n\n The \u201cdot by dot\u201d result makes this sharper.\u00a0Pfau, Merrill, and Bowman (2024)<\/a>\u00a0showed that meaningless filler tokens \u2014 repeated dots \u2014 can help transformers solve certain algorithmic tasks that they cannot solve without intermediate tokens. Their conclusion is that additional tokens can provide computational benefits independent of the semantic content of those tokens. The content was empty; but the computation was useful.<\/p>\n\n\n\n There is further evidence from the faithfulness literature.\u00a0Lanham et al. (2023)<\/a>\u00a0systematically measured how much models actually condition on their own chain-of-thought when predicting final answers. Their findings are revealing: models show large variation across tasks in how strongly they rely on their CoT, sometimes heavily conditioning on it, sometimes largely ignoring it. Most telling, as models become larger and more capable, they tend to produce\u00a0less<\/em>\u00a0faithful reasoning \u2014 suggesting that stronger models may generate explanatory text while following a predetermined answer pathway. Such a chain-of-thought begins to look more like a post-hoc rationalisation, rather than a causal driver of the answer.<\/p>\n\n\n\n Taken together, these results build a compelling case that the internal recursion \u2014 the additional forward passes \u2014 contributes significantly to reasoning competence, potentially more so than the semantic content of the reasoning trace itself.<\/p>\n\n\n\n However, it would be premature to conclude that the reasoning trace is merely a vehicle for computation. There are important confounders that complicate this narrative.<\/p>\n\n\n\n The scratchpad as external memory.<\/strong> A transformer\u2019s hidden activations during one forward pass are transient. Without a scratchpad, the model must compress all relevant intermediate computations into the current residual stream. With a scratchpad, it can write intermediate results into the token sequence, and those tokens become part of the context \u2014 in implementation terms, they produce key-value cache entries that later tokens can attend to. The scratchpad creates an addressable external memory.<\/p>\n\n\n\n Consider arithmetic:<\/p>\n\n\n\n <\/p>\n\n\n\n The token \u201ccarry 1\u201d is not just an explanation. It is a stored variable. Later computation attends to it. By pinning intermediate variables into token space, the model does not need to maintain all intermediate state implicitly in a single compressed hidden representation. It can externalise state into the context and retrieve it later through attention. This is a genuine benefit of the content<\/em> of the trace, not just its length.<\/p>\n\n\n\n Commitment and discretisation.<\/strong> There is another subtle role: the scratchpad turns soft latent possibilities into explicit symbolic commitments. Before writing an intermediate result, the model may have a distributed representation containing several possible latent paths. Once it writes \u201cThe carry is 1,\u201d future predictions are conditioned on that discrete commitment. The model has selected a path.<\/p>\n\n\n\n This can help \u2014 it reduces the space of downstream possibilities and makes the remaining computation simpler. But it can also harm. If the model commits to the wrong intermediate result, subsequent tokens build a coherent but incorrect continuation. The reasoning thus becomes confidently wrong because the commitment was wrong.<\/p>\n\n\n\n This trade-off between commitment and flexibility points to something important. Research on latent reasoning suggests that continuous latent representations can maintain multiple possible reasoning paths simultaneously rather than committing to one.\u00a0Hao et al. (2024)<\/a>\u00a0showed in COCONUT that continuous thought vectors can encode multiple alternative next steps at once, enabling the model to perform what is essentially\u00a0breadth-first search<\/strong>\u00a0over the solution space \u2014 exploring many possibilities in parallel rather than committing to a single depth-first path as explicit CoT does. This idea has been further developed at ICLR 2026 by work on\u00a0continuous chain-of-thought (\\(CoT^2\\))<\/a>, which establishes theoretically how continuously-valued tokens enable models to track multiple discrete traces in parallel within a single inference pass.<\/p>\n\n\n\n Scratchpad tokens are not necessarily faithful reports of reasoning, but they can be active components of reasoning.<\/p>\n<\/blockquote>\n\n\n\n So the picture could be genuinely complex. If the reasoning trace is not just a vehicle for forward passes, but neither is it a faithful transcript of internal computation, then the truth could sit uncomfortably in between. Understanding where exactly it sits may be crucial for designing the optimal form of reasoning architecture. This is one of our future research directions to investigate.<\/p>\n\n\n\n In the meantime, it is useful to think of explicit CoT and latent reasoning as two ends of a spectrum. <\/p>\n\n\n\n On one end, chain-of-thought externalises everything: intermediate steps are visible tokens that provide computation, memory, and discrete commitments \u2014 but at the cost of path commitment and potentially unfaithful traces. On the other end, latent reasoning internalises everything: iterative refinement happens entirely in hidden representation space, with no visible tokens, no discrete commitments, and the ability to maintain multiple reasoning paths simultaneously.<\/p>\n\n\n\n Neither extreme is necessarily the right answer. The optimal form of reasoning may sit somewhere in between \u2014 perhaps combining the computational depth of latent iteration with selective externalisation of key intermediate state. Our goal is to verify where the balance lies, and then find it.<\/p>\n\n\n\n In the meantime, studying the latent extreme is instructive. This is what looped transformers do.<\/p>\n\n\n\n The Tiny Recursive Model (TRM) is a concrete instantiation of this idea. Instead of generating explicit reasoning tokens, TRM maintains two latent state vectors (\\(z_H\\) and \\(z_L\\)\u200b) and iterates them through a learned update function. The \u201cscratchpad\u201d is the hidden state itself, updated through 3 outer cycles and 4\u20136 inner cycles. No tokens are emitted during reasoning \u2014 the refinement happens entirely in representation space.<\/p>\n\n\n\n The scale context is striking. TRM has roughly 7 million<\/em> parameters. For reference, GPT-2 Small has 117 million. This is not a large model thinking hard; it is a tiny model thinking repeatedly. And yet it outperforms models orders of magnitude larger on ARC-AGI, a benchmark specifically designed to test abstract reasoning and generalisation.<\/p>\n\n\n\n The recursive process \u2014 not scale \u2014 appears to be the key ingredient.<\/p>\n\n\n\n This raises a natural question. If the power comes from the loop, does the specific operator being looped matter? TRM uses standard Transformer blocks: attention layers for cross-position communication, MLPs for per-position computation. But attention is not the only option.<\/p>\n\n\n\n Mamba-2, a state space model, processes sequences through a recurrent state update:<\/p>\n\n\n\n \\(h_t = a_th_{t-1} + B_tx_t\\)<\/p>\n\n\n\n where the parameters are input-dependent, allowing the model to selectively propagate or forget information. This recurrence is itself a form of iterative refinement. There is a conceptual elegance to putting an inherently iterative operator inside an iterative loop \u2014 recurrence within recurrence.<\/p>\n\n\n\n The practical question is whether Mamba-2 can enter the design space of operators for recursive reasoning without degrading capability. We tested this directly.<\/p>\n\n\n\n\n
Reasoning as Conditional Distribution Shaping<\/h2>\n\n\n\n
Reasoning is not convergence<\/h3>\n\n\n\n
Three levels of computation<\/h3>\n\n\n\n
\n
The latent-variable view<\/h3>\n\n\n\n
What Is Chain-of-Thought Actually For?<\/h2>\n\n\n\n
CoT as additional computation<\/h3>\n\n\n\n
Does the reasoning trace actually matter?<\/h3>\n\n\n\n
But that is not the full picture<\/h3>\n\n\n\n
37 + 48
7 + 8 = 15, write 5 carry 1.
3 + 4 + 1 = 8.
Answer: 85.<\/pre>\n\n\n\n\n
From Scratchpad to Latent Loop<\/h2>\n\n\n\n
Swapping the Operator: Results on ARC-AGI<\/h2>\n\n\n\n
![\"\"](\"https:\/\/fin.ai\/research\/wp-content\/uploads\/2026\/05\/image-1-1024x681.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/fin.ai\/research\/wp-content\/uploads\/2026\/05\/image-2-1024x737.png\")
<\/figure>\n\n\n\n