Infinite-Parameter LLMs: Generating Weights From Live Data, Not the Prompt

Researchers from Cambridge — Jinli Hu, Ross M. Clarke, Yichuan Zhang, and José Miguel Hernández-Lobato — published "Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data" on arXiv on September 16, 2026. The proposal: instead of stuffing everything a model needs to know about the current conversation into the prompt, where it's re-read every request and discarded when the session ends, compile that live context directly into a compact set of generated weights via a hypernetwork — updated continuously as the session proceeds, using a Bayesian belief rather than a single frozen read. It drew a substantive 117-point Hacker News discussion, much of it correctly distinguishing this from a shipped product and asking the right questions about what it would actually change if it works. Here's the mechanism, honestly assessed.

TL;DR

The problem, stated precisely

Scaling laws hold that language models get more capable with more parameters and more training data, and Mixture-of-Experts architectures have ridden that relationship hard — activating a fraction of an enormous, statically stored parameter bank per token. That entire framework is built on static pretraining data. A model in actual deployment, though, faces something different: much of what would make it more useful right now isn't in its training set at all, it's in the live interaction currently happening — facts a user just supplied, a correction they just gave. A conventional model's weights are frozen after training, so none of that can be learned in the ordinary sense. Instead, it gets placed in the prompt via retrieval or instruction, re-read on every single request, and thrown away the moment the request ends.

That's a real, familiar inefficiency to anyone who's built a RAG pipeline or a long-running agent conversation: the same background facts get re-fed into the context window turn after turn, consuming tokens and compute on every pass, with nothing persisting once the session closes.

The proposed mechanism

The paper's answer, inspired by MoE's own logic, is the Infinite-Parameter LLM: a compact hypernetwork that turns data supplied at run time into a low-rank modulation of a shared base network, so the feed-forward weights actually used for generation are produced from live data rather than pulled from a fixed, pretrained bank. Where prior weight-generation approaches read the run-time context once and then freeze the resulting weights for the rest of the session, this paper's contribution is carrying a Bayesian belief over the hypernetwork's latent code and updating that belief online as the session continues — so the effective weight the model uses is continuously re-derived from an evolving belief, not fixed after one initial read.

The claimed payoff, if the approach works as described: the model's stored footprint (the hypernetwork itself) stays fixed and small, but the space of weights it can effectively compile is, in principle, unbounded — hence "infinite-parameter." For the specific category of knowledge currently carried in the prompt, the paper argues carrying it in generated weights instead is amortized in compute (paid once via generation rather than re-processed every request), frees the context window for other use, persists across turns without re-supplying it, and can generalize better than in-context use for that same information.

What this is not

It's worth being precise about scope, because Hacker News discussion of the paper split cleanly between people engaging with the actual proposal and people extrapolating well past it. This is not a released model, a product, or a demonstrated system at any meaningful scale — the paper itself specifies an evaluation protocol for testing the approach against in-context learning and retrieval, which is a research-paper way of saying "here's how you'd check if this actually works," not a claim that it's been checked and confirmed at scale yet.

One sharp, technically grounded comment on the discussion thread cut through some of the more speculative framing: what the paper actually demonstrates is closer to an architectural mechanism for updating weights online via projection through a smaller matrix — a dynamic, generalized version of LoRA (low-rank adaptation, an existing and widely used fine-tuning technique) — rather than a change to model architecture or training approach at the scale the "infinite-parameter" name might suggest to a casual reader. Weights changing during a session isn't itself new (LoRA adapters, prompt tuning, and various continual-learning methods already do versions of this); the paper's actual novel contribution is the online-updated Bayesian belief mechanism specifically, layered on top of that broader idea.

The speculative extrapolations, and why they're premature

A meaningful share of the Hacker News discussion jumped straight to consequences the paper doesn't claim: a "web 4.0" built on vector databases instead of text, a global network where any individual's discovery instantly updates every deployed model, continuous learning as a step toward machine consciousness. These are the kinds of extrapolations that show up whenever a paper touches "models that update at run time" — understandable given how consequential continual learning would be if it worked at scale, but worth separating from what this specific paper actually establishes, which is a proposed mechanism and an evaluation protocol, not a proof that any of those downstream scenarios follow.

The more grounded concerns raised in the same discussion are worth taking seriously on their own terms, independent of the paper's actual scope: if a system's weights genuinely update from run-time data, that reopens questions about data poisoning (an attacker feeding false "corrections" specifically to corrupt behavior) and stability (whether a continuously-updating system could drift into unwanted patterns — catastrophic forgetting, or a system that inadvertently steers toward its input's biases over time, sometimes discussed as a system falling into an unwanted "attractor state") that a frozen-weights deployment doesn't face in the same way. These are reasonable questions for the research direction generally, not established problems with this specific paper's proposal, which hasn't been deployed anywhere these failure modes could actually manifest.

How this differs from simply fine-tuning on the conversation

A natural question is why this needs to be a hypernetwork at all, rather than just fine-tuning the base model's own weights directly on whatever the session has produced so far. The distinction matters for a few concrete reasons the paper's framing implies, even where it doesn't spell each one out explicitly. Fine-tuning a full model, even with parameter-efficient methods, typically requires gradient computation and backpropagation through the model being updated — expensive per-session overhead that doesn't scale to updating weights continuously, turn by turn, inside an interactive conversation. A hypernetwork sidesteps that by learning, during its own separate training phase, a general mapping from "data of this kind" to "the right weight modulation" — so at inference time, producing an updated weight set is a forward pass through a small network, not a full optimization loop.

The Bayesian belief piece adds a second distinction worth separating out. A naive hypernetwork that reads context once and generates weights is still doing a single, static mapping — new information arriving later in the conversation would need a fresh full read of everything supplied so far to update correctly. Carrying an evolving belief over the hypernetwork's latent code instead means each new piece of information can update that belief incrementally, closer to how a running estimate updates with new evidence rather than being recomputed from scratch — the kind of design choice that matters a great deal for whether an approach like this could actually be practical at conversational timescales rather than being a theoretical curiosity.

Honest limitations

• This is a proposal with an evaluation protocol, not a demonstrated result at scale — no benchmark numbers showing it beats in-context learning or retrieval are presented as already-confirmed in the paper as summarized here.
• The relationship to existing techniques (LoRA, prior weight-generation methods) is closer than the "infinite-parameter" framing suggests — the genuinely new piece is the online Bayesian belief update mechanism specifically, not the general idea of context-conditioned weights.
• Safety and stability questions (poisoning, drift) are reasonable concerns for the research direction, not established issues with this paper, which describes an evaluation protocol rather than a deployed system exhibiting these failure modes.
• Published September 16, 2026 — too recent for any independent reproduction or follow-up critique to exist yet.

What this means for builders

If you're building RAG pipelines, long-running agent conversations, or anything that repeatedly re-feeds the same background context into a model's prompt on every turn, this is worth tracking as a research direction rather than acting on today — the core problem it targets (context that should persist but currently gets discarded or re-processed every request) is a real, familiar pain point, similar in spirit to what MEMORY.md-style persistent memory files solve at the prompt-engineering layer rather than the weight layer. Whether a hypernetwork-generated-weights approach ends up being the mechanism that fixes it, versus better retrieval, better prompt caching, or an entirely different architecture, is exactly what the paper's own proposed evaluation protocol is designed to test — worth watching for the actual benchmark results in a follow-up paper before treating this as more than an interesting, early-stage idea.

Related on explainx.ai

• MEMORY.md: long-term persistent memory for AI agents
• What is loop engineering for AI agents?
• RAG vs. MCP: a complete comparison
• Claude Code Projects: shared memory across coordinated threads
• What is recursive self-improvement in AI?
• Scalable oversight: RLHF, Constitutional AI, weak-to-strong generalization
• Paper: arXiv:2609.18842

This post summarizes a research paper published September 16, 2026 and its subsequent Hacker News discussion. The architecture and claims described are the authors' own proposal and evaluation protocol; no independent benchmark results confirming the approach outperforms in-context learning or retrieval at scale were available at time of writing.