Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
Recorded: Sept. 17, 2026, 6:09 p.m.
| Original | Summarized |
[2609.18842] Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
Skip to main content Search Log in Search arXiv Press Enter to search · Advanced search Computer Science > Artificial Intelligence arXiv:2609.18842 (cs) [Submitted on 16 Sep 2026] Abstract:The scaling laws hold that a language model grows more capable with more parameters and more training data, and Mixture-of-Experts (MoE) architectures have ridden these laws to remarkable results, activating only a fraction of an enormous stored parameter bank for each token. That success is built on static pretraining data. A deployed model faces a different world, where much of the data that would make it more useful is not in its training set but in the live interaction it is currently handling, such as the facts a user supplies or the corrections they give. A conventional model cannot learn from this data, because its weights are frozen after training. Instead, the knowledge and behaviour supplied at run time are placed in the prompt, by retrieval or instruction, and re-read on every request only to be discarded once the request ends. We ask how an architecture could learn from live interaction by writing it into its weights. Taking inspiration from MoE, we propose the \textbf{Infinite-Parameter LLM}. A compact hypernetwork turns the data given at run time into a low-rank modulation of a shared base network, so the feed-forward weights are generated from live data rather than stored in a fixed bank. Where prior weight generators read the context once and freeze, we carry a Bayesian belief over the generator's latent code and update it online, so the effective weight is re-derived from that evolving belief as the session proceeds rather than fixed after one read. The stored footprint stays fixed, yet the weights the model can compile are effectively infinite. For the knowledge and behaviour supplied at run time, carrying them in the weights rather than the prompt is amortized in compute, frees the context window, persists across turns, and can generalise better than in-context use. We specify an evaluation protocol that tests exactly this against in-context learning and retrieval. Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG) Cite as: Focus to learn more arXiv-issued DOI via DataCite (pending registration) Submission history From: Jinli Hu Dr [view email] [v1]
Full-text links: View a PDF of the paper titled Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data, by Jinli Hu and 2 other authorsView PDFHTML (experimental)TeX Source view license < prev | new Change to browse by: References & Citations NASA ADSGoogle Scholar export BibTeX citation BibTeX formatted citation loading... Data provided by: Bookmark
Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv (What is alphaXiv?) Links to Code Toggle CatalyzeX Code Finder for Papers (What is CatalyzeX?) DagsHub Toggle DagsHub (What is DagsHub?) GotitPub Toggle Gotit.pub (What is GotitPub?) Huggingface Toggle Hugging Face (What is Huggingface?) ScienceCast Toggle ScienceCast (What is ScienceCast?) Demos Demos Replicate Toggle Replicate (What is Replicate?) Spaces Toggle Hugging Face Spaces (What is Spaces?) Spaces Toggle TXYZ.AI (What is TXYZ.AI?) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower (What are Influence Flowers?) Core recommender toggle CORE Recommender (What is CORE?) Author About arXivLabs arXivLabs: experimental projects with community collaborators Which authors of this paper are endorsers? | We gratefully acknowledge support from About Major funding support from |
The paper introduces the concept of the Infinite-Parameter LLM, designed to address the limitation of conventional language models where weights are fixed after training, preventing them from learning effectively from live, run-time interactions. This limitation arises because deployed models often need to incorporate knowledge and behavioral adjustments supplied during live exchanges, such as supplied facts or corrections, which are not present in the static training dataset. The authors propose an architecture, drawing inspiration from Mixture-of-Experts (MoE) models, to enable the model to learn dynamically from this live data and embed it into its weights. The core mechanism of the Infinite-Parameter LLM utilizes a compact hypernetwork to generate the feed-forward weights as a low-rank modulation of a shared base network, rather than storing these weights in a fixed bank. In previous approaches, weight generators read context once and were frozen, whereas this method maintains a continuous update mechanism. Specifically, the architecture carries a Bayesian belief over the generator's latent code, allowing this belief to be updated online as the session progresses. This process ensures that the effective weights are continuously re-derived from this evolving belief throughout the interaction, rather than being fixed after a single read. This design maintains a fixed stored footprint while effectively allowing the model to compile an infinite set of trainable weights. By incorporating the knowledge and behavior supplied during run-time directly into the weights, the model achieves several advantages over traditional in-context learning methods. This integration of runtime data into the weights is amortized in terms of computational cost while simultaneously freeing up the context window and persisting the learned information across multiple turns. Furthermore, the dynamic adaptation offers the potential for superior generalization compared to relying solely on in-context learning or retrieval techniques. To rigorously validate these claims, the authors specify an evaluation protocol designed to precisely test the performance of the Infinite-Parameter LLM against established methods for in-context learning and retrieval. |