---
title: "Memorilla - 16 memory tokens let a frozen LLM read thousands of documents at once"
description: "Stanford's Memorilla compresses document collections into 16 learned tokens for a frozen LLM. FactKG accuracy climbs from 80.43% to 84.33% with 108x fewer tokens. Open source, Apache 2.0."
date: 2026-11-08
reading_time: "4 min read"
url: https://myargoquest.com/blog/memorilla
markdown: https://myargoquest.com/blog/memorilla.md
publisher: Argo (myargoquest.com)
---

# Memorilla - 16 memory tokens let a frozen LLM read thousands of documents at once

# Memorilla: 16 memory tokens let a frozen LLM read thousands of documents at once

An AI agent that remembers everything you have ever told it is the goal. The obstacle is obvious: context windows are finite, document collections grow, and feeding every relevant document into every call is expensive and slow.

Stanford's SNAP group, led by Jure Leskovec, published a solution called Memorilla. It replaces thousands of document tokens with 16 learned memory tokens, and the frozen LLM becomes more accurate on factual verification than if it had read every document in full.

![A six-step pipeline: documents through a frozen encoder into a memory store, then a trainable memory module reads the store plus the query and produces 16 memory tokens for the frozen LLM](https://myargoquest.com/blog/memorilla-1.jpg)
*Documents are encoded once. The memory module compresses query-relevant information into 16 tokens. The frozen LLM reads only those tokens.*

## How it works

Memorilla separates the act of remembering from the act of recalling. A frozen embedding encoder (Qwen3-Embedding-4B) processes each incoming document, such as a conversation or a book chapter, into an embedding that lives in a persistent store. When a question arrives, a small trainable memory module (194 million parameters, about 2% of the decoder size) reads the question and every embedding in the store. It produces 16 query-dependent vectors that the frozen Qwen3-8B decoder reads alongside the question.

The key constraints that guided the design are: every query reads the whole collection, not a pre-selected subset; what memory returns depends on the question; the token budget is fixed regardless of collection size; and new documents can be added without re-encoding the past.

![Two-panel comparison: FactKG score 80.43% with full context vs 84.33% with 16 memorilla tokens](https://myargoquest.com/blog/memorilla-2.jpg)
*On FactKG, Memorilla scores 84.33% against 80.43% for full document context, with 108x fewer document tokens.*

## The results

The paper reports results across eight benchmarks covering personalization, factual reasoning, and long-document QA. Memorilla achieved the best average score among all evaluated methods.

| Metric | Full context | Memorilla (16 tokens) | Improvement | |---|---|---|---| | FactKG accuracy | 80.43% | 84.33% | +3.9% | | Document tokens | baseline | 108x fewer | 108x reduction | | NarrativeQA compression | baseline | 4,000x | 4,000x reduction | | Latency (512 docs, H100) | 19.9s | 0.74s | 27x faster | | TextWorld agent success | 18.6% | 24.2% | +5.6% |

![Six metric cards: FactKG 84.33%, 108x compression, 4000x on NarrativeQA, 0.74s latency, 194M params, agent 24.2%](https://myargoquest.com/blog/memorilla-3.jpg)
*Memorilla improvements across accuracy, compression, latency, and agent performance.*

## The four design requirements

The paper is explicit about four requirements that distinguish Memorilla from prior work. Collection-wide access means every query considers every document, not just the top-N from a retriever. Query-dependent recall means the same collection serves many different queries. The fixed token budget ensures the LLM never sees more than 16 memory tokens regardless of collection size. Reusable embeddings mean documents are encoded once and the store grows without re-encoding history.

![Four labeled boxes: collection-wide access, query-dependent recall, fixed token budget, reusable embeddings](https://myargoquest.com/blog/memorilla-4.jpg)
*Memorilla's four design requirements distinguish it from RAG and other memory approaches.*

## How it compares

The practical alternative to Memorilla is retrieval-augmented generation (RAG), which picks the top-N passages and feeds them into the context. RAG is simpler, well understood, and works. Memorilla differs in that it considers every document, compresses into a fixed budget, and stays fast regardless of collection size. On the other hand, Memorilla can lose fine-grained facts that a dedicated retriever would surface, and it has only been tested with a Qwen encoder and a Qwen decoder.

![A comparison table across five dimensions: context window, retrieval, and Memorilla](https://myargoquest.com/blog/memorilla-5.jpg)
*Memorilla shows near-constant latency and a fixed 16-token budget regardless of collection size, unlike context window or retrieval approaches.*

## What it means for agent memory

The most relevant result for agent development is the TextWorld experiment. An agent that used Memorilla to recall its own past game sessions improved its success rate from 18.6% to 24.2%. That is a 30% relative improvement from remembering what it did before, without any changes to the model or the agent architecture.

For an agent system like Argo, where agents run long-lived sessions across multiple projects and document stores, the same pattern applies: encode every completed task output into a Memorilla store, and the agent draws on the entire history through 16 tokens per query.

## The limits

The paper lists its own limitations. Memorilla can lose fine-grained facts. It was tested only with a Qwen encoder and Qwen decoder. Transfer across model families is untested. It may not carry over to coding tasks. The memory module needs training for each setup, which plain retrieval does not.

The training recipe is three stages: stage 1 on Wikipedia, stage 2 on a mixture, stage 3 on multitask data. All checkpoints are on Hugging Face at memorilla/Memorilla-Qwen3-8B, Apache 2.0.

## The repo

Code: github.com/snap-stanford/memorilla. Paper: arXiv 2610.21375. Weights on Hugging Face: memorilla/Memorilla-Qwen3-8B.
