The picture problem: multimodal AI is where the memory wall gets worse, and who is tearing it down

Our own numbers tell a clean story for text. The memory one 128K-token conversation occupies in the leading open-weight models fell from 18.7 GB in January 2025 to 6.25 GB on 1 October 2026, by the desk’s TCC input series. Architecture did that, not hardware; HBM per flagship NVIDIA GPU has not moved since July 2025. The uncomfortable footnote is that the series measures text. The moment a model has to look at the world, the arithmetic changes, and this is where the next memory crunch is already forming.

What’s happening

• A model reading a page of text pays roughly one cache entry per word. A model watching video pays per patch, per frame, per second. Streaming video work from Shanghai Jiao Tong University describes the key-value cache growing linearly with time and names the consequence: memory use and response delay that break online uses like driving and robotics [LiveVLM, arXiv 2505.15269, DAC 2026].
• The fix the field has converged on is not more memory. It is deciding what to forget. LiveVLM keeps a fixed 12,000-token budget, scores visual tokens by how much other visual tokens attend to them, and retrieves only the pages that matter at question time; on a 24 GB consumer GPU it reports 3.02x lower peak memory than a prior method at 256 frames [arXiv 2505.15269].
• A second camp argues that forgetting is the wrong verb. RetentiveKV, from Zhejiang University and Alibaba, keeps evicted visual tokens as a compressed continuous state instead of deleting them, because a patch that looked unimportant when the frame arrived can become the answer three questions later. It reports 5x cache compression with memory at a 5 percent budget falling from 2.24 GiB to 0.23 GiB on an A100 [arXiv 2605.04075].
• Training-free visual memory banks are now a research category of their own; a CVPR 2026 paper reports long-video understanding past 1,000 frames on a single RTX 3090 by compressing the cache iteratively and retrieving from it [CVPR 2026, FlexMem].
• On the text side, the leanest architectures in our basket already behave like a memory hierarchy: DeepSeek-V4-Pro holds recent context in full detail and older context at 4x and 128x compression, at 1.3 GB per 128K conversation; Meta’s Muse Glimmer 30B, a multimodal model, uses 13 global layers and 39 local 2,048-token windows and needs 1.83 GB [desk computation from published configurations, 1 October 2026].

The mechanism, in one paragraph

Every one of these designs makes the same bet: information should be stored at a resolution that matches how likely it is to be needed. Recent tokens stay sharp. Older tokens are summarized. The summary of a summary is cheaper still. This is a multi-resolution memory, and it is the reason text got 3x leaner in 21 months without a single extra gigabyte of HBM. Vision is harder because the raw input is larger by two or three orders of magnitude, so the compression has to start earlier, inside the encoder, and it has to survive questions the model has not seen yet.

Who is in front

China holds the deployment lead. The two most concrete streaming-video results above come from Shanghai Jiao Tong University and Fudan (LiveVLM) and from Zhejiang University with Alibaba (RetentiveKV), and the lean text architectures that top our basket, DeepSeek-V4 and Kimi-K3, are Chinese. Alibaba’s Qwen3-VL line is the base model most of this research is tested on, which is its own kind of lead.

Korea owns the cleanest statement of the hierarchy idea. The Block Transformer, by Namgyu Ho, Sangmin Bae, Se-Young Yun and colleagues at KAIST with Google DeepMind coauthors, decodes coarse blocks across the whole sequence first and individual tokens only inside a block, and reports 10 to 20x inference throughput at equal quality [NeurIPS 2024, arXiv 2406.02657]. It is a two-level memory. The open question in Ho’s own framing is whether the levels keep paying off past two.

The United States set the vocabulary for forgetting. Beidi Chen’s H2O work at Carnegie Mellon, with collaborators at Stanford and elsewhere, showed in 2023 that a small set of “heavy hitter” tokens carries most of the attention mass and that the rest of the cache can be dropped [NeurIPS 2023, arXiv 2306.14048]. Nearly every eviction method in the video papers above descends from that observation. The fixed-state alternative, where memory is a recurrent state of constant size rather than a growing cache, comes from Albert Gu and Tri Dao’s Mamba line, which NVIDIA now ships at frontier scale in Nemotron 3 Ultra, the second leanest model in our basket at 1.52 GB per conversation.

Brazil is not in the model race, and it would be dishonest to pretend otherwise. What Brazil has is the mathematics. Odemir Martinez Bruno’s group at the University of São Paulo in São Carlos has spent two decades on multi-scale fractal descriptors for texture and shape recognition, the discipline of describing an image by how its detail changes across scales [arXiv 1304.1568, 1412.7963]. Leandro Villas at Unicamp, president of ITEC, points newcomers to that work and to Michael Barnsley, whose fractal image compression of the 1980s and 1990s first encoded a picture as a set of self-similar transforms rather than as pixels. Barnsley’s method lost to wavelets and then to learned codecs. The idea underneath it did not lose, and it is worth saying plainly where this desk’s interest lies.

The fractal question

Every lean architecture in our basket is multi-scale. None is self-similar. DeepSeek uses two compression ratios; Block Transformer uses two levels; Gemma and Muse Glimmer use one local window and one global view. A fractal memory would use the same rule at every level: compress the last minute the way you compress the last hour, the way you compress the last year, with the same transform applied recursively. Nobody has published a working version for language or vision models. That is either because it does not work, or because the people who understand iterated function systems and the people who build attention kernels have never sat in the same room. The desk’s view is that the second explanation is at least as likely as the first, and that the São Carlos tradition is one of the few places where the first half of that room already exists. We will keep testing the hypothesis against the numbers, and we will say so if the numbers refuse it.

What it means

The materials consequence is direct. If multimodal memory followed the text curve, demand for HBM per deployed model would keep falling even as usage grows, and the HBM4 supply crunch, with contract prices reported heading from about $600 a stack this year toward $1,300 next year, would ease on the demand side. If it does not, and video agents ship with linearly growing caches, every robot, every car and every always-on assistant becomes a standing claim on SK hynix, Samsung and Micron output, and the 432 GB AMD put on the MI455X looks less like a luxury than a floor. The gap between our TCC and TCC-X readings, 218.5 against 327.5 on 1 October, is a measure of how much the industry is already hedging that bet with hardware.

What to watch

• Whether Alibaba ships RetentiveKV-style retained state inside a production Qwen-VL release rather than as a research overlay; a config.json that exposes it will move our basket.
• KAIST’s follow-ups to Block Transformer: a three-level or deeper hierarchy with published numbers would be the first evidence that the fractal direction pays.
• The first open-weight video model from any lab whose cache size per hour of video is published alongside the model; today none is, and until one is, “multimodal memory” cannot enter the TCC basket.
• Brazil’s rare-earth and tantalum story is the usual place this desk looks for a Brazilian angle. On memory, the angle is São Carlos, and whether anyone funds the collaboration.

Sources: LiveVLM, Shanghai Jiao Tong University and Fudan University, arXiv 2505.15269 (DAC 2026); RetentiveKV, Zhejiang University and Alibaba, arXiv 2605.04075; FlexMem, CVPR 2026; Block Transformer, KAIST, NeurIPS 2024, arXiv 2406.02657; H2O, NeurIPS 2023, arXiv 2306.14048; O. M. Bruno et al., multi-scale fractal descriptors, arXiv 1304.1568 and 1412.7963; Tantalum Strategy memory-index basket, 1 October 2026, tantalum.info/indexes/methodology.