HiVe-Mem Combines Graph Expansion and Evidence Composition for Token-Efficient Multimodal Agents

Summary:HiVe-Mem introduces a two-dimensional memory framework for multimodal agents operating over long interaction histories. Its horizontal expansion builds a relational graph over episodes using textual and visual anchors, allowing semantic retrieval seeds to recover complementary information from neighboring memories. Its vertical evidence composition retains compact summaries, captions, grounded visual views, raw images, and interaction traces, then uses a lightweight trainable selector to assembl

---

Core Design: Separating Memory Retrieval from Evidence Use

Conventional multimodal-agent memory systems usually compress every interaction episode into a fixed summary and retrieve it through text or vector similarity. This design has two structural weaknesses: semantic neighbors may omit genuinely relevant episodes, while a single representation cannot expose the evidence needed by different queries.

HiVe-Mem separates memory access into two dimensions:

• Horizontal Memory Expansion broadens retrieval coverage.
• Vertical Evidence Composition selects the most useful evidence form for each query from every retrieved episode.

Retrieval therefore determines which historical episodes may be relevant, while evidence selection determines what each episode actually contributes to the current answer.

Horizontal Expansion: Recovering Complementary Episodes Through a Graph

HiVe-Mem organizes episodes in a relational graph whose connections are derived from textual and visual anchors. Semantic retrieval first produces seed nodes, after which related neighbors are added to recover information omitted by direct vector matching.

Formally, the retrieved set can be viewed as expanding from (R) to (R^+ = R \cup N(R)), where (N(R)) contains graph neighbors of the seed episodes. This is more controlled than simply increasing Top-K because expansion is constrained by learned or constructed relationships rather than adding arbitrary candidates.

Visual anchors are especially useful for episodes that are related through objects, scenes, or actions but do not have high text-level similarity. The tradeoff is a higher ingestion and maintenance burden: the system must store image and text anchors, update graph links, and preserve references to the underlying media.

Vertical Composition: Exposing Multiple Evidence Views per Memory

Each retrieved episode retains several evidence forms:

• A compact summary for a quick event overview;
• Captions for semantic object and scene recognition;
• Grounded visual views for checking specific regions or details;
• Raw images to avoid information loss introduced by captioning;
• Interaction history for preserving action order, tool calls, and state transitions.

A lightweight trainable selector scores each evidence type independently for the current query and episode, then assembles a query-adaptive subset. In simplified form, the selected set is (E_m(q)={e_i \mid s_i(q,m)\geq\tau_m}), where (m) is a retrieved episode and (s_i) is the compatibility score for evidence type (i).

This avoids concatenating every representation into the prompt. It also avoids forcing the final answer to depend solely on lossy summaries. However, selector errors are consequential: an incorrect gate can discard both a valid summary and the raw visual evidence needed to verify it.

Engineering Trade-offs: Fewer Tokens Do Not Mean Equal System Savings

The paper reports roughly 90% lower token consumption than the most token-efficient competing multimodal memory baseline. For agents that retain long histories, original images, or complete tool traces, this can reduce inference latency, memory pressure, and API expenditure.

Total production cost nevertheless includes several components:

1. Graph construction and maintenance: New episodes require textual and visual anchor extraction, relationship updates, and index consistency.
2. Multimodal storage: Summaries are cheap, while raw images, grounded crops, and full traces require more object storage and I/O.
3. Selector inference: Evidence scoring runs on every query. The abstract does not disclose selector size or whether it requires a separate visual encoder.
4. Graph-expansion latency: Additional candidate episodes must still be scored and assembled into the final prompt.
5. Historical write cost: Preserving multiple evidence forms increases ingestion-time encoding and storage expenses.

The approach is most compelling for long-running multimodal agents that accumulate heterogeneous textual and visual interactions. Its benefits may be limited when histories are short or rarely contain images.

Benchmark Results and Evidentiary Limits

HiVe-Mem is evaluated on three multimodal memory benchmarks and across multiple base models, with the paper reporting consistent task improvements. The token-efficiency comparison is made against the strongest competing baseline in token usage, indicating that the gains are not simply caused by feeding every method a longer context.

Several details required for a full reproduction remain unavailable in the abstract:

• How visual tokens and offline processing costs are counted;
• The absolute accuracy gains, confidence intervals, and statistical significance;
• Selector parameter count, training-set size, and training cost;
• Graph and raw-media storage overhead;
• Generalization to closed-source models and production tool traces.

The 90% claim should therefore not be interpreted as a 90% reduction in end-to-end latency. If raw images are encoded into many visual tokens, or graph maintenance and evidence selection are excluded from the comparison, the system-level savings may be materially smaller.

Practical Integration Guidance

Teams building long-term multimodal agents can adopt the architecture incrementally:

1. Assign a unified event ID so summaries, captions, grounded views, images, and traces remain aligned.
2. Use hierarchical retrieval: vector or keyword search for seeds, followed by constrained graph expansion rather than indiscriminately increasing Top-K.
3. Keep raw evidence behind references: use summaries for routine retrieval and load original media only when verification requires it.
4. Train a lightweight evidence selector with fallback behavior so a systematic failure in one evidence type does not eliminate all useful signals.
5. Measure complete lifecycle cost, including context tokens, visual tokens, retrieval latency, encoding time, storage, and historical write cost.
6. Test failure modes explicitly, including noisy graph links, faulty visual anchors, conflicts between captions and raw images, and unseen query types.

HiVe-Mem does not introduce a new foundation model. Its contribution is a different access protocol for multimodal agent memory: expand recall through a relational graph, then use query-adaptive selection to control which evidence reaches the model. If subsequent code and complete benchmarks validate both the token accounting and long-term maintenance cost, the framework could become a practical option for multimodal agents that must balance retrieval coverage, evidence quality, and context budgets.

---

Source: Hugging Face Daily Papers (2026-11-09)

— END —

Tags: #Multimodal Agents Follow #Long-Term Memory Follow #Retrieval-Augmented Generation Follow #Evidence Selection Follow #Token Efficiency Follow