ZICQ
中 Log in / Sign up
Newsroom Agentic #Continual-ARC #ARC-AGI #Continual Learning #Agent Evaluation #Parametric Learning

Continual-ARC Evaluates Lifelong Learning: Harness Memory Trails Weight Updates

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:Continual-ARC reorganizes three ARC-AGI editions into a lifelong task stream. Previously encountered tasks return with fresh instances after dormant gaps while new tasks continue to arrive. Both memory-based and weight-based learners consume a shared feedback budget covering demonstrations, harness actions, submissions, and paid retries. After the stream ends, the learner is frozen and evaluated separately on fresh instances of encountered tasks for memorization and on unseen tasks for generaliz


Continual-ARC's central result is that persistence in an Agent Harness is not equivalent to skill acquisition. Under its lifelong-learning protocol, most harness-based methods trail a simple baseline that caches task demonstrations, while weight-based learning produces substantial gains on unseen tasks.

What the benchmark changes

Existing evaluations of continual agent learning commonly focus on a single applied domain, expose each task only once, and assume one family of learner. As a result, memory-based systems and systems that update model weights are rarely compared under the same resource constraints.

Continual-ARC constructs a lifelong stream from three ARC-AGI editions. Its tasks require no domain knowledge, reducing contamination from a model's prior expertise. Encounters are not one-shot: previously observed tasks return with fresh instances after dormant gaps, while new tasks continue to arrive. The evaluation therefore follows a working lifetime containing acquisition, interruption, recurrence, and transfer.

Technical mechanism

A shared feedback budget

For each instance, a learner can pay for demonstrations, act inside the harness, submit an answer, and retry after an incorrect submission; paid retries also consume the budget. Conceptually, the cost can be tracked as:

C_total = Σ(c_demo + c_action + c_submit + c_retry)

with each component translated under the task's budget rules and the learner ultimately evaluated according to the share of feedback consumed. The supplied abstract does not disclose the complete scoring formula, demonstration prices, action prices, or cross-task normalization procedure. The benchmark therefore provides a common interaction-feedback scale, not necessarily a full conversion of GPU hours, memory, storage, and training cost.

This is still an important design improvement. External memory, program libraries, and model weights use different storage mechanisms, but all must compete for opportunities to complete tasks under one interaction economy. Charging for incorrect retries discourages unbounded trial-and-error and forces a system to distinguish cheap validation from expensive exploration.

Frozen post-stream evaluation

After the stream ends, the learner is frozen and tested in two ways:

  • On fresh instances of encountered tasks, measuring retention and memorization;
  • On unseen tasks, measuring transfer and generalization.

Freezing prevents continued updates from contaminating the evaluation and separates retention from abstraction. A system may replay an old task successfully while failing to acquire a reusable rule. Conversely, a parametric learner may improve generalization without entirely avoiding forgetting.

Learner families under comparison

The paper compares several approaches:

| Learning method | Weights updated | Primary question | |---|---:|---| | Agent Harness with frozen weights | No | Can a tool-using harness solve the tasks? | | Agent Harness with persistence | No | Does external state substitute for skill acquisition? | | Two prompting floors | Usually no | What can simple prompt policies achieve? | | Program-induction method | No | Can feedback produce reusable executable programs? | | Qwen3.5-9B trained to imitate GLM-5.3 Flash | Yes | Can parametric learning transfer across tasks? |

The weight-based result comes from one specific imitation-learning implementation. It does not prove that every parameter-update method dominates every memory system. It does, however, test whether weight consolidation can produce stronger transfer than external harness state under the same ARC lifecycle.

Experimental findings

Four qualitative trends stand out:

  1. Persistence generally did not improve memorization. Without explicit expert guidance, retaining historical state did not automatically improve task reproduction.
  2. Most harness methods trailed the demonstration-cache floor. Across all tested models, directly caching each task's demonstrations was more effective than most harness-based approaches.
  3. No harness method improved on unseen tasks. Local improvements on encountered tasks did not reliably transfer to new tasks.
  4. Weight-based learning achieved substantial gains. The parametric learner showed meaningful improvement on unseen tasks, producing the paper's most counterintuitive result.

One plausible interpretation is that harness persistence stores experiences rather than extracting skills. The system must learn which trajectories are worth retaining, how to abstract failure causes, and when to compress experience into a reusable rule. A demonstration cache bypasses experience selection by preserving known effective behavior, creating a strong lower bound. Weight updates may instead compress shared regularities across tasks and therefore support transfer.

The abstract does not report exact scores, number of task streams, ordering seeds, confidence intervals, or training costs. The qualitative ordering can be assessed, but not its magnitude or robustness to budget pricing, task order, and the particular training recipe.

Engineering trade-offs

External memory and harnesses

External memory requires no retraining and remains easy to audit and roll back. It fits small sets of facts, tool arguments, and exceptional procedures. Its costs, however, grow with the number of tasks. Retrieval failures break execution, exact replay does not create new rules, and placing entire trajectories in context raises token costs and attention interference.

Parametric learning

Parameter updates compress knowledge into shared weights, avoid carrying a full interaction history at inference time, and can generalize across tasks. The corresponding costs are training compute, checkpoint management, rollback complexity, and catastrophic-forgetting control. Because the paper tests a specific imitation-based approach, the findings do not establish that parametric learning wins universally.

Hybrid systems

A practical architecture need not choose one mechanism. Exact, low-frequency facts can remain in external memory; recurring and stable skills can be consolidated through LoRA, adapters, or controlled fine-tuning; and the harness can handle retrieval, validation, and execution. The three stores must be coordinated to avoid saving the same experience in memory, a program library, and model weights, which would invalidate any attempt to compare resource consumption honestly.

How objective is the benchmark?

ARC tasks require no domain knowledge, reducing differences caused by pretrained expertise. Returning to a task with fresh instances avoids mistaking verbatim recall for skill learning. Freezing the learner and evaluating encountered and unseen tasks separately also identifies memorization and transfer more clearly than a single aggregate score.

External validity remains an open question. ARC tasks are abstract and rule-oriented, whereas web operation, repository maintenance, robotics, and enterprise workflows introduce noise, long-horizon dependencies, and irreversible actions. The abstract also omits the dormancy lengths, task order, budget allocation, number of repetitions, and construction criteria for unseen tasks. If those choices were tuned against particular systems, benchmark overfitting remains possible.

Guidance for developers and deployment

  1. Establish a demonstration-cache baseline first. A continual-learning agent should demonstrate that it can outperform direct expert-demonstration caching before adding complex memory or retrieval machinery.
  2. Separate state by type. Track episodic memory, executable programs, and model checkpoints independently, with explicit write conditions, provenance, and eviction policies.
  3. Maintain a feedback-cost ledger. Record demonstrations, tool calls, submissions, retries, GPU time, storage, and parameter-update cost alongside task success.
  4. Measure retention and transfer independently. Freeze the system after the stream, then test old tasks with fresh instances, new task families, and realistic environment variants across multiple orders and seeds.
  5. Consolidate parameters selectively. Update weights only for recurring, stable, verifiable skills; retain rollback checkpoints and regression gates so one update cannot silently overwrite unrelated capabilities.
  6. Match the mechanism to the task. Store rare exceptions and exact procedures in external memory, consolidate repeated abstract rules into weights, and keep exploration and validation in the harness.

Continual-ARC's main contribution is not a new learning algorithm but a sharper evaluation boundary between remembering and learning. Its results challenge the assumption that an agent becomes a lifelong learner merely by accumulating context, preserving trajectories, or extending its history. For engineering teams, the benchmark offers a more rigorous way to decide whether external memory, program induction, or parametric consolidation is worth its operational cost.


Source: Hugging Face Daily Papers (2026-11-11)

— END —

Tags: #Continual-ARC #ARC-AGI #Continual Learning #Agent Evaluation #Parametric Learning

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…