KDASO Dynamically Optimizes Skill Libraries for Liability Adjudication
Summary:KDASO is an automated skill-optimization framework for liability adjudication. It combines domain knowledge, recent case data, and LLM execution traces to continuously induce, refine, and route interpretable skills. Neighborhood-conditioned induction helps the skill library track evolving scenarios, trajectory-based refinement reduces the semantic gap between written rules and model behavior, and context-budgeted routing prevents critical skills from being omitted as the library grows. The pape
The problem is not whether an LLM reasons, but which rules it executes
Liability adjudication requires a model to map event descriptions to traffic responsibility, legal liability, or platform rules. A larger context alone does not guarantee interpretability: static skill libraries become stale as regulations and case distributions change, while apparently reasonable skill descriptions may not translate into faithful model behavior. KDASO (Knowledge-Data Dual-Driven Automated Skill Optimization) therefore shifts the optimization target from prompts or model weights to an executable skill library that evolves with real task streams.
The pipeline has three components: neighborhood-conditioned skill induction, execution-aligned refinement, and context-budgeted routing. They address stale knowledge, the semantic execution gap, and context omission, respectively.
Inducing skills from recent cases rather than copying static knowledge
KDASO first generates skills using recent adjudication examples. Each new example is not processed in isolation: the system consults semantically related historical cases and skills, extracts their shared variables, and induces a reusable rule covering the new pattern. This reduces overfitting to the wording of a single case and allows skills to represent local regularities in the case space.
Conceptually, this turns skill induction into a local conditional-generation problem. The model must compare a new example with its neighborhood, identify common variables, and produce a structured skill containing applicability conditions, factual patterns, and adjudication actions. The abstract does not explain how neighborhoods are defined—text embeddings, case graphs, entity relations, or rule retrieval—or whether skills are executable code, natural-language rules, or both. Those choices directly affect reuse boundaries and error propagation.
Neighborhood-conditioned induction also introduces a deployment risk. If the current batch distribution drifts, recent cases may not represent the newest regulations or platform policies. A production system should therefore attach version, timestamp, source, and scope metadata to every skill rather than maintaining the library through similarity alone.
Refining skills from execution traces
The second stage analyzes the skills an LLM invoked, the factual evidence it used, and whether its final conclusion was consistent with the selected skill conditions. When a skill's description and execution behavior diverge, KDASO modifies the description or executable logic rather than merely adding a constraint to the final answer.
This is more useful than ordinary self-generation over successful outputs because the optimization signal comes from execution discrepancies, not just final labels. Trajectories can distinguish selection errors from content errors: the former call for routing changes, while the latter require rewriting the skill. In this sense, trajectories act as both diagnostic records and regression-test artifacts.
Trajectory refinement is nevertheless vulnerable to reward hacking. An optimizer that only observes final correctness may preserve accidental successes as malformed skills. A stronger implementation should use causal comparisons—evaluating the case with the candidate skill, without it, and with a conflicting skill—and invoke rule validators to check formal constraints. LLM self-evaluation alone cannot establish that a revised skill better reflects expert logic.
Routing under a context budget
As the library grows, loading every rule increases prompt cost and can obscure the rules that matter. KDASO assigns a context budget and routes the most relevant skills into the reasoning context. The trade-off is recall versus context occupancy: an overly narrow router omits applicable rules, while an overly broad one adds tokens, latency, and rule conflicts.
A practical router must jointly optimize candidate retrieval, deduplication, priority ordering, and token budgets. High-risk rules should have a mandatory-loading path, and low-confidence routing should fall back to broader retrieval or secondary verification. Optimizing only an internal routing metric may fail to improve task accuracy if loaded skills conflict or a single verbose skill already consumes the budget.
Experimental results and evaluation limits
The paper evaluates KDASO across ride-hailing adjudication, legal liability determination, and logical reasoning judgment. The reported gains over general-purpose and domain-specific baselines suggest that combining dynamic skill maintenance, trajectory refinement, and budget-aware routing can outperform a fixed skill library. An 8.96% improvement in conceptually similar transfer settings indicates that induced skills are not necessarily tied to their source-case distribution. Stable gains during continued optimization further support the framework's use in long-running systems.
The abstract does not disclose the exact baselines, metric definitions, dataset sizes, model versions, confidence intervals, or significance tests. It also does not specify whether the 8.96% figure is absolute or relative, or whether transfer tasks remain in-distribution. Liability decisions also involve asymmetric consequences, so aggregate accuracy may conceal serious false negatives or false positives. Production assessment should separately report adjudication accuracy, recall, false-decision rate, calibration error, and failures on high-risk cases.
Deployment guidance: treat the skill library as an auditable software dependency
The most practical deployment model is not unrestricted model-driven rule editing, but a gated skill supply chain:
- Maintain a versioned registry: Record provenance, applicable scope, dependencies, creation time, and evaluation score for every skill.
- Gate automatic revisions: Require domain review before changes affecting legal responsibility or other high-consequence decisions enter production.
- Preserve offline regression suites: Include historical disputes, edge cases, conflicting rules, and failed trajectories so one update does not silently break another scenario.
- Make routing observable: Log candidate skills, selected skills, selection rationale, and discarded rules to distinguish induction, refinement, and routing failures.
- Measure marginal value: Archive or down-weight skills that are rarely selected or have no causal contribution, preventing unbounded library growth.
KDASO reframes knowledge maintenance in LLM systems as something closer to continuous software integration: generated skills are candidate changes, trajectory replay is testing, and routing budgets constrain deployment. Its broader value is not limited to adjudication accuracy, but lies in offering a sustainable and auditable mechanism for evolving rules in high-risk AI systems.
— END —Source: ArXiv AI (cs.AI) (2026-11-06)
Tags: #KDASO #Skill Optimization #Liability Adjudication #Trajectory Feedback #Context Routing
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments