Indegene Builds an LLM Agent for Governed Enterprise Data Quality

Summary:Indegene has built a production-oriented Data Quality Agent within its NEXT Unified Data Platform on AWS. The system uses Claude Sonnet 4.5 on Amazon Bedrock, Bedrock AgentCore, Strands Agents, and two RAG knowledge bases to combine table metadata, profiling results, anomaly signals, existing rules, external standards, and internal enterprise definitions. It can propose column-level, table-level, and natural-language-to-SQL validation rules before generating executable SQL and Python. The archi

---

The Release: Turning Data Quality Engineering into an Agentic Workflow

Indegene has introduced a Data Quality Agent inside its NEXT Unified Data Platform on AWS. Rather than replacing conventional rule engines such as AWS Glue Data Quality, the agent targets the expensive, expertise-intensive part of the workflow: understanding a dataset's context, proposing appropriate validation rules, and translating approved rules into executable code.

A traditional process separates profiling, rule design, SQL or Python development, testing, approval, and maintenance. Indegene's implementation turns these tasks into a collaborative workflow. The agent gathers context and drafts rules and code; data engineers and business specialists retain final approval authority.

Architecture: Six Context Sources and Two Knowledge Bases

Before recommending a rule, the agent combines six context sources:

1. Table and column metadata from AWS Glue Data Catalog;
2. Actual profiling results, including null rates, cardinality, ranges, and frequent values;
3. Anomalies detected against historical baselines;
4. Existing active rules retrieved from the rule repository;
5. External quality standards and validation practices;
6. Internal glossaries, standard operating procedures, approved rules, historical anomalies, and expert feedback.

The external knowledge base is periodically refreshed through approved standards, controlled web search, and scraping. The internal knowledge base captures organization-specific definitions. Documents are stored in Amazon S3, vectorized, and retrieved through Amazon Bedrock Knowledge Bases backed by Aurora PostgreSQL with pgvector. AgentCore Gateway exposes both knowledge bases to the agent as tools.

The important design choice is not a larger context window. It is making retrieved evidence part of rule generation. For an NPI column, for example, the system can combine the required ten-digit format, trusted reference values, and approved enterprise rules. This reduces the gap between business definitions and executable checks, but it also means that stale terminology, conflicting policies, or incorrect approvals can be retrieved and amplified. Document versioning, source precedence, and expiry policies therefore remain essential.

From Profiling to Execution

After a user selects a supported source such as Redshift or Snowflake and a target table, a Python Lambda function computes column statistics, patterns, and table-level signals. Results are stored in DynamoDB. Anomaly detection compares current profiling values with historical baselines to flag null-rate spikes, cardinality changes, and out-of-range values.

The agent then proposes rules in three modes:

• Column Level: completeness, type, length, range, format, and uniqueness checks;
• Table Level: duplicate records, cross-column consistency, referential integrity, and business consistency;
• Custom Query: natural-language business conditions translated into SQL and refined conversationally.

For an HCP product-performance table, the agent can generate both structural rules, such as a non-null and unique HCP_ID or non-negative prescription volume, and business rules, such as market share totaling 100% within a territory and reporting month or HCP specialty being consistent with product eligibility.

Once a rule is approved, Claude Sonnet 4.5 generates SQL, Python, or PySpark. AgentCore Code Interpreter performs syntax checks, target-schema validation, and sample-data test runs where supported. Approved code is stored in S3 and can be executed by Lambda, Glue, Athena, or Redshift. DynamoDB holds rule metadata, including identifiers, type, column, status, author, severity, and version, providing a centralized governance surface.

Why Use a Separate Validation Agent?

The architecture uses one LLM to generate rules and another to review the resulting rules and scripts. This generator–validator separation can reduce errors within a single generation pass, but it also creates correlated failure risk: two models may share similar blind spots or training biases.

An LLM reviewer should therefore supplement rather than replace deterministic validation. Safety-critical paths still need SQL parsing, column-existence checks, read-only permissions, cost estimates, sample replay, and differential testing against approved baseline rules. Rules that write data, touch sensitive information, or materially affect business reporting should run with least-privilege permissions and sandbox isolation.

Engineering Trade-offs: Latency, Cost, and Governance

Indegene reports profiling times of 45–90 seconds for approximately one million rows and two to five minutes for ten million rows. Rule generation takes 20–45 seconds per table, SQL generation takes three to eight seconds per rule, and standard rule execution takes 20–30 seconds. Asynchronous orchestration reduces interactive waiting, but complex SQL can still stretch end-to-end latency to several minutes.

Lambda is well suited to event-driven profiling and smaller tables, but full scans over very large datasets are constrained by concurrency, timeouts, and cost. A better approach is column-selective statistical profiling: read metadata first, identify candidate columns, and run expensive exact checks only where needed. Indegene claims metadata-first profiling reduces compute by roughly 45–51%, although the actual benefit depends on data distribution.

Bedrock prompt caching can reuse stable system prompts, retrieved knowledge, and schema context, with Indegene reporting up to 90% lower input-token cost. This works only when the cache prefix remains stable. Injecting large and changing sample datasets into every request can sharply reduce the cache hit rate. Total cost also includes vector retrieval, context tokens, code generation, validation runs, anomaly analysis, and human review; pricing a single model call is insufficient.

Interpreting the Reported Evaluation

According to Indegene's estimates, rule-suggestion accuracy ranges from 80% to 90%, 60% to 75% of suggestions are accepted without modification, schema-grounded SQL passes validation on the first attempt in 90% to 97% of cases, and custom SQL passes in 80% to 90%. The reported hallucination rate is below 5–8%. New-table onboarding reportedly fell from two to five working days to two to four hours, while rule-definition and implementation effort per dataset fell from roughly 30 hours to four.

These figures demonstrate architectural feasibility but do not constitute a general benchmark. They come from the vendor's own production deployments, without a disclosed independent test set, annotation protocol, customer-by-customer breakdown, or confidence intervals. The term “accuracy” may combine rule applicability, logical correctness, and business relevance. Moreover, the fact that 20% to 35% of suggestions require human edits exposes the gap between acceptance and correctness.

A stronger evaluation would disclose precision and recall by rule type, leave-one-customer-out testing, severity-classification error, escaped-invalid-rule rates, manual edit counts, end-to-end cost per rule, and comparisons against static rule generators and retrieval-only baselines.

Deployment Guidance

1. Start with low-risk column rules: enable format, length, completeness, and range checks before exposing cross-domain business logic.
2. Treat knowledge governance separately: assign owners, versions, expiration dates, and conflict precedence to glossaries, SOPs, and approved rules.
3. Keep deterministic controls: parse generated SQL, verify permissions, estimate cost, replay samples, and regression-test results.
4. Control the feedback loop: store only expert-approved, structured corrections in memory; do not automatically feed rejected model output back into the knowledge base.
5. Set automation boundaries from risk: high-confidence basic rules may be deployed automatically, while critical business and sensitive-data rules always require SME approval.
6. Build a reproducible benchmark: use historical incidents to measure missed defects, false positives, rule escapes, and manual modifications rather than relying only on generation speed.

NEXT UDP's value lies in treating data quality rules as versioned and auditable enterprise assets rather than adding another chat interface for SQL generation. Whether it succeeds still depends on knowledge quality, deterministic validation, least-privilege execution, and human review being embedded in routine engineering workflows.

---

Source: AWS Machine Learning Blog (2026-11-04)

— END —

Tags: #Data Quality Agent Follow #RAG Follow #AWS Bedrock Follow #Healthcare Data Governance Follow #Agentic Workflow Follow