LlamaParse Pen Tests Against OWASP LLM Top 10, Uncovering Hidden-Text Extraction Risks

Summary:LlamaIndex commissioned Trace to run a four-day agentic dynamic application security assessment of LlamaParse. The grey-box test covered parsing, classification, field extraction, indexing, retrieval, deployed agents, and the MCP server against all ten OWASP Top 10 risks for LLM applications. One notable finding involved a deliberately crafted invoice with white-on-white hidden lines. Although a human reader saw only the ordinary invoice, the extractor returned altered totals and banking detail

---

Why Conventional API Penetration Testing Is Not Enough

LlamaParse does not merely handle a few strings submitted through a form. It processes complete documents supplied by suppliers, applicants, policyholders, and other external parties. Those files can enter parsing, classification, extraction, and indexing, and can later be retrieved by search systems, customer-deployed agents, or the MCP server. Each stage is a potential entry point for untrusted input.

Traditional API penetration tests often reduce a security question to a binary check, such as whether an endpoint rejects a request without authentication. LLM systems behave differently: outputs are probabilistic, and the input can alter model behavior. A single call therefore cannot establish a reliable result. Source-code scanning is also insufficient because the behavior of a model is not fully represented in application logic.

An agentic tester must adapt after each response, reproduce effective attacks, and demonstrate them with working exploits. Human review remains necessary for high-impact findings.

Test Design and Coverage

Trace began with the access an attacker would obtain after signing up: working credentials, but no source code or cloud configuration. Test accounts were placed in separate organizations so that tenant boundaries could be tested from both sides.

The scope included:

• The web application, sign-in flows, and public APIs;
• Document parsing, classification, and field extraction;
• Index construction and retrieval;
• Customer-deployed AI agents;
• The LlamaParse MCP server;
• All ten OWASP Top 10 risks for LLM applications, together with standard web application risks.

The agents read the documentation, created a GitHub repository, wrote Python code, generated test PDFs, uploaded them, executed the workflows, and iteratively modified the code. Every finding was reviewed by an OSCP-certified engineer. Exploits were validated before remediation and re-tested before inclusion in an audit-ready report.

The value of agentic DAST is not merely attack-string generation. It turns security claims into reproducible exploitation evidence, which is particularly important for prompt injection, cross-tenant leakage, and excessive agency.

A Hidden-Text Invoice Exposes a Data-Integrity Failure

The test created an invoice whose visible page showed a normal total and bank account. Additional fields were printed in white on a white background. The hidden content contained no instruction; it looked like a corrected version of the same business fields.

A person opening the PDF saw only the ordinary invoice. The extraction system read the page text and returned the altered total and banking details. This was not a prompt-injection exploit. It exposed a more basic trust mismatch: the model could observe more than the business user had reviewed and approved.

The scenario maps directly to accounts-payable workflows. A supplier can submit an invoice into an automated payment pipeline. If the system trusts extracted values while the review interface fails to expose or flag hidden content, an attacker may redirect a payment without presenting an obvious visual anomaly.

Traditional OCR and PDF parsers face similar document-integrity problems, but LLM pipelines add another layer of propagation. Hidden content can influence classification, summaries, extraction, RAG retrieval, and downstream agent decisions.

The Boundary Between Extraction and Business Trust

LlamaIndex's stated position is that the parser should report what exists on the page, while the customer's pipeline decides which values are trustworthy. That division makes sense because only the customer knows the business rules: amount thresholds, approved account lists, supplier master data, contractual terms, and approval policies.

The boundary, however, must not become an excuse for weak safeguards. The parsing layer should preserve evidence that hidden content existed, including visual-text differences, page numbers, coordinates, and confidence scores. The business layer should treat every extracted value as untrusted data rather than a fact already validated by a model.

Production pipelines should add:

1. Deterministic validation for amounts, bank accounts, identity documents, and contract clauses;
2. Field-to-page and field-to-coordinate provenance for human review;
3. Detection of hidden, low-contrast, and overlapping content;
4. Cross-checks against independent trusted sources such as supplier master data;
5. A deterministic policy engine and human approval before payment or account-change actions.

Tenant Isolation, Index Poisoning, and Agent Permissions

Parsing is only the first stage. Once malicious content enters a shared index, it can influence retrieval results or propagate through agents and MCP tools into data leakage and unauthorized actions.

Document ingestion should therefore be treated as an untrusted data ingress rather than an ordinary file upload. Teams need separate authorization boundaries for index writes, project access, retrieval results, and agent tools. Tools that modify account details, trigger payments, or mutate indexes should also enforce parameter constraints, approval gates, and rollback paths.

Models should not be used as the only line of defense. They are useful for extracting fields from unstructured pages, but deterministic rules, data lineage, and business authorization are better suited to deciding whether a field is consistent with the visible invoice or belongs to an approved supplier.

Enterprise Controls and Deployment Recommendations

LlamaIndex states that its managed service does not train on customer data by default, and that its model providers operate under zero-data-retention and no-training agreements. Uploaded files are cached for 48 hours by default and can be deleted through the API or UI; do_not_cache=true skips caching for sensitive files. Deployment options include managed SaaS in North America or the EU, single-tenant SaaS, deployment in a customer's AWS, Azure, or GCP account, and self-hosting. SAML/OIDC SSO and role-based access control are also available.

These controls address residency, retention, and identity management, but they do not replace application security. Engineering teams should also:

• Enforce least-privilege writes to shared indexes and retain source labels;
• Enforce separate authorization boundaries for parsing, extraction, retrieval, and agent tools;
• Include external model APIs, managed parsing, and MCP servers in vendor risk management;
• Preserve audit logs for document parsing and critical field changes;
• Add hidden-text, prompt-injection, cross-tenant retrieval, and excessive-agency cases to regression suites;
• Integrate continuous security testing into the release process.

Limits of the Assessment

This was a four-day, point-in-time assessment and does not prove that future versions will be free of defects. Agentic testing broadens exploration, but coverage remains constrained by model capability, testing budgets, refusal behavior, and account design. Subtle multi-step business-logic flaws may remain undiscovered.

The practical approach is defense in depth: combine agentic testing with manual code review, tenant-isolation tests, supply-chain scanning, permission regression, and deterministic business-rule validation. The conclusions must also be reassessed when models, parsers, indexing systems, or MCP permissions change.

---

Source: LlamaIndex Blog (2026-11-11)

— END —

Tags: #LlamaParse Follow #Penetration Testing Follow #OWASP LLM Follow #Document Security Follow #MCP Follow