Automating the Exception: How a Second LLM Judge Drives Straight-Through Processing

Document-centric workflows have been difficult to automate and required human intervention. Attempts to automate document handling often failed or did not scale, because legacy intelligent document processing (IDP) systems were fragile. They often required manually retraining models on dozens of documents just to identify specific fields—only to repeat the process whenever a format changes. The result was a costly cycle of maintenance and manual data entry.

Modern generative AI and agentic document handling solutions, like Appian’s DocCenter, are designed to break that cycle and increase straight-through processing (STP)—the ability to process documents from intake to completion without manual intervention.

DocCenter achieves high accuracy rates, typically 95%–99%. DocCenter does this by moving beyond single-model extraction to overcome generative AI shortcomings and hallucinations. A second LLM as a judge independently reviews and validates extracted data, helping catch errors caused by layout shifts before they reach downstream processes. Using a second, independent large language model to audit and validate the output of the initial extraction model drastically reduces exception handling, improves reliability, and drives higher straight-through processing rates.

This blog will explore how DocCenter document processing tool incorporates a second LLM as a judge to validate extracted data, reduce errors, and improve straight-through processing in document workflows.

If you ask a single LLM to extract data from a dense, complex document and simultaneously grade its own extraction work, you can fall prey to AI bias and hallucinations. For example, if the extraction LLM misreads an ink-smudged digit on a physical shipping manifest, asking a follow-up question like "Are you absolutely sure about that value?" rarely yields a different result because it evaluates the data using the same contextual weights.

By introducing a second LLM, the extracted result can be independently evaluated—and critically, more documents pass straight through without human intervention. The second model doesn't just add accuracy; it reduces the volume of exceptions that require manual review, directly increasing your straight-through processing rate and reducing human touch required. In internal testing, the AI Reviewer independently caught over half of all extraction errors before they reached a human reviewer—reducing the manual reconciliation workload by more than 50%.

Rather than relying on a single AI model to both extract and verify data, DocCenter separates those responsibilities across independent AI components, enabling a more objective view of extracted information. The primary model focuses on the heavy-lifting—discovery and mapping—while the secondary model acts as an unbiased auditor. The second LLM does not simply re-extract the text from scratch. Instead, it is given a completely unique prompt, a distinct cognitive persona, and a specialized task: cross-reference the original source document directly against the first model's raw extraction payload to flag discrepancies.

While adding a second model introduces another call, DocCenter’s AI Reviewer prompt is highly optimized for rapid JSON evaluation. For asynchronous processes—like invoicing or insurance claims—the few seconds of execution time are a good tradeoff compared to hours of manual review.

You do not need to spend hours writing prompt engineering templates or crafting rigid system logic to deploy this architecture. DocCenter automatically builds the comprehensive prompt framework for you, injecting the necessary system roles, tasks, and output constraints. When the AI Reviewer processes an incoming document, its system instructions execute a strict algorithmic playbook:

1. Analyze the source document provided.
2. Review the extracted data as provided.
3. Only return fields referenced in the output format, making sure the output JSON matches the Output Format schema exactly.
4. For each field referenced in the output schema, create an object containing four critical validation keys:

• extractedValue: This should be the value originally extracted from the document before the review.

• suggestedValue: This should be the correct value as found in the document. If the original extraction was wrong, you must provide the correct value here. If the correct value should be blank, return back "".

• isCorrect: A boolean (true or false). Set to true if the originally extracted data matches the document, and false otherwise.

• reasoning: A brief string explaining your decision. If correct, state where you found the information. If incorrect, explain the discrepancy.

Because of this strict programmatic structure, the output returned by the judge isn't conversational fluff—it is machine-readable data that DocCenter parses instantly and surfaces in our Reconciliation interface for that particular field. If the primary extraction model successfully finds an invoice number but accidentally pulls an incorrect date due to formatting drift, the AI Reviewer's payload cleanly captures the error:

{
  "invoice_number": {
    "extractedValue": "INV-2026-899",
    "suggestedValue": "INV-2026-899",
    "isCorrect": true,
    "reasoning": "The invoice number matches the value at the top right of the document."
  },
  "invoice_date": {
    "extractedValue": "2026-10-26",
    "suggestedValue": "2026-10-27",
    "isCorrect": false,
    "reasoning": "The extracted date was 2026-10-26, but the document clearly shows 2026-10-27 in the summary block."
  }
}

While no large language model is entirely immune to hallucinations, separating extraction from validation drastically minimizes the impact of generative errors. In an open-ended extraction task, a single model is highly susceptible to confirmation bias—if it misreads a value, it will defend that value upon self-reflection.

The AI Reviewer operates under a constrained, comparative sandbox. It is not tasked with open-ended generation; instead, it performs strict deterministic and semantic cross-referencing between the source document and the primary payload. By restricting the second LLM’s cognitive boundaries to validation, verification, and explicit reasoning, the likelihood of a hallucination escaping detection is significantly reduced. The AI Reviewer turns a blind spot into an audited trail.

Until recently, setting up a multi-model validation pipeline required custom engineering, API integrations, and infrastructure to orchestrate between models. In DocCenter, this is packaged in a simple, interactive wizard designed for day-to-day use by operations teams.

When configuring the AI Reviewer, the user interacts with an intuitive modal UI where they can complete setup in three simple steps:

• Navigate to the Straight-through processing tab. Instead of writing validation code, you are presented with the field setup you created originally for extraction. You simply check off the boxes next to the fields you want the second LLM to audit.

• Select your LLM model. Pick from a drop-down menu the specific large language model configuration you want to act as your judge.

• Refine, don't write. DocCenter auto-generates the base prompt for you. You don't have to start from scratch; you can immediately preview the prompt and refine the role of the AI Reviewer or any additional instructions you want to add.

While the AI Reviewer serves as your structural judge, DocCenter also features a separate validation functionality that combines deterministic expression rules with national language validations. This lets users input their own custom business validations for an extra layer of security.

For instance, you can type plain English sentences such as: "Ensure the state field matches a valid US postal abbreviation." Together, these two layers harden your document pipeline against errors before any data enters your downstream enterprise systems.

A common misconception regarding straight-through processing and zero-touch automation is that the ultimate goal is to eliminate human oversight completely. At Appian, we view it differently: zero-touch processing isn't about replacing humans; it is about rescuing them from mundane, repetitive tasks so they can achieve more.

By introducing the AI Reviewer, humans focus only on high-value exceptions and strategic edge cases. When the judge flags a discrepancy ("isCorrect": false), DocCenter flags that document so an operations expert can review. Crucially, the human doesn't have to hunt for the error. The UI displays the AI Reviewer’s explicit reasoning string right next to the target field, letting the expert know exactly where the discrepancy lies.

The human remains the final authority, reviewing the judge's logical reasoning trail alongside the conflicting extractions to make the definitive call. This ensures that the human retains control, resolving exceptions with the necessary context and authority, while the AI handles the heavy lifting of identifying the discrepancy. This shifts the human's role from a tedious data-entry clerk to a strategic decision-maker who resolves vendor disputes, manages high-value anomalies, and applies contextual judgment where it matters most.

A Fortune 500 health tech company relied on a manual process where 120 managers navigated 17 different systems, causing delivery delays and high costs. By deploying DocCenter, the company gained a 360-degree view of order status and achieved 60% zero-touch business processing with over 50,000 monthly documents, reducing overhead costs by 60%.

By democratizing advanced multi-LLM architectures through a walk-up usable interface, DocCenter bridges the gap between complex generative AI theory and practical, day-to-day operations. You don't need a specialized engineering department to build a smart, self-correcting document workflow—you just need a few clicks in a workspace dialog.

Want to see the AI Reviewer and natural language validation features live? Contact our team today to schedule a demo of DocCenter.