Validation & Retry Loops

Even with tool use, production extraction produces semantic errors that schemas cannot catch. A validation-retry loop wraps each call: validate the output, and if it fails, send Claude a corrective retry. Done well, this converts one-shot failures into a resilient, self-correcting pipeline.

The retry-with-error-feedback pattern

An effective retry returns three things to the model: the original document (source to re-examine), the failed extraction (what it produced), and the specific validation error (exactly what was wrong). The specific error is the critical piece — without it, the model has no guidance and usually reproduces the same mistake.

messages.append({"role": "user",
  "content": "That was wrong. Try again."})

messages.append({"role": "user", "content":
  f"Error: {err}\nRe-examine the document and "
  "fix only the fields in this error."})

from pydantic import BaseModel, model_validator

class Invoice(BaseModel):
    line_items_total: float
    stated_total: float

    @model_validator(mode="after")
    def totals_match(self):
        if round(self.line_items_total, 2) != round(self.stated_total, 2):
            raise ValueError(
                f"line_items_total {self.line_items_total} != "
                f"stated_total {self.stated_total}"
            )
        return self

def extract_with_retry(document, max_attempts=3):
    messages = [{"role": "user", "content": document}]
    for attempt in range(max_attempts):
        raw = call_claude_tool(messages)  # returns tool_use input dict
        try:
            return Invoice(**raw)
        except ValueError as err:
            messages.append({"role": "assistant", "content": str(raw)})
            messages.append({
                "role": "user",
                "content": (
                    "Your extraction failed validation.\n"
                    f"Error: {err}\n"
                    "Re-examine the original document and correct only the "
                    "fields involved in this error."
                ),
            })
    raise RuntimeError("Extraction failed after retries; route to human review")

When retries help and when they don't

The exam heavily tests distinguishing fixable from unfixable failures. Retries work when the information exists but was processed incorrectly; they cannot conjure information that is not there.

Build validation into the schema

• Calculated vs. stated totals — extract both the model's computed sum and the document's stated total; any discrepancy auto-flags without extra logic.
• Conflict-detection booleans — add a flag for contradictory source data (e.g., '30 days' in one section, 'net 60' in another) so the model surfaces the conflict instead of silently picking one.

Syntax, semantics, and Pydantic's dual role

Schema syntax errors (malformed JSON, wrong types) are eliminated by tool use. Semantic validation — cross-field arithmetic, date ordering, business rules — cannot be expressed by a JSON schema and needs external logic. This is where Pydantic earns its place: a Pydantic model enforces structure *and* semantics in one object, and its ValidationError produces machine-readable, per-field messages that drop straight into the retry prompt.