AI Output Validator - Catching What the Model Shouldn't Say

Why I Built This

Large language models are being dropped into customer-facing products at a pace that outstrips the tooling to secure them. A model that leaks a user’s SSN in a support response, gives unqualified medical advice, or fabricates a citation that doesn’t exist creates real liability. Most teams catch these problems through manual review or after a user reports them.

I built a validator that sits between the model’s output and the end user. It scans every response for PII leakage, toxicity, policy violations, and hallucinated claims before anything reaches the customer. The offline checks (PII, toxicity, policy) work without any API key. Hallucination detection uses the LLM-as-judge pattern, where a second model call evaluates the first one’s output for factual accuracy.

Zero dependencies. Python standard library only.

PII Detection

The validator scans for 10 categories of personally identifiable information using regex patterns:

• Social Security Numbers (with and without dashes)
• Credit card numbers (Visa, Mastercard, Amex, Discover with Luhn-aware patterns)
• Email addresses
• US phone numbers
• IP addresses
• Dates of birth
• Medical record numbers
• AWS access keys
• Generic API keys and tokens
• Passwords exposed in plaintext

Each pattern uses context-aware matching. A credit card regex alone would flag random 16-digit numbers, so the patterns look for surrounding context like “card number”, “CC:”, or common prefixes. The goal is catching real leakage, not flagging every long number in the output.

Policy Violation Checks

These are configurable rules that catch outputs that cross professional boundaries:

Unqualified Medical Advice – Detects phrases like “you should take”, “I recommend taking”, or “increase your dosage” that cross from information into medical direction without appropriate disclaimers.

Unqualified Legal Advice – Catches “you should sue”, “you have a strong case”, or “file a lawsuit” without legal qualifications.

Unqualified Financial Advice – Flags specific investment recommendations like “buy this stock” or “invest in” without disclaimers.

Authority Impersonation – Detects when the model claims to be a doctor, lawyer, licensed professional, or government official.

False Certainty – Catches absolute claims like “100% guaranteed”, “absolutely certain”, or “impossible to fail” that misrepresent confidence levels.

The LLM-as-Judge Pattern

Regex catches known PII formats and policy violations, but hallucinations are different. A fabricated statistic or a made-up citation looks syntactically identical to a real one. You need semantic understanding to evaluate factual accuracy.

The validator implements the LLM-as-judge pattern: it sends the model’s output to a second model with instructions to act as a factual accuracy auditor. The judge model evaluates:

• Whether specific claims are verifiable or likely fabricated
• Whether cited sources, URLs, or references actually exist
• Whether statistics and numbers appear plausible or invented
• An overall reliability assessment

The judge returns structured JSON that the validator parses into findings with severity levels. This works with any OpenAI-compatible API, so you can use OpenAI, Ollama (free, local), Azure, or any other provider.

# The judge prompt asks for structured evaluation
prompt = """You are a factual accuracy auditor. Analyze this LLM output 
and identify any claims that appear hallucinated, fabricated, or unverifiable.

For each suspicious claim, provide:
- The exact claim text
- Why it appears unreliable  
- Severity (critical/high/medium/low)

Return your analysis as JSON."""

Risk Scoring

Every finding gets a severity level (critical, high, medium, low) and the validator calculates a composite risk score from 0 to 10. The score uses weighted severity values:

• Critical findings (SSNs, passwords): weight 4
• High findings (medical advice, authority claims): weight 3
• Medium findings (IP addresses, false certainty): weight 2
• Low findings (informational): weight 1

A single critical PII finding pushes the score above the typical threshold. This gives teams a single number to use in automated pipelines – anything above a 7 gets routed to human review, anything above a 9 gets blocked.

Demo Output

AI Output Validator Report
==================================================
Input length: 1,033 chars
Checks run:   pii, toxicity, policy
Risk score:   10.0/10
Findings:     10

[1] Social Security Number Detected    CRITICAL
    Category: pii
    Match: [REDACTED]

[2] Medical Record Number Detected     CRITICAL
    Category: pii
    Match: [REDACTED]

[3] Password Exposure Detected         CRITICAL
    Category: pii
    Match: [REDACTED]

[4] Email Address Detected             HIGH
    Category: pii

[5] Unqualified Medical Advice         HIGH
    Category: policy

...and 5 more findings

How It Fits Into a Pipeline

The validator is designed to sit in the response path of any LLM-powered application:

1. User sends a query to your application
2. Your application sends the query to the LLM
3. The LLM returns a response
4. The validator scans the response before it reaches the user
5. If the risk score exceeds your threshold, the response is blocked or flagged

The offline checks (PII, toxicity, policy) add minimal latency since they’re regex-based. The hallucination check adds one additional API call, which you can skip for low-risk use cases with the --no-llm flag.

Tech Stack

• Python 3.10+ (standard library only)
• urllib for OpenAI-compatible API calls (no requests/httpx)
• LLM-as-judge for hallucination detection
• Regex-based PII and policy scanning

What I Learned

The most interesting design decision was the hybrid approach. Regex-based checks are fast, deterministic, and work offline, but they can only catch patterns with known formats. LLM-based checks can evaluate semantic meaning and catch hallucinations, but they’re slower, non-deterministic, and require an API key.

Combining both gives you a baseline of reliable offline detection with an optional deeper analysis layer. The offline checks alone catch the most dangerous issues (PII leakage, authority impersonation), while the LLM judge handles the subtler problems that require understanding context.

The LLM-as-judge pattern itself is worth studying. Using one model to evaluate another model’s output is becoming a standard technique in AI safety, and building it from scratch helped me understand both its power and its limitations. The judge can miss hallucinations it doesn’t have knowledge about, and it can itself hallucinate problems that don’t exist. It’s a defense layer, not a guarantee.