Secret Scanner - Catching Leaked Credentials Before They Ship

Why I Built This

Leaked secrets in source code are one of the most common and preventable security incidents. A developer hardcodes an AWS key during testing, pushes to GitHub, and within minutes automated scrapers have already found it. GitGuardian’s 2024 report found over 12.8 million new secret occurrences in public commits.

I built a scanner that catches these before they leave the developer’s machine. It runs as a CLI tool or a git pre-commit hook, uses regex pattern matching for known secret formats, and Shannon entropy analysis to flag strings that look suspicious even without matching a known pattern.

Zero dependencies. Python standard library only.

What It Detects

The scanner ships with 20+ built-in detection rules covering the services developers leak credentials for most often:

Critical severity – immediate exposure risk:

• AWS Access Keys and Secret Keys
• GitHub Personal Access Tokens (classic and fine-grained)
• Slack Bot/User Tokens
• Stripe Secret Keys
• Database connection strings with embedded credentials
• Private keys (RSA, EC, DSA, OpenSSH)
• Supabase Service Role Keys

High severity – significant exposure:

• Google API Keys
• SendGrid and Twilio API Keys
• Hardcoded passwords
• GitHub OAuth tokens
• Slack webhook URLs

Medium severity – needs review:

• Generic API key and secret assignments
• High-entropy strings flagged by Shannon analysis
• Sensitive env vars outside .env files

How Entropy Detection Works

Regex patterns catch known formats like ghp_ or AKIA, but what about proprietary tokens or keys from services the scanner doesn’t have a rule for?

That’s where Shannon entropy comes in. Entropy measures the randomness of a string. A normal variable name like userProfileController has low entropy because it uses common letter patterns. A secret like xK9mR2pL5wQ8nJ4vT7cY has high entropy because the characters are distributed more uniformly.

The scanner calculates entropy for strings found in suspicious contexts (variable assignments with keywords like “key”, “token”, “secret”) and flags anything above the threshold. Hex strings use a lower threshold (3.5 bits) since their character set is smaller; base64-like strings use 4.5 bits.

def shannon_entropy(data: str) -> float:
    if not data:
        return 0.0
    freq = {}
    for char in data:
        freq[char] = freq.get(char, 0) + 1
    length = len(data)
    entropy = 0.0
    for count in freq.values():
        prob = count / length
        entropy -= prob * math.log2(prob)
    return entropy

False Positive Reduction

A secret scanner that fires on every long string is useless because developers stop reading the output. I built in several layers of filtering:

• UUID detection – Strings matching UUID format are skipped
• SHA-256 hashes – 64-character hex strings are likely file hashes, not secrets
• Placeholder values – Strings like “changeme”, “example”, “placeholder” are ignored
• Environment variable references – os.environ.get("SECRET_KEY") is safe; SECRET_KEY = "abc123" is not
• Lock file exclusion – package-lock.json, yarn.lock, and similar files are skipped entirely
• Inline suppression – Add secret-scanner:ignore or nosec as a comment to mark intentional patterns

Pre-commit Hook

The scanner installs as a git pre-commit hook with one command:

python scanner.py --install-hook

After installation, every git commit automatically scans staged files. If any critical or high severity findings are detected, the commit is blocked with details on what was found and where to fix it.

Output Formats

Text – Color-coded terminal output with severity levels, file locations, matched patterns (redacted), and remediation advice.

JSON – Structured output for CI pipelines and dashboards. Includes summary counts by severity and full finding details.

SARIF – Static Analysis Results Interchange Format. Upload directly to GitHub Code Scanning for findings to appear as annotations on pull requests.

# GitHub Actions integration
python scanner.py --format sarif --severity high . > results.sarif

Custom Rules

Teams can add their own detection patterns through a config file:

{
  "custom_patterns": [
    {
      "id": "internal-api",
      "name": "Internal API Token",
      "pattern": "INTERNAL_[A-Z]+_TOKEN\\s*=\\s*['\"]([a-zA-Z0-9_-]{32,})['\"]",
      "severity": "high",
      "description": "Internal service token detected.",
      "remediation": "Move to vault."
    }
  ]
}

Demo Run

Secret Scanner Results
==================================================
Files scanned: 4
Lines scanned: 30
Findings: 20

By severity:
  CRITICAL: 8
  HIGH: 2
  MEDIUM: 10

[1] Database Connection String        CRITICAL
    demo-project/.env:1
    Fix: Move to environment variable or secrets manager.

[2] GitHub Personal Access Token      CRITICAL
    demo-project/.env:2
    Fix: Revoke at github.com/settings/tokens.

[3] Stripe Secret Key                 CRITICAL
    demo-project/api.js:3
    Fix: Roll the key immediately in Stripe Dashboard.

[4] AWS Access Key ID                 CRITICAL
    demo-project/api.js:8
    Fix: Rotate the key in IAM console.

...and 16 more findings

Tech Stack

• Python 3.10+ (standard library only)
• Shannon entropy analysis
• SARIF 2.1.0 output spec
• Git hooks integration

What I Learned

The hardest part of building a secret scanner isn’t the detection. It’s the false positive management. Every regex pattern that’s too broad generates noise that makes developers ignore the tool entirely. Every pattern that’s too narrow misses real leaks. Finding the right balance took iterating on keyword pre-filters, entropy thresholds, and context-aware exclusions.

The entropy approach is particularly interesting because it’s language-agnostic. It doesn’t care if the secret is an AWS key or a proprietary internal token. If the string has enough randomness in a suspicious context, it gets flagged. That’s the safety net that catches what the regex rules miss.