pii-scrubbing

name	pii-scrubbing
version	1.0.0
category	security
description	Systematically scrub Personally Identifiable Information (PII) from a codebase — inventory real domains, hostnames, credentials, and email addresses across all files; patch them to placeholders; create .example templates for gitignored configs; rewrite git history to remove PII from past commits; and verify the result is clean.
tags	["pii","security","git-filter-repo","history-rewrite","credentials","secrets"]

PII Scrubbing — Systematic Codebase Cleanup

Trigger

Use this skill when:

• Asked to "remove PII from the repo" or "scrub real URLs/hostnames/credentials"
• Preparing a repo for open-source publication
• A security review finds real domains, passwords, or emails in code/docs
• Asked to create .example templates for config files with secrets
• Asked to remove PII from git history

Workflow

Phase 1: Inventory

Search the repo systematically for PII patterns. Run ALL of these searches:

Pattern	What it catches
grep -r 'realdomain.com' .	Real infrastructure domains
grep -r 'customer-host.com' .	Real server hostnames
grep -r 'user:pass@' .	Credentials embedded in URLs
grep -rE '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}' .	Email addresses
grep -r 'sk-[a-zA-Z0-9]{20,}' .	API keys
grep -rE '\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b' .	IP addresses (check each — 127.x.x.x is safe)
`grep -r 'password	secret
grep -rE ':1[0-9]{4}[0-9]*' .	Port numbers (check if they're custom infrastructure)
find . -name '*cred*' -o -name '*secret*' -o -name '*token*' -o -name '*.pwd' 2>/dev/null	Credential files in working tree

Gitignore audit for credential files: Every credential file found in the working tree must be gitignored. For each one:

git check-ignore <filename>
# Returns the path if gitignored (safe), nothing if NOT gitignored (CRITICAL)

Also verify the .gitignore patterns that catch them. Common patterns: *cred*, *secret*, *token*, *.pem, *.key. Run grep -n 'cred\|secret\|token\|pem\|key' .gitignore to confirm coverage. If a credential file isn't covered by any existing pattern, add one.

PII categories to flag (any one of these is actionable):

Category	Examples	Treatment
Infrastructure domain	your-infra-domain.org, customer-app.com	Replace with your-domain.com
Production hostname	mweb-stage.customer.or.kr, api.customer.org	Replace with your-customer-host.com
Agent hostname	gisu-host, joseph-host	Replace with your-agent-host
Real credentials	user:pass, sk-abc..., T1tus!nbox_2026	Replace with your-credential-placeholder
Real email addresses	[e-mail], [e-mail]	Replace with [e-mail]
Real file paths with user	/home/luke, /Users/luke	Replace with /path/to/app or $HOME
Client project names	clientco-royalty, client-app	Evaluate: is this identifying? If yes, use client-project
Company/organization names	CLIENTCO, AcmeCorp	Replace with ExampleCorp or generic ClientName
Real person names in paths	joseph/, luke/	Rename to generic descriptor (operator/, user/). Scan directory listing, not just file contents
Real person names in content	Luke, Joseph (not biblical/historical figures)	Replace with the user, the operator, or a role descriptor
Multiple TLD variants (same base domain)	client.org, ,

Phase 2: Patch Working Tree

For each file with PII, determine the right treatment:

A) Scripts with hardcoded default URLs:

# Before
URL = os.environ.get("SOME_VAR", "https://realdomain.com:13004")

# After
URL = os.environ.get("SOME_VAR", "https://your-domain.com:13004")

Pattern: change the fallback default to your-domain.com. The env var override still works for actual deployments.

B) Config templates:

# Before
MOSES_INBOX_URL="https://realdomain.com:13004"

# After
MOSES_INBOX_URL="${DOMAIN_URL:-https://your-domain.com:13004}"

Or replace directly with placebo values and document that the user must set their actual values.

C) Example curl commands in docs/code:

# Before
curl https://realdomain.com:13004/api/send

# After
curl https://your-domain.com:13004/api/send

D) Credentials in example code:

# Before
curl -sk -u "user:password123"

# After
curl -sk -u "user:your-password"

E) Email addresses in skill references:

# Before
[e-mail]

# After
[e-mail]

Phase 3: Create .example Templates for Gitignored Files

For any config file that contains real secrets and is gitignored:

1. Create filename.example with placeholder values
2. Ensure the real file is in .gitignore
3. Add !*.example exception to .gitignore if .example extension is generally gitignored
4. Commit both the .gitignore fix and the .example file

Phase 3b: Template + Setup Script Pattern (for programmatic configs)

For JSON configs or structured data that need programmatic generation (not manual editing), use a three-file pattern instead of .example:

File	Location	Purpose
Template	src/name.template.json (repo)	Placeholder values like {{CORTEX_DOMAIN}}
Setup script	ops/scripts/setup-name.sh (repo)	Reads env vars or prompts, substitutes placeholders, writes real file outside repo
Real config	~/.hermes/state/name.json (outside repo)	Actual URLs/credentials, never committed
Consumer code	Python/sh scripts	Reads private path first, falls back to template if not found

Structure in the repo:

Repo (public):
  src/agent-registry.template.json           ← {{PLACEHOLDER}} values only
  ops/scripts/setup-agent-registry.sh         ← creates real file at ~/.hermes/state/

Filesystem (gitignored / outside repo):
  ~/.hermes/state/agent-registry.json        ← real URLs, NEVER committed

Consumer code pattern:
  # 1. Check ~/.hermes/state/ first
  # 2. Fall back to src/*.template.json if not found
  # 3. This lets new installs run with defaults, production uses real config

When to use this pattern:

• Config file is JSON or other structured format (not plain key=value)
• Values need validation beyond "is it non-empty"
• Multiple URL/credential pairs need filling (e.g. agent registry with 5+ agents)
• Same setup process runs on multiple machines

When to use .example pattern instead:

• Simple key=value configs (.env, .ini)
• Files the user will manually edit once
• Self-explanatory values that don't need validation

Proven in Hermes Cortex:

• models.env — env vars with CORTEX_DOMAIN override
• hermes-inbox.conf — created by setup with CORTEX_INBOX_* vars
• agent-registry.json — multi-agent registry created by setup-agent-registry.sh

Non-interactive deployment:

# Set env vars, then run the script — no prompts
CORTEX_DOMAIN=myhost.com CORTEX_HEALTH_PORT=13007 \
  ESTHER_DOMAIN=otherhost.com ESTHER_HEALTH_PORT=13007 \
  bash ops/scripts/setup-agent-registry.sh

For .env files specifically:

• Audit docker-compose*.yml for all ${VAR:?...} and ${VAR:-...} references
• Collect required and optional variables
• Create deploy/.env.example with:
  • Comment header explaining purpose and where to put the file
  • Quick-seed commands (e.g., openssl rand -hex 32 for generating secrets) — embed these in the example so users can copy-paste them
  • Required vars listed first with empty values (the user MUST fill these)
  • Optional vars commented out with defaults documented
  • Hermes integration vars as a separate commented section if they go in a different file (~/.hermes/.env)
• Ensure !.env.example exception is in .gitignore AFTER .env.*:

  .env
  .env.local
  .env.*
  !.env.example  # ← the negation must come AFTER .env.*
  

• See: deploy/.env.example in hermes-cortex for a worked example

For service-specific config files (e.g., ops/scripts/inbox.conf.example):

• Follow the same pattern: real values → placeholders, document what the user should replace

Phase 4: Rewrite Git History

Use git-filter-repo to scrub PII from ALL past commits:

1. Install git-filter-repo (if needed):

   pip3 install git-filter-repo
   

2. Create replacements file (/tmp/pii-replacements.txt):

   literal:realdomain.org==>your-domain.com
   literal:realdomain.com==>your-domain.com
   literal:customer.host.name==>your-customer-host
   literal:RealPassword123!==>your-password-placeholder
   

3. Clone to a temp directory (run filter-repo on a clone, not the working repo):

   cd /tmp
   rm -rf repo-clean 2>/dev/null
   git clone /path/to/working-repo repo-clean
   cd repo-clean
   

4. Run filter-repo:

   git-filter-repo --replace-text /tmp/pii-replacements.txt --force
   

5. Verify — search for every pattern:

   git grep -i "realdomain\|password123\|other-pattern" .
   # Should return EXIT 1 (no matches)
   

6. Handle branch protection (GitHub):

   • If main is protected: push to main-sanitized first, then ask the user to temporarily disable branch protection
   • Then force-push the cleaned history:

     git remote add origin https://github.com/user/repo.git
     git push --force --all origin
     

Phase 4b: Cover Commit Messages (Separate Pass)

⚠️ --replace-text only covers file contents and filenames — it does NOT touch commit messages. Commit messages need a dedicated --replace-message pass.

After --replace-text completes, run a second filter-repo pass with the same replacements file:

cd /tmp/repo-clean
git filter-repo --force --refs HEAD --replace-message /tmp/pii-replacements.txt

This rewrites the history again, applying the same literal replacements to commit subjects and bodies. Without this, commit messages like fix: add Gisu to mirror chain for CLIENTCO images still leak the real project name.

When to use: Always. Unless you've verified there's no PII in any commit message (via git log --oneline --grep), assume there is.

Phase 4c: File Renaming via --filename-callback

When obfuscating a project name that appears in filenames (e.g. clientco-development-patterns.md → acme-development-patterns.md), use --filename-callback.

The callback receives bytes, not str — decode/encode accordingly:

git filter-repo --force --refs HEAD \
  --replace-text /tmp/replacements.txt \
  --filename-callback 'return filename.replace(b"clientco", b"acme").replace(b"CLIENTCO", b"ACME").replace(b"Clientco", b"Acme")'

After renaming, update the replacement rules file to include the case variants:

CLIENTCO==>ACME
Clientco==>Acme
clientco==>acme

Phase 4d: Case Variants

A project name can appear in three case forms:

• UPPERCASE: CLIENTCO — in titles, headings, CISAC codes
• Titlecase: Clientco — in running text ("Clientco-royalty monorepo")
• lowercase: clientco — in URLs, paths, repo names (clientco-works, clientco-royalty)

--replace-text does case-sensitive exact match — clientco==>acme will NOT match CLIENTCO or Clientco. Add ALL variants to the replacements file:

CLIENTCO==>ACME
Clientco==>Acme
clientco==>acme

Same for --filename-callback — each variant needs its own .replace() call.

Phase 5: Verify — Check Both Content AND Commit Messages

Comprehensive check after all changes:

Check	How	Expected
No PII in file contents	git grep -i "realdomain|original-pattern" .	Exit 1 (no matches)
No PII in filenames	find . -iname "*oldproject*"	Empty
No PII in commit messages	git log HEAD --oneline --grep="oldproject|other-pattern"	Empty
No PII in past commits	git-filter-repo output shows 0 matches	Clean run
Real configs gitignored	git check-ignore path/to/config.json	Exit 0 (ignored)
Example templates tracked	git ls-files includes .example files	Listed

Verification gotcha: After cloning from a local repo that's already been rewritten, git log --all --grep= shows false positives from refs/remotes/origin/main (the stale remote tracking ref). Always verify against HEAD explicitly: git log HEAD --oneline --grep=... — not --all.

Ordering Rules in Replacements File

Place more specific patterns FIRST in the replacements file, more general patterns LAST. Filter-repo applies rules sequentially — a vague match can consume a URL before the specific domain rule fires.

Wrong ordering:

clientco==>acme                              # ← fires first on "clientco.or.kr" → "acme.or.kr" ❌
clientco.or.kr==>client-domain.com           # ← never reached

Correct ordering:

clientco.or.kr==>client-domain.com           # ← fires first, catches full domain ✅
mweb-stage.clientco.or.kr==>your-gisu-host   # ← even more specific should come first
clientco==>acme                              # ← now safe: only matches non-domain uses

Pitfalls

• Don't run git-filter-repo on the working repo directly — it removes remotes and you lose the origin. Always clone to tmp first.

• Don't forget .example gitignore exceptions — if .gitignore has .env.*, .env.example will be ignored too. Add !.env.example AFTER the pattern.

• Don't stop at the working tree — PII in past commits is still visible via git blame and accessible to anyone who clones history. Always use git-filter-repo for a full scrub.

• Don't miss the second domain variant — if the user has realdomain.com and realdomain.org, search for BOTH. Users often have one domain for infrastructure and another for email/SSL.

• Don't miss commit messages — --replace-text is for file contents only. Commit messages need a separate --replace-message pass. This is the #1 oversight — without it, the commit log still leaks.

• Don't use --all in verification — git log --all --grep= includes stale remote tracking refs that predate the filter. Use git log HEAD --grep= to verify the actual branch being pushed.

• Don't forget filename callbacks return bytes — --filename-callback in filter-repo receives bytes, not str. Use filename.replace(b"old", b"new") not filename.replace("old", "new").

• Don't forget case variants — clientco is three different strings (clientco, Clientco, CLIENTCO). Add all to the replacements file.

• Don't order replacements wrong — put specific full-domain patterns before generic project-name patterns. A clientco==>acme rule that fires before clientco.or.kr==>client-domain.com produces acme.or.kr instead of client-domain.com.

• Don't force-push protected branches — check with first, or push to an unprotected branch and ask for protection to be temporarily lifted.

References

• deploy/.env.example — reference template for Langfuse docker-compose env vars
• src/agent-registry.json.example — reference template for agent registry with placeholder URLs
• ops/scripts/inbox.conf.example — reference template for inbox config
• skills/software-development/public-contribution/SKILL.md — PII→placeholder mapping table for contributions