| name | pii-anonymization |
| description | Use or extend PII anonymization in genai-tk — covers the shared core library (genai_tk.extra.nlp), the AnonymizationMiddleware for agents, the anonymize_files_flow Prefect ETL flow, custom entity recognizers (COMPANY, PRODUCT, PROJECT), and the SensitivityRouterMiddleware. Use when adding privacy controls to a workflow, an agent profile, or a batch pipeline. |
PII Anonymization in genai-tk
Read First
docs/nlp.md — full NLP reference (spaCy, Presidio, classifiers, French support)docs/middleware-pii-and-routing.md — agent middleware referencegenai_tk/extra/nlp/presidio.py — PresidioDetector, PresidioDetectorConfig, CustomRecognizerConfiggenai_tk/extra/nlp/anonymization.py — anonymize_text(), make_fake_value(), AnonymizationConfiggenai_tk/agents/langchain/middleware/anonymization_middleware.py — LangChain agent middlewaregenai_tk/workflow/prefect/flows/anonymize_flow.py — Prefect ETL flow
Architecture
genai_tk/extra/nlp/
presidio.py ← PresidioDetector, PresidioDetectorConfig, CustomRecognizerConfig, DetectedEntity
anonymization.py ← anonymize_text(), make_fake_value(), AnonymizationConfig
Consumers:
agents/langchain/middleware/anonymization_middleware.py ← AnonymizationMiddleware (agent)
workflow/prefect/flows/anonymize_flow.py ← anonymize_files_flow (ETL batch)
Both consumers call the same anonymize_text() function — identical behaviour at ETL time and agent runtime.
Key Types
| Type | Module | Purpose |
|---|
PresidioDetectorConfig | extra.nlp.presidio | Which fields to detect, custom recognizers, spaCy model, score threshold |
CustomRecognizerConfig | extra.nlp.presidio | One regex-based custom entity (entity name + patterns + context words) |
PresidioDetector | extra.nlp.presidio | Runs Presidio analysis; AnalyzerEngine is cached per-config (singleton) |
AnonymizationConfig | extra.nlp.anonymization | Wraps detector config + Faker settings + fuzzy-deano config |
anonymize_text() | extra.nlp.anonymization | Stateless: detect → fake → return (anonymized_text, mapping) |
make_fake_value() | extra.nlp.anonymization | Maps entity type → Faker call |
AnonymizationMiddleware | agents.langchain.middleware.anonymization_middleware | LangChain AgentMiddlware; before_model anonymizes, after_model deanonymizes; thread-safe |
SensitivityRouterMiddleware | agents.langchain.middleware.sensitivity_router_middleware | Routes sensitive calls to a safe LLM |
Use Case 1 — Add Anonymization to an Agent Profile (YAML)
langchain_agents:
privacy_agent:
name: "Privacy Agent"
type: react
llm: default
middlewares:
- class: genai_tk.agents.langchain.middleware.anonymization_middleware:AnonymizationMiddleware
analyzed_fields: [PERSON, EMAIL_ADDRESS, PHONE_NUMBER, CREDIT_CARD]
faker_seed: 42
fuzzy_deanonymize: true
fuzzy_threshold: 85
Run: cli agents langchain -p privacy_agent --chat
With custom domain entities (companies, products):
middlewares:
- class: genai_tk.agents.langchain.middleware.anonymization_middleware:AnonymizationMiddleware
analyzed_fields: [PERSON, EMAIL_ADDRESS]
faker_seed: 42
custom_recognizers:
- entity_name: COMPANY
patterns:
- "(?i)\\b(Acme Corp|Globex|Tech Solutions)\\b"
context: [company, firm, organization]
- entity_name: PRODUCT
patterns:
- "(?i)\\b(WidgetPro|CloudMaster)\\b"
context: [product, service, platform]
Use Case 2 — Batch Anonymize Files (Prefect ETL)
workflows:
anonymize_docs:
run: genai_tk.workflow.prefect.flows.anonymize_flow.anonymize_files_flow
inputs:
base_dir: "${paths.data_root}/raw"
output_dir: "${paths.data_root}/anonymized"
pathspecs: ["**/*.txt", "**/*.md"]
analyzed_fields: [PERSON, EMAIL_ADDRESS, PHONE_NUMBER]
faker_seed: 42
save_mapping: true
uv run cli workflow run anonymize_docs --dry-run
uv run cli workflow run anonymize_docs
Use Case 3 — Programmatic / Scripting
from faker import Faker
from genai_tk.extra.nlp import AnonymizationConfig, CustomRecognizerConfig, PresidioDetector, PresidioDetectorConfig, anonymize_text
config = PresidioDetectorConfig(
analyzed_fields=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD"],
custom_recognizers=[
CustomRecognizerConfig(
entity_name="COMPANY",
patterns=[r"(?i)\b(Acme Corp|Tech Solutions)\b"],
context=["company", "firm"],
),
],
score_threshold=0.4,
)
detector = PresidioDetector(config=config)
Faker.seed(42)
faker = Faker(["en_US"])
anonymized, mapping = anonymize_text(
"John Smith at Acme Corp: john@acme.com",
detector=detector,
faker=faker,
)
anonymized2, mapping = anonymize_text(chunk2, detector=detector, faker=faker, mapping=mapping)
French / Multilingual PII
Set language in config or per-detector:
nlp:
default_language: fr
default_model: fr_core_news_sm
models:
en: en_core_web_sm
fr: fr_core_news_sm
from genai_tk.extra.nlp import PresidioDetector, PresidioDetectorConfig
detector = PresidioDetector(config=PresidioDetectorConfig(language="fr"))
entities = detector.detect("Appelez Jean Dupont au 06 12 34 56 78")
Install French model: python -m spacy download fr_core_news_sm
If the model is not installed, you get a clear ImportError with install instructions.
Supported Entity Types
Presidio built-ins: PERSON, EMAIL_ADDRESS, PHONE_NUMBER, CREDIT_CARD, LOCATION, IBAN_CODE, US_SSN, IP_ADDRESS, URL, DATE_TIME, NRP, ORG
Domain-specific (via CustomRecognizerConfig): COMPANY → faker.company(), PRODUCT → faker.bs(), PROJECT → faker.bs()
Custom fallback: any unknown entity type → faker.bothify(text="ABCD####")
Combining with SensitivityRouterMiddleware
middlewares:
- class: genai_tk.agents.langchain.middleware.anonymization_middleware:AnonymizationMiddleware
analyzed_fields: [PERSON, EMAIL_ADDRESS, PHONE_NUMBER]
faker_seed: 42
- class: genai_tk.agents.langchain.middleware.sensitivity_router_middleware:SensitivityRouterMiddleware
safe_llm: ollama_local
sensitive_source_patterns: ["**/hr/**", "**/confidential/**"]
Tests
GENAITK_PROFILE=pytest uv run pytest tests/unit_tests/extra/test_anonymization_core.py -q
GENAITK_PROFILE=pytest uv run pytest tests/unit_tests/agents/langchain/middleware/ -q
Avoid
- Do NOT store the anonymization mapping without encryption — it maps fake values back to real PII.
