| name | unf |
| description | Universal Numeric Fingerprint (UNF) generator. For strings, it splits into words and sorts them alphabetically to provide order-invariant fingerprints. Supports dataframes and files too. |
♾️ UNF Skill
The UNF (Universal Numeric Fingerprint) skill provides a robust mechanism for generating consistent data identifiers. Unlike traditional file hashes (like MD5 or SHA-256), a UNF is format-agnostic—meaning the same data values will produce the same hash regardless of whether they are stored in CSV, Parquet, SAS, or Stata.
🌟 Key Features
- Semantic Hashing: For strings, it splits content into words and sorts them alphabetically. This ensures that "temperature is celcius" and "celcius is temperature" produce the same UNF.
- Vector Hashing: Fingerprint entire data columns (Polars Series).
- Format Invariance: Identical data in different file formats (e.g., CSV vs. Parquet) yields the same UNF.
- Column-Order Invariance: Dataframes with reordered columns produce the same hash.
- Dataverse Alignment: Designed for parity with the canonical Dataverse UNF implementation.
🛠️ Components
unf_hash.py: CLI tool to compute a UNF for a string or file.
dartfx-unf: High-performance Python implementation using the Polars engine.
🚀 Usage
Hash a simple string
python3 .gemini/skills/unf/scripts/unf_hash.py "Data for fingerprinting"
Hash a data file (CSV, Parquet, etc.)
python3 .gemini/skills/unf/scripts/unf_hash.py data/dataset.csv
Get a detailed JSON report
python3 .gemini/skills/unf/scripts/unf_hash.py --json data/dataset.parquet
📐 Specification