m62-altitude-onboarding

name

m62-altitude-onboarding

description

Extract entity data from household document folders (PDFs, Word docs, images, spreadsheets) and update the Altitude (Altcore) platform via API. Queries Altitude first to find existing households and their universe of entities (Individuals, LegalEntities, AccountFinancials, Contacts, TangibleAssets, Households), extracts data from documents, matches and merges against existing records (filling empty fields, flagging conflicts), creates relationships, and uploads documents to the correct entity. Use this skill whenever the user mentions Altitude, Altcore, onboarding families, extracting entity data from documents, updating households, processing client folders, or uploading documents. Also trigger when the user has a folder of family documents (trusts, LLCs, tax returns, IDs, insurance, estate plans, bank statements) and wants to populate a wealth management platform.

Altitude Document Extraction & Entity Update

⛔ CRITICAL RULE: You MUST read EVERY SINGLE FILE in the household folder. Not most files. Not the important-looking files. ALL files. Write a file tracker (altitude_review/file_tracker.md) listing every file. Mark each READ as you go. Do NOT proceed to Phase 4 until the tracker shows 100% READ. If you read 22 out of 60 files, you have failed. This is the #1 cause of extraction failure — see "Zero-Skip Rule" in Phase 3.

This skill extracts entity data from household document folders and updates the Altitude platform. It follows a query-first, match-and-merge approach — never blindly creating entities. Every change is reviewed before pushing.

Prerequisites

Required Tools

This skill runs cross-platform (macOS, Linux, Windows). The following tools must be installed and on the user's PATH before running. Verify each at the start of Step 0 with shutil.which(...) and fail fast with a clear message if anything is missing — do NOT attempt to install tooling automatically.

Tool	Why	macOS	Linux	Windows
Python 3.9+	Script runtime for .docx/.xlsx/.eml/large PDFs	brew install python	apt install python3	winget install Python.Python.3.12 (avoid the Microsoft Store stub — it silently redirects to a non-functional alias)
pip packages	Document parsing	pip install pypdf python-docx openpyxl requests	same	same
qpdf	Decrypt password-protected PDFs	brew install qpdf	apt install qpdf or dnf install qpdf	winget install qpdf.qpdf or choco install qpdf or scoop install qpdf
poppler (pdftotext)	Text-first PDF extraction (REQUIRED, not optional) — the default PDF read strategy uses pdftotext -layout before falling back to Claude's Read tool. Avoids the 2000px image-dimension limit that scanned-PDF pages can hit.	brew install poppler	apt install poppler-utils	winget install oschwartz10612.Poppler or choco install poppler
tesseract (OCR)	Fallback for scanned PDFs where pdftotext returns empty (i.e., pure image PDFs — trust documents, deeds, handwritten notes). Pipe pdftoppm -r 150 → tesseract to get text.	brew install tesseract	apt install tesseract-ocr	winget install UB-Mannheim.TesseractOCR or choco install tesseract
pandoc (optional)	Cross-platform .docx → text	brew install pandoc	apt install pandoc	winget install JohnMacFarlane.Pandoc
curl	Occasional API examples (all scripted work uses requests)	built-in	built-in	built-in on Windows 10 1803+ (C:\Windows\System32\curl.exe)

Verify with this snippet (use PYTHON from Cross-Platform Setup below):

# check_prereqs.py
import shutil, socket, sys
missing = []
# Required tools
for tool in ("qpdf", "pdftotext"):        # pandoc + tesseract are optional-but-recommended
    if not shutil.which(tool):
        missing.append(tool)
# Python packages
try:
    import pypdf, docx, openpyxl, requests  # noqa: F401
except ImportError as e:
    missing.append(f"python package: {e.name}")
# DNS reachability check (fail fast if the user is on a restricted network)
try:
    socket.gethostbyname("api.m62.live")
except socket.gaierror:
    missing.append("DNS: cannot resolve api.m62.live (check network or set up hosts override — see Step 0.c)")
if missing:
    sys.exit(f"Missing prerequisites: {', '.join(missing)}")
print("All prerequisites OK")

Windows-Specific Notes

1. Python alias trap: Windows 10+ ships a python.exe stub that opens the Microsoft Store instead of running Python. Verify with python --version. If it opens the Store, disable the alias under Settings → Apps → Advanced app settings → App execution aliases and install real Python from python.org or winget.
2. Long paths (MAX_PATH 260): household folders with deep nesting can exceed Windows' legacy 260-character path limit. Either enable long paths (reg add HKLM\SYSTEM\CurrentControlSet\Control\FileSystem /v LongPathsEnabled /t REG_DWORD /d 1 /f as admin, then reboot) or place household folders at a short root like C:\cl\ instead of the default Documents tree.
3. File paths in prompts: when passing file paths to sub-agents, use forward slashes or raw strings in Python (r"C:\cl\Smith" or "C:/cl/Smith"). Mixing backslashes with regular strings causes \n, \t, \r escapes to fire unexpectedly.
4. PowerShell execution policy: running .ps1 scripts may be blocked by the default Restricted policy. For the refresh scripts, either run with powershell -ExecutionPolicy Bypass -File tools\refresh-api-spec.ps1 or set the policy once with Set-ExecutionPolicy -Scope CurrentUser RemoteSigned.
5. No bash-isms: Do not write &&, ||, $(...) command substitution, ${VAR} expansion, single-quote heredocs, or python -c "..." with embedded newlines. Always write scripts to a .py file and run them with python script.py.
6. Line endings: Python handles CRLF/LF transparently. If you write a .py helper script on Windows, don't worry about line endings.

Step 0: Load Saved Configuration + Authenticate

Do this FIRST before anything else.

Altitude implements a full OAuth 2.1 + PKCE + Dynamic Client Registration authorization server — the exact same protocol Claude uses for its MCP/connector integrations (RFC 8414 / RFC 7591 / RFC 9728 / RFC 7636). This is the preferred interactive auth mode for the skill: the user signs in on Altitude's own hosted login page in a browser, approves the client, and the skill receives a JWT access token via a local loopback callback.

The access token returned by OAuth is a standard Altitude JWT — it works for every REST endpoint (/api/v1/individual, /api/v1/household, /api/v1/document, etc.), not just MCP endpoints, despite the mcp:read/mcp:write scope names.

Auth modes supported:

Mode	Header used on every request	When to use
OAuth (browser)	Authorization: Bearer <access_token>	Default for interactive use. Altitude-hosted login, optional MFA, refresh tokens.
API Key	X-API-Key: ak_live_...	Automation, CI, long-lived server integrations. No browser needed.
JWT (direct)	Authorization: Bearer <id_token>	Fallback: user pastes a JWT obtained out-of-band (e.g., from the Altitude UI session).

0.a — Config file schema

{HOME_DIR}/.altitude/config.json (where HOME_DIR is $HOME on macOS/Linux or %USERPROFILE% on Windows). The config supports all three modes via an authMode discriminator:

{
  "authMode": "oauth" | "api_key" | "jwt",
  "baseUrl": "https://api.m62.live",
  "firmName": "Wellington Advisors",

  "apiKey": "ak_live_xxxxxxxx",                    // if authMode=api_key
  "jwt": "eyJhbGciOiJIUzUxMi...",                  // if authMode=jwt (manual paste)

  // if authMode=oauth — populated by the OAuth flow below:
  "oauth": {
    "clientId": "550e8400-e29b-...",
    "accessToken": "eyJhbGciOiJIUzUxMi...",
    "refreshToken": "k8f3...",
    "tokenType": "Bearer",
    "expiresAt": "2026-04-18T18:00:00Z",
    "scope": "mcp:read mcp:write",
    "email"                     
  

Security rules (enforce strictly):

1. NEVER write the password to disk. OAuth is specifically designed so the skill never sees the password — the browser handles that directly with Altitude.
2. Keep the config file chmod 600 on Unix; on Windows, NTFS per-user ACLs under %USERPROFILE% provide equivalent protection.
3. When the accessToken is within 5 minutes of expiry, silently refresh via POST /oauth/token with grant_type=refresh_token. If the refresh fails (revoked, expired), fall back to the full browser auth flow.

0.b — If config exists and credentials are current

• authMode=api_key + apiKey set → smoke-test with GET /api/v1/authenticate → use
• authMode=oauth + accessToken not expired → use immediately
• authMode=oauth + accessToken expired but refreshToken valid → refresh silently
• Any other state → run the appropriate auth flow below

0.c — Auth Mode 1: OAuth (browser, recommended for interactive use)

This is the Claude-connector flow. The skill acts as a public OAuth client:

1. Discover endpoints — GET {baseUrl}/.well-known/oauth-authorization-server returns:

   {
     "issuer": "https://api.m62.live",
     "authorization_endpoint": "https://api.m62.live/oauth/authorize",
     "token_endpoint": "https://api.m62.live/oauth/token",
     "registration_endpoint": "https://api.m62.live/oauth/register",
     "scopes_supported": ["mcp:read", "mcp:write"],
     "grant_types_supported": ["authorization_code", "refresh_token"],
     "response_types_supported": ["code"],
     "code_challenge_methods_supported": ["S256"],
     "token_endpoint_auth_methods_supported": ["none"]
   }
   

Complete script — write this to a temp file and run it. It handles the whole flow including the loopback server:

# altitude_oauth_login.py
import base64, hashlib, http.server, json, os, pathlib, secrets, socketserver
import sys, threading, urllib.parse, urllib.request, webbrowser

BASE = sys.argv[1]  # e.g., https://api.m62.live

# 1. Discover endpoints
meta_url = f"{BASE}/.well-known/oauth-authorization-server"
meta = json.loads(urllib.request.urlopen(meta_url, timeout=10).read())

# 2. Load or register client
home = pathlib.Path(os.environ.get("USERPROFILE") or os.environ["HOME"])
cfg_path = home / ".altitude" / "config.json"
cfg_path.parent.mkdir(exist_ok=True)
cfg = json.loads(cfg_path.read_text()) if cfg_path.exists() else {}
client_id = cfg.get("oauth", {}).get("clientId")

# 3. Pick a free loopback port
with socketserver.TCPServer(("127.0.0.1", 0), None) as s:
    port = s.server_address[1]
redirect_uri = f"http://127.0.0.1:{port}/callback"

if not client_id:
    reg_body = json.dumps({
        "client_name": "M62 Altitude Onboarding Skill",
        "redirect_uris": [redirect_uri],
        "grant_types": ["authorization_code", "refresh_token"],
        "token_endpoint_auth_method": "none",
        "response_types": ["code"],
    }).encode()
    req = urllib.request.Request(meta["registration_endpoint"], data=reg_body,
                                 headers={: })
    reg = json.loads(urllib.request.urlopen(req, timeout=).read())
    client_id = reg[]
    ()


cv = base64.urlsafe_b64encode(secrets.token_bytes()).rstrip().decode()
cc = base64.urlsafe_b64encode(hashlib.sha256(cv.encode()).digest()).rstrip().decode()
state = secrets.token_urlsafe()


result = {}
 (http.server.BaseHTTPRequestHandler):
     ():   
     ():
        qs = urllib.parse.urlparse(.path).query
        params = (urllib.parse.parse_qsl(qs))
        result.update(params)
        body = 
        .send_response()
        .send_header(, )
        .send_header(, ((body)))
        .end_headers()
        .wfile.write(body)
        threading.Thread(target=.server.shutdown, daemon=).start()

server = http.server.HTTPServer((, port), Handler)
threading.Thread(target=server.serve_forever, daemon=).start()


auth_url = meta[] +  + urllib.parse.urlencode({
    : , : client_id, : redirect_uri,
    : , : state,
    : cc, : ,
})
()
: webbrowser.(auth_url)
 Exception: ()


 time
deadline = time.time() + 
  result  time.time() < deadline:
    time.sleep()
server.shutdown()

   result:
    sys.exit()
 result.get() != state:
    sys.exit()
    result:
    sys.exit()


token_body = urllib.parse.urlencode({
    : ,
    : result[], : cv,
    : redirect_uri, : client_id,
}).encode()
req = urllib.request.Request(meta[], data=token_body,
    headers={: })
tok = json.loads(urllib.request.urlopen(req, timeout=).read())


 datetime
expires_at = datetime.datetime.utcnow() + datetime.timedelta(seconds=tok[] - )
cfg[] = 
cfg[] = BASE
cfg.setdefault(, {}).update({
    : client_id,
    : tok[],
    : tok.get(),
    : tok.get(, ),
    : expires_at.isoformat() + ,
    : tok.get(),
})
cfg_path.write_text(json.dumps(cfg, indent=))
 os.name == :
    os.chmod(cfg_path, )
()

0.d — OAuth token refresh (automatic)

When the access token is close to expiry, refresh silently:

# altitude_oauth_refresh.py
import json, os, pathlib, urllib.parse, urllib.request, datetime, sys
home = pathlib.Path(os.environ.get("USERPROFILE") or os.environ["HOME"])
cfg = json.loads((home / ".altitude" / "config.json").read_text())
base = cfg["baseUrl"]
oa = cfg["oauth"]
body = urllib.parse.urlencode({
    "grant_type": "refresh_token",
    "refresh_token": oa["refreshToken"],
    "client_id": oa["clientId"],
}).encode()
req = urllib.request.Request(f"{base}/oauth/token", data=body,
    headers={"Content-Type": "application/x-www-form-urlencoded"})
try:
    tok = json.loads(urllib.request.urlopen(req, timeout=10).read())
except urllib.error.HTTPError as e:
    # Refresh failed (token revoked, expired) — caller should re-run full OAuth flow
    sys.exit(f"REFRESH_FAILED:{e.code}")
expires_at = datetime.datetime.utcnow() + datetime.timedelta(seconds=tok["expires_in"] - 30)
oa["accessToken"] = tok["access_token"]
if tok.get("refresh_token"):  # refresh rotation
    oa["refreshToken"] = tok["refresh_token"]
oa["expiresAt"] = expires_at.isoformat() + "Z"
(home / ".altitude" / "config.json").write_text(json.dumps(cfg, indent=2))

0.e — Auth Mode 2: API Key (automation)

User pastes the key (it starts with ak_live_ for production or ak_test_ for dev). Smoke-test with GET {baseUrl}/api/v1/authenticate — 200 means the key is valid. Save with authMode="api_key".

0.f — Auth Mode 3: Direct JWT paste (fallback)

If the user already has a JWT (from the Altitude UI's browser session, for example), they can paste it directly. Save with authMode="jwt" and jwt=<token>. This mode has no refresh capability — when the JWT expires, prompt for a new paste or switch to OAuth.

0.f.5 — DNS reachability test + loopback fallback

Run this test at Step 0 before any API calls. On some networks (corporate DNS, split-horizon, DNS rebinding filters) api.m62.live fails to resolve via the system resolver even though the service is reachable by IP. This has caused 100% of API calls in the skill to fail with connection timeouts in prior runs.

# altitude_dns_probe.py — run first, cache result
import socket, subprocess, json, os, pathlib, sys
def try_system_dns():
    try:
        ip = socket.gethostbyname("api.m62.live")
        return ("system", ip)
    except socket.gaierror:
        return None
def try_public_dns():
    for server in ("1.1.1.1", "8.8.8.8", "9.9.9.9"):
        try:
            out = subprocess.check_output(
                ["dig", f"@{server}", "api.m62.live", "+short", "+time=3"],
                text=True, timeout=5
            ).strip().splitlines()
            ips = [x for x in out if x and not x.startswith(";")]
            if ips: return ("public", ips[0])
        except Exception: pass
    return None

result = try_system_dns() or try_public_dns()
if not result:
    sys.exit("DNS: cannot resolve api.m62.live via any method. Check network/VPN/firewall.")

method, ip = result
home = pathlib.Path(os.environ.get("USERPROFILE") or os.environ[])
probe_file = home /  / 
probe_file.parent.mkdir(exist_ok=)
probe_file.write_text(json.dumps({: method, : ip, : }))
()

If method == "system" → system DNS works, use normal Python requests or curl.

If method == "public" → system DNS is broken but public DNS has the IP. Every subsequent API call must override. Two patterns:

1. curl with --resolve (simplest, reliable):

   curl --resolve "api.m62.live:443:<IP>" -H "X-API-Key: $KEY" "https://api.m62.live/api/v1/..."
   

2. Python requests with connection patching (cleaner for scripts):

   # altitude_http.py
   import json, os, pathlib, requests
   from urllib3.util import connection
   home = pathlib.Path(os.environ.get("USERPROFILE") or os.environ["HOME"])
   probe = json.loads((home / ".altitude" / "dns_probe.json").read_text())
   if probe["method"] == "public":
       _orig = connection.create_connection
       def _patched(addr, *args, **kwargs):
           host, port = addr
           if host == "api.m62.live":
               addr = (probe["ip"], port)
           return _orig(addr, *args, **kwargs)
       connection.create_connection = _patched
   # Now use requests normally — DNS patching is transparent
   

On Windows with curl.exe, the same --resolve flag works. PowerShell's Invoke-WebRequest does not support --resolve; use curl.exe or a Python script via PowerShell instead.

Re-probe every hour (IP can change). Cache the IP in dns_probe.json with a TTL check.

0.g — Pick the right header per request

The skill's helper emits the correct header automatically based on authMode:

# altitude_auth.py — load once, reuse everywhere
import json, os, pathlib, datetime
home = pathlib.Path(os.environ.get("USERPROFILE") or os.environ["HOME"])
cfg = json.loads((home / ".altitude" / "config.json").read_text())

def ensure_fresh():
    """Refresh OAuth token if within 5 minutes of expiry."""
    if cfg.get("authMode") != "oauth": return
    exp = datetime.datetime.fromisoformat(cfg["oauth"]["expiresAt"].rstrip("Z"))
    if exp - datetime.datetime.utcnow() < datetime.timedelta(minutes=5):
        import subprocess, sys
        subprocess.check_call([sys.executable, "altitude_oauth_refresh.py"])
        # reload
        global cfg
        cfg = json.loads((home / ".altitude" / "config.json").read_text())

def headers():
    ensure_fresh()
    mode = cfg.get("authMode", "api_key")
    if mode == "api_key":
        return {"X-API-Key": cfg["apiKey"]}
    if mode == "oauth":
        return {"Authorization": f"Bearer {cfg['oauth']['accessToken']}"}
    if mode == "jwt":
        return {: }
     RuntimeError()

 ():  cfg[]
 ():  cfg.get(, )

0.h — Prompt the user to choose

If no usable config exists, ask:

How should I authenticate to Altitude?

1. OAuth (browser, recommended) — I'll open Altitude's login page in your browser. You sign in there; I never see your password. I'll cache a short-lived access token + refresh token.
2. API Key (automation) — you paste an ak_live_... key. Good for CI or long-running integrations where no human is present.
3. JWT paste (fallback) — paste a JWT obtained from your existing Altitude browser session.

And: "Which environment? Production (https://api.m62.live) or Development (http://localhost:8080)?"

Then run the script for the chosen mode, save config, and proceed.

0.i — Backwards compatibility

Config files without authMode but with apiKey set should be treated as authMode=api_key for transparent upgrade. Write out an updated config with authMode set on the next run.

Cross-Platform Setup

Detect the operating system and set platform-appropriate defaults. Do this ONCE at the start and reuse throughout:

import platform, shutil, os, tempfile

OS = platform.system()  # "Windows", "Darwin", "Linux"

# Python command
PYTHON = "python" if OS == "Windows" else "python3"

# Temp directory (NEVER hardcode /tmp/)
TMPDIR = tempfile.gettempdir()  # e.g., C:\Users\X\AppData\Local\Temp on Windows, /tmp on Unix

# Word doc converter
if shutil.which("textutil"):
    DOCX_CMD = "textutil -convert txt"          # macOS
elif shutil.which("pandoc"):
    DOCX_CMD = "pandoc -t plain -o"             # Cross-platform
else:
    DOCX_CMD = None  # Fall back to python-docx (see below)

# PDF decryptor
QPDF = shutil.which("qpdf")
# Install if missing:
#   macOS:   brew install qpdf
#   Windows: choco install qpdf  OR  winget install qpdf  OR  scoop install qpdf
#   Linux:   apt install qpdf    OR  dnf install qpdf

Save these values and use them for all subsequent commands. When this skill says python3, use PYTHON. When it says /tmp/, use TMPDIR. When it says textutil, use DOCX_CMD.

Full OpenAPI Spec

The full Altitude OpenAPI specification is available at api-docs/api.json relative to this skill's directory. If you encounter an endpoint or schema not covered in the reference files, search the full spec: Glob pattern "**/m62-altitude-onboarding/**/api.json" then use Grep to find specific endpoints or schema definitions.

Additional Requirements

• firmId (UUID) for the target firm — typically discovered during Phase 1 when querying Altitude

---

Workflow Overview

Phase 1:   Query Altitude       → Find existing household + its full entity universe
Phase 2:   Scan Documents       → Classify ALL files, create read-tracking checklist
Phase 3:   Extract Entities     → PARALLEL agents read files, write extraction caches
Phase 3M:  Merge Extractions    → Combine all agent caches into unified extraction
Phase 3.5: Cross-Doc Validation → Name enrichment, relationship inference, absence tracking
Phase 3.7: Self-Audit           → Adversarial review: any unread files? any unnamed people? any missing entities?
Phase 4:   Match & Merge        → Match extracted entities to existing ones, diff fields
Phase 5:   Review               → Show user what will change (fills + conflicts)
Phase 6:   Push Updates         → PATCH existing entities, POST new ones (with approval)
Phase 7:   Upload Documents     → Associate each document with its correct entity

Parallel Extraction Strategy

Phase 3 uses parallel sub-agents to avoid context exhaustion. The orchestrator (you) NEVER reads document contents directly. Instead, you spawn extraction agents that each handle a subset of files and write their results to disk.

Batching rules — split by subdirectory, then by count. Hard cap: 25 files per batch.

1. Group files by subdirectory first. Each top-level folder in the household directory becomes a candidate batch (e.g., Identification/, LLC/, Tax Documents/, Financial Statements/, Insurance/, Estate Planning/).

2. Cap every batch at 25 files. Any group exceeding 25 gets split:

   • 26-50 files → 2 batches of ~18-25
   • 51-75 files → 3 batches of ~17-25
   • 76+ → more splits, 25-file cap Never let a single batch exceed 30 files — sub-agent context pressure becomes severe beyond that, and image-heavy PDFs compound the load.

3. If a subdirectory has < 4 files, merge it with another small directory into one batch.

4. Target batch size: 15-20 files is the sweet spot. Very small families (< 10 files total) use 1 batch; larger families get 5-10 batches running in parallel.

5. Parallelism budget: 5-8 concurrent agents is the default target. 10+ concurrent agents has hit output-size limits in practice; split into waves if needed.

6. Imbalance is OK — don't force-balance batches. A batch of 16 mixed files + a batch of 22 all-statements is fine. Grouping by document type (all statements together, all trust docs together) is more valuable than perfect file-count parity, because agents can apply type-specific heuristics (statement period parsing, trust role extraction).

Historical precedent:

• Glickman (85 files) → 5 batches of 10-21 files, completed in ~9 minutes
• Boro-Hamilton (215 files) → 8 batches of 16-37 files; the 37-file batch strained context and retried once. Cap of 25 would have prevented the retry.

Example batching for a 40-file household:

Batch 1 (Agent A): Identification/ (3 files) + Onboarding/ (1 file) = 4 files
Batch 2 (Agent B): LLC/ (8 files) + Estate Planning/ (4 files) = 12 files
Batch 3 (Agent C): Tax Documents/ files 1-10 = 10 files
Batch 4 (Agent D): Tax Documents/ files 11-14 + Financial Statements/ (5 files) + Insurance/ (3 files) = 12 files

Each extraction agent receives:

• Its file list (absolute paths)
• The household name and folder path
• The extraction field definitions (from this skill's Phase 3 entity fields section)
• The document type patterns (from references/document_type_patterns.md)
• Instructions to write output to altitude_review/extraction_cache_batch_{N}.jsonl

Each extraction agent produces:

• One JSONL file: altitude_review/extraction_cache_batch_{N}.jsonl
• One tracker section: altitude_review/file_tracker_batch_{N}.md

After ALL agents complete, the orchestrator:

1. Reads all extraction_cache_batch_*.jsonl files
2. Reads all file_tracker_batch_*.md files
3. Merges into unified extraction_cache.jsonl and file_tracker.md
4. Verifies 100% file coverage before proceeding to Phase 3.5

Spawn agents using the Agent tool:

Agent(
  prompt="[extraction agent prompt with file list and instructions]",
  description="Extract batch N ({directory_name})",
  mode="bypassPermissions"
)

Launch ALL extraction agents in a single message so they run in parallel. Do NOT launch them sequentially — that defeats the purpose.

---

Phase 1: Query Altitude — Get Existing Household Universe

Before touching any documents, query Altitude to understand what already exists.

Step 1.1: Search for the household

GET /api/v1/household/search?searchFor={household_name}&size=50
X-API-Key: {api_key}

or with JWT:

GET /api/v1/household/search?searchFor={household_name}&size=50
Authorization: Bearer {token}

If a matching household is found, record its id. If multiple matches, ask the user which one. If no match, note that this is a new household (will need POST later).

Step 1.2: Get the household's full relationship graph

Query outgoing relationships (household → members) and incoming relationships:

GET /api/v1/household/{householdId}/relationships/from
X-API-Key: {api_key}

Or via the standalone entity relationship endpoint:

GET /api/v1/entity-relationship/from/HOUSEHOLD/{householdId}
X-API-Key: {api_key}

This returns all EntityRelationshipDto entries — every individual, legal entity, account, contact, and their relationship types (MEMBER, OWNERSHIP, TRUSTEE, BENEFICIARY, ADVISOR, etc.). Record:

• All individual IDs + basic info
• All legal entity IDs + entity types
• All account (AccountFinancial) IDs + account types
• All contact IDs + job titles
• Relationship metadata (type, role, percentage, effectiveFrom, effectiveTo)

Step 1.3: Fetch full details for each entity

Account graph traversal — DO NOT trust household.totalAccountCount. In practice the household count often exceeds the number of accounts reachable via direct HOUSEHOLD → ACCOUNT_FINANCIAL relationships, because most accounts hang off trusts and LLCs, not the household itself. In a $1.22B household with 48 accounts, fewer than a dozen were directly owned by the household — the rest were inside trust/LLC sub-graphs.

Traversal algorithm (implement this before moving past Phase 1):

# altitude_account_graph.py — recursively discover all accounts reachable from household
visited_entities = set()   # (entity_type, entity_id) pairs we've already expanded
all_accounts = {}          # account_id -> basic info

def expand(entity_type, entity_id):
    key = (entity_type, entity_id)
    if key in visited_entities: return
    visited_entities.add(key)
    rels = api_get(f"/api/v1/entity-relationship/from/{entity_type}/{entity_id}")
    for r in rels:
        if r["targetEntityType"] == "ACCOUNT_FINANCIAL":
            all_accounts[r["targetEntityId"]] = {
                "id": r["targetEntityId"],
                "name": r["targetEntityName"],
                "ownerType": entity_type,
                "ownerId": entity_id,
                "ownerName": "<look up from existing cache>",
            }
        elif r["targetEntityType"] in ("LEGAL_ENTITY", "INDIVIDUAL"):
            expand(r["targetEntityType"], r["targetEntityId"])  # recurse

expand("HOUSEHOLD", household_id)

This expands Household → its individuals and legal entities → each of their outgoing relationships → any sub-LEs they hold → all the way down to every leaf ACCOUNT_FINANCIAL. The number of accounts discovered should match or exceed household.totalAccountCount. If the discovered count is LOWER, flag as open question (the household counter may include hard-deleted or orphan accounts).

Account search fallback — some accounts may not be wired into the relationship graph (orphan accounts created directly). Also search by household name tokens AFTER graph traversal to catch these:

GET /api/v1/account-financial/search?searchFor={householdNameToken}&size=100

For each individual in the household:

GET /api/v1/individual/{id}

For each legal entity in the household:

GET /api/v1/legal-entity/{id}

For each account discovered via the traversal above:

GET /api/v1/account-financial/{id}

For each contact in the household:

GET /api/v1/contact/{id}

For tangible assets, query by owner:

GET /api/v1/tangible-asset/by-owner/INDIVIDUAL/{individualId}
GET /api/v1/tangible-asset/by-owner/LEGAL_ENTITY/{legalEntityId}

For liabilities (query by individual/household):

GET /api/v1/liability/by-individual/{individualId}
GET /api/v1/liability/by-household/{householdId}

For insurance policies (query by individual/household/legal entity):

GET /api/v1/insurance-policy/by-individual/{individualId}
GET /api/v1/insurance-policy/by-household/{householdId}
GET /api/v1/insurance-policy/by-legal-entity/{legalEntityId}

Store all of this as the "Altitude Universe" — the complete current state of the household in Altitude. This is the baseline for comparison.

Step 1.4: Search for accounts and contacts by name

Additionally, search for any accounts and contacts by name pattern:

GET /api/v1/account-financial/search?searchFor={account_name_pattern}&size=50
GET /api/v1/contact/search?searchFor={contact_name_pattern}&size=50

Step 1.5: Build the Altitude Universe index

Create an in-memory index for matching:

{
  "household": { "id": "...", "name": "...", "firmId": "..." },
  "individuals": [
    {
      "id": "...", "firstName": "...", "lastName": "...", "ssn": "...",
      "dateOfBirth": "...", "email": "...", "phone": "...", "addressLegal": "..."
    }
  ],
  "legal_entities": [
    {
             
             
    
  
   
    
             
         
    
  
   
    
             
         
    
  
   
    
             
         
    
  
   
    
             
           
         
    
  
   
    
             
           
       
    
  
    ... 

Save as {household_folder}/altitude_review/altitude_universe.json for reference.

---

Phase 2: Scan & Classify Documents

Step 2.0: OneDrive / cloud-sync hydration pre-scan (REQUIRED before spawning agents)

OneDrive, Dropbox, iCloud and Box store files as "dataless placeholders" until accessed — reading one triggers a download. In extraction sub-agents, a cloud-stub read times out after tens of seconds (default socket timeout), wasting compute. Detect unhydrated files before spawning agents and report them to the user for bulk hydration in Finder/Explorer before the expensive extraction runs.

# altitude_hydration_scan.py
import os, subprocess, sys
from pathlib import Path

HOUSEHOLD = sys.argv[1]  # absolute path to household folder
STUB_THRESHOLD_SECS = 3  # a 1-byte read taking > 3s is almost certainly a cloud stub

def is_cloud_stub(path: str) -> bool:
    try:
        subprocess.check_output(
            ["dd", f"if={path}", "bs=1", "count=1", "of=/dev/null"],
            stderr=subprocess.DEVNULL, timeout=STUB_THRESHOLD_SECS,
        )
        return False
    except (subprocess.TimeoutExpired, subprocess.CalledProcessError):
        return True

stubs = []
for root, _, files in os.walk(HOUSEHOLD):
    if "altitude_review" in root: continue
    for f in files:
        if f.startswith(".DS_Store"): continue
        p = os.path.join(root, f)
        if is_cloud_stub(p): stubs.append(p)

if stubs:
    print(f"❌ {len(stubs)} cloud-stub files detected (read times out):")
    for s in stubs: print(f"  {s}")
    ()
    ()
    ()
    ()
    ()
    ()
    sys.exit()
:
    ()

If stubs are found: present the list to the user in a compact form (grouped by parent directory, counts), ask them to hydrate, then re-run this scan before proceeding. Do NOT launch extraction agents if any unhydrated files remain — they will consume hundreds of seconds of agent time timing out on reads.

If all files are hydrated: proceed to document classification below.

Step 2.1: Classify Documents

List all files recursively in the household folder. Classify each document using the patterns in references/document_type_patterns.md. Key classification rules:

Extraction priority:

• Tier 1 (extract first): Onboarding sheets, IDs (DLs, passports), trust agreements, LLC operating agreements, articles of organization/incorporation, EIN letters, account applications
• Tier 1.5 (NEVER SKIP — compact, data-dense IRS forms): Form 1098 (mortgage → creates liability + tangible asset), Schedule K-1 (ownership percentages — authoritative), Form 1099-DIV/INT/B/R (account + custodian validation), Form W-2 (employer + income), Form 5498 (IRA details). A single 1098 produces 1 liability, 1 tangible asset, and 2+ relationships. A single K-1 produces an entity + ownership relationship with exact percentage.
• Tier 2: Personal tax returns (1040), entity tax returns (1065/1120/1120S/1041), account statements, insurance policy declarations, property tax bills, beneficiary designations
• Tier 3: Financial statements, meeting notes, presentations, valuations, estate planning flowcharts
• Tier 4 (skip): Duplicates ("Copy of", "zDupes"), receipts, .msg files, spreadsheets with personal notes

Document-to-entity association — each document maps to an entity type for upload: Read references/document_entity_association.md for the complete mapping of which document types associate with which Altitude entity type and what documentSubType to use.

---

Phase 3: Extract Entities from Documents

PDF Reading — TEXT-FIRST by default

⚠ CRITICAL: Do NOT start with Claude's Read tool on PDFs. Claude's Read tool rejects images with any dimension >2000px, and many scanned PDFs (trust documents, deeds, handwritten notes, high-res scans) include pages that trip this limit. The result is "image exceeds 2000px dimension limit" errors that abort whole extraction batches.

Required reading order for every PDF:

1. Text-first via pdftotext (poppler) — works on any PDF with embedded text:

   pdftotext -layout -nopgbrk "file.pdf" - | head -c 200000 > /tmp/extracted.txt
   

   Then Read /tmp/extracted.txt. This is fast, safe, and avoids the image limit entirely.

2. If pdftotext returns mostly blank or gibberish → the PDF is a scan. Render and OCR:

   # Render pages 1-5 at 150 dpi (cap pages for large scans)
   pdftoppm -r 150 -f 1 -l 5 "file.pdf" /tmp/scan_page
   # Each page becomes /tmp/scan_page-1.png, scan_page-2.png, etc.
   for png in /tmp/scan_page-*.png; do
     tesseract "$png" "${png%.png}" -l eng
   done
   cat /tmp/scan_page-*.txt > /tmp/extracted.txt
   

   Then Read /tmp/extracted.txt.

3. Only fall back to Claude's Read tool on the raw PDF as a last resort — and only if the file is < 5 MB (to avoid loading many high-res pages). If Read fails with the 2000px error, mark the file status=FAILED_IMAGE_TOO_LARGE in the tracker and move on. Do not loop.

Use pypdf for page-index scanning — this is still the best way to find the data-rich pages in a 200-page tax return without loading every page's content:

Large PDF Strategy (20+ pages)

Tax returns and combined statements are often 50-200+ pages. Reading only the first few pages will miss K-1 summaries, W-2s, 1099s, Schedule H, and passthrough entity details buried deep in the document. Use this two-pass strategy, built on the text-first foundation above:

Pass 1 — Page Index Scan (fast, text-only):

Use PYTHON from the Cross-Platform Setup section for all Python invocations. On Windows, write multi-line scripts to a temp .py file instead of using -c to avoid shell quoting issues.

# page_scan.py — write this to a temp file, then run: python page_scan.py
import sys
from pypdf import PdfReader
reader = PdfReader(sys.argv[1])
print(f'Total pages: {len(reader.pages)}')
for i, page in enumerate(reader.pages):
    text = (page.extract_text() or '')[:150].replace('\n', ' | ')
    print(f'  Page {i+1}: {text}')

Run: {PYTHON} page_scan.py "file.pdf" (where {PYTHON} is python on Windows and python3 on macOS/Linux, per Cross-Platform Setup above).

This produces a one-line summary per page. Scan the output for keywords that signal data-rich pages:

Keyword	What It Signals	Action
K-1, Schedule K-1	Partnership/LLC ownership + income	Read full page — TAX_K1 checklist
W-2, Wage and Tax	Employer name, wages, SSN	Read full page — TAX_W2 checklist
1099, 1099-DIV, 1099-INT, 1099-B, 1099-R	Account/custodian validation, income	Read full page — TAX_1099 checklist
1098, Mortgage Interest	Mortgage lender, balance, property	Read full page — TAX_1098 checklist
Schedule E, Passthrough	Entity names, EINs, income types	Read full page
Schedule H, Household Employ	Domestic staff, household employer EIN	Read full page
Schedule A, Itemized	Mortgage interest, charitable, taxes	Skim for amounts
Schedule C, Profit or Loss	Sole proprietorship business	Read full page
Sign Here, Occupation, Preparer	Occupations, CPA name/phone	Read full page (usually page 2 of 1040)
8879, e-file	SSNs, AGI confirmation, preparer	Read full page
Entity names (trust names, LLC names)	Entity K-1 details	Read full page
LESSER, TRUST, or any family surname	Related trust/entity income	Read full page

Pass 2 — Targeted Deep Read: For each flagged page, extract text with pdftotext -f N -l N "file.pdf" (where N is the page number) to a temp file and Read that. Only use Claude's Read tool on the original PDF for the flagged pages if pdftotext returns empty for that specific page (indicating a scanned page). For a typical 200-page return, you'll usually need to read 15-25 key pages.

Minimum pages to ALWAYS read from a personal 1040 return:

1. Cover letter (page 1) — preparer firm, client address
2. Form 8879 — SSNs, AGI, preparer name
3. Form 1040 pages 1-2 — income summary, dependents, occupations, preparer, filing status
4. Schedule E page 2 — ALL passthrough entity names + EINs
5. Passthrough income detail pages — entity-by-entity breakdown
6. Schedule H (if present) — household employment
7. Any pages with K-1, W-2, 1098, or 1099 keywords

Password-protected PDFs: Tax returns are often password-protected. The password is frequently in the filename (e.g., "pass 701431"). Decrypt before reading:

qpdf --password=PASSWORD --decrypt input.pdf decrypted.pdf

Write the decrypted file to the same directory as the input, or to a temp directory (use Python tempfile.mkdtemp() if needed — do NOT hardcode /tmp/).

If qpdf is not installed:

• macOS: brew install qpdf
• Windows: choco install qpdf or winget install qpdf or scoop install qpdf
• Linux: apt install qpdf or dnf install qpdf

⛔ CRITICAL: Zero-Skip Rule — THE #1 CAUSE OF EXTRACTION FAILURE

EVERY file in the household folder MUST be opened and read. NO EXCEPTIONS.

This is the single most important rule in this skill. In testing, 100% of extraction failures trace back to files that were not read. Not "low quality" files. Not "redundant" files. Files that were simply never opened. An Operating Agreement that contains ownership percentages. A 1099 that reveals an account number. A DocuSign certificate that identifies the employer. An email signature with an attorney's contact info.

You WILL be tempted to skip files. You will think "I already have the EIN from the onboarding sheet, I don't need to read the EIN letter." You will think "The amendments just change the address, I already know the address." You will think "The Sunbiz is just a state filing." Every one of these thoughts leads to missed data. Every document contains something — a name, an address, a date, a registered agent, a formation date — that cannot be found anywhere else.

Do not classify files as low priority and skip them. Do not read 22 out of 60 files and call it done. Read ALL 60. If the context window gets full, save your extraction progress to disk and continue in a follow-up pass.

For each file, at minimum:

• PDFs: Read at least page 1. If it's a multi-page form (tax return, statement), use the Large PDF Strategy above to find all data-rich pages.
• Images (.jpg, .png): Read with Claude's vision. Even a property photo confirms a real asset exists.
• Word docs (.docx): Convert using the platform's DOCX_CMD (see Cross-Platform Setup). Fallback chain: textutil (macOS) → pandoc (cross-platform) → python-docx (write a docx_read.py script — see Standard Document Extraction below for the exact script). If all fail, flag for user — don't silently skip.
• Emails (.eml): Parse headers + body. Extract attachments and process them too.

Enforce with a tracking file: After Phase 2 classification, write a file checklist to altitude_review/file_tracker.md with every file path. As you read each file, update the tracker with status (READ/SKIPPED) and a one-line summary of what was extracted. Before Phase 4, parse the tracker and verify ZERO files have status other than READ. If any files remain unread, you MUST read them before proceeding. This is not optional.

Example tracker format:

| # | File | Status | Extracted |
|---|------|--------|-----------|
| 1 | Identification/DL.png | READ | Brett Adam, Michaela Dylan, DOBs, addresses |
| 2 | LLC/Operating Agreement.pdf | READ | Members: Brett 60%, Michaela 40% |
| 3 | Tax/1099-INT.pdf | PENDING | |

For folders with 10+ files, use the Parallel Extraction Strategy (see Workflow Overview). Spawn one Agent per batch. Each agent handles its assigned files independently and writes results to its own extraction_cache_batch_{N}.jsonl file. The orchestrator merges after all agents complete. For folders with < 10 files, a single agent handles all files.

Extraction Cache (REQUIRED — each agent writes its own)

After reading EACH file, each extraction agent appends what it learned to its own cache file: altitude_review/extraction_cache_batch_{N}.jsonl (one JSON object per line, append-only). The orchestrator later merges all batch files into altitude_review/extraction_cache.jsonl.

Each line captures everything extracted from a single file:

{"file": "Identification/DL.png", "readAt": "2026-03-19T22:00:00Z", "fileNumber": 1, "entities": {"individuals": [{"name": "Brett Adam Podolsky", "dob": "1988-02-19", "gender": "M", "dlNumber": "P342-061-88-059-0", "dlState": "FL", "dlExpiry": "2031-02-19", "address": "3985 NW 53rd St, Boca Raton, FL 33496"}]}, "relationships": [], "contacts": [], "accounts": [], "notes": "Both Brett and Michaela DLs on same image"}
{"file": "LLC/Hercules/Operating Agreement.pdf", "readAt": "2026-03-19T22:01:00Z", "fileNumber": 2, "entities": {"legalEntities": [{"name": "Hercules Lender LLC", "type": "LLC", "managementType": "MEMBER_MANAGED", "opAgreementDate": "2022-09-29"}]}, "relationships": [{"source": "Brett Podolsky", "target": "Hercules Lender LLC", "type": "OWNERSHIP", "percentage": 50, "role": "Managing Member"}], "contacts": [{"name": "Jason Evans", "role": "Registered Agent", "address": "2300 NW Corporate Blvd Suite 215, Boca Raton FL 33431"}], "accounts": [], "notes": "Principal: 20352 Hacienda Ct"}

Why this matters:

• Resumability: If context resets at file 150 of 292, the next session reads the cache and picks up at file 151 — no re-reading of the first 150 files
• Cross-document validation: Later files can check against earlier extractions ("is this the same trust?") without re-reading the source documents
• Subagent handoff: One agent extracts (writes cache), another matches/merges (reads cache)
• Audit trail: Every extracted field traces to a specific file

Cache rules:

• Append after EACH file, not at the end of a batch — partial progress is saved
• Include ALL extracted data, not just summaries — names, dates, numbers, addresses, percentages
• Use fileNumber to track progress — on resume, skip files with fileNumber ≤ max in cache
• For files with no extractable data, still append a line with empty entities and a note explaining why
• The cache is the source of truth for Phase 4 matching — read it instead of relying on context memory

On resume, check for existing cache (use os.path.join for cross-platform paths):

import json, os
cache_path = os.path.join('altitude_review', 'extraction_cache.jsonl')
existing = []
try:
    with open(cache_path) as f:
        existing = [json.loads(line) for line in f if line.strip()]
    last_file = max(e['fileNumber'] for e in existing)
    print(f"Resuming from file {last_file + 1} ({len(existing)} files already cached)")
except FileNotFoundError:
    print("No cache found, starting fresh")

Standard Document Extraction

For each Tier 1 and Tier 2 document, extract structured data. Use Claude's native tools:

• PDFs: Use Claude Read tool natively (supports text and scanned PDFs with vision)

Windows note: Do NOT use python -c "..." with embedded newlines or nested quotes. Windows cmd and PowerShell mangle multi-line argv strings and single-quote escaping differently from bash. The cross-platform pattern is: write the script to a temp .py file with the Write tool, then run it as python script.py. This avoids every shell-quoting pitfall on every OS. All snippets below use this pattern.

• Word docs (.docx): Use the platform's DOCX_CMD (see Cross-Platform Setup):
  • macOS: textutil -convert txt file.docx then read the .txt
  • Cross-platform: pandoc file.docx -t plain (pipe or redirect output)
  • Python fallback (works everywhere): write docx_read.py below, then python docx_read.py "file.docx"

# docx_read.py
import sys
from docx import Document
for p in Document(sys.argv[1]).paragraphs:
    print(p.text)

Install python-docx if needed: pip install python-docx

• Images (.jpg, .png): Use Claude Read tool — Claude can see images natively (multimodal)
• Spreadsheets (.xlsx): Write xlsx_read.py below, then python xlsx_read.py "file.xlsx". Install if needed: pip install openpyxl

# xlsx_read.py
import sys, openpyxl
wb = openpyxl.load_workbook(sys.argv[1], data_only=True)
for sheet in wb.sheetnames:
    ws = wb[sheet]
    print(f"=== {sheet} ({ws.max_row} rows x {ws.max_column} cols) ===")
    for row in ws.iter_rows(values_only=True):
        print(row)

• Emails (.eml): Write eml_read.py below, then python eml_read.py "file.eml". Extract entity data from the email body (e.g., account confirmations, policy updates, advisor correspondence). Process any saved attachments as their native file type.

# eml_read.py — prints headers + body, saves attachments to temp dir
import email, os, sys, tempfile
with open(sys.argv[1], 'rb') as f:
    msg = email.message_from_binary_file(f)
for h in ('From', 'To', 'Date', 'Subject'):
    print(f"{h}: {msg[h]}")
att_dir = os.path.join(tempfile.gettempdir(), 'eml_attachments')
os.makedirs(att_dir, exist_ok=True)
for part in msg.walk():
    ctype = part.get_content_type()
    if ctype == 'text/plain':
        body = part.get_payload(decode=True)
        if body:
            print(body.decode(errors='replace'))
    fn = part.get_filename()
    if fn:
        out_path = os.path.join(att_dir, fn)
        with open(out_path, 'wb') as out:
            out.write(part.get_payload(decode=True))
        print(f"Saved attachment: {out_path}")

For each document, extract:

Individuals:

• Core: firstName, lastName, preferredName (known-by/English name — e.g., "Tina" for legal "Dong"), dateOfBirth, ssn, gender, maritalStatus, citizenship
• Contact: email, phoneNumberPrimary, phoneNumberSecondary, faxNumber
• Address: addressLine1, addressLine2, city, state, postalCode, country, addressType, addressEmployer
• Financial: netWorth, netWorthLiquid, annualIncome, sourceOfFunds, riskTolerance, timeHorizon
• Tax: taxStatus, taxIdType (SSN/ITIN/EIN/FOREIGN_TIN/FOREIGN_ENTITY_TIN/VAT/GST/BUSINESS_NUMBER), taxIdIssuingCountry
• Regulatory: accreditedInvestorStatus, isPoliticallyExposedPerson
• Beneficiary: transferOnDeath beneficiaries
• Lifecycle: dateOfDeath, lifecycleStatus
• Profession: occupation, employer

Legal Entities:

• Core: legalName, entityType, formationDate, jurisdiction, incorporationState, incorporationCountry
• Tax: taxId (EIN), taxClassification, fiscalYearEnd
• Governance: llcManagementType, primarySigners, nominee, eligibility
• Registered Agent: agent name and address
• Authorized Shares: corpAuthorizedShares, corpIssuedShares
• Compliance: KYC/AML tracking, regulatoryStatus, affiliations
• Trust-specific (if entityType=TRUST):
  • Trustees: trustee names and roles
  • Grantor: grantor name and revocable status (revocable/irrevocable)
  • Beneficiaries: primary and contingent beneficiary names
  • Settlor and Trustee names, roles
  • Situs (jurisdiction), governing law
  • Investment Advisor, Distribution Advisor, Crummey Power Holders
  • Powers of Appointment, Spendthrift Provision, GST exemption status
  • Insurance Provisions, Amendment tracking (number, date), Restatement status
  • Pour-over trust name, trust aliases

Account Financials:

• Core: name, accountNumber, accountCategory (INDIVIDUAL/ENTITY), subCategory
• Type: wrapper (IRA/401k if applicable), taxStatus
• Custodian: custodianId, custodian name
• Ownership: ownershipType (PERCENT_BASED default)
• Connection: providerDetails (sourceSystemName, sourceSystemAccountId)

Contacts:

• Core: firstName, lastName, email, phone, jobTitle
• Note: Contact DTO has NO companyName/organization field. Company info goes in biography or via ADVISOR entity relationship.
• Address: addressLine1, addressLine2, city, state, postalCode, country

Tangible Assets:

• Core: name, category (LUXURY/VEHICLE/REAL_PROPERTY/COLLECTIBLE/OTHER)
• Asset Type: assetType (CAR, MOTORCYCLE, BOAT, YACHT, AIRCRAFT, HELICOPTER, RV, PRIMARY_RESIDENCE, VACATION_HOME, RENTAL_PROPERTY, COMMERCIAL_PROPERTY, LAND, FARM_RANCH, TIMESHARE, WATCH, JEWELRY, HANDBAG, FASHION, LUXURY_OTHER, ART, WINE, ANTIQUE, MEMORABILIA, COINS, STAMPS, BOOKS, MUSICAL_INSTRUMENT, COLLECTIBLE_OTHER, EQUIPMENT, LIVESTOCK, FURNITURE, OTHER_TYPE)
• Identification: serialOrIdentifier, VIN, parcel number
• Valuation: currentValue, valuationDate, valuationSource, taxBasis, taxBasisDate, acquisitionType (PURCHASE/AUCTION/GIFT/INHERITANCE/COMMISSION/EXCHANGE/PRIZE), purchaseDate, purchasePrice
• Status: status (ACTIVE/SOLD/TRANSFERRED/DONATED/DISPOSED/STOLEN/LOST)
• Location: location, storageFacility, custodianName, custodianContact
• Insurance: isInsured (Boolean), primaryInsurancePolicyNumber, insuredValue, insuranceExpirationDate (Note: full insurance details are on the InsurancePolicy entity, not TangibleAsset)
• Estate: includedInEstate, estateDisposition, designatedBeneficiaryId, estateNotes
• Type-specific: vehicleDetails (make/model/year), realPropertyDetails (address/square footage), luxuryDetails (brand/model), collectibleDetails (artist/medium)
• Maintenance: maintenanceRecords, provenance chain, condition

Insurance Policies:

• Core: name, policyCategory (LIFE/UMBRELLA/LONG_TERM_CARE/DISABILITY/HEALTH/AUTO/HOMEOWNERS/FLOOD/CYBER/COLLECTIONS/WINDSTORM/OTHER), policyNumber, carrierName, policyStatus (ACTIVE/LAPSED/CANCELLED/PAID_UP/SURRENDERED/MATURED/PENDING)
• Financial: coverageAmount, annualPremium, paymentFrequency (MONTHLY/BI_WEEKLY/QUARTERLY/ANNUAL/INTEREST_ONLY), deductible
• Dates: effectiveDate, expirationDate, applicationDate, issueDate, firstPaymentDate
• Description: description (max 4000)
• Life-specific (policyCategory=LIFE): lifePolicyType (TERM/WHOLE_LIFE/UNIVERSAL/VARIABLE_UNIVERSAL/INDEXED_UNIVERSAL/SURVIVORSHIP/GROUP_TERM), deathBenefit, cashValue, cashValueAsOfDate, loanBalance, termLengthYears, termExpirationDate, isConvertible, conversionDeadline, isIlitOwned, ilitLegalEntityId, isSecondToDie, secondInsuredIndividualId, guaranteedDeathBenefit, riders, surrenderChargeSchedule, dividendOption (CASH/PREMIUM_REDUCTION/ACCUMULATE_AT_INTEREST/PAID_UP_ADDITIONS/ONE_YEAR_TERM)
• Umbrella-specific (policyCategory=UMBRELLA): excessLiabilityCoverage, underlyingAutoRequired, underlyingHomeRequired, underlyingPoliciesDescription, coversRentalProperties, coversWatercraft, uninsuredMotorist
• LTC-specific (policyCategory=LONG_TERM_CARE): dailyBenefitAmount, benefitPeriodDescription, benefitPeriodMonths, eliminationPeriodDays, inflationProtectionType (NONE/SIMPLE/COMPOUND_3_PERCENT/COMPOUND_5_PERCENT/CPI_LINKED/FUTURE_PURCHASE_OPTION), coversHomeCare, coversAssistedLiving, coversNursingFacility, coversAdultDayCare, sharedBenefitRider, isPartnershipQualified, remainingBenefitPool
• Disability-specific (policyCategory=DISABILITY): monthlyBenefitAmount, benefitPeriodDescription, eliminationPeriodDays, isOwnOccupation, ownOccupationPeriodDescription, costOfLivingAdjustment, futureIncreaseOption, residualDisabilityRider, isGroupPolicy, isTaxableBenefit

Liabilities:

• Core: name, liabilityType (MORTGAGE/HOME_EQUITY_LOC/STUDENT_LOAN/MARGIN_LOAN/PLEDGED_ASSET_LINE/CREDIT_LINE/CREDIT_CARD/AUTO_LOAN/PERSONAL_LOAN/BUSINESS_LOAN/OTHER), liabilityStatus (CURRENT/DELINQUENT/IN_DEFERMENT/IN_FORBEARANCE/PAID_OFF/DEFAULTED/CHARGED_OFF)
• Lender: lenderName, accountNumber
• Balance: originalBalance, currentBalance, balanceAsOfDate, creditLimit, availableCredit
• Rate: interestRate, interestRateType (FIXED/VARIABLE/HYBRID), indexRateDescription, rateCap, rateFloor
• Payment: monthlyPayment, minimumPayment, paymentFrequency (MONTHLY/BI_WEEKLY/QUARTERLY/ANNUAL/INTEREST_ONLY), nextPaymentDate
• Term: originationDate, maturityDate
• Collateral: isSecured, collateralDescription, linkedTangibleAssetId, linkedAccountFinancialId
• Tax: isInterestDeductible, interestDeductionType (MORTGAGE_INTEREST/INVESTMENT_INTEREST/STUDENT_LOAN_INTEREST/BUSINESS_INTEREST/NONE), interestPaidYtd, interestPaidPriorYear
• Description: description (max 4000)

Estate Planning (nested on Individual via PATCH):

• Will: hasWill, willDate, executorName, contingentExecutorName, attorneyName, jurisdiction
• Healthcare: hasHealthcareDirective, directiveDate, agentName, alternateAgentName, hasLivingWill
• Financial POA: hasFinancialPoa, type (DURABLE/SPRINGING/LIMITED/GENERAL), agentName, poaDate
• Guardianship: hasGuardianDesignation, guardianName, alternateGuardianName
• Marital: hasPrenuptialAgreement, prenuptialDate, hasPostnuptialAgreement
• Estate Review: lastReviewDate, estimatedEstateValue, estimatedEstateTaxLiability, lifetimeGiftExclusionUsed

Charitable Profile (nested on Individual/LegalEntity via PATCH):

• Extract philanthropic interests, giving history, donor-advised fund info from charitable documents
• Individual: nested under philanthropicProfile field in Individual PATCH
• LegalEntity: nested under charitableDetails field in LegalEntity PATCH

Tag every extracted field with its _source document path.

IMPORTANT: For each document, use the document-type-specific extraction checklist in references/document_type_patterns.md. These checklists ensure you don't miss middle names, occupations, relationship inferences, entity hierarchies, or absence-as-data signals.

Step 3.4: Sensitive Data Detection (mandatory, first-class artifact)

Before merging extraction caches, scan for sensitive data that must NEVER enter API payloads or the main extraction cache. This includes:

• Credit card numbers (PCI DSS)
• Plaintext passwords / passphrases
• Social media / banking login credentials
• Full SSNs in document body text (outside structured extraction fields)
• Passport numbers, driver's license numbers beyond the structured DL fields
• Wire/ACH routing + account number pairs for unrelated parties
• Private encryption keys or API tokens pasted into documents

Write a separate artifact: altitude_review/sensitive_data.json — lists every finding with file, a short description, and a redaction recommendation. This artifact is visible to the user in the review but is NEVER merged into create_payloads.json, extraction_cache.jsonl, or any API-facing file.

Schema:

[
  {
    "id": 1,
    "file": "Onboarding/Fischer Travel Participation Agreement.pdf",
    "fileNumber": 14,
    "type": "credit_card",
    "description": "Visa ending 9115, expiry 9/29, CVV visible",
    "recommendation": "Rotate card immediately. Redact before uploading document OR upload only the redacted version. Never store in any Altitude field.",
    "severity": "high"
  },
  {
    "id": 2,
    "file": "Family Office/FO Tracker/SRB SIN PW.pdf",
    "type": "credential",
    "description": "Social Insurance Number password (Canadian SIN auth)",
    "recommendation": "Move to password manager (1Password/Bitwarden). Do not upload raw file to Altitude.",
     
  

Severity levels:

• critical — PCI data (credit cards), plaintext banking passwords → warn user explicitly in Phase 5 review
• high — passport #s, SIN passwords, full SSN in body text → flag for redaction
• medium — partial credentials, personal-notes-with-secrets → flag, low action required
• low — reference notes, flagged for awareness

Extraction agents must emit sensitive findings to this artifact, NOT to notes or entity fields. Each batch writes its own sensitive_data_batch_{N}.json which the orchestrator merges.

In Phase 5 review, include a dedicated "Sensitive Data Found" section at the top (above normal entity updates) so the user sees it immediately. If any critical severity items exist, the review should recommend pausing Phase 6 until rotation is confirmed.

Document upload policy: files containing sensitive data should either be (a) redacted before upload, (b) excluded from upload with a note, or (c) uploaded with a warning tag so downstream consumers know the file needs handling. Default: do NOT upload files with severity=critical unless the user explicitly overrides.

Step 3.5: Cross-Document Validation Pass

After extracting from ALL documents, run these mandatory checks before proceeding to Phase 4:

0. ⛔ CRITICAL: Latest-date-wins field resolution — When the same field appears on the same entity in multiple documents with different values, the value from the most recent source document wins. This is the single most important merge rule — it supersedes the "most complete value" heuristic below when values actually differ.

   Determine each document's "as-of date" using this priority (first match wins):

   1. Explicit "As of" date printed on the document (e.g., "As of 3/31/2026")
   2. Document execution/signing/effective date (e.g., restated trust date, policy effective date)
   3. Filing or issue date (e.g., 1099 tax year = Dec 31 of that year; deed recording date)
   4. Statement period end date (e.g., "November 2025 statement" → 2025-11-30)
   5. Filename-embedded date patterns: YYYY.MM.DD, YYYY-MM-DD, MM.DD.YY, YYYY_MM (e.g., Certificate of IconTrust 2025.05.15.pdf → 2025-05-15; DL_2024.docx → 2024-01-01 as month/day unknown fallback)
   6. File mtime (filesystem modification time) — only as a last resort, as OneDrive/ Dropbox sync often rewrites mtime to the download time

   Persist asOfDate on every cache entry: each JSONL line in extraction_cache_batch_{N}.jsonl must include an asOfDate field (ISO format YYYY-MM-DD) so Phase 4 can resolve conflicts deterministically.

   Apply to every scalar field — address, email, phone, marital status, employer, occupation, trustee, beneficiary, policy status, account balance, valuation, etc. Exceptions (older value wins):

   • dateOfBirth, ssn, formationDate, taxId — immutable; first confirmed value wins, later contradictions are conflicts to flag
   • originalBalance, originationDate, purchaseDate, purchasePrice — historical values, don't overwrite with later docs
   • firstName, lastName at birth — flag middle/preferred name additions as enrichment rather than replacement

   Apply to amendments & restatements: a "Restated Trust" or "Second Amendment" supersedes the original trust agreement for ALL trustee/grantor/beneficiary fields. The original becomes historical (set on old relationships). Filename tokens to watch: , , , , , , , , (prefer over ).

Step 3.7: Self-Audit Pass (Adversarial Review)

Before proceeding to Phase 4, act as your own auditor. Pretend someone else did the extraction and you are checking their work. Go through these checks:

Document coverage audit:

• List every file in the folder. Is every single one marked as READ? If any file was skipped, read it now. Common misses: .docx files that failed to convert, 1099 cover pages dismissed as "just a cover letter," photos, emails.

Entity completeness audit — for each entity type, ask:

• Individuals: Are there any NAMED PEOPLE in any document who are not yet in my entity list? Check: estate planning docs name guardians, trustees, beneficiaries, executors. Insurance docs name agents. Tax docs name preparers. Emails name senders. LLC docs name attorneys and managers. Every named person is either an Individual or a Contact.
• Legal Entities: Did every entity get created? Common miss: when two spouses each have their OWN trust (not one shared trust), that's TWO legal entities. Check LLC operating agreements for managing members that are THEMSELVES entities (entity-to-entity chains).
• Accounts: Did every 1099 reveal an account number + custodian? Did every bank/institution mentioned in the onboarding sheet get an account created? Did every account statement get its account number extracted?
• Insurance Policies: Were all policies from the insurance summary captured? Are there additional policy documents in the folder not covered by the summary?
• Tangible Assets from Insurance: Did the auto policy list specific vehicles? Create a TangibleAsset (VEHICLE) for each. Did the homeowners policy cover a property? Ensure the property TangibleAsset has isInsured, insuredValue, primaryInsurancePolicyNumber set. Did a collections/valuable articles policy schedule individual items? Each is a TangibleAsset.
• Liability ↔ Asset Links: For every mortgage, auto loan, boat loan, or secured loan — is there a corresponding TangibleAsset? If yes, record linkedTangibleAssetId. If the asset doesn't exist yet, create it first.
• Contacts: Did EVERY professional mentioned in ANY document get a Contact entity? Check: attorneys (estate, corporate, LLC formation — these are often different people), CPAs, insurance agents, financial advisors (from 1099s and statements), CFOs, and loan counterparties.

Relationship completeness audit:

• Does every Individual have at least one relationship (OWNERSHIP from household, or PARENT/CHILD)?
• Does every LegalEntity have at least one relationship (OWNERSHIP, TRUSTEE, GRANTOR)?
• Does every Account have an OWNERSHIP relationship to its owner?
• Is there a SPOUSE relationship if the documents show married individuals?
• Are PARENT→CHILD relationships created for BOTH parents, not just one?
• Does every Contact have at least one professional relationship (ADVISOR/ATTORNEY/ACCOUNTANT)?

Data quality audit:

• Do any two entities have conflicting addresses? (Flag for user)
• Are SSNs 9 digits with no dashes?
• Are phone numbers in E.164-compatible format (digits only)?
• Are dates in ISO format (YYYY-MM-DD)?
• Is any sensitive data (passwords, credit cards) accidentally included in entity fields?

---

Phase 4: Match & Merge — The Core Logic

This is the critical phase. Read references/match_merge_rules.md for detailed rules.

Step 4.1: Deduplicate extracted entities (cross-document merge)

The same person or entity may appear in multiple documents. Merge extracted records using these identity signals:

Individuals — match if ANY of:

• SSN matches exactly (definitive)
• Full name similarity ≥ 0.85 AND (DOB matches OR address matches)
• Full name similarity ≥ 0.65 AND DOB matches AND address matches

Legal Entities — match if ANY of:

• EIN/Tax ID matches exactly (definitive)
• Legal name similarity ≥ 0.7 AND entity type matches

Accounts — match if ANY of:

• Account number matches exactly (definitive)
• Account name similarity ≥ 0.8 AND custodian matches

Contacts — match if ANY of:

• Email matches exactly (definitive)
• Phone matches exactly (definitive)
• Full name similarity ≥ 0.85 AND job title matches

Tangible Assets — match if ANY of:

• Address/parcel number matches (for real property)
• VIN/serial number matches
• Name similarity ≥ 0.8 AND category matches AND owner matches

Insurance Policies — match if ANY of:

• Policy number matches exactly (definitive)
• Name similarity ≥ 0.8 AND carrier name matches (case-insensitive)
• Carrier + coverage amount + policy category all match (probable)

Liabilities — match if ANY of:

• Account number + lender name matches exactly (definitive)
• Name similarity ≥ 0.8 AND lender name matches (case-insensitive)
• Lender + liability type + current balance within 5% tolerance (probable)

When merging across documents for the same field:

1. Different values → latest-date-wins (see Phase 3.5 Step 0). Record winner, loser, and both asOfDates for Phase 5 review.
2. One null, one non-null → take the non-null value (no date check needed).
3. Different levels of specificity (e.g., "Denver" vs "Denver, CO 80202") → prefer the more specific value only if its source is newer or equal in age; otherwise latest-date-wins.
4. Immutable fields (ssn, dateOfBirth, formationDate, taxId) → first confirmed value wins; flag any later contradiction as a hard conflict, don't overwrite silently.

Always track all source documents + their as-of dates on every field.

Step 4.2: Match extracted entities to Altitude Universe

For each merged extracted entity, attempt to match it to an existing Altitude entity:

Individual matching against Altitude:

1. SSN exact match (if both have SSN) → definitive match
2. firstName + lastName exact match (case-insensitive) → strong match
3. firstName + lastName fuzzy match (≥ 0.85 similarity) + DOB match → strong match
4. lastName match + DOB match → probable match (flag for confirmation)
5. No match → candidate for new entity creation

Legal Entity matching against Altitude:

1. EIN/taxId exact match → definitive match
2. legalName exact match (case-insensitive) → strong match
3. legalName fuzzy match (≥ 0.8 similarity) + entityType match → strong match
4. No match → candidate for new entity creation

Account matching against Altitude:

1. accountNumber exact match → definitive match
2. Account name fuzzy match (≥ 0.85 similarity) + custodian match → strong match
3. No match → candidate for new entity creation

Contact matching against Altitude:

1. email exact match → definitive match
2. phone exact match → definitive match
3. firstName + lastName exact match (case-insensitive) + jobTitle match → strong match
4. FIRM-WIDE contact search (do this before creating any new Contact): Query GET /api/v1/contact/search?searchFor={firstName}+{lastName}&size=50 to find existing Contacts across OTHER households in the same firm. A JPM banker serving Verita may already exist under a different household — reuse, don't duplicate. If found:
   • Add the new household as an additional client relationship on the existing Contact (relationship: HOUSEHOLD→CONTACT, type ADVISOR/ATTORNEY/etc.)
   • Merge any new fields (if the existing Contact has no email and you have one, PATCH)
   • Do NOT create a duplicate Contact
5. No match anywhere → candidate for new entity creation

Firm-wide dedup applies especially to: JPM bankers, attorneys (Kirkland & Ellis, Venable LLP, etc.), CPAs (large firms serve multiple clients), insurance agents, Verita's own staff (they work across every household). These should be shared Contacts, not per-household duplicates.

Tangible Asset matching against Altitude:

1. serialOrIdentifier exact match → definitive match
2. Name + category + owner match → strong match
3. Address match (for real property) → strong match
4. No match → candidate for new entity creation

Insurance Policy matching against Altitude:

1. policyNumber exact match → definitive match
2. name + carrierName match (case-insensitive) → strong match
3. carrierName + coverageAmount + policyCategory match → probable match
4. No match → candidate for new entity creation

Liability matching against Altitude:

1. accountNumber + lenderName exact match → definitive match
2. name + lenderName match (case-insensitive) → strong match
3. lenderName + liabilityType + currentBalance within 5% → probable match
4. No match → candidate for new entity creation

Step 4.3: Field-level diff against Altitude

For each matched entity, compare every field. Altitude records carry an updatedAt timestamp (the last write time for that entity). Treat updatedAt as the Altitude value's effective date when resolving conflicts.

For each field in the extracted entity:
  altitude_value     = existing_altitude_entity[field]
  altitude_asof      = existing_altitude_entity.updatedAt   # last API write
  extracted_value    = extracted_entity[field].value
  extracted_asof     = extracted_entity[field].asOfDate     # from Phase 3.5 Step 0

  IF altitude_value is null/empty AND extracted_value is not null/empty:
    → FILL: Queue this field for automatic update (safe to copy)

  ELIF altitude_value is not null/empty AND extracted_value is not null/empty:
    IF altitude_value == extracted_value:
      → MATCH: Values agree, no action needed
    ELIF field is immutable (ssn, dateOfBirth, formationDate, taxId):