refresh-genie-best-practices

# Refresh Genie Best Practices This skill drives an interactive review that keeps the evaluator's local best-practices reference in sync with the official Databricks Genie documentation. It only proposes changes that matter for **Genie space design and configuration** (the scope of this agent) and applies them only after the user approves. Keep relevance high and noise low. Most runs will surface few or zero changes. **Reporting "no relevant updates" is a correct and expected outcome.** Do not invent updates to look productive. When in doubt about whether a finding matters to a Genie space author, drop it. Less is more. ## Files this skill touches All paths are relative to the repo root. - `genie_validator/prompts/best_practices.md` - the authoritative reference (sections 1-6). The thing you update. - `genie_validator/prompts/evaluator_system.md` - the evaluator system prompt. It cites `best_practices.md` **by section number** (sections 1-6, 1-3, 2, 3g, 3h, 4, 5, 6). If you renumber or restructure sections, update these references in the same pass. - `genie_validator/prompts/org_conventions.md` - customer-specific conventions (naming, fiscal calendar, currency/timezone). NOT driven by public docs. Read-only here: only flag conflicts, never auto-edit. - `genie_validator/prompts/report_system.md` - read-only sanity check; it only renders the reference date string. ## Load-bearing constraints (do not break) 1. **Date line format.** Line 3 of `best_practices.md` is exactly `Last updated: YYYY-MM-DD`. It is parsed by `genie_validator/nodes.py` (`re.search(r"Last updated:\s*(\d{4}-\d{2}-\d{2})", ...)`) into `_BEST_PRACTICES_DATE` and shown in the report. Keep this exact format. Update it **last**, and only if at least one edit was applied. 2. **Preserve section numbering.** The evaluator prompt references sections by number. Prefer **additive edits inside the existing section** (1-6 and the 3a-3h subsections). Only restructure if unavoidable, and then mirror every changed reference in `evaluator_system.md`. 3. **No em dashes.** Per repo and global convention, do not use em dashes in any edited markdown. Use a comma, colon, parentheses, or a sentence break. 4. **Match house style.** Concise, imperative bullets that read like the surrounding `best_practices.md` content. ## Procedure ### 1. Load local state Read `best_practices.md` in full (note the `Last updated:` date and the current 1-6 / 3a-3h structure), `org_conventions.md`, and skim `evaluator_system.md` to see which section numbers it cites. Open by telling the user the current `Last updated:` date and how stale that is relative to today, so they can gauge how much drift to expect. ### 2. Fetch the docs WebFetch these six authoritative pages: - https://docs.databricks.com/aws/en/genie/ - https://docs.databricks.com/aws/en/genie/set-up - https://docs.databricks.com/aws/en/genie/tune-quality - https://docs.databricks.com/aws/en/genie/monitor - https://docs.databricks.com/aws/en/genie/best-practices - https://docs.databricks.com/aws/en/genie/troubleshooting Optionally consult https://docs.databricks.com/llms.txt to discover **newly added** adjacent Genie pages not in the list above. Only follow links that are clearly about Genie space authoring. If a fetch fails (restricted egress, etc.), say so explicitly and continue with whatever fetched. Do not silently skip a source. ### 3. Diff against local For each practice the docs describe, classify it: **already covered** / **new** / **changed** (docs now say something different from local). Apply a hard scope filter. KEEP only items that affect Genie **space design and configuration**, for example: - Tables/views in the space, descriptions, hard limits, knowledge-store budget. - Instructions: SQL expressions (measures/filters/fields), example SQL queries, SQL functions, general text instructions, and the instruction priority hierarchy. - Joins and relationship cardinality; synonyms; format assistance; entity matching. - Benchmark question design. - UC data-layer expectations that the evaluator checks (types, PK/FK, naming, descriptions). DROP (out of scope): end-user querying tips, UI click-throughs, billing/quotas, permissions/governance plumbing, release-note chatter, and anything not actionable as an evaluation rule the agent could check against a space config. ### 4. Present a prioritized proposal Present surviving candidates as a single Markdown table, sorted by priority (High first). Use these exact columns: | ID | Prio | Docs | Current local state | Recommendation | |----|------|------|---------------------|----------------| - **ID**: short stable handle (e.g. `A`, `B`, `C`). - **Prio**: `High` / `Medium` / `Low`, by how much the change affects space-design accuracy. - **Docs**: what the docs say now, in one terse phrase. Name the source page in parentheses (e.g. "(tune-quality)"). - **Current local state**: what `best_practices.md` says today: not covered / partial / conflicts. One terse phrase. - **Recommendation**: `Include -> §<section>` or `Skip`, plus a 3-6 word reason. Write for a Genie space author practitioner: sentence fragments, no marketing language, no restating the obvious. Keep each cell short enough to scan. Omit source URLs and "new vs changed" prose from the table; if the user wants detail on a row, give it on request. If nothing clears the scope filter, skip the table: state plainly that the local reference is already aligned with the docs, no changes recommended, and stop here. That is a valid outcome. Then ask the user which IDs to apply. Do not edit anything before they choose. ### 5. Apply approved edits For each approved item: - Edit `best_practices.md` additively within the matching section to preserve numbering and house style. - If a change genuinely requires restructuring sections, update the corresponding `section N` references in `evaluator_system.md` in the same pass so they still point at real headings. ### 6. Org-conventions conflict check Re-read `org_conventions.md` against the applied changes. If a newly added universal practice conflicts with a customer convention (rare), surface the conflict to the user and let them decide. Never edit `org_conventions.md` automatically. ### 7. Bump the date Only if at least one edit was applied, update line 3 of `best_practices.md` to `Last updated: <today in YYYY-MM-DD>`. ### 8. Summarize Report what changed (with section locations), what was skipped and why, and any follow-ups in `evaluator_system.md` or conflicts flagged in `org_conventions.md`. ## Post-edit checks (only when edits were applied) - `grep -n "Last updated:" genie_validator/prompts/best_practices.md` shows the new `YYYY-MM-DD`. - Smoke import still works and the date parses (not "unknown"): `uv run python -c "import genie_validator.nodes as n; print(n._BEST_PRACTICES_DATE)"` - Every `section N` reference in `evaluator_system.md` still maps to a real heading in `best_practices.md`.