refresh-genie-best-practices

name

refresh-genie-best-practices

description

Interactively refresh the local Genie best-practices reference from the official Databricks Genie docs. TRIGGER when the user asks to "refresh / update Genie best practices", "check the Databricks docs for new best practices", "are our best practices up to date", or wants to sync genie_validator/prompts/best_practices.md with the latest documentation. Scope is limited to practices that affect Genie space design and configuration.

Refresh Genie Best Practices

This skill drives an interactive review that keeps the evaluator's local best-practices reference in sync with the official Databricks Genie documentation. It only proposes changes that matter for Genie space design and configuration (the scope of this agent) and applies them only after the user approves.

Keep relevance high and noise low. Most runs will surface few or zero changes. Reporting "no relevant updates" is a correct and expected outcome. Do not invent updates to look productive. When in doubt about whether a finding matters to a Genie space author, drop it. Less is more.

Files this skill touches

All paths are relative to the repo root.

• genie_validator/prompts/best_practices.md - the authoritative reference (sections 1-6). The thing you update.
• genie_validator/prompts/evaluator_system.md - the evaluator system prompt. It cites best_practices.md by section number (sections 1-6, 1-3, 2, 3g, 3h, 4, 5, 6). If you renumber or restructure sections, update these references in the same pass.
• genie_validator/prompts/org_conventions.md - customer-specific conventions (naming, fiscal calendar, currency/timezone). NOT driven by public docs. Read-only here: only flag conflicts, never auto-edit.
• genie_validator/prompts/report_system.md - read-only sanity check; it only renders the reference date string.

Load-bearing constraints (do not break)

1. Date line format. Line 3 of best_practices.md is exactly Last updated: YYYY-MM-DD. It is parsed by genie_validator/nodes.py (re.search(r"Last updated:\s*(\d{4}-\d{2}-\d{2})", ...)) into _BEST_PRACTICES_DATE and shown in the report. Keep this exact format. Update it last, and only if at least one edit was applied.
2. Preserve section numbering. The evaluator prompt references sections by number. Prefer additive edits inside the existing section (1-6 and the 3a-3h subsections). Only restructure if unavoidable, and then mirror every changed reference in evaluator_system.md.
3. No em dashes. Per repo and global convention, do not use em dashes in any edited markdown. Use a comma, colon, parentheses, or a sentence break.
4. Match house style. Concise, imperative bullets that read like the surrounding best_practices.md content.

Procedure

1. Load local state

Read best_practices.md in full (note the Last updated: date and the current 1-6 / 3a-3h structure), org_conventions.md, and skim evaluator_system.md to see which section numbers it cites. Open by telling the user the current Last updated: date and how stale that is relative to today, so they can gauge how much drift to expect.

2. Fetch the docs

WebFetch these six authoritative pages:

• https://docs.databricks.com/aws/en/genie/
• https://docs.databricks.com/aws/en/genie/set-up
• https://docs.databricks.com/aws/en/genie/tune-quality
• https://docs.databricks.com/aws/en/genie/monitor
• https://docs.databricks.com/aws/en/genie/best-practices
• https://docs.databricks.com/aws/en/genie/troubleshooting

Optionally consult https://docs.databricks.com/llms.txt to discover newly added adjacent Genie pages not in the list above. Only follow links that are clearly about Genie space authoring.

If a fetch fails (restricted egress, etc.), say so explicitly and continue with whatever fetched. Do not silently skip a source.

3. Diff against local

For each practice the docs describe, classify it: already covered / new / changed (docs now say something different from local).

Apply a hard scope filter. KEEP only items that affect Genie space design and configuration, for example:

• Tables/views in the space, descriptions, hard limits, knowledge-store budget.
• Instructions: SQL expressions (measures/filters/fields), example SQL queries, SQL functions, general text instructions, and the instruction priority hierarchy.
• Joins and relationship cardinality; synonyms; format assistance; entity matching.
• Benchmark question design.
• UC data-layer expectations that the evaluator checks (types, PK/FK, naming, descriptions).

DROP (out of scope): end-user querying tips, UI click-throughs, billing/quotas, permissions/governance plumbing, release-note chatter, and anything not actionable as an evaluation rule the agent could check against a space config.

4. Present a prioritized proposal

Present surviving candidates as a single Markdown table, sorted by priority (High first). Use these exact columns:

ID	Prio	Docs	Current local state	Recommendation

• ID: short stable handle (e.g. A, B, C).
• Prio: High / Medium / Low, by how much the change affects space-design accuracy.
• Docs: what the docs say now, in one terse phrase. Name the source page in parentheses (e.g. "(tune-quality)").
• Current local state: what best_practices.md says today: not covered / partial / conflicts. One terse phrase.
• Recommendation: Include -> §<section> or Skip, plus a 3-6 word reason.

Write for a Genie space author practitioner: sentence fragments, no marketing language, no restating the obvious. Keep each cell short enough to scan. Omit source URLs and "new vs changed" prose from the table; if the user wants detail on a row, give it on request.

If nothing clears the scope filter, skip the table: state plainly that the local reference is already aligned with the docs, no changes recommended, and stop here. That is a valid outcome.

Then ask the user which IDs to apply. Do not edit anything before they choose.

5. Apply approved edits

For each approved item:

• Edit best_practices.md additively within the matching section to preserve numbering and house style.
• If a change genuinely requires restructuring sections, update the corresponding section N references in evaluator_system.md in the same pass so they still point at real headings.

6. Org-conventions conflict check

Re-read org_conventions.md against the applied changes. If a newly added universal practice conflicts with a customer convention (rare), surface the conflict to the user and let them decide. Never edit org_conventions.md automatically.

7. Bump the date

Only if at least one edit was applied, update line 3 of best_practices.md to Last updated: <today in YYYY-MM-DD>.

8. Summarize

Report what changed (with section locations), what was skipped and why, and any follow-ups in evaluator_system.md or conflicts flagged in org_conventions.md.

Post-edit checks (only when edits were applied)

• grep -n "Last updated:" genie_validator/prompts/best_practices.md shows the new YYYY-MM-DD.
• Smoke import still works and the date parses (not "unknown"): uv run python -c "import genie_validator.nodes as n; print(n._BEST_PRACTICES_DATE)"
• Every section N reference in evaluator_system.md still maps to a real heading in best_practices.md.