run-solution-analyzer

# Run Solution Analyzer This skill covers running the Solutions Analyzer locally against this clone of Azure-Sentinel. Each script writes CSVs or docs to a directory you choose; the scripts themselves are agnostic to where that directory lives. If your workflow includes publishing the generated CSVs/docs to additional repositories, GitHub Pages, or Kusto, those targets are user-specific and should be recorded in agent memory (workspace or user scope) rather than in this skill. ## Scripts Overview | Script | Purpose | Invocation | | ------------------------------------ | ------------------------------------------------------------------ | ------------------------------------------------- | | `map_solutions_connectors_tables.py` | Mapper: generates all CSV data | Run directly | | `generate_connector_docs.py` | Doc generator: markdown + HTML pages, static + interactive indexes | Run directly | | `generate_asim_browser.py` | ASIM browser: interactive HTML schema browser | Run directly | | `upload_to_kusto.py` | Upload CSVs to Azure Data Explorer (Kusto) cluster | Run directly | | `collect_table_info.py` | Fetch table reference data from Microsoft docs | Auto-invoked by mapper (`--force-refresh=tables`) | | `collect_asim_fields.py` | Fetch ASIM field/schema definitions from Microsoft docs | Auto-invoked by mapper (`--force-refresh=asim`) | | `generate_interactive_docs.py` | Generate `index.html` and HTML entity pages | Auto-invoked by `generate_connector_docs.py` | All scripts run from `Tools/Solutions Analyzer`. ## Default Output Location By default the mapper writes CSVs into `Tools/Solutions Analyzer/` in the current branch. The doc generator and ASIM browser require an explicit `--output-dir` so they never pollute the source tree. ## Sync the Working Branch With `origin/master` **Always do this before any analyzer run.** A working branch that has drifted from master makes the mapper analyze a stale source tree — every connector merged to master after the divergence point can appear as a new "added" connector in subsequent diffs, even though no mapper logic changed. ```powershell git fetch origin master git merge origin/master --no-edit ``` - Use `git merge`, not `git rebase`, so subsequent `git log --first-parent origin/master` analysis (used by `compare_runs.py` and friends) still reflects merge dates correctly. - Resolve any conflicts before running the mapper. - Do this even for incremental/docs-only runs: docs link to GitHub paths that may have moved. After merging, verify the working tree is clean of unintended changes: ```powershell git status --short | Select-Object -First 20 ``` ## Cache Invalidation Model The mapper's analysis cache is keyed by `(analysis_type, file_path)` and stores the source file `mtime`. When a source file is modified (including via `git merge`), its mtime changes and the entry is automatically re-analyzed on the next run. **Source-content changes do NOT require `--force-refresh`.** `--force-refresh` exists for two distinct reasons: 1. **Analyzer logic changed** (mapper code modified) — invalidate cached results for the affected analysis types so they get recomputed from current source. This is the **local-only** case. 2. **Network-fetched data is stale** — re-fetch tables / marketplace / ASIM schema fields from external sources. This is the **network** case and is reserved for explicit full refresh. | Type | Triggers network fetch? | When to refresh | | ------------- | ---------------------------------------------------------- | ------------------------------------------------------ | | `parsers` | No | Mapper parser-analysis logic changed | | `solutions` | No | Mapper solution-analysis logic changed | | `standalone` | No | Mapper standalone-content logic changed | | `asim` | **Yes** — re-runs `collect_asim_fields.py` against MS docs | ASIM analyzer logic changed AND network refresh wanted | | `marketplace` | Yes — Azure Marketplace API | Marketplace data is stale | | `tables` | **Yes** — re-runs `collect_table_info.py` against MS docs | Table reference info is stale | Note: `asim` is hybrid — it invalidates the local ASIM-parser analysis cache **and** re-runs `collect_asim_fields.py`. There is currently no flag to do only the local half. ### After a `git merge` from master Default: just run the mapper with no flags. Mtime-based invalidation re-analyzes every file that changed during the merge: ```powershell python map_solutions_connectors_tables.py ``` Network-fetched CSVs (`tables_reference.csv`, `asim_fields.csv`, `asim_entity_fields.csv`, `asim_logical_types.csv`, `asim_vendors_products.csv`, marketplace cache) will be reused from cache. That's expected — they shouldn't drift just because master moved forward. ## Full Mapper Run (Invalidating All Caches) Use only when explicitly asked to do a full/clean run, when "invalidate all caches" is requested, or when analyzer logic was modified across multiple analysis types. ```powershell cd "Tools/Solutions Analyzer" python map_solutions_connectors_tables.py --force-refresh=all ``` - `--force-refresh=all` clears all 6 cache types (asim, parsers, solutions, standalone, marketplace, tables) AND re-runs `collect_asim_fields.py` and `collect_table_info.py` against the network. - `--force-refresh=all-offline` clears `parsers`, `solutions`, `standalone`, AND `asim` (which still re-runs `collect_asim_fields.py` despite the name; only `marketplace`/`tables` are skipped). - Multiple types: `--force-refresh=parsers,solutions,standalone` (purely local, no network calls). - Run with `isBackground: false` and `timeout: 0`. - Do NOT truncate/filter output (no `Select-Object` piping). **Known issue (as of 2026-04-28):** `collect_asim_fields.py` fails with HTTP 404 because the Microsoft Sentinel docs were migrated out of `MicrosoftDocs/azure-docs`. The mapper logs a warning and proceeds, leaving `asim_fields.csv`, `asim_entity_fields.csv`, `asim_logical_types.csv`, and `asim_vendors_products.csv` empty/stale. Until the script is updated to fetch from the new source, avoid `--force-refresh=all` and `--force-refresh=asim` unless you accept stale ASIM schema CSVs. ## Generate Documentation ```powershell python generate_connector_docs.py --output-dir <DOCS_DIR> [--skip-input-generation] [--html-output-dir <HTML_DIR>] [--html-docs-path <REL_PATH>] [--html-index-url <URL>] ``` - `--output-dir` — **mandatory**, never omit. - `--skip-input-generation` — skip the embedded mapper invocation when CSVs are already fresh. - `--html-output-dir`, `--html-docs-path`, `--html-index-url` — control where `index.html` and the interactive entity pages are written and how they link back to the index. These are typically only set when publishing to a static-site / GitHub Pages location; defaults are sensible for a local preview. - Run with `isBackground: false` and `timeout: 0`. Do NOT truncate output. ## Generate ASIM Browser ```powershell python generate_asim_browser.py --output-dir <DIR> [--docs-base-path <REL_PATH>] [--index-url <URL>] [--link-extension <EXT>] ``` `--docs-base-path` and `--index-url` make sense only when the browser is deployed alongside an interactive docs index. For a standalone local preview, `--output-dir` alone is enough. ## When to Re-Run the Mapper Before Docs Run mapper before generating docs if: - The mapper script itself was modified. - Any override YAML file in the `overrides/` folder was modified. - You specifically need to refresh the CSV data. - You are explicitly asked to run the mapper. Otherwise, just run the doc generator with `--skip-input-generation`. ## Caching and Logging - **Cache:** `Tools/Solutions Analyzer/.cache/` - **Logs:** `Tools/Solutions Analyzer/.logs/` - **Log file:** `Tools/Solutions Analyzer/.logs/map_solutions_connectors_tables.log` ### Watching Mapper Progress While It Runs The mapper writes timestamped progress lines (`[ N.Ns] ...`) to its own log file in real time. To see live status during a run, tail that file directly: ```powershell Get-Content "Tools/Solutions Analyzer/.logs/map_solutions_connectors_tables.log" -Tail 12 ``` **Do NOT rely on `... | Tee-Object -FilePath <log> | Select-Object -Last N` for live progress.** PowerShell pipelines buffer until the upstream command finishes, so the Tee'd file stays empty until the entire run completes (including the post-processing/CSV-write phase, which can be many minutes after the per-parser progress lines were emitted). The mapper's own `.logs/map_solutions_connectors_tables.log` is unbuffered and updates within seconds. ## Upload to Kusto ```powershell python upload_to_kusto.py -c <cluster-url> -d <database> --solution-analyzer --source-dir ./ ``` - `--solution-analyzer` uploads the standard set of CSVs defined in `SOLUTION_ANALYZER_FILES`. - `--source-dir ./` uses local CSVs instead of downloading from GitHub. - Without `--source-dir`, downloads CSVs from the public Azure-Sentinel GitHub repo. - `--dry-run` shows what would be uploaded without making changes. - Requires Azure CLI authentication (`az login`). The cluster URL and database name are environment-specific and not part of this skill. ## User-Specific Publishing Pipelines If you maintain a publishing pipeline (e.g. CSV output worktree, separate docs repository, GitHub Pages site, specific Kusto cluster), record the concrete paths, URLs, and the order of copy/commit/push steps in agent memory — for example, in `/memories/repo/` for a workspace-scoped note or in `/memories/` for a cross-workspace one. The agent should consult that memory to resolve any placeholders above before running.