update-org-data-integrations

# Update Org-Data Integrations ## Overview Updates existing integrations or creates new ones for the `yarn start org-data` command, using upstream field definitions and reference sample events from Elastic repos as the source of truth. ## Before You Start Ask the user to provide: 1. **Integration name** -- which integration to update or create (e.g. `okta`, `crowdstrike`, or a new one) 2. **Source repo and path** -- where the upstream package definitions live. One of: - **Integrations repo**: `/Users/johndoe/repos/integrations/packages/<name>/` - **Beats repo**: `/Users/johndoe/repos/beats/x-pack/filebeat/module/<name>/` (or auditbeat, etc.) - **Cloudbeat repo**: ask user for the path - **Endpoint-package repo**: `/Users/johndoe/repos/endpoint-package/` -- for the Elastic Defend (endpoint) integration. This package lives in its own dedicated repository, not in the integrations repo. If the user points to a specific folder, use that directly. ## Workflow Copy this checklist and track progress: ``` Task Progress: - [ ] Step 1: Read upstream package definitions - [ ] Step 2: Read the existing integration (or base class for new) - [ ] Step 3: Update or create the integration class - [ ] Step 3b: Ensure cross-integration correlation - [ ] Step 4: Register the integration (new integrations only) - [ ] Step 5: Add detection rules - [ ] Step 6: Verify the changes compile - [ ] Step 7: Lint and format the code - [ ] Step 8: Update this skill with session learnings ``` ### Step 1: Read Upstream Package Definitions Depending on the source repo, read these files: **From the integrations repo** (`/Users/johndoe/repos/integrations/packages/<name>/`): - `manifest.yml` -- package name, version, description - `data_stream/*/fields/*.yml` -- ECS and custom field definitions - `data_stream/*/sample_event.json` -- reference sample documents - `data_stream/*/manifest.yml` -- data stream type and dataset name **From the beats repo** (`/Users/johndoe/repos/beats/x-pack/filebeat/module/<name>/`): - `manifest.yml` -- module metadata - `*/config/*.yml` -- dataset configuration - `*/_meta/fields.yml` -- field definitions - `*/_meta/sample_event.json` -- sample events (if available) **From the endpoint-package repo** (`/Users/johndoe/repos/endpoint-package/`): - `package/endpoint/manifest.yml` -- package name (`endpoint`), version, title ("Elastic Defend"), conditions - `package/endpoint/data_stream/*/manifest.yml` -- per-data-stream type (logs or metrics) and dataset name - `package/endpoint/data_stream/*/fields/fields.yml` -- full field definitions (generated from custom schemas/subsets) - `package/endpoint/data_stream/*/sample_event.json` -- reference sample documents (all streams except `collection`) - `schemas/examples/v1/` -- additional realistic example events (e.g. `process_created_windows.json`, `malware_alert.json`, `network_outbound_connection_attempt_windows.json`) - `custom_schemas/` -- custom ECS-extension field definitions (source of truth for non-ECS fields) - `custom_subsets/elastic_endpoint/` -- subset definitions that control which fields each data stream uses The endpoint package has 16 data streams: | Data Stream | Type | Index Pattern | | ---------------- | ------- | --------------------------------------------- | | alerts | logs | `logs-endpoint.alerts-default` | | process | logs | `logs-endpoint.events.process-default` | | file | logs | `logs-endpoint.events.file-default` | | network | logs | `logs-endpoint.events.network-default` | | security | logs | `logs-endpoint.events.security-default` | | registry | logs | `logs-endpoint.events.registry-default` | | library | logs | `logs-endpoint.events.library-default` | | device | logs | `logs-endpoint.events.device-default` | | api | logs | `logs-endpoint.events.api-default` | | actions | logs | `logs-endpoint.actions-default` | | action_responses | logs | `logs-endpoint.action.responses-default` | | heartbeat | logs | `logs-endpoint.heartbeat-default` | | metadata | metrics | `metrics-endpoint.metadata-default` | | metrics | metrics | `metrics-endpoint.metrics-default` | | policy | metrics | `metrics-endpoint.policy-default` | | collection | logs | `logs-endpoint.diagnostic.collection-default` | Note: The endpoint package is typically a prerelease version (e.g. `9.4.0-prerelease.0`), so the integration class should set `prerelease = true`. Extract from the upstream definitions: - All **data stream names** and their index patterns (e.g. `logs-okta.system-default`) - The **field mappings** -- which ECS fields and custom fields each data stream uses - The **sample event structure** -- realistic document shapes with reference field values - The **Fleet package name** -- exact name used for `installPackage` ### Step 2: Read the Existing Integration **For updates**, read the existing integration file: ``` src/commands/org_data/integrations/<name>_integration.ts ``` Identify what needs to change: field names, data stream configs, document structure, etc. **For new integrations**, read the base class and one existing integration as a reference: - `src/commands/org_data/integrations/base_integration.ts` -- abstract base class - Pick a similar existing integration as a template (e.g. `slack_integration.ts` for audit log integrations, `entra_id_integration.ts` for entity analytics integrations) ### Step 3: Update or Create the Integration Class Every integration class must: 1. **Extend `BaseIntegration`** from `./base_integration` 2. **Set `packageName`** -- the Fleet package name (from `manifest.yml`) 3. **Set `displayName`** -- human-readable name for CLI output 4. **Set `dataStreams`** -- array of `DataStreamConfig` objects: ```typescript readonly dataStreams: DataStreamConfig[] = [ { name: 'Data Stream Name', index: 'logs-<package>.<dataset>-default' }, ]; ``` 5. **Implement `generateDocuments(org, correlationMap)`** -- returns `Map<string, IntegrationDocument[]>` keyed by index name 6. **Set `prerelease = true`** if the package requires `?prerelease=true` for Fleet API Key patterns for `generateDocuments`: - Use `org.employees` to iterate over organization members - Use `correlationMap` to link identities across integrations (e.g. `correlationMap.oktaUserIdToEmployee`) - Use `this.getRandomTimestamp()` for realistic time distribution - Each document must include `'@timestamp'` and the correct `data_stream` - **Produce raw/pre-pipeline documents** -- see Step 3c below for the full rules - **Use stable Employee/Device fields for correlated identifiers** -- see Step 3b below for the full rules on `user.id`, `host.mac`, `host.ip`, etc. **Endpoint-specific guidance:** - The `packageName` is `'endpoint'` and `displayName` should be `'Elastic Defend'` - The endpoint package has many data streams; select the most relevant ones for the organization use case (typically: `process`, `file`, `network`, `security`, `alerts`, and optionally `registry`, `library`, `api`) - Documents should include `agent.type: 'endpoint'` and `event.module: 'endpoint'` - The existing `privileged_user_monitoring` command already generates endpoint process events and can serve as a reference for document structure (see `src/commands/privileged_user_monitoring/sample_documents.ts`) ### Step 3b: Ensure Cross-Integration Correlation Every integration must use **stable, pre-generated identity fields** from the `Employee` and `Device` objects so that the same user or device can be correlated across multiple integrations in Elastic Security. Never generate random identifiers (SIDs, UIDs, MACs, IPs) per event with `faker`. #### Stable fields on Employee These fields are generated once in `org_data_generator.ts` and must be reused by every integration that references users: | Field | Description | Example | | ----------------- | ----------------------------------------------------- | ------------------------------------------------ | | `userName` | Primary user identifier across all integrations | `reese.doyle` | | `email` | Corporate email address | `reese.doyle@acmecrm.com` | | `windowsSid` | Windows SID (shared domain prefix + per-employee RID) | `S-1-5-21-1199716861-1301547593-1626275133-1001` | | `unixUid` | Unix UID (integer, starts at 1000) | `1000` | | `oktaUserId` | Okta user ID | `00uAbCdEfGhIjKlMn` | | `entraIdUserId` | Microsoft Entra ID (Azure AD) user UUID | `a1b2c3d4-...` | | `duoUserId` | Cisco Duo user ID | `DUABCDEFGHIJKLMNOPQR` | | `onePasswordUuid` | 1Password member UUID | `a1b2c3d4-...` | | `employeeNumber` | HR employee number | `123456` | | `githubUsername` | GitHub username (Engineering + Executive only) | `reese-doyle` | | `gitlabUserId` | GitLab numeric user ID (Engineering + Executive only) | `42518` | #### Stable fields on Device | Field | Description | Example | | --------------------- | ---------------------------------------------------------------------------------------------------------- | ----------------------------- | | `id` | Device UUID, used as `host.id` | `276e59a0-...` | | `macAddress` | Stable MAC address (dash-separated) | `8a-d3-02-ed-99-a2` | | `ipAddress` | Stable IPv4 address | `234.22.230.186` | | `crowdstrikeAgentId` | CrowdStrike Falcon agent ID | `e045e02b...` | | `crowdstrikeDeviceId` | CrowdStrike device ID | `efb573dc...` | | `defenderDeviceId` | Microsoft Defender for Endpoint machine id (`host.id` after MDE ingest; distinct from Elastic `device.id`) | `c114cb1c0b...` (40-char hex) | | `serialNumber` | Hardware serial number | `A1B2C3D4E5F6` | | `elasticAgentId` | Elastic Agent UUID for local workstation agent | `c3f1a9d2-...` | #### Stable fields on Host | Field | Description | Example | | ---------------- | -------------------------------------- | ------------------------ | | `id` | Host UUID | `a2b3c4d5-...` | | `name` | Server hostname | `api-server-prod-a1b2c3` | | `elasticAgentId` | Elastic Agent UUID for the server host | `d4e5f6a7-...` | #### Stable fields on Service `org.services` is a catalog of platform services, SaaS apps, the org's identity provider, and (for medium / enterprise sizes) org-owned microservices. It's the source of truth for ECS `service.entity.id`. Generate services from `src/commands/org_data/data/services.ts` -- never invent service IDs in an integration. | Field | Description | Example |