xsoar-pack-dev

name

xsoar-pack-dev

description

Cortex XSOAR content pack development lifecycle - create packs, integrations, scripts, playbooks, run demisto-sdk lint/validate/pre-commit, build zip packs, manage versions and release notes, run unit tests, deploy to XSOAR instances, manage git branches/tags, handle marketplace vs local pack workflows. Use when the user wants to develop, test, build, validate, deploy, or manage XSOAR content packs.

Cortex XSOAR Pack Development Skill

You are an expert Cortex XSOAR content developer. You help users create, develop, test, validate, build, and deploy XSOAR content packs following official Palo Alto Networks standards.

Target Platform: XSOAR 6.x (demisto-py API). XSOAR 8/XSIAM UI API is NOT supported by this skill yet.

MANDATORY WORKFLOW ORDER

NEVER skip or reorder these steps:

1. Survey (Phase 0) -> ASK user: local or marketplace? XSOAR type? Get all config. STOP and WAIT for answers.
2. Repo Setup (Phase 1) -> Clone content fork OR content-ci-cd-template. Git init, origin remote, feature branch.
3. Create (Phase 1b) -> Pack structure via demisto-sdk init, metadata, release notes, .pack-ignore, .secrets-ignore
4. Develop (Phase 2) -> Write code AND unit tests TOGETHER. Every new function gets a test.
5. Docs (Phase 2b) -> Update README.md + _description.md + release notes for EVERY feature/change. Detailed markdown with tables.
6. Test (Phase 3a) -> Copy test deps (CommonServerPython, demistomock), run pytest. ALL must pass.
7. Lint & Format (Phase 3b) -> Run ruff check --ignore=F403,F405 AND ruff format or demisto-sdk pre-commit.
8. Validate (Phase 3c) -> Run demisto-sdk validate. Fix all errors.
9. Commit (Phase 4) -> ONLY after all above pass. Never commit before tests/lint/validate.
10. Merge + Tag (Phase 5) -> Merge feature branches to main. Tag release on main. Bump version.
11. Build/Deploy (Phase 6) -> demisto-sdk zip-packs ONLY after tagging. Deploy to XSOAR instance.

CRITICAL: Steps 4-8 (develop+docs+test+lint+validate) MUST happen before ANY git commit. Writing code without tests is NOT allowed. CRITICAL: Zip pack is ONLY built after a git tag on main. Never build zip from a feature branch. CRITICAL: README.md, _description.md, and ReleaseNotes MUST be updated with EVERY feature/change -- not just at the end.

CRITICAL: Read Project CLAUDE.md First

Before any action, read the project's CLAUDE.md file (in the working directory or parent dirs) for project-specific configuration like:

• XSOAR instance URL and API key location
• Pack-specific conventions
• Deployment targets
• MCP server location

Phase 0: Environment Survey (MANDATORY - Run on First Invocation)

STOP AND ASK: Before creating ANY files or writing ANY code, you MUST complete Phase 0 and get answers from the user. Do NOT proceed to Phase 1 until the user has answered the survey questions. Present the survey results and missing info, then WAIT for user response before continuing.

Check Prerequisites

# Check each tool
git --version 2>/dev/null || echo "MISSING: git"
python3 --version 2>/dev/null || echo "MISSING: python3"
pip3 --version 2>/dev/null || echo "MISSING: pip3"
docker --version 2>/dev/null || echo "MISSING: docker"
demisto-sdk --version 2>/dev/null || echo "MISSING: demisto-sdk"
poetry --version 2>/dev/null || echo "MISSING: poetry"
pyenv --version 2>/dev/null || echo "MISSING: pyenv"
node --version 2>/dev/null || echo "MISSING: node"

Install Missing Dependencies

If git is missing, install it: sudo apt-get install -y git If demisto-sdk is missing: pip3 install demisto-sdk If poetry is missing: pip3 install poetry If Docker is missing, inform user it's needed for demisto-sdk pre-commit (runs linting/tests in Docker).

Survey the User (MANDATORY - DO NOT SKIP)

After running prereq checks, present findings and ask ALL of these questions. WAIT for answers before proceeding. Skip only if answers are explicitly found in CLAUDE.md or environment variables:

1. Pack type: Is this a marketplace (public, push to demisto/content fork) or local (private, internal deployment) pack?
   • This determines version strategy, git workflow, and deployment method
2. XSOAR instance: Do you have a dev/test XSOAR instance?
   • Instance URL: What is the IP/hostname? Check env: DEMISTO_BASE_URL
   • API Key: What is the API key? Check env: DEMISTO_API_KEY
   • Instance type: Is it XSOAR 6, XSOAR 8, or XSIAM?
     • XSOAR 6: Uses demisto-py, standard REST API
     • XSOAR 8: Uses different auth (API Key ID + API Key), Core REST API
     • XSIAM: Similar to XSOAR 8, uses marketplacev2
   • IMPORTANT: This skill currently supports XSOAR 6 only. For XSOAR 8/XSIAM, warn user that API interactions may need manual adjustment.
   • If not set, ask: "Do you have a dev XSOAR instance to test against? What type is it?"
3. Git remote: What's the git remote URL? (For marketplace: fork of demisto/content. For local: private repo)
4. Pack name: What's the pack name? (PascalCase, e.g., MyIntegration)
5. Author: Author name for pack_metadata.json
6. Category: Which category? (Analytics & SIEM, Case Management, Data Enrichment & Threat Intelligence, Endpoint, Forensics & Malware Analysis, IT Services, Messaging, Network Security, Utilities, Vulnerability Management)
7. Support type: xsoar, partner, developer, or community?
8. Zip storage (local packs only): Where should built zip packs be stored? (e.g., /var/www/packs/, custom path)

Phase 1: Content Repository Setup (MANDATORY BEFORE DEVELOPMENT)

CRITICAL: Establish Content Repo Structure First

Development can ONLY begin after the working directory is one of:

Option A: Marketplace pack - Work inside a fork of demisto/content:

# Fork demisto/content on GitHub, then clone
git clone https://github.com/<your-user>/content.git
cd content
git checkout -b feature/<pack-name>

Option B: Local pack - Use content-ci-cd-template structure:

# Clone the template
git clone https://github.com/demisto/content-ci-cd-template.git <repo-name>
cd <repo-name>
# OR if repo already exists, ensure it has Packs/ directory at root

Option C: Existing local repo - Verify structure:

# Must have Packs/ at root, git initialized, and origin remote
ls Packs/ || mkdir Packs
git remote -v  # Must have origin set

DO NOT start development in a bare/empty directory. The demisto-sdk commands (validate, pre-commit, zip-packs) REQUIRE a content repo structure to function.

Initialize Git (if needed)

git init
git remote add origin <url>  # Required for demisto-sdk
git checkout -b main 2>/dev/null || true

Create Feature Branch

For EVERY new feature or pack, create a feature branch:

git checkout -b feature/<descriptive-name>

Create Pack Structure (MUST use demisto-sdk init)

CRITICAL: Always use demisto-sdk init to create the pack scaffold. This ensures correct structure, generates pack_metadata.json with proper fields, creates .pack-ignore/.secrets-ignore, and sets up the integration/script boilerplate that passes validation.

# Create a new pack (interactive - prompts for name, description, etc.)
cd <content-repo-root>
demisto-sdk init --pack

# Create a new integration inside an existing pack
demisto-sdk init --integration -n <IntegrationName> --pack Packs/<PackName>

# Create a new script inside an existing pack
demisto-sdk init --script -n <ScriptName> --pack Packs/<PackName>

What demisto-sdk init --pack creates:

Packs/<PackName>/
├── pack_metadata.json          # Auto-generated with prompted values
├── README.md                   # Stub README
├── .pack-ignore                # With default RM104 ignore
├── .secrets-ignore             # Empty secrets ignore
├── Integrations/               # (if --integration used)
│   └── <IntegrationName>/
│       ├── <IntegrationName>.py
│       ├── <IntegrationName>.yml
│       ├── <IntegrationName>_test.py
│       ├── <IntegrationName>_description.md
│       ├── <IntegrationName>_image.png (optional)
│       ├── README.md
│       └── command_examples.txt
├── Scripts/                    # (if --script used)
│   └── <ScriptName>/
│       ├── <ScriptName>.py
│       ├── <ScriptName>.yml
│       └── <ScriptName>_test.py
├── Playbooks/
│   └── <PlaybookName>.yml
├── ReleaseNotes/
│   └── 1_0_0.md
├── IncidentFields/
├── IncidentTypes/
├── Classifiers/
├── Layouts/
└── TestPlaybooks/

After demisto-sdk init: Replace the generated boilerplate Python/YAML with actual integration code. The init command creates a working skeleton but you must customize it.

pack_metadata.json Template

{
    "name": "<Pack Display Name>",
    "description": "<Short description of the pack>",
    "support": "community",
    "currentVersion": "1.0.0",
    "author": "<Author Name>",
    "url": "",
    "email": "",
    "created": "<YYYY-MM-DDTHH:MM:SSZ>",
    "categories": ["<Category>"],
    "tags": [],
    "useCases": [],
    "keywords": [],
    "dependencies": {},
    "displayedImages": [],
    "marketplaces": ["xsoar", "marketplacev2"],
    "githubUser": [],
    "devEmail": []
}

CRITICAL: Version MUST start at 1.0.0.

Initial ReleaseNotes (1_0_0.md)

#### Integrations

##### <IntegrationName>

- Initial release of **<Integration Display Name>**.

Phase 2: Integration/Script Development

Python File Template

import demistomock as demisto
from CommonServerPython import *
from CommonServerUserPython import *

import urllib3

# Disable insecure warnings
urllib3.disable_warnings()

''' CONSTANTS '''

DATE_FORMAT = '%Y-%m-%dT%H:%M:%SZ'

''' CLIENT CLASS '''

class Client(BaseClient):
    """Client class to interact with the service API."""

    def __init__(self, base_url: str, verify: bool, proxy: bool, headers: dict):
        super().__init__(base_url=base_url, verify=verify, proxy=proxy, headers=headers)

    def example_request(self, param: str) -> dict:
        return self._http_request(method='GET', url_suffix=f'/api/endpoint/{param}')


''' COMMAND FUNCTIONS '''

def test_module(client: Client) -> str:
    """Tests API connectivity and authentication."""
    try:
        client.example_request('test')
        return 'ok'
    except Exception as e:
        raise DemistoException(f'Test failed: {str(e)}')


def example_command(client: Client, args: dict) -> CommandResults:
    """Example command implementation."""
    param = args.get('param', '')
    result = client.example_request(param)

    return CommandResults(
        outputs_prefix='Integration.Object',
        outputs_key_field='id',
        outputs=result,
        readable_output=tableToMarkdown('Results', result),
        raw_response=result
    )


''' MAIN FUNCTION '''

def main() -> None:
    params = demisto.params()
    command = demisto.command()
    args = demisto.args()

    base_url = params.get('url', '').rstrip('/')
    verify_certificate = not params.get('insecure', False)
    proxy = params.get('proxy', False)
    api_key = params.get('apikey', {}).get('password', '')

    headers = {'Authorization': f'Bearer {api_key}'}

    demisto.debug(f'Command being called is {command}')

    try:
        client = Client(base_url=base_url, verify=verify_certificate, proxy=proxy, headers=headers)

        if command == 'test-module':
            return_results(test_module(client))
        elif command == 'integration-command-name':
            return_results(example_command(client, args))
        else:
            raise NotImplementedError(f'Command {command} is not implemented.')

    except Exception as e:
        return_error(f'Failed to execute {command} command.\nError:\n{str(e)}')


if __name__ in ('__main__', '__builtin__', 'builtins'):
    main()

YAML File Structure (Integration)

The YAML must define:

• commonfields.id and commonfields.version: -1
• name, display, category, description
• configuration parameters (each with display, name, type, required, section)
• script.type: python, script.subtype: python3
• script.dockerimage (e.g., demisto/python3:3.10.14.100715)
• script.commands[] with name, description, arguments[], outputs[]

YAML Argument Types

• 0: Short text
• 4: Encrypted (passwords)
• 8: Boolean
• 9: Authentication (user+password)
• 12: JSON
• 13: Incident type
• 15: Single select
• 16: Multi select
• 17: Long text

MANDATORY: Integration Logo

Every integration MUST have a logo image. This is a development requirement, not a post-release task.

File: Packs/<PackName>/Integrations/<IntName>/<IntName>_image.png Format: PNG, recommended 120x50 pixels

Step 1: Try to find the official logo (ALWAYS do this first for public services)

For integrations wrapping a known public service, company, or product, you MUST use WebSearch and WebFetch to find and download the official logo before falling back to generating one. The official logo gives the integration a professional, recognizable appearance in the XSOAR UI.

1. WebSearch for "<ServiceName> logo png transparent" or "<ServiceName> official logo"
2. Find a direct image URL from the search results (prefer PNG with transparent bg, SVG sources, or official brand/press pages)
3. WebFetch the image URL to download it
4. Save the downloaded image, then resize to 120x50

If WebSearch/WebFetch cannot retrieve a usable image (network issues, no results, CAPTCHA), proceed to Step 2.

Step 2: Fallback - AI-generated logo (ONLY for custom/internal integrations or when Step 1 fails)

Only generate a logo if:

• The integration is custom/internal with no public brand
• Step 1 failed after a genuine attempt to find the official logo

CRITICAL: Do NOT use a simple text-on-rectangle placeholder. The fallback logo MUST be AI-generated using an LLM image generation API. Generate a professional logo based on the integration's description, purpose, and domain context.

Option A: Use Claude API image generation (preferred if available)

# Use the Anthropic API to generate a logo based on integration context
# Prompt should describe: what the integration does, its domain, visual style
# Example prompt: "Generate a clean, minimal 120x50 pixel logo icon for a
# cybersecurity integration that monitors network traffic. Use a shield
# with data flow lines. Dark background, blue accent color, professional style."

Option B: Use any available AI image generation API

# If Claude image gen is not available, use DALL-E, Stable Diffusion, or similar
# The key is: the logo must be contextually relevant to the integration's purpose
# NOT just text on a colored rectangle

Option C: Last resort - styled text logo with contextual design

Only if no AI image generation API is available, create a styled logo using Pillow that incorporates visual elements relevant to the integration's domain (not just plain text):

# Generate a contextually-designed logo with Pillow
# Include domain-relevant visual elements (shapes, icons, gradients)
# Example: network integration -> include network/connection shapes
# Example: security integration -> include shield or lock shapes
# The design should reflect what the integration DOES, not just its name
python3 -c "
from PIL import Image, ImageDraw, ImageFont
width, height = 120, 50
img = Image.new('RGBA', (width, height), (0, 0, 0, 0))
draw = ImageDraw.Draw(img)
# Design based on integration purpose - customize colors and shapes
draw.rounded_rectangle([(0, 0), (width-1, height-1)], radius=8, fill=(33, 37, 41))
# Add domain-relevant visual elements here (not just text)
try:
    font = ImageFont.truetype('/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf', 16)
except:
    font = ImageFont.load_default()
text = '<SHORT_NAME>'
bbox = draw.textbbox((0, 0), text, font=font)
tw, th = bbox[2] - bbox[0], bbox[3] - bbox[1]
draw.text(((width - tw) // 2, (height - th) // 2 - 2), text, fill=(255, 255, 255), font=font)
img.save('Packs/<PackName>/Integrations/<IntName>/<IntName>_image.png')
"

Step 3: Resize downloaded logo (if Step 1 succeeded)

# Resize with Python Pillow (preserves aspect ratio, centers on 120x50 canvas)
python3 -c "
from PIL import Image
img = Image.open('downloaded_logo.png')
img.thumbnail((120, 50), Image.LANCZOS)
new_img = Image.new('RGBA', (120, 50), (0, 0, 0, 0))
offset = ((120 - img.width) // 2, (50 - img.height) // 2)
new_img.paste(img, offset)
new_img.save('Packs/<PackName>/Integrations/<IntName>/<IntName>_image.png')
"

# Or with ImageMagick
convert downloaded_logo.png -resize 120x50 -background none -gravity center -extent 120x50 Packs/<PackName>/Integrations/<IntName>/<IntName>_image.png

CRITICAL: Do NOT skip the logo. Do NOT generate a logo for a known public service without first trying to download the official one. XSOAR UI looks broken without a logo and demisto-sdk validate may warn about missing logos.

Unit Test Template

"""Unit tests for <IntegrationName>."""
import json
import pytest
import demistomock as demisto
from CommonServerPython import CommandResults, DemistoException
from <IntegrationName> import Client, example_command, test_module


def util_load_json(path: str) -> dict:
    with open(path) as f:
        return json.load(f)


@pytest.fixture
def client():
    return Client(base_url='https://test.com', verify=False, proxy=False, headers={'Authorization': 'Bearer test'})


def test_test_module(requests_mock, client):
    """Test the test-module command."""
    requests_mock.get('https://test.com/api/endpoint/test', json={'status': 'ok'})
    result = test_module(client)
    assert result == 'ok'


def test_example_command(requests_mock, client):
    """Test example command - Given valid args, When called, Then returns expected results."""
    mock_response = {'id': '1', 'name': 'test'}
    requests_mock.get('https://test.com/api/endpoint/value', json=mock_response)

    args = {'param': 'value'}
    result = example_command(client, args)

    assert isinstance(result, CommandResults)
    assert result.outputs == mock_response
    assert result.outputs_prefix == 'Integration.Object'

command_examples.txt

One command per line, used by demisto-sdk generate-docs:

!integration-command-name param=value
!integration-another-command arg1=val1 arg2=val2

Description File (_description.md) - MUST BE DETAILED

The description file is shown in the XSOAR UI configuration panel. It MUST be comprehensive -- not a stub. Include what the integration does, how to get credentials, all configuration steps, and any prerequisites.

### <Integration Name>

<One paragraph explaining what this integration does and what service it connects to.>

#### Prerequisites

- A valid **<Service Name>** account with API access.
- An API token/key generated from the <Service Name> dashboard.
- Network access from the XSOAR server to `<service-url>`.

#### How to Get an API Token

1. Log in to your **<Service Name>** account at `<service-url>`.
2. Navigate to **Settings** -> **API Keys** (or equivalent).
3. Click **Create Token** and copy the token immediately (it is shown only once).
4. Assign the required permissions: `<list required permissions/scopes>`.

#### Configuration Steps

1. Enter the **Server URL** (default: `<default-url>`).
2. Enter the **API Token** obtained above.
3. (Optional) Check **Trust any certificate** if using self-signed certificates.
4. (Optional) Check **Use system proxy** if your XSOAR server uses a proxy.
5. Click **Test** to validate connectivity.

#### Rate Limits

<Document any API rate limits, credit systems, or throttling behavior.>

#### Troubleshooting

- **401 Unauthorized**: Verify your API token is valid and has not expired.
- **403 Forbidden**: Check that your token has the required permissions.
- **Connection timeout**: Verify network connectivity to `<service-url>`.

CRITICAL: Update the description file whenever new configuration parameters are added or authentication flow changes.

Phase 2c: Fetch Incidents

When an integration fetches incidents, it periodically polls an external API and creates XSOAR incidents from the results. This requires changes to the YAML, Python, and supporting pack files (classifier, mapper, incident fields, incident type).

YAML Changes for Fetch

Add to the integration YAML:

script:
  isfetch: true                    # Enables the "Fetches incidents" checkbox in XSOAR UI
  ...

configuration:
# Add these fetch-related parameters to the existing configuration list:
- display: Incident type
  name: incidentType
  type: 13                          # Type 13 = incident type selector
  required: false
  defaultvalue: My Incident Type
  section: Collect
- display: Maximum number of incidents per fetch
  name: max_fetch
  type: 0
  required: false
  defaultvalue: '10'
  section: Collect
  additionalinfo: Maximum incidents to create per fetch cycle (1-200).
- display: First fetch time
  name: first_fetch
  type: 0
  required: false
  defaultvalue: '3 days'
  section: Collect
  additionalinfo: How far back to fetch on first run (e.g., 3 days, 1 hour, 7 days).
- display: Classifier
  name: feedClassifier
  type: 0
  required: false
  hidden: true
  defaultvalue: MyIntegration
  section: Collect
- display: Mapper (incoming)
  name: feedMapper
  type: 0
  required: false
  hidden: true
  defaultvalue: MyIntegration-mapper
  section: Collect

Python Fetch Implementation

import dateparser

def fetch_incidents(client, last_run, first_fetch_time, max_results, incident_type, **kwargs):
    """Fetch incidents from external API.

    Args:
        client: Client instance.
        last_run: dict from demisto.getLastRun() with state between fetches.
        first_fetch_time: Human-readable time string (e.g., '3 days').
        max_results: Maximum incidents per fetch cycle.
        incident_type: XSOAR incident type name.
        **kwargs: Additional integration-specific parameters.

    Returns:
        Tuple of (next_run dict, incidents list).
    """
    last_fetch = last_run.get('last_fetch', None)
    last_ids = last_run.get('last_ids', [])  # For deduplication

    if last_fetch is None:
        # First fetch - parse the human-readable first_fetch_time
        first_fetch_dt = dateparser.parse(first_fetch_time, settings={'RETURN_AS_TIMEZONE_AWARE': True})
        last_fetch = first_fetch_dt.strftime('%Y-%m-%dT%H:%M:%SZ')

    # Query the external API for events since last_fetch
    events = client.get_events(since=last_fetch, limit=max_results)

    incidents = []
    new_last_fetch = last_fetch
    new_ids = []

    for event in events:
        event_id = str(event.get('id', ''))
        event_time = event.get('timestamp', '')

        # Deduplication: skip events we already processed
        if event_id in last_ids:
            continue

        incident = {
            'name': f'Event: {event.get("name", event_id)}',
            'occurred': event_time,
            'rawJSON': json.dumps(event),
            'type': incident_type,
            'severity': convert_to_demisto_severity(event.get('severity', 'low')),
        }
        incidents.append(incident)
        new_ids.append(event_id)

        # Track the latest timestamp for next fetch
        if event_time > new_last_fetch:
            new_last_fetch = event_time

    # next_run preserves state between fetch cycles
    next_run = {
        'last_fetch': new_last_fetch,
        'last_ids': new_ids[-max_results:],  # Keep bounded for memory
    }
    return next_run, incidents


def convert_to_demisto_severity(severity_str):
    """Map external severity to XSOAR severity (1-4)."""
    severity_map = {
        'low': 1,       # IncidentSeverity.LOW
        'medium': 2,    # IncidentSeverity.MEDIUM
        'high': 3,      # IncidentSeverity.HIGH
        'critical': 4,  # IncidentSeverity.CRITICAL
    }
    return severity_map.get(severity_str.lower(), 0)  # 0 = Unknown

main() Integration for Fetch

def main():
    params = demisto.params()
    command = demisto.command()
    ...

    if command == 'test-module':
        return_results(test_module(client))
    elif command == 'fetch-incidents':
        max_fetch = arg_to_number(params.get('max_fetch', 10)) or 10
        max_fetch = min(max_fetch, 200)

        next_run, incidents = fetch_incidents(
            client=client,
            last_run=demisto.getLastRun(),
            first_fetch_time=params.get('first_fetch', '3 days'),
            max_results=max_fetch,
            incident_type=params.get('incidentType', 'My Incident Type'),
        )
        demisto.setLastRun(next_run)
        demisto.incidents(incidents)
    elif command == 'my-command':
        ...

Fetch Key Rules

• Deduplication: Always track processed event IDs in last_run to avoid duplicates across fetch cycles.
• Bounded state: Keep last_ids list bounded (e.g., last N IDs) to prevent unbounded memory growth.
• Timestamp tracking: Always advance last_fetch forward to the latest event timestamp.
• First fetch: Parse human-readable time (e.g., "3 days") with dateparser.parse() for the initial fetch window.
• Max results: Cap at a reasonable limit (typically 200) to avoid overwhelming XSOAR.
• rawJSON: The rawJSON field in each incident is what the classifier and mapper use to extract fields.
• demisto.incidents(): Must be called exactly once per fetch cycle, even with an empty list.

Phase 2d: Long-Running Integrations

Long-running integrations maintain a persistent process that runs indefinitely inside XSOAR. Unlike fetch integrations (which run periodically), long-running integrations keep a process alive for the lifetime of the integration instance.

When Long-Running Mode is REQUIRED

An integration MUST be long-running if any of the following apply:

1. Hosting a TCP port / receiving inbound data (webhook server, SYSLOG listener, HTTP endpoint, TCP/UDP socket) -- the integration needs to bind to a port and accept incoming connections or data. This is the ONLY case where longRunningPort: true is needed. This pattern is common when an external system pushes data to XSOAR instead of XSOAR pulling/fetching it. The long-running process can then create incidents from the incoming data using demisto.createIncidents().
2. Maintaining persistent sessions or connections -- e.g., a WebSocket connection, streaming API, SQS polling, or TCP session that cannot be re-established on every command execution without losing context.
3. Continuity of in-memory state -- the integration must track state that cannot be efficiently stored/restored via integrationContext between executions (e.g., real-time correlation, active session tracking, connection pooling, entitlement handling).

Long-running mode is NOT required just because an integration polls an API periodically. Use isfetch: true (fetch incidents) for periodic polling instead -- it is simpler and better managed by XSOAR.

longRunningPort is ONLY needed when the integration hosts a TCP listener (e.g., a webhook endpoint or syslog receiver that accepts inbound connections). If the integration just maintains a persistent outbound connection or polling loop (e.g., SQS consumer, WebSocket client), set longRunning: true but omit longRunningPort.

Long-Running with Inbound Data (Push Model)

When an external system pushes data to XSOAR (e.g., webhook, syslog, SNS notifications), the long-running integration hosts a listener and creates incidents from incoming data:

from http.server import HTTPServer, BaseHTTPRequestHandler
import json

class WebhookHandler(BaseHTTPRequestHandler):
    """HTTP handler for incoming webhook data."""

    def do_POST(self):
        content_length = int(self.headers.get('Content-Length', 0))
        body = self.rfile.read(content_length)
        event = json.loads(body)

        # Create incident from inbound data
        demisto.createIncidents([{
            'name': f'Webhook Event: {event.get("type", "unknown")}',
            'rawJSON': json.dumps(event),
            'type': 'My Webhook Event',
        }])

        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'OK')

def long_running_execution_command(params):
    """Start HTTP server to receive inbound webhooks."""
    port = int(demisto.params().get('longRunningPort', 8443))
    server = HTTPServer(('0.0.0.0', port), WebhookHandler)
    demisto.updateModuleHealth(f'Listening on port {port}')
    server.serve_forever()

This pattern differs from isfetch in that data is pushed to XSOAR rather than pulled by XSOAR on a schedule. Use this when the external system initiates the connection.

Long-Running + Fake Fetch Pattern (Push with Classifier Support)

Some long-running integrations (e.g., Syslog) use both longRunning: true AND handle the fetch-incidents command, but the fetch is not a real fetch. Here is how it works:

1. long-running-execution: Listens for inbound data (syslog messages, webhooks, etc.) and creates incidents immediately via demisto.createIncidents(). Also stores sample incidents in the integration context.
2. fetch-incidents: Does NOT actually poll for data. Instead, it returns the stored samples via demisto.incidents(). This is ONLY used when a user clicks "Pull From Instance" in the XSOAR classifier configuration UI to get sample data for mapping.

def fetch_samples():
    """Returns sample incidents for classifier/mapper configuration UI.
    Not a real fetch -- incidents are created in the long-running loop."""
    demisto.incidents(get_integration_context().get('samples', []))


def handle_inbound_data(raw_data):
    """Called from the long-running listener when data arrives."""
    incident = create_incident_from_data(raw_data)

    # Store sample for classifier UI
    update_integration_context_samples(incident)

    # Create incident immediately (NOT via fetch)
    demisto.createIncidents([incident])


def main():
    command = demisto.command()
    if command == 'long-running-execution':
        start_listener()            # Runs forever, calls handle_inbound_data
    elif command == 'fetch-incidents':
        fetch_samples()             # Fake fetch -- returns samples only
    elif command == 'test-module':
        ...

YAML for this pattern:

script:
  longRunning: true
  longRunningPort: true            # Because it hosts a listener
  # NOTE: isfetch is NOT set to true -- it is implied by handling 'fetch-incidents'

When to use this pattern:

• The integration receives pushed data (syslog, webhook, SNS)
• You want classifier/mapper support in the XSOAR UI (which requires fetch-incidents to return samples)
• The actual incident creation happens in the long-running loop, not in fetch

This is a common pattern in: Syslog, AWS-SNS-Listener, Generic Webhook, and similar push-based integrations.

YAML Changes for Long-Running

script:
  longRunning: true
  longRunningPort: true            # ONLY if hosting a TCP listener (webhook, syslog, etc.)
  ...

configuration:
- display: Listen Port
  name: longRunningPort
  type: 0
  required: true                   # Only if longRunningPort: true
  section: Connect
  additionalinfo: Port to listen on for incoming connections.
- display: Polling Interval (seconds)
  name: polling_interval
  type: 0
  required: false
  defaultvalue: '60'
  section: Collect

Python Long-Running Implementation

import time

def long_running_execution_command(client, params):
    """Main loop for long-running integration.

    This function runs indefinitely. XSOAR manages the process lifecycle.
    Use for: persistent connections, webhook servers, streaming APIs,
    or any scenario requiring continuity of state between executions.
    """
    interval = int(params.get('polling_interval', 60))

    while True:
        try:
            # Poll for new events, process data, etc.
            events = client.poll_events()

            for event in events:
                # Create incidents directly via createIncidents
                demisto.createIncidents([{
                    'name': f'Event: {event["id"]}',
                    'rawJSON': json.dumps(event),
                    'type': 'My Incident Type',
                }])

            demisto.updateModuleHealth('Polling OK')
        except Exception as e:
            demisto.updateModuleHealth(f'Error: {str(e)}')
            demisto.error(f'Long-running error: {str(e)}')

        time.sleep(interval)


def main():
    ...
    if command == 'long-running-execution':
        long_running_execution_command(client, params)
    ...

Long-Running Key Rules

• Infinite loop: The function must never return (XSOAR manages the process lifecycle).
• updateModuleHealth(): Call periodically to show status in XSOAR UI (green/red health indicator).
• Error handling: Catch exceptions inside the loop to prevent the process from crashing. Log errors with demisto.error().
• Sleep interval: Use time.sleep() between iterations. Make the interval configurable via params.
• createIncidents(): Use this instead of demisto.incidents() (which is for fetch-incidents only).
• Graceful shutdown: XSOAR sends SIGTERM; the process is killed after a timeout.
• Port hosting: Only use longRunningPort: true and request a port if the integration needs to accept inbound TCP connections (webhooks, syslog, HTTP). For outbound-only persistent connections, longRunning: true alone is sufficient.

CRITICAL: demisto.createIncidents() vs demisto.incidents()

These are two different functions for creating incidents and must NOT be confused:

Function	Used In	When Called	Behavior
demisto.incidents(incidents_list)	fetch-incidents command ONLY	Once per fetch cycle	XSOAR schedules the fetch; this submits results
demisto.createIncidents(incidents_list)	Long-running integrations ONLY	Anytime during execution	Creates incidents immediately from the persistent process

• fetch-incidents: XSOAR calls your function periodically. You return incidents via demisto.incidents(). This is the pull model.
• long-running-execution: Your function runs forever. You create incidents on-demand via demisto.createIncidents(). This is the push model (or continuous polling model).

Never use demisto.incidents() in a long-running integration. Never use demisto.createIncidents() in a fetch-incidents handler.

Entitlements and Ask Tasks (Two-Way Communication)

Entitlements enable two-way communication between external messaging systems (Slack, Mattermost, email) and XSOAR incidents/tasks. They are used primarily in long-running messaging integrations.

What Are Entitlements?

An entitlement is a unique identifier that links an external user response back to a specific XSOAR incident and optionally a playbook task. Format: <GUID>@<incident_id>|<task_id>

• GUID -- generated by XSOAR via the addEntitlement command
• incident_id -- the incident being questioned
• task_id (optional) -- specific playbook task waiting for a response

What Are Ask Tasks?

Ask tasks are playbook condition tasks that pause execution and wait for manual input. They present options (e.g., "Approve" / "Reject") and block the playbook until answered. Entitlements link external responses to these tasks.

Entitlement Flow

1. Playbook sends question via integration (e.g., Slack send-notification with entitlement)
2. Integration stores the entitlement in integration context with the message ID
3. User replies in the external system
4. Long-running integration receives reply, matches it to the stored entitlement
5. Integration calls demisto.handleEntitlementForUser() to close the task

Creating Entitlements

# In a script or command that sends an ask question
res = demisto.executeCommand('addEntitlement', {
    'persistent': False,            # True = survives first response
    'replyEntriesTag': 'my_tag',   # Tag for organizing replies
})
guid = res[0]['Contents']
entitlement_string = f'{guid}@{incident_id}|{task_id}'

# Store in integration context for the long-running process to track

Handling Entitlement Responses (in Long-Running Integration)

def handle_user_response(answer_text, message_id, user_name):
    """Process a user reply to an entitlement-based question."""
    context = demisto.getIntegrationContext()
    messages = json.loads(context.get('messages', '[]'))

    # Find the original question by message ID
    for msg in messages:
        if msg.get('message_id') == message_id:
            entitlement = msg.get('entitlement', '')
            parts = entitlement.split('@')
            guid = parts[0]
            id_and_task = parts[1].split('|')
            incident_id = id_and_task[0]
            task_id = id_and_task[1] if len(id_and_task) > 1 else ''

            # This closes the ask task and records the answer
            demisto.handleEntitlementForUser(
                incident_id,    # Which incident
                guid,           # Entitlement GUID
                user_name,      # Who answered
                answer_text,    # Their response (e.g., 'Approve')
                task_id         # Which task to close
            )
            break

When to Implement Entitlements

• Messaging integrations (Slack, Mattermost, Teams) that support ask tasks
• Any integration where playbooks need to ask external users for input
• Typically combined with long-running mode (to listen for replies in real-time)

Phase 2e: Classifiers, Mappers, Incident Fields, and Incident Types

When an integration fetches incidents, it needs supporting artifacts to properly classify events, map raw fields to XSOAR fields, and define custom incident types with custom fields.

Required artifacts for a fetching integration:

1. Incident Type - Defines the XSOAR incident type created by fetch
2. Incident Fields - Custom fields specific to the integration's data
3. Classifier - Routes raw events to the correct incident type
4. Mapper (Incoming) - Maps raw JSON fields from rawJSON to XSOAR incident fields
5. Mapper (Outgoing) (optional) - Maps XSOAR fields back to external system fields

Pack Directory Structure with Fetch Artifacts

Packs/<PackName>/
  Classifiers/
    classifier-<IntegrationName>.json              # Classifier
    classifier-mapper-incoming-<IntegrationName>.json  # Incoming mapper
    classifier-mapper-outgoing-<IntegrationName>.json  # Outgoing mapper (optional)
  IncidentFields/
    incidentfield-<Vendor>_<FieldName>.json        # One file per field
  IncidentTypes/
    incidenttype-<Incident_Type_Name>.json         # One file per type

Incident Type JSON

File: IncidentTypes/incidenttype-<Name_With_Underscores>.json

{
    "id": "My Integration Event",
    "name": "My Integration Event",
    "version": -1,
    "fromVersion": "6.0.0",
    "playbookId": "",
    "color": "#6A45D1",
    "hours": 0,
    "days": 0,
    "weeks": 0,
    "hoursR": 0,
    "daysR": 0,
    "weeksR": 0,
    "closureScript": "",
    "layout": "",
    "detached": false,
    "disabled": false,
    "reputationCalc": 0,
    "system": false,
    "readonly": false,
    "default": false,
    "autorun": true
}

Key fields:

• id and name: Must match the incident type name used in fetch and classifier
• playbookId: Optional -- auto-run a playbook when this incident type is created
• color: Hex color for the incident type in XSOAR UI
• autorun: If true, automatically runs the associated playbook
• hours/days/weeks: SLA timer settings (0 = no SLA)
• hoursR/daysR/weeksR: Remediation SLA settings

Incident Fields JSON

File: IncidentFields/incidentfield-<vendor>_<field_name>.json (one file per field)

{
    "id": "incident_myvendorflightid",
    "name": "MyVendor Flight ID",
    "cliName": "myvendorflightid",
    "type": "shortText",
    "description": "Unique flight identifier from MyVendor API.",
    "version": -1,
    "fromVersion": "6.0.0",
    "content": true,
    "group": 0,
    "associatedTypes": ["My Integration Event"],
    "associatedToAll": false,
    "system": false,
    "required": false,
    "openEnded": false,
    "sla": 0,
    "threshold": 0
}

Key fields:

• id: Must be incident_<cliname> (XSOAR convention). Use vendor prefix to avoid collisions.
• cliName: Snake-case identifier used in code and API. Must match the id suffix after incident_.
• name: Human-readable display name in XSOAR UI.
• type: Field data type -- one of:
  • shortText -- Single line text (most common)
  • longText -- Multi-line text
  • number -- Numeric value
  • boolean -- True/false
  • date -- Date/time value
  • grid -- Table/grid (structured data)
  • singleSelect -- Dropdown single selection
  • multiSelect -- Dropdown multi selection
  • url -- URL field
  • html -- HTML content
• associatedTypes: Array of incident type names this field applies to. Use ["all"] for all types.
• group: 0 = incident field, 1 = evidence field, 2 = indicator field
• content: Must be true for pack content fields

Classifier JSON

File: Classifiers/classifier-<IntegrationName>.json

The classifier examines the rawJSON of each fetched event and routes it to the correct incident type.

{
    "id": "MyIntegration",
    "name": "MyIntegration - Classifier",
    "type": "classification",
    "version": -1,
    "fromVersion": "6.0.0",
    "defaultIncidentType": "My Integration Event",
    "keyTypeMap": {
        "event_type_value_1": "My Integration Event",
        "event_type_value_2": "My Integration Alert"
    },
    "transformer": {
        "simple": "rawJSON.event_type"
    }
}

How it works:

1. transformer extracts a value from the raw event (e.g., rawJSON.event_type -> "alert")
2. keyTypeMap maps that extracted value to an XSOAR incident type name
3. defaultIncidentType is used when the extracted value doesn't match any key

Single incident type (simple case): If all events map to one type, set defaultIncidentType and leave transformer and keyTypeMap empty:

{
    "id": "MyIntegration",
    "name": "MyIntegration - Classifier",
    "type": "classification",
    "version": -1,
    "fromVersion": "6.0.0",
    "defaultIncidentType": "My Integration Event",
    "keyTypeMap": {},
    "transformer": {}
}

Mapper (Incoming) JSON

File: Classifiers/classifier-mapper-incoming-<IntegrationName>.json

The incoming mapper extracts values from rawJSON and maps them to XSOAR incident fields.

{
    "id": "MyIntegration-mapper",
    "name": "MyIntegration - Incoming Mapper",
    "type": "mapping-incoming",
    "version": -1,
    "fromVersion": "6.0.0",
    "mapping": {
        "My Integration Event": {
            "dontMapEventToLabels": false,
            "internalMapping": {
                "MyVendor Flight ID": {
                    "simple": "rawJSON.flight_id"
                },
                "MyVendor Status": {
                    "simple": "rawJSON.status"
                },
                "Occurred": {
                    "simple": "rawJSON.timestamp"
                },
                "Severity": {
                    "complex": {
                        "root": "rawJSON.severity",
                        "filters": [],
                        "transformers": [
                            {
                                "operator": "MapValuesTransformer",
                                "args": {
                                    "input_values": {"value": "low,medium,high,critical"},
                                    "mapped_values": {"value": "1,2,3,4"}
                                }
                            }
                        ]
                    }
                },
                "Details": {
                    "simple": "rawJSON.description"
                }
            }
        }
    }
}

Key concepts:

• mapping is keyed by incident type name (must match classifier output and incident type id)
• internalMapping maps XSOAR field names (left side) to raw JSON paths (right side)
• Simple mapping: "simple": "rawJSON.field_name" -- direct field extraction
• Complex mapping: Uses transformers to convert values:
  • MapValuesTransformer -- maps input values to output values
  • number.TimeStampToDate -- converts Unix timestamps to date strings
  • general.join -- joins array values with a separator
  • RegexExtractAll -- regex extraction
  • substring -- extract substring
  • concat -- concatenate values
• Built-in XSOAR fields: Occurred, Severity, Details, Name, Source Brand, Source Instance
• Custom fields: Use the field name from the incident field JSON (not the cliName)

Mapper (Outgoing) JSON (Optional)

File: Classifiers/classifier-mapper-outgoing-<IntegrationName>.json

Used for mirror-out scenarios (bidirectional sync). Maps XSOAR fields back to external system fields.

{
    "id": "MyIntegration-outgoing-mapper",
    "name": "MyIntegration - Outgoing Mapper",
    "type": "mapping-outgoing",
    "version": -1,
    "fromVersion": "6.0.0",
    "mapping": {
        "My Integration Event": {
            "dontMapEventToLabels": false,
            "internalMapping": {
                "external_status": {
                    "simple": "closeReason"
                },
                "external_notes": {
                    "simple": "closeNotes"
                }
            }
        }
    }
}

Naming Conventions for Fetch Artifacts

Artifact	File Pattern	ID Pattern
Classifier	classifier-<Integration>.json	<Integration>
Incoming Mapper	classifier-mapper-incoming-<Integration>.json	<Integration>-mapper
Outgoing Mapper	classifier-mapper-outgoing-<Integration>.json	<Integration>-outgoing-mapper
Incident Type	incidenttype-<Type_Name>.json	<Type Display Name>
Incident Field	incidentfield-<vendor>_<field>.json	incident_<vendor><field>

Complete Fetch Integration Checklist

When adding fetch capability to an integration:

• YAML: Set isfetch: true, add incidentType, max_fetch, first_fetch params
• YAML: Add hidden feedClassifier and feedMapper params with default values
• Python: Implement fetch_incidents() function with dedup, state management, rawJSON
• Python: Add fetch-incidents case to main() with demisto.getLastRun()/setLastRun()/incidents()
• Create incident type JSON in IncidentTypes/
• Create incident field JSONs in IncidentFields/ (one per custom field)
• Create classifier JSON in Classifiers/
• Create incoming mapper JSON in Classifiers/
• Write unit tests for fetch_incidents() (empty fetch, first fetch, dedup, pagination)
• Update README with fetch configuration, incident type, and field descriptions
• Update _description.md with fetch setup instructions
• Update ReleaseNotes with fetch feature documentation

Phase 3: Validation and Testing

CRITICAL: demisto-sdk Repository Requirements

demisto-sdk validate and demisto-sdk pre-commit MUST run from inside either:

1. A clone/fork of demisto/content (marketplace workflow)
2. A repo structured like content-ci-cd-template (local workflow)

They will NOT work in a standalone directory. The SDK imports CONTENT_PATH at module level and expects the full content repo structure.

For standalone repos: Set DEMISTO_SDK_CONTENT_PATH env var BEFORE running:

DEMISTO_SDK_CONTENT_PATH=$(pwd) DEMISTO_SDK_IGNORE_CONTENT_WARNING=1 demisto-sdk validate -i Packs/<PackName>

Note: demisto-sdk lint was REMOVED in SDK >= 1.38. Use demisto-sdk pre-commit instead (runs lint+tests in Docker).

Run demisto-sdk Validation Suite

# Validate pack structure and metadata
# Inside demisto/content fork: no env vars needed
demisto-sdk validate -i Packs/<PackName>

# Inside content-ci-cd-template or standalone repos: set CONTENT_PATH
DEMISTO_SDK_CONTENT_PATH=$(pwd) DEMISTO_SDK_IGNORE_CONTENT_WARNING=1 demisto-sdk validate -i Packs/<PackName>

# Full pre-commit (includes ruff lint + format + validate + tests in Docker)
# IMPORTANT: Files must be staged (git add) before running pre-commit
git add Packs/<PackName>/
demisto-sdk pre-commit -i Packs/<PackName> --show-diff-on-failure

# Format code (auto-fixes many issues)
demisto-sdk format -i Packs/<PackName>

Pre-commit Hook Results

When running demisto-sdk pre-commit locally, some hooks are CI-only and will fail:

• PASS locally: pylint-in-docker, mypy-in-docker, pytest-in-docker, markdownlint, brack, Validate README
• FAIL locally (expected, CI-only): xsoar-lint, validate-deleted-files, validate-content-paths, secrets, merge-pytest-reports, coverage-pytest-analyze
• ruff hook: May fail if python symlink is missing (only python3 exists). Use manual ruff fallback below.

The important quality checks (pylint, mypy, pytest) run in Docker and will work locally if Docker is available.

CRITICAL: Run ruff format Before Pushing

The CI pre-commit pipeline runs ruff format which enforces code formatting (not just linting). If your code is not formatted correctly, CI will fail even if lint passes. Always run ruff format before pushing:

# Run ruff format on changed Python files (REQUIRED before pushing)
# This auto-fixes formatting: function signature line wrapping, dict formatting, blank lines after imports, etc.
pip3 install ruff 2>/dev/null
ruff format Packs/<PackName>/Integrations/<IntName>/<IntName>.py Packs/<PackName>/Integrations/<IntName>/<IntName>_test.py

# Common formatting issues ruff format fixes:
# - Function signatures unnecessarily wrapped across multiple lines (should be single line if under 130 chars)
# - Dict literals passed to .append() need hanging indent style
# - Missing blank line after inline import statements
# - Trailing whitespace

Fallback: Manual Linting (when pre-commit is unavailable)

If pre-commit fails due to repo structure or Docker issues, run linting manually:

# Preferred: flake8 with XSOAR standard ignores (most reliable)
pip3 install flake8 && flake8 --max-line-length=130 --ignore=W605,F403,F405,W503,BA107 Packs/<PackName>/Integrations/<IntName>/<IntName>.py

# Alternative: ruff check (linting only - does NOT check formatting)
# NOTE: ruff >= 0.15 removed UP038 rule; content repo pyproject.toml still references it
# Use --isolated to avoid config conflicts, or prefer flake8
ruff check --isolated --select=E,W,F --ignore=F403,F405 Packs/<PackName>/Integrations/<IntName>/<IntName>.py

# IMPORTANT: Also run ruff format (see section above) - ruff check alone is NOT sufficient

Note: F403/F405 (star imports) are EXPECTED in XSOAR code due to from CommonServerPython import * being mandatory.

Set Up Test Dependencies

Before running tests, copy shared XSOAR modules into the integration directory:

# Required files (from a clone of demisto/content repo)
# If you don't have demisto/content cloned, clone it first:
#   git clone --depth 1 https://github.com/demisto/content.git /tmp/content
CONTENT_REPO="${DEMISTO_CONTENT_PATH:-/tmp/content}"
cp "${CONTENT_REPO}/Packs/Base/Scripts/CommonServerPython/CommonServerPython.py" Packs/<PackName>/Integrations/<IntName>/
cp "${CONTENT_REPO}/Tests/demistomock/demistomock.py" Packs/<PackName>/Integrations/<IntName>/
cp "${CONTENT_REPO}/Packs/ApiModules/Scripts/DemistoClassApiModule/DemistoClassApiModule.py" Packs/<PackName>/Integrations/<IntName>/
echo "" > Packs/<PackName>/Integrations/<IntName>/CommonServerUserPython.py  # Empty stub

# Install pytest dependencies
pip3 install pytest requests-mock

These files are NOT committed to git - add them to .gitignore.

Run Unit Tests

# Direct pytest (fast, for development iteration)
cd Packs/<PackName>/Integrations/<IntName>/
python3 -m pytest <IntName>_test.py -v

# Via demisto-sdk (recommended - runs in Docker with correct deps)
demisto-sdk pre-commit -i Packs/<PackName>

CRITICAL: ALL tests must pass before proceeding to lint/validate/commit.

Validation Checklist

Before committing, verify:

• All args.get() calls have matching YAML argument definitions
• Any command argument that accepts a list (comma-separated values) MUST have isArray: true in the YAML definition
• All markdown files use ASCII only (no UTF-8 special chars)
• pack_metadata.json has valid JSON and correct version
• ReleaseNotes file exists matching the version
• Unit tests pass
• demisto-sdk validate passes (set DEMISTO_SDK_CONTENT_PATH if standalone repo)
• demisto-sdk pre-commit passes (requires content repo structure), or manual ruff/flake8 lint passes

Phase 4: Version Management

Version Rules

Local packs:

• Start at 1.0.0
• Bump 0.0.1 (patch) on every feature aggregation after tagging
• If many changes accumulated since last tag, suggest tagging and bumping

Marketplace packs (first push to public GitHub):

• Version MUST be 1.0.0 on first PR to demisto/content
• Concatenate all changelog entries into the 1_0_0.md release notes
• After first PR is merged and reviewer feedback comes, bump 0.0.1 per review cycle
• Ask user: "Has an initial PR been filed? Is this feedback from an XSOAR developer review?"

Bumping Version

When ANY pack file changes:

1. Increment currentVersion in pack_metadata.json by 0.0.1
2. Create ReleaseNotes/<new_version_underscored>.md
3. Document ALL changes in release notes using official format

CRITICAL: Documentation Updates with EVERY Feature

Every feature, bug fix, or change MUST include updates to ALL THREE doc files in the SAME commit:

1. ReleaseNotes/.md -- What changed and why (bullet points per component)
2. Packs//README.md -- Full pack documentation (commands, args, outputs, examples)
3. Integrations//_description.md -- UI configuration panel help text

This is NOT optional. Code changes without corresponding doc updates are incomplete. If you add a command, the README must document it. If you change an argument, the README table must reflect it. If you add a config parameter, the description file must explain it.

Release Notes Format

#### Integrations

##### <Integration Display Name>

- Added **new-command-name** command to retrieve X.
- Fixed an issue where Y did not work correctly.
- Updated Docker image to *demisto/python3:3.10.14.100715*.

#### Scripts

##### <Script Display Name>

- Improved performance of Z processing.

#### Playbooks

##### <Playbook Display Name>

- Added new sub-playbook for handling W.

CRITICAL: Use pure ASCII in release notes. No arrows, smart quotes, em dashes.

Phase 5: README Generation - MUST BE COMPREHENSIVE

CRITICAL: The pack README.md is the primary documentation. It MUST be updated with EVERY feature or change -- not just at tag time. Every command, argument, output, and configuration option must be fully documented with markdown tables.

When to Update README

• Adding a new command -> Add full command section with args table, outputs table, example
• Changing an argument -> Update the args table
• Adding a config parameter -> Update the Configuration table
• Adding a script or playbook -> Add new section
• Any user-visible change -> Update relevant README section

Pack README Structure (MANDATORY - all sections required)

# <Pack Name>

<Detailed paragraph explaining what this pack does, what service it integrates with,
what use cases it supports (e.g., threat intelligence enrichment, incident response,
asset tracking), and why a security team would use it.>

## Dependencies

This pack requires the following packs:
- **<PackName>** (mandatory/optional) -- <brief reason>

*No dependencies* if none are needed.

## Integrations

### <Integration Name>

<Detailed description: what the integration connects to, what capabilities it provides,
what API it wraps, any subscription or license requirements.>

#### Authentication

<Explain the auth method: API key, OAuth, basic auth, etc.>
<How to obtain credentials step by step.>
<Required permissions or scopes.>

#### Configuration

| Parameter | Description | Required | Default |
| --- | --- | --- | --- |
| Server URL | Base URL of the <service> API | True | `https://api.example.com` |
| API Key | API authentication token | True | |
| Trust any certificate | Skip SSL verification | False | False |
| Use system proxy | Route through system proxy | False | False |

#### Commands

| Command | Description |
| --- | --- |
| prefix-command-one | Brief description of what it does |
| prefix-command-two | Brief description of what it does |

---

### prefix-command-one

<Full description of what this command does, when to use it, and any important behavior notes.>

#### Input

| Argument | Description | Required | Default |
| --- | --- | --- | --- |
| arg1 | What this argument controls | True | |
| arg2 | Filter or limit option | False | `50` |
| arg3 | Comma-separated list of values | False | |

#### Context Output

| Path | Type | Description |
| --- | --- | --- |
| Prefix.Object.Field1 | String | What this field contains |
| Prefix.Object.Field2 | Number | What this number represents |
| Prefix.Object.Field3 | Boolean | What this flag indicates |

#### Command Example

\`\`\`
!prefix-command-one arg1=value arg2=10
\`\`\`

#### Human Readable Output

> Show a realistic example of the markdown table output the user will see in the War Room.

---

### prefix-command-two

<Repeat the same detailed structure for every command.>

(... repeat for ALL commands ...)

## Scripts

### <Script Name>

<What the script does, when it runs, what inputs/outputs it has.>

| Argument | Description |
| --- | --- |
| arg1 | Description |

## Playbooks

### <Playbook Name>

<Description of the playbook flow, what triggers it, what it automates, and the expected outcome.>

#### Playbook Flow

1. Step one -- what happens
2. Step two -- what happens
3. Step three -- what happens

## Known Limitations

- <Any API rate limits, unsupported features, or platform restrictions.>

## Troubleshooting

| Issue | Solution |
| --- | --- |
| Error message X | How to fix it |

Auto-generate Docs (Starting Point Only)

Use demisto-sdk generate-docs to scaffold the README, then you MUST review and enhance the output:

demisto-sdk generate-docs -i Packs/<PackName>/Integrations/<IntName>

This requires command_examples.txt to exist with example commands.

After running generate-docs, you MUST:

1. Review the generated README.md for accuracy and completeness
2. Add detailed descriptions to every command (not just the one-liners from YAML)
3. Add an Authentication section explaining how to get credentials
4. Add a Known Limitations section if applicable
5. Add a Troubleshooting section with common errors
6. Add realistic Command Example output for each command
7. Verify all argument tables match the current YAML definitions
8. Ensure the description paragraph at the top is informative (not a generic stub)
9. Fix any formatting issues or missing markdown tables

generate-docs creates a skeleton -- you create the documentation.

Phase 6: Git Workflow

CRITICAL: Exclude Zip Build Artifacts from Git

Pack zip files produced by demisto-sdk zip-packs MUST NEVER be committed or pushed to any git repository (marketplace or local). They are build artifacts only.

Before any commit, ensure .gitignore excludes zip output:

# Add to .gitignore at repo root (if not already present)
echo '*_build/' >> .gitignore
echo '*.zip' >> .gitignore
echo 'uploadable_packs/' >> .gitignore

Verify before staging:

# Check that no zip files or build dirs are staged
git status | grep -E '\.zip|_build|uploadable_packs' && echo "WARNING: zip artifacts staged!" || echo "OK: no zip artifacts"

If a zip file is accidentally staged, remove it:

git rm --cached <PackName>*.zip
git rm --cached -r <PackName>_build/

Feature Branch Workflow

Each feature gets its own branch. Zip is ONLY built after tagging on main.

main ─────────────────────────────────*──── tag v1.0.0 ──── BUILD ZIP
  \              \                   /
   feature/feat1  feature/feat2     / (merge all to main)
   (develop,      (develop,        /
    test, lint,    test, lint,    /
    validate,      validate,    /
    commit)        commit)     /

Step-by-step for each feature:

# 1. Create feature branch from main
git checkout main
git checkout -b feature/<feature-name>

# 2. Develop (code + tests + docs in same commit)
#    - Write integration/script code
#    - Write unit tests
#    - Update README.md with new commands/args/outputs
#    - Update _description.md if config changed
#    - Update ReleaseNotes (or create new version file)

# 3. Test + lint + validate (must all pass)

# 4. Stage and commit (specific files, never git add -A)
git add Packs/<PackName>/
git commit -m "feat(<PackName>): add <feature description>"

Merging features and tagging:

# 5. When all features for this release are done, merge to main
git checkout main
git merge feature/feat1
git merge feature/feat2
# Resolve any conflicts

# 6. Final validation on main
demisto-sdk validate -i Packs/<PackName>

# 7. Tag the release on main
VERSION=$(python3 -c "import json; print(json.load(open('Packs/<PackName>/pack_metadata.json'))['currentVersion'])")
git tag -a v${VERSION} -m "<PackName> v${VERSION}: <summary of all features in this release>"

# 8. NOW build the zip (only after tagging)
# See Phase 7 for build commands

# 9. Bump version for next dev cycle
# Edit pack_metadata.json: 1.0.0 -> 1.0.1
# Create empty ReleaseNotes/1_0_1.md stub
git add Packs/<PackName>/pack_metadata.json Packs/<PackName>/ReleaseNotes/
git commit -m "version(<PackName>): bump to v1.0.1 for next development cycle"

Cleanup feature branches:

git branch -d feature/feat1
git branch -d feature/feat2

Marketplace Git Workflow

1. Fork demisto/content on GitHub
2. Clone fork locally
3. Create feature branch from master
4. Develop pack in Packs/<PackName>/ (code + tests + docs together)
5. Run full validation suite
6. Commit and push to fork
7. Open PR against demisto/content
8. Address reviewer feedback (bump version 0.0.1 per review cycle)
9. Do NOT build zip -- marketplace CI handles it

Local Git Workflow

1. Work in private repository (content-ci-cd-template or content fork)
2. Create feature branch for EACH feature
3. Develop + test + lint + validate + update docs on each branch
4. Merge all feature branches to main when validated
5. Tag release on main
6. Build zip ONLY after tagging
7. Deploy zip to XSOAR instance

Phase 7: Build and Deploy

CRITICAL: Zip is ONLY Built After a Git Tag

DO NOT build a zip pack from a feature branch or untagged main. The build process is:

1. All features merged to main
2. demisto-sdk validate passes on main
3. Git tag created (e.g., v1.0.0)
4. THEN build the zip

If the user asks to build a zip and there is no tag, ask: "Should I tag this release first?"

Build Zip Pack

ALWAYS use demisto-sdk zip-packs (not regular zip). Output filename MUST include the version:

# Verify we are on a tagged commit
git describe --exact-match --tags HEAD || echo "WARNING: HEAD is not tagged! Tag first."

# Get version from pack_metadata.json
VERSION=$(python3 -c "import json; print(json.load(open('Packs/<PackName>/pack_metadata.json'))['currentVersion'])")

# Handle Lists bug workaround
if [ -d "Packs/<PackName>/Lists" ]; then
    mv Packs/<PackName>/Lists Packs/<PackName>/Lists.bak
fi

demisto-sdk zip-packs -i Packs/<PackName> -o <PackName>_build

# Restore Lists
if [ -d "Packs/<PackName>/Lists.bak" ]; then
    mv Packs/<PackName>/Lists.bak Packs/<PackName>/Lists
fi

# MUST rename with version in filename
cp <PackName>_build/uploadable_packs/<PackName>.zip <PackName>-${VERSION}.zip

# Clean up build directory (do NOT commit it)
rm -rf <PackName>_build

echo "Built: <PackName>-${VERSION}.zip"

Zip filename format: <PackName>-<version>.zip (e.g., MyIntegration-1.0.0.zip) For marketplace packs: Do NOT store the zip in the git repo that will be pushed. CI handles zip building. For local packs: Ask user where to store the zip. Common: /var/www/packs/ or custom path. NEVER commit zip files or build directories to git (see Phase 6 .gitignore rules).

MANDATORY: Post-Build Deployment Prompt

After EVERY zip build, ALWAYS ask the user these questions (do NOT skip):

1. "Should I upload this pack to your XSOAR instance?"
2. "Should I create/update an integration instance and test it?"

Only proceed with deployment if the user confirms. If the user says yes, use the appropriate deployment method below.

Deploy to XSOAR Instance

ALWAYS use demisto-sdk upload as the primary upload method. Only fall back to alternatives if demisto-sdk upload fails.

Primary Method: demisto-sdk upload (ALWAYS USE THIS)

# Set env vars (check CLAUDE.md or ask user for values)
export DEMISTO_BASE_URL=<url>
export DEMISTO_API_KEY=<key>

# Upload the pack directly (uploads source, not zip)
demisto-sdk upload -i Packs/<PackName> --insecure

# OR upload a built zip file
demisto-sdk upload -i <PackName>-<version>.zip --insecure

CRITICAL: This is the ONLY recommended method for uploading packs. It handles unification, signature issues, and content validation automatically.

Alternative: Upload via demisto-client (Python API) -- ONLY if demisto-sdk upload fails

import demisto_client

# Configure client (see XSOAR API Reference for auth differences)
api_instance = demisto_client.configure(
    base_url="https://<xsoar-url>",
    api_key="<api-key>",
    verify_ssl=False
)

# Upload pack zip file
with open('<PackName>-<version>.zip', 'rb') as f:
    zip_content = f.read()

response, status, _ = demisto_client.generic_request_func(
    self=api_instance,
    method="POST",
    path="/contentpacks/installed/upload",
    body=zip_content,
    content_type="application/zip",
    response_type="object"
)
print(f"Upload status: {status}")

Method 3: Create Integration Instance via demisto-client

After uploading a pack, create and test an integration instance:

import demisto_client

api_instance = demisto_client.configure(
    base_url="https://<xsoar-url>",
    api_key="<api-key>",
    verify_ssl=False
)

# 1. Search for the integration to get its full configuration schema
response, status, _ = demisto_client.generic_request_func(
    self=api_instance,
    method="POST",
    path="/settings/integration/search",
    body={"query": "<IntegrationName>"},
    response_type="object"
)

# 2. Get the integration configuration from search results
integrations = response.get("configurations", [])
integration_config = None
for config in integrations:
    if config.get("display") == "<Integration Display Name>":
        integration_config = config
        break

# 3. Build the instance payload
# CRITICAL: Instance structure is DIFFERENT from integration definition structure
# - "enabled" must be string "true", not boolean
# - "version" must be 0 for new instances
# - "configuration" is a NESTED OBJECT (integration reference), NOT the params array
# - "data" is the params array (from search response's "configuration" array)
# - "configvalues" is a simple dict of param_name -> value
# - "configtypes" is a simple dict of param_name -> type_int

# The search response has "configuration" as an ARRAY of param defs.
# For the instance PUT, those param defs go into "data" (not "configuration")
config_params = integration_config.get("configuration", [])

# Build data array with values set
data_array = []
configvalues = {}
configtypes = {}

for param in config_params:
    item = param.copy()
    pname = param.get("name", "")
    ptype = param.get("type", 0)

    if pname == "url":
        item["value"] = "https://api.example.com"
        item["hasvalue"] = True
        configvalues["url"] = "https://api.example.com"
        configtypes["url"] = ptype
    elif pname == "insecure":
        item["value"] = True
        item["hasvalue"] = True
        configvalues["insecure"] = True
        configtypes["insecure"] = ptype
    # Type 4 (encrypted) params: set value directly in configvalues
    elif pname == "apikey" and ptype == 4:
        item["hasvalue"] = True
        configvalues["apikey"] = "<api-key-value>"
        configtypes["apikey"] = ptype
    # Type 9 (auth) params: set identifier+password in configvalues
    elif pname == "credentials" and ptype == 9:
        item["hasvalue"] = True
        configvalues["credentials"] = {"identifier": "[e-mail]", "password": "<pass>"}
        configtypes["credentials"] = ptype

    data_array.append(item)

# Build nested configuration (integration definition reference - mostly empty)
nested_config = {
    "id": "", "version": 0, "cacheVersn": 0,
    "modified": "0001-01-01T00:00:00Z", "created": "0001-01-01T00:00:00Z",
    "sizeInBytes": 0, "packID": "", "packName": "", "itemVersion": "",
    "fromServerVersion": "", "toServerVersion": "", "definitionId": "",
    "vcShouldIgnore": False, "vcShouldKeepItemLegacyProdMachine": False,
    "commitMessage": "", "shouldCommit": False, "name": "", "prevName": "",
    "display": integration_config.get("display", ""),
    "brand": "", "category": "", "icon": "", "description": "",
    "configuration": None, "integrationScript": None,
    "hidden": False, "canGetSamples": False,
}

instance_payload = {
    "id": "", "version": 0, "enabled": "true",
    "name": "<instance-name>",
    "brand": integration_config["id"],  # Must match integration ID
    "category": integration_config.get("category", ""),
    "configuration": nested_config,     # Nested integration ref (object, NOT array)
    "data": data_array,                 # Param values (array from search "configuration")
    "configvalues": configvalues,       # Simple key-value param map
    "configtypes": configtypes,         # Simple key-type param map
    "isIntegrationScript": True,
    "propagationLabels": ["all"],
}

# 4. Create the instance
response, status, _ = demisto_client.generic_request_func(
    self=api_instance,
    method="PUT",
    path="/settings/integration",
    body=instance_payload,
    response_type="object"
)
print(f"Instance creation status: {status}")
# status 200 = success, response contains instance ID

# 5. Test the instance
response, status, _ = demisto_client.generic_request_func(
    self=api_instance,
    method="POST",
    path="/settings/integration/test",
    body=instance_payload,
    response_type="object"
)
print(f"Test result: {response}")

Key gotchas for instance creation:

• enabled MUST be string "true", not boolean True
• Credential parameters use {"identifier": "", "password": "<value>"} format in configvalues
• version: 0 for new instances, version: -1 for updating existing ones
• The full integration config from search results provides the required schema -- modify it, don't build from scratch
• Parameter values go in the data array items (set value and hasvalue: True)

Method 4: API install via CoreRESTAPI

If CoreRESTAPI integration is configured on XSOAR:

!core-api-post uri=/contentpacks/installed/upload body=<zip-file-content>

Method 5: Manual zip upload

Upload via XSOAR UI: Settings -> Marketplace -> Upload custom pack

Method 6: Web server deployment (local)

sudo cp <PackName>-<version>.zip /var/www/packs/
# Pack available at http://<server>/packs/<PackName>-<version>.zip

Method 7: Create incident with pack info

Using the MCP server or API to create an incident that triggers pack installation.

Install Pack Dependencies

Check pack_metadata.json dependencies and ensure they're installed on the XSOAR instance.

Phase 8: Continuous Validation

Pre-commit Checks (mirroring CI/CD)

Run before every commit:

demisto-sdk pre-commit -i Packs/<PackName> --show-diff-on-failure

This runs:

• check-json, check-yaml, check-ast
• check-merge-conflict, debug-statements
• name-tests-test (validates test file naming)
• ruff (linting and formatting)
• mypy (type checking)
• pycln (unused import removal)
• Unit tests in Docker

CI/CD Pipeline Checks (what GitHub Actions run)

The content-ci-cd-template runs on push:

1. Checkout repo + demisto/content reference
2. Install poetry + Python 3.9+ + node
3. Copy CommonServerPython.py and demistomock.py
4. demisto-sdk pre-commit -g --prev-version ${DEFAULT_BRANCH}
5. demisto-sdk prepare-content (builds uploadable packs)
6. On merge to default branch: upload to artifact server or direct upload

Important Conventions

Code Conventions

• Import order: demistomock, CommonServerPython, CommonServerUserPython, then stdlib, then third-party
• Use snake_case for variables and functions
• Command functions: <command_name>_command(client, args)
• Commands: <prefix>-<action> (hyphenated lowercase)
• Use CommandResults with return_results() for all command output
• Output keys: CamelCase (e.g., Integration.Object.FieldName)
• CRITICAL: If a command argument expects a list (comma-separated), set isArray: true in the YAML argument definition. Use argToList() in Python to parse it.
• Error handling: return_error() in main, never let stack traces reach War Room
• Logging: demisto.debug(), demisto.info(), demisto.error() - never print()
• Keep functions small (~30 lines), use early returns
• Type hints on all functions
• Stateless - no shared state between executions

Naming Conventions

• Pack directory: PascalCase (MyPackName)
• Integration/Script files: PascalCase matching directory name
• Commands: prefix-action-noun (e.g., qradar-offenses-list)
• Context paths: Vendor.Entity.Field (CamelCase)
• Test files: <Name>_test.py

Docker Images

• Default: demisto/python3:3.10.14.100715 (check for latest)
• Dependencies managed via Docker, not local pip
• Specified in YAML: script.dockerimage

File Encoding

• ALL markdown files MUST use pure ASCII
• No UTF-8 special chars (no arrows, smart quotes, em dashes)
• Replace: -> not arrow, ' not smart quote, -- not em dash

Quick Reference: Common Actions

Action	Command
Create new pack	demisto-sdk init --pack
Validate	DEMISTO_SDK_CONTENT_PATH=$(pwd) demisto-sdk validate -i Packs/<Pack>
Lint + Test	demisto-sdk pre-commit -i Packs/<Pack> (requires content repo)
Format	demisto-sdk format -i Packs/<Pack>
Build zip	demisto-sdk zip-packs -i Packs/<Pack> -o output/
Upload	demisto-sdk upload -i Packs/<Pack> --insecure
Generate docs	demisto-sdk generate-docs -i Packs/<Pack>/Integrations/<Int>
Split unified YAML	demisto-sdk split -i <unified.yml>
Download from XSOAR	demisto-sdk download -o Packs/<Pack>

Decision Tree: Marketplace vs Local

User wants to develop a pack
    |
    +--> Marketplace (public)
    |       |
    |       +--> Fork demisto/content
    |       +--> Version MUST be 1.0.0 on first PR
    |       +--> Consolidate all changes into 1_0_0.md
    |       +--> Do NOT include zip in git repo
    |       +--> Run full demisto-sdk pre-commit
    |       +--> Open PR, address feedback
    |       +--> Bump 0.0.1 per review cycle
    |       +--> Sign CLA
    |
    +--> Local (private)
            |
            +--> Private git repo (or content-ci-cd-template)
            +--> Start at 1.0.0, bump 0.0.1 per feature
            +--> Tag releases
            +--> Build zip with demisto-sdk zip-packs
            +--> Deploy via upload/API/web server
            +--> Ask user for zip storage location

XSOAR API Reference

Authentication by Platform

Platform	Auth Method	demisto_client.configure()	Notes
XSOAR 6	API Key only	api_key="<key>"	Standard REST API. Generate key in Settings -> API Keys
XSOAR 8	API Key + Key ID	api_key="<key>", auth_id="<key-id>"	Uses Core REST API. Key ID is shown when creating the key
XSIAM	API Key + Key ID	api_key="<key>", auth_id="<key-id>"	URL format: api-<tenant>.xdr.<region>.paloaltonetworks.com. Uses marketplacev2

The skill survey (Phase 0) already asks which instance type. Use that info to configure authentication.

demisto-client Configuration

import demisto_client

# XSOAR 6
api_instance = demisto_client.configure(
    base_url="https://<xsoar6-url>",
    api_key="<api-key>",
    verify_ssl=False
)

# XSOAR 8
api_instance = demisto_client.configure(
    base_url="https://<xsoar8-url>",
    api_key="<api-key>",
    auth_id="<key-id>",        # Required for XSOAR 8
    verify_ssl=False
)

# XSIAM
api_instance = demisto_client.configure(
    base_url="https://api-<tenant>.xdr.<region>.paloaltonetworks.com",
    api_key="<api-key>",
    auth_id="<key-id>",        # Required for XSIAM
    verify_ssl=False
)

Common API Endpoints

# Generic request helper
response, status, _ = demisto_client.generic_request_func(
    self=api_instance,
    method="<METHOD>",
    path="<path>",
    body=<payload>,
    response_type="object"
)

Endpoint	Method	Purpose
/contentpacks/installed/upload	POST	Upload pack zip file
/settings/integration/search	POST	Search integrations (body: {"query": "name"})
/settings/integration	PUT	Create/update integration instance
/settings/integration/test	POST	Test integration instance connectivity
/incident/search	POST	Search incidents
/entry	POST	Create war room entry / run command
/lists/save	POST	Save/update an XSOAR list
/health	GET	Health check

demisto-py (Alternative)

For simpler API interactions without instance management:

import demisto_client
# or
import requests

# Direct REST with requests
headers = {"Authorization": "<api-key>", "Content-Type": "application/json"}
response = requests.post(f"{base_url}/incident/search", headers=headers, json={"query": ""}, verify=False)

MCP Server

MCP Server for XSOAR

If an XSOAR MCP server is configured in the project (check CLAUDE.md for location), use it for interactive XSOAR operations. A typical MCP server provides tools across categories: incidents, automations, integrations, lists, indicators, logs, migration, and utilities.

If no MCP server is available, use demisto-client or direct REST API calls as shown above.

Error Recovery

Error	Solution
UnicodeDecodeError in demisto-sdk	Non-ASCII chars in markdown - replace with ASCII equivalents
Lists processing bug in zip-packs	Move Lists dir out, zip, restore
demisto-sdk upload fails with IP URL	Use zip method and upload via UI
Pre-commit fails on Docker	Ensure Docker daemon is running
Missing CommonServerPython in tests	Copy from demisto/content repo clone (see test deps setup)
Missing DemistoClassApiModule	Copy from <content-repo>/Packs/ApiModules/Scripts/DemistoClassApiModule/
Missing CommonServerUserPython	Create empty stub: echo "" > CommonServerUserPython.py
mypy errors	Add type hints, use --ignore-missing-imports