sdl-log-parser

name	sdl-log-parser
author	Prithvi Moses <[e-mail]>
description	Use whenever the user wants to author, edit, debug, validate, or explain a SentinelOne Singularity Data Lake (SDL) log parser — the augmented-JSON files at /logParsers/ that extract fields from raw log text before ingestion. Trigger on "SDL parser", "Skylight parser", "log parser", "parser editor", "write a parser", "test parser", or any pasted raw log the user wants turned into structured fields. Also trigger on parser-DSL keywords like `formats:`, `patterns:`, `lineGroupers:`, `rewrites:`, `discardAttributes:`, `aliasTo:`, `{parse=...}`, `{regex=...}`. Especially trigger when the user pastes a raw log (CEF, syslog, JSON, key=value, multi-line, CSV) and asks to extract fields, normalize timestamps, or drop noise. If the project is SDL/Singularity/Scalyr and the user says "parse this log", use this skill. Always validates end-to-end via putFile → HEC ingest (parser applied) → query. NOT for PowerQuery (use powerquery), NOT for plain ingest without a parser (use sdl-api).

SentinelOne SDL Log Parser Authoring

This skill turns raw log samples into deployed, validated SDL parser definitions. A parser is an augmented-JSON file at /logParsers/<name> on the SDL tenant that extracts fields from each ingested line. The parser editor and the Test Parser button in the console run the parser client-side in JavaScript; this skill mirrors that workflow programmatically and finishes by ingesting a sample through the deployed parser to confirm the actual ingest path works.

Sandbox proxy blocked? If putFile or HEC ingest calls to *.sentinelone.net fail with a connection or proxy error inside the Claude sandbox, use the s1-secops-mcp server instead. It runs locally via node and bypasses the sandbox proxy entirely. Setup: add it to claude_desktop_config.json (see s1-secops-mcp/README.md). Use sdl_put_file to deploy the parser and hec_ingest to run the ingest validation step — both execute from your machine, not the sandbox.

Two hard rules (every parser, every time)

These two rules apply to every parser you write or edit, regardless of source, regardless of whether you're starting from scratch or copying from the catalog. Violating either produces a non-conformant parser even if extraction works.

Rule 1: The 4 mandatory attributes are always present.

attributes: {
  "dataSource.category": "security",   // hardcoded constant, never any other value
  "dataSource.name":     "Corelight",  // specific product/service name
  "dataSource.vendor":   "Corelight",  // parent vendor / company name
  "metadata.version":    "1.0.0"       // semver; bump on substantive parser changes
}

dataSource.category is fixed at "security". Not application, not network, not system, not audit. The catalog has parsers that violate this; you do not.

metadata.version is mandatory and follows semver (MAJOR.MINOR.PATCH, with optional pre-release suffix like -rc1 or -beta1). Increment it on every build. Patch bump for fixes, minor for additive changes, major for breaking schema changes. It can also be set inside mappings via a constant op when the source itself carries a vendor schema version you'd rather propagate per-event:

{ constant: { field: "metadata.version", value: "<your-current-version>" } }

A constant in mappings overrides the parser-root attributes: value when it fires, which is useful when one parser handles multiple vendor schema versions and you want each event tagged with the right one.

Rule 2: Every OCSF field name comes from references/ocsf-schema-documentation.md.

Before emitting any OCSF dotted path (src_endpoint.ip, actor.user.name, metadata.product.vendor_name, etc.), grep references/ocsf-schema-documentation.md for the exact string. Do not invent field names. Do not copy from a catalog parser without verifying. Do not rely on memory. The reference is the single source of truth for the ~25,759 documented OCSF field names across the 7 categories.

The rest of this skill assumes both rules are upheld. If either is unclear for a given parser, stop and resolve before continuing.

Critical pitfalls (read before authoring)

These are the DSL traps that cost the most iteration cycles when authoring parsers on the current S1 tenant. They are repeated in the long gotcha list below with more context, but if you're about to write a new parser, internalize these first. Items 1-9 are tenant-validated as of 2026-05-26; items 10-12 as of 2026-06-22.

1. Key=value catch-all REQUIRES leading .*. The canonical form is ".*$_=identifier$=$_=quoteOrSpace$" with repeat: true. Without the .* the format anchors at position 0 of the message and fires zero times on any line starting with a syslog priority <NNN>, a CEF header, an XML envelope, or a timestamp — i.e. virtually every real log source. The leading .* lets the engine scan forward to the first key=value pair and then repeat: true walks the rest. This is the same pattern the built-in keyValue parser uses. Also: {parse=keyValue} is NOT a valid sub-parser directive; built-in parsers like keyValue are only usable via root-level aliasTo:.
2. v1 mappings are FIRST-MATCH-WINS, which means renames must be duplicated into every per-subtype block alongside the constants. A catch-all predicate: "true" rename block placed at the end NEVER fires for events that matched an earlier per-subtype block — only the truly-unmatched events get those renames. If you have per-subtype constant blocks for OCSF class/activity fan-out (e.g. subtype=='forward' → activity_id=6), you must put the full rename list inside EACH of those blocks. Verbose but correct.
3. v0 constants: block is REJECTED on the current tenant. putFile returns 400: Syntax error at position -1: Assertion failed: Expected a field whenever a v0 mapping body contains a constants: array. v0 renames: and copies: work fine; only constants: fails. Practical impact: prefer v1 ({ constant: { field, value } }) for any new parser that needs OCSF class fan-out. Stay on v0 only when extending an existing v0 parser that doesn't add new constants.
4. Same-name rename silently NULLS the field. Writing { rename: { from: "action", to: "action" } } to "keep" a captured raw field under its own name actually deletes the value. The KV catch-all already extracts raw vendor field names directly into the event — to "keep" one, OMIT the rename. Only write a rename when the target name is genuinely different from the source.
5. Unquoted numerics from quoteOrSpace are auto-typed as NUMBER columns. A FortiGate srcport=44321 sentbyte=8421 line going through the KV catch-all produces srcport: 44321 (number), sentbyte: 8421 (number) — queryable with avg() / sum() / > N without number() wrapping. Quoted values stay as strings even if they look numeric (level="6" → string). Caveat: SDL column type-locking still applies, so wrap numerics with number() in dashboard panels and STAR rule bodies if the column has ever been written as string in this tenant's history.

Two related queryability rules (not DSL traps, but they matter for any consumer of the parser):

6. The raw message field is preserved on every event regardless of parser activity. SDL never lets a parser overwrite message; the raw event body always survives. So parse "field=$x$" from message queries continue to work alongside the typed columns the parser produces. This is the backward-compat escape hatch during a parser rollout.
7. When two ingest paths produce OCSF for the same source, the schema can drift. A common case: upstream collector (Observo etc.) flattens type=traffic, subtype=forward to just event.type='forward' while the downstream parser keeps both fields populated. Consumers (dashboards, STAR rules, hunts) should write subtype filters as (event.subtype='forward' OR event.type='forward') until the rollout fully converges.
8. Trailing .* at the END of a format string suffers the leading-.*-strip too. A field marker like $process.cmd_line{regex=.*}$ placed at the end of a format string captures the EMPTY STRING. The SDL stripping rule applies whether .* starts the regex or IS the entire regex. Use [\\s\\S]+ (one-or-more) or [\\s\\S]* (zero-or-more) for "everything that remains on the line" semantics — those forms don't begin with . and survive intact.
9. Regex alternation matches in DECLARATION ORDER, not longest-leftmost. regex=CRON|CROND is a bug — CRON matches the prefix of CROND and leaves D dangling. Put longer alternatives first: regex=CROND|CRON|crond|cron. Applies to named patterns, inline {regex=...}, and any other alternation. When alternatives share a prefix, sort by length descending.
10. gron expands arrays into [N]-indexed attributes; index with [N], NOT .N. Do not believe "gron drops arrays." $unmapped.{parse=gron}$ emits one queryable attribute per array leaf: unmapped.signInEventTypes[0], unmapped.additionalDetails[0].key, unmapped.targetResources[0].modifiedProperties[0].newValue. The trap that makes arrays look dropped is querying with dot-index (unmapped.targetResources.0.id → null); the real key is bracket-indexed (unmapped.targetResources[0].id). gron/dottedJson = each element its own attribute (use for "extract every field"); strict* = array kept as one JSON value for array_from_json(). Scalar/object flattening is identical across all three. Caveat: a raw PowerQuery columns/filter clause can't type the [, so validate [N] fields with powerquery_schema_discover, not a columns projection. To lift a whole array subtree to OCSF, rename_tree { from: "unmapped.targetResources[0].modifiedProperties", to: "entity.data" }. Tenant-validated 2026-06-22 (Microsoft Entra ID). See references/parse-directives.md.
11. copy first-of-N treats an empty string "" as "exists" and lets it win. copy {from: ["userId","servicePrincipalId"]} where userId is "" copies the empty string, not the fallback — silently breaking identity-coalesce (e.g. service-principal / managed-identity sign-ins whose userPrincipalName/userId are empty but servicePrincipalName/servicePrincipalId hold the identity). Fix by dropping empty strings upstream (the collector/Lua serializer should omit "" values), or drop the empty source before the copy. Tenant-validated 2026-06-22.
12. On a LIVE source, a "still broken" event is usually pre-propagation, not a parser bug. Parser activation is ~3-5 min and a continuously-ingesting source keeps producing events through the OLD version during that window; SDL never re-parses historical events. Bump metadata.version every deploy and confirm via | group count() by metadata.version (or powerquery_schema_discover) which version parsed a given event before chasing a "bug." Related: if the parser sets event time from a log field, events are stamped at the log's own time, so widen validation query windows (24h/7d) or filter by a nonce field instead of time, or you get zero rows and a false negative.

When to use this skill

Use this skill when the user wants to turn raw log text into structured SDL events. Common triggers:

• "Write a parser for this log."
• "Why isn't <field> showing up after ingest?"
• A pasted log line — CEF, syslog, JSON-per-line, key=value, multi-line, vendor-specific.
• Edits to an existing parser ("add a rewrite that masks the password", "tag every event with dataset=fortinet").
• Migration ("alias the new parser to the old one").

Do not use this skill for PowerQuery authoring (use powerquery) or for plain-text ingestion that does not need a parser (use sdl-api directly).

Workflow

Follow this loop. Skipping steps 1 (catalog check) and 5 (end-to-end validation) are the two most common reasons a parser goes live with surprises, so do not skip them.

1. Check the ai-siem catalog first. Before writing anything, search https://github.com/Sentinel-One/ai-siem/tree/main/parsers for the vendor/product. The repo has ~150 community + marketplace parsers; most common sources (Cloudflare, FortiGate, Palo Alto, Corelight, AWS RDS, Okta, Abnormal, Juniper, pfSense, Cisco ASA, etc.) are already there. If a parser exists, download it and diff against the user's requirements — far less work than authoring from scratch. See references/ai-siem-catalog.md for the recipe and a map of template parsers by shape.
2. Inspect the sample. Read every line the user pasted. Look for: leading priority/timestamp/host (syslog framing), the body shape (CEF | JSON | key=value | positional CSV | freeform), any obvious sub-structures (uri=..., embedded JSON, base64), and whether multiple physical lines belong to one logical event (stack traces, MySQL slow queries, Postgres SQL).
3. Decide on strategy. See Strategy decision tree below. Most logs are one of: alias an existing built-in, single line format with $field$ markers, repeating key/value catch-all, JSON parser with discardAttributes, line-grouper for multi-line, or a mapper for OCSF-style restructuring.
4. Draft the parser. Use the templates in examples/ as starting points, or (from step 1) a catalog parser as a base. Always start minimal — extract the outer frame first (timestamp + host + body), then add fragment formats or {parse=...} directives to crack the body open.
5. Validate end-to-end. Use the sdl-api skill to deploy the parser, ingest the sample, and query for the extracted fields. See Validation below for the exact recipe. Do not call this skill done before this step succeeds.
6. Iterate. If a field is missing or wrong, identify which format/rewrite was responsible, edit, redeploy, re-ingest with a fresh Nonce, re-query.
7. Hand off. Show the user the final parser file (path + version), the sample they gave, and the parsed fields the query returned.

Required default attributes on every parser (MANDATORY)

Four attributes are MANDATORY on the top-level attributes: block of every parser, regardless of source. These are non-negotiable:

"dataSource.category": "security",    // ALWAYS hardcoded to "security": fixed constant, never change
"dataSource.name":     "Corelight",   // specific product/service name (example value)
"dataSource.vendor":   "Corelight",   // parent vendor / company name (example value)
"metadata.version":    "1.0.0"        // semver; required on every parser

These four are required by the downstream SDL pipeline (Marketplace routing, parser catalog, content-pack grouping). Omitting any of them produces a non-conformant parser even when field extraction is correct.

dataSource.category MUST always be hardcoded to "security". This is not a taxonomy pick, it is a fixed constant. Never substitute another value (application, network, system, audit) regardless of the source type. Empirical audit of the ai-siem catalog (2026-04) shows ~17% of community parsers violate this rule with values like application, network, system, audit. Fix on copy.

dataSource.vendor is the parent company (Cisco for Umbrella, Microsoft for Azure AD). dataSource.name is the specific product (Cisco Umbrella, Azure AD). For single-product vendors both can be the same string (Corelight / Corelight).

metadata.version is semver (MAJOR.MINOR.PATCH with optional -rcN / -betaN suffix). Always increment on a new build so content-pack tooling can detect drift. Patch for fixes, minor for additive changes, major for breaking schema changes. May also be emitted from inside mappings via:

{ constant: { field: "metadata.version", value: "<your-current-version>" } }

A mappings.constant overrides the parser-root attributes: value when its predicate fires. This is the right pattern when one parser handles multiple vendor schema versions and you want each event labeled with the schema-version it actually came from.

Catalog-parser audit step. When you start from a catalog parser, the very first edits you make are to confirm all four fields are present, that dataSource.category is "security", and that metadata.version exists and reflects the change you're about to make. About 1/3 of the community parsers in Sentinel-One/ai-siem are missing dataSource.category outright; many that include it use a non-security value. Fix before doing anything else.

Default output schema: OCSF (mandatory reference)

Unless the user explicitly asks for vendor-native names, emit OCSF-shaped events. This means:

• Name captured fields with their OCSF dotted paths (e.g. src_endpoint.ip, dst_endpoint.port, connection_info.protocol_num, app_name, actor.user.name).
• Tag every event with the class metadata on attributes: class_uid, class_name, category_uid, category_name, and metadata.product.vendor_name / metadata.product.name / metadata.log_provider, all IN ADDITION TO the four mandatory attributes above.
• Put the per-event subtype (activity_id + activity_name) on the format, not the top-level attributes, because one parser often handles multiple subtypes (SESSION_CREATE vs SESSION_CLOSE).

OCSF mapping MUST always be done using references/ocsf-schema-documentation.md. This file is the authoritative SentinelOne community-documented OCSF field catalog (7 categories, 96 articles, ~25,759 field entries across every event class). Before emitting any OCSF field name, grep this file for the exact dotted path. Do not invent field names, do not copy from memory, do not trust catalog parsers to have the right name (many community parsers in ai-siem use vendor-native or stale OCSF names that need correcting). Workflow:

1. Pick the OCSF class (see Quick picker below).
2. Open references/ocsf-schema-documentation.md and grep for the class number / category to find the field block.
3. Copy the dotted path verbatim into your parser.
4. If unsure between two candidate fields, confirm with the user rather than guessing.

references/ocsf-mapping.md covers the two authoring idioms (capture-directly-into-dotted vs capture-vendor-then-rename) and the most common class-specific tables (Network Activity, Authentication, File Activity).

Why this matters: downstream PowerQuery hunts, STAR rules, dashboards, and Marketplace integrations assume OCSF. Vendor-native names force every consumer to learn each source format and break portability. A wrong dotted path (source.ip instead of src_endpoint.ip, dst.port instead of dst_endpoint.port) is silently wrong: it ingests, but every downstream consumer fails to match.

Quick picker (use this to find the class number, then look up fields in ocsf-schema-documentation.md):

• Network firewall / NAT / flow → 4001 Network Activity
• HTTP / web / proxy → 4002 HTTP Activity
• DNS → 4003 DNS Activity
• DHCP → 4004 DHCP Activity
• RDP / SSH session → 4005 RDP Activity
• TLS / SSL handshake → 4006 SSH Activity (TLS uses connection_info under 4001/4002)
• Email → 4009 Email Activity
• Authentication → 3002 Authentication
• Account change → 3001 Account Change
• API activity → 6003 API Activity
• File system ops → 1001 File System Activity
• Kernel ops → 1003 Kernel Activity
• Memory ops → 1004 Memory Activity
• Module ops → 1005 Module Activity
• Process ops → 1007 Process Activity
• Registry ops → 201001 (Windows Registry)
• Detection finding → 2004 Detection Finding
• Compliance finding → 2003 Compliance Finding
• Vulnerability finding → 2002 Vulnerability Finding
• Inventory / device → 5001 Device Inventory Info
• Email / file finding → 2007 (Email Finding) / 2006 (File Hosting Finding)

When the source could reasonably belong to multiple classes (proxy logs, EDR alerts), confirm with the user rather than picking silently.

Top-level parser structure

A parser file is augmented JSON (unquoted keys allowed, // and /* */ comments allowed). Top-level keys:

{
  timezone: "UTC",                                  // fallback for timestamp parsing
  attributes: {
    // REQUIRED defaults on every parser
    "metadata.version":    "1.0.0",
    "dataSource.category": "security",    // always "security" — hardcoded, never change
    "dataSource.name":     "Juniper SRX",
    "dataSource.vendor":   "Juniper",
    // OCSF class metadata + vendor product metadata
    "metadata.product.vendor_name": "Juniper",
    "metadata.product.name":        "SRX",
    "metadata.log_provider":        "rt_flow",
    "class_uid":      4001,
    "class_name":     "Network Activity",
    "category_uid":   4,
    "category_name":  "Network Activity"
  },
  patterns:   { tsPattern: "\\w+\\s+\\d+\\s+[0-9:]+" },  // named regex shortcuts
  lineGroupers: [ { start: "...", continueThrough: "..." } ],  // multi-line glue
  formats: [
    {
      id: "happy-path",
      attributes: { activity_id: 1, activity_name: "Open" },  // per-subtype tagging
      format: "$time=tsPattern$ $src_endpoint.hostname$ $body$",
      discard: false,
      halt: true,                                   // stop after first match
      repeat: false                                 // re-apply until no progress
    }
  ],
  // Advanced (not always present):
  mappings: { mappings: [ /* rename vendor-native fields to OCSF dotted names */ ] }
}

formats is the primary engine. Everything else either feeds into it (patterns, lineGroupers) or runs after it (rewrites, mappings, discardAttributes).

Field-matcher syntax inside a format string

$fieldName=patternName{parse=...}{regex=...}{attrWhitelist=...}{attrBlacklist=...}{timezone=...}$

Rules that bite people often:

• $field$ alone uses the quotable pattern by default, which stops at whitespace or quote boundaries.
• $field{regex=\\d+}$ inlines a regex — backslash metacharacters double-escape (\\d not \d).
• Two adjacent $a$$b$ with no delimiter between them require a pattern or regex on $a$ so it knows when to stop.
• message is reserved — it's the raw event text. You cannot capture into message from a format. You can discardAttributes: ["message"] to drop it.
• A literal $ in the log requires \$$ (escape the dollar, then close the field).
• Any space in the format matches \s+. To match a single literal space use \\ (backslash-space).

For the full directive list see references/syntax.md and references/parse-directives.md.

Canonical reference parsers by input shape

When you've identified the shape of the source log, jump straight to the canonical catalog parser for that shape and start from it. These are battle-tested in production:

Input shape	Canonical parser	What to learn from it
Pure JSON-per-line (flat or nested)	community/abnormal_security_logs-latest/	${parse=gron}$ capture, then mappings to OCSF
JSON with envelope (<ts> <host> <json>)	community/json_generic-latest/ (or examples/02-json-with-envelope.json)	dottedJson on the body
JSON, nested, multi-class	sentinelone/marketplace-cloudflare-latest/	gron + format IDs + enum cast + tree ops, all in v1 mappings
Already-OCSF JSON	community/okta_ocsf_logs-latest/	Pass-through with light renames
CEF over syslog	community/generic_access_logs-latest/ (or examples/01-cef-over-syslog.json)	Pipe-delimited header + KV extension
LEEF over syslog	community/leef_template_logs-latest/	Multi-format header + KV body
Pure key=value freeform	sentinelone/marketplace-fortinetfortigate-latest/	repeat: true catch-all, attrBlacklist, observables array
Key=value with record-type fan-out to many OCSF classes	examples/12-linux-auditd-ocsf.json	type=-anchored record-type capture (avoids the nametype= collision), KV catch-all, v1 first-match-wins mappings fanning one source to Process 1007 / File 1001 / Auth 3002 / Finding 2004 / Network 4001, EXECVE arg-vector → process.cmd_line via computeFields replace()
Positional CSV (100+ columns)	sentinelone/marketplace-paloaltonetworksfirewall-latest/	commaSeparatedvalues, skipNumericConversion, attr[N] indexed access in v1 mappings
Positional space-delimited	sentinelone/marketplace-awsvpcflowlogs-latest/	intermittentTimestamps, fixed columns
Pipe-delimited (non-CEF)	sentinelone/marketplace-zscalerinternetaccess-latest/	pipeSeparatedValues parse directive
Multi-line stack / SQL	community/sql_database_logs-latest/ and sentinelone/marketplace-awsrdslogs-latest/	lineGroupers start/continueThrough, format-id sentinels for sub-shapes
Windows Event XML	community/microsoft_windows_eventlog-latest/	XML with \\t / \\n escapes, per-EventID sub-parsers (4624, 4625, 4720, 4728, 1102)
HTTP access logs	community/apache_http_logs-latest/	Built-in accessLog alias-or-extend
pfSense / iptables freeform firewall	community/pfsense_firewall_logs-latest/	Frame → subtype → protocol cascade with discard: true for IPv6
Rewrites-only legacy style	community/okta_logs-latest/	Minimal-diff edits when you can't migrate to mappings
Gron-capture + everything-in-mappings	community/PARSER_TEMPLATE/ (and examples/08-gron-capture-template.json)	The most general scaffold; use when you want all transformations in one block

When in doubt, marketplace-cloudflare-latest/ is the most complete reference for "modern v1 parser doing everything right" (gron, format IDs, enum cast, tree ops, OCSF tagging). Start there if you're not sure.

Strategy decision tree

Apply in order — first match wins:

1. Is there a parser in the ai-siem catalog? Search Sentinel-One/ai-siem for the vendor/product name. If so, start from that parser (see references/ai-siem-catalog.md for the fetch recipe and per-shape template map). On copy, immediately audit and fix:
   • The 4 mandatory attributes (dataSource.category hardcoded to "security", dataSource.name, dataSource.vendor, metadata.version). About 1/3 of community parsers miss dataSource.category outright; many that include it use application/network/system/audit instead of "security". Force them to "security". Add metadata.version if missing, bump it if you're changing the parser.
   • All OCSF field names against references/ocsf-schema-documentation.md. Catalog parsers frequently use vendor-native names (source_ip, dst_port) where OCSF dotted paths (src_endpoint.ip, dst_endpoint.port) are required. Rewrite via mappings.rename or change the capture name directly.
   • Stale tenant-specific attributes (e.g. hardcoded site.id).
   • class_uid should be an integer, not a string.
   • Bump metadata.version.
2. Is there a built-in parser? Web access logs → accessLog. Pure JSON-per-line → json or dottedJson. Syslog → systemLog. Key=value pairs → keyValue. Heroku logplex → heroku-logplex. MySQL/Postgres → their dedicated parsers. CloudFront/ELB/S3/Redshift → AWS parsers. See references/builtin-parsers.md. If a built-in fits, recommend it and (optionally) just author an alias parser: { aliasTo: "json" }.
3. Is the log JSON-per-line but with a vendor envelope? Use one format that captures the envelope and applies {parse=json} (or dottedJson/escapedJson/urlEncodedJson/base64EncodedJson) on the embedded body. Add discardAttributes: ["message"] to save storage.
4. Is the whole line an OCSF/JSON blob you want flattened? Use the gron-capture-then-mappings idiom: format: "$unmapped.{parse=gron}$" at the top, then rename/copy/cast everything in a mappings block. See examples/08-gron-capture-template.json and the community/PARSER_TEMPLATE/ reference.
5. Is the body a sequence of key=value pairs? Use the two-format key/value idiom. The repeating catch-all format MUST be exactly ".*$_=identifier$=$_=quoteOrSpace$" (with the leading .*) and have repeat: true. Without the .* the format only anchors at position 0 of the message and fires zero times on any line that starts with a syslog priority <NNN>, a CEF header, an XML envelope, or anything else that isn't bare identifier=. The _ field name means "use the captured token as the field name". {parse=keyValue} is NOT a valid sub-parser directive — it returns Unknown parser "keyValue". The keyValue built-in can only be referenced via aliasTo: "keyValue" at the parser root (one-line alias parser), not via {parse=...}. Canonical references: built-in keyValue parser; examples/03-key-value.json; sentinelone/marketplace-fortinetfortigate-latest/ (FortiGate syslog, key=value with quoted strings).
6. Is it CEF or LEEF? Treat the header as a positional pipe-delimited line, then apply key/value catch-all to the extension. See examples/01-cef-over-syslog.json.
7. Is it positional / CSV / TSV? Use commaSeparatedValues / commaSeparatedvalues or pipeSeparatedValues parse directives with skipNumericConversion: true, and name the columns in a mappings block by positional index (attr[N]). Palo Alto Networks Firewall in ai-siem is the canonical reference.
8. Does the source emit multiple distinct event shapes (e.g. MySQL error vs MySQL general vs Postgres from one RDS stream)? Give each format an id: and fan mapping entries out with predicate: "<id>='true'". See the AWS RDS marketplace parser.
9. Are events multi-line? Add a lineGroupers block with start + one of continueThrough / continuePast / haltBefore / haltWith. Then write formats against the joined event. Beware HEC ingest newline splitting (see ingest-path gotchas). Canonical multi-line examples: parsers/community/sql_database_logs-latest/ (Postgres SQL) and parsers/sentinelone/marketplace-awsrdslogs-latest/ (RDS mysql/postgres mixed stream).
10. Is the source Windows Event Log XML? Use the per-EventID sub-parser idiom from parsers/community/microsoft_windows_eventlog-latest/: a lineGroupers block joins multi-line XML, then one format per EventID (4624, 4625, 4720, 4728, 1102, etc.) extracts EventID-specific fields. Remember the \\t / \\n double-escape gotcha. Class fan-out: 4624/4625 → class_uid: 3002 Authentication, 4720/4728 → class_uid: 3001 Account Change, 1102 → class_uid: 6004 Web Resources Activity (or whatever fits).
11. Do you need to restructure to OCSF or another schema? Use mappings (gron-style mappers). See references/mappers.md for the two equivalent syntaxes (v1 singular / v0 marketplace). For new parsers prefer v1; only stay on v0 when extending an existing v0 parser. Always look up dotted paths in references/ocsf-schema-documentation.md before emitting them.
12. None of the above? Hand-write a line format. Look for stable delimiters and use named patterns when delimiters are insufficient (e.g., a timestamp containing spaces). Drop events you never want (discard: true on the format) rather than letting them fall through to a looser format.

Common gotchas (memorize)

• Double-escape every regex metacharacter. \\d, \\s, \\., \\\\. The augmented-JSON layer eats one backslash before the regex engine sees it.
• severity is a RESERVED field coerced to 0–6 integer. The ingest pipeline maps info/warn/error strings to 0–6 using its own vocabulary. OCSF labels like "Informational" / "Medium" collide and get rewritten. For the OCSF string, emit severity_name (or any non-colliding name); for the integer, emit severity_id directly (int, 0–6). Tenant-validated April 2026.
• gron-captured dotted keys are FLAT, not nested. A source JSON key like "user.email": "alice" becomes unmapped.user.email as a single flat field — mappings must reference it with from: "unmapped.user.email" (no escaping). from: "unmapped.user\\.email" does NOT match on current tenants, contrary to the escape-the-dots pattern in ai-siem's PARSER_TEMPLATE.conf.
• Saved parsers apply only to newly ingested events. Historical logs are not re-parsed. So during iteration you must re-ingest the sample after every parser edit.
• A parser=<name>-tagged source with NO /logParsers/<name> file ingests raw (passthrough). If events carry parser='auditd' but dataSource.name / class_uid are null, the named parser file does not exist yet: the collector tags the events but SDL has nothing to apply. Deploy /logParsers/<name> and NEW events get parsed. After deploy, the parser's root attributes (e.g. dataSource.name) are the canary that the parser is RUNNING AT ALL — if they are still null on fresh events the parser has not propagated yet (not a format bug); if they are populated but a captured field is null, that IS a format/mapping bug. metadata.version is the canary for WHICH version is live: bump it on every deploy so you can tell from the live stream when a specific change has propagated (poll | group c=count() by metadata.version until the new version appears). Propagation observed at ~3-5 min on this tenant. Tenant-validated 2026-06-01.
• halt: true stops the format engine after a match. Use it for mutually exclusive formats (e.g., TCP vs UDP variants in pfSense). Without it, all matching formats merge their captures.
• repeat: true reapplies the same format from where it left off. Required for the key/value catch-all idiom, otherwise only the first pair is captured.
• discardAttributes MUST be per-format for scratch fields, NOT parser-root. A discardAttributes: [...] block at the parser root does NOT reliably drop fields created by named captures inside a format string ($_xxx{regex=...}$ scratch markers leak through to the event). Put discardAttributes: ["_pre", "_tail", ...] ON THE FORMAT OBJECT itself, alongside format:. The built-in json parser shows the shape: formats: [ { format: "$json{parse=json}$", repeat: true, discardAttributes: ["message"] } ]. Parser-root scope only works for fields produced by {parse=json} / {parse=dottedJson} flat expansion (vendor noise keys like flipkart_not_secure), not for format captures. Tenant-validated 2026-05-13.
• There is NO from: directive on a format entry. Only id, attributes, format, discard, halt, repeat, and rewrites are honored. A from: "<somefield>" next to format: is silently ignored, and the format then runs against the raw line; a greedy regex like [\s\S]+ will clobber every targeted field with the entire log message on every event. The from: key only exists in mappings ops ({rename: {from, to}}, {copy: {from, to}}). To run a format-like rewrite against a specific captured field, use rewrites: on the same format with input: "<fieldname>", or do the work in a mappings op.
• The parser only sees the message field. Ingest-time attributes (app_id, cluster, region, custom enrichments from collectors / HEC ingest) are NOT visible to format strings, format-level discard: filters, or rewrites regex inputs. They ARE visible to mappings predicates and mappings op from: paths since mappings run after parsing. Encode any sentinel you need to gate a format on into the message body or anchor on a string the message itself contains.
• v1 mappings is FIRST-MATCH-WINS. Mapping entries are tried in declaration order; only the first entry whose predicate matches an event runs. A predicate: "true" catch-all therefore MUST go LAST in the list — placed first, it shadows every per-app/per-class entry below it (renames don't apply, class constants stay generic). Pattern: most specific predicates first, generic catch-all last. If two predicates would both fire on the same event and you want both effects, fold their transformations into a single entry. Tenant-validated April 2026.
• Fragment-format anchors must live INSIDE the prefix's regex, not as separate format-string text. The format $_pre{regex=[\s\S]*}$/actuator/health$_suf{regex=[\s\S]*}$ (greedy [\s\S]* followed by literal text in the format string between the field markers) does NOT match — the engine commits each field's regex independently and won't backtrack across literal-text gaps. Working form: $_pre{regex=[\s\S]*\/actuator\/health}$$_suf{regex=[\s\S]*}$ (literal anchor inside the prefix's regex). Same shape as Phase-10 IOC formats ([\s\S]*X-Forwarded-For=[\[]? baked into the prefix).
• Greedy [\s\S]*ANCHOR lands on the LAST occurrence of ANCHOR. That is the right behaviour when the captured token follows the rightmost anchor on the line (Dropwizard [\s\S]*\[dw-[0-9]+ - is unique to the request-line bracket). It fails when the anchor is too generic: [\s\S]*\[ followed by a UT-[0-9]+ capture fails on any line whose last [ is something else (e.g. [#033[36m...] ANSI brackets). Two fixes: (a) make the anchor multi-character and discriminating ([\s\S]*\[UT-), accepting that the literal is consumed by the anchor and your capture holds only the trailing portion ($user_task_id_num{regex=[0-9]+}$); or (b) use a lazy quantifier ([\s\S]*?) plus a discriminating literal when the FIRST occurrence is the right one.
• Greedy [\\s\\S]*<anchor> collides with COMPOUND field names that contain the anchor as a substring. Anchoring a record-type capture on bare type= ($_pre{regex=[\\s\\S]*type=}$$rectype{regex=[A-Z0-9_]+}$) silently mis-fires on auditd PATH records because nametype=NORMAL (and objtype=) contain type= — the greedy match lands on the LAST type= (the one inside nametype=), captures NORMAL, and the rest of the format then fails. Fix: widen the anchor with the delimiter that precedes the REAL token, e.g. type= with a leading space ($_pre{regex=[\\s\\S]* type=}$), since the real record-type follows a space (audispd: type=PATH) whereas nametype=/objtype= are preceded by a letter. General rule: when your anchor is a short substring that also appears inside longer field names, disambiguate it with a neighbouring delimiter. Tenant-validated 2026-06-01 on a Linux auditd parser: bare type= parsed SYSCALL/EXECVE/PROCTITLE but dropped every PATH record; type= fixed all record types.
• Per-event parse limit is just under 50,000 chars. Long events get truncated before parsing.
• Line groupers max 100,000 chars per joined event. Above that, the grouper still emits but truncated.
• intermittentTimestamps: true is required for logs (like MySQL general query log) that emit a timestamp only on the second they roll over. Without it, every line missing a timestamp gets the ingest time.
• Regex alternation cannot wrap a $...$ token. ($a$|$b$) is invalid. Use multiple line fragments instead.
• SDL regex engine does NOT support lookarounds. (?=...), (?!...), (?<=...), (?<!...) all fail silently. Use explicit token delimiters or split into multiple formats.
• SDL strips the leading .* from a regex before passing it to the Java engine. This means using .*? (non-greedy) at the start of a regex produces an orphaned ? which the engine rejects as "Dangling meta character '?' near index 0". Use plain .* and rely on normal greedy matching. If you need non-greedy behaviour, restructure the capture into two segments (see below).
• Trailing .* at the END of a format string is also stripped (same root cause as leading .*). A format ending in $_body{regex=.*}$ or $process.cmd_line{regex=.*}$ (the field marker is the last thing on the line, regex is exactly .*) silently captures the EMPTY STRING instead of "everything to end of line". The SDL leading-.*-strip rule applies regardless of whether the .* is the entire regex or merely starts it; with the .* stripped, the regex becomes empty and the field defaults to a zero-length capture. Tenant-validated 2026-05-26 on a linuxLogsParser sudo format: COMMAND=$process.cmd_line{regex=.*}$ captured "" (empty), whereas the same format with regex=[\\s\\S]+ captured the full command-line including embedded spaces. Use [\\s\\S]+ (one or more, including newlines) or [\\s\\S]* (zero or more, including newlines) at any field marker where you want "everything that remains on the line" semantics. Both forms avoid the leading-.* strip and survive intact.
• Regex alternation matches in DECLARATION ORDER, not longest-leftmost. regex=CRON|CROND is a bug: CRON is tried first and matches the literal prefix of CROND, leaving D to fail against the next literal in the format string. The fix is to put the LONGER alternative FIRST: regex=CROND|CRON|crond|cron. This applies anywhere alternation appears (named patterns, inline {regex=...}, regex in rewrites). Tenant-validated 2026-05-26: linuxLogsParser cron_cmd format with CRON|CROND matched zero CROND events; same format with CROND|CRON matched every cron variant. General rule: when alternatives share a prefix, sort by length descending.
• [\s\S]*? (lazy) at the start of a prefix regex silently matches almost nothing — same family of bugs as .*?, but no error. A fragment format like $_pre{regex=[\s\S]*?}$$field{regex=PATTERN}$$_suf{regex=[\s\S]*}$ will fire on a tiny fraction of events instead of the expected matches, because the SDL regex engine does not backtrack across field boundaries when the prefix is lazy. The fix is the same shape as the multi-char-anchor rule: use greedy [\s\S]* plus a discriminating literal anchor inside the prefix regex itself (e.g. [\s\S]*\"merchant_transaction_id\":\" to anchor on a JSON key before capturing a CL transaction id). Tenant-validated 2026-05-13. If you actually need first-match semantics, anchor on a literal that only appears once per line (e.g. \[log\., \"url\":\"), do not reach for *?.
• Two-segment sequential capture pattern for JSON values inside a raw string. When a log line is a raw JSON string (with \" escaping) and you need to extract a field value, use a two-segment format with a $$ transition. First segment grabs everything up to and including the opening quote of the value into a scratch field; second segment captures the value up to the closing quote. Example for extracting severity_name before SDL coerces it: $_ps{regex=.*\\\"severity\\\":\\\"}$$severity_name{regex=[^\\\"]*}$. The $$ boundary: the first $ closes seg 1, the second $ opens seg 2. The scratch field (_ps) must be in the format's OWN discardAttributes: ["_ps"] array (per-format, alongside format:) — parser-root discardAttributes does not reliably drop format-capture scratch fields on this tenant. This pattern works for any quoted string value in a JSON-encoded log line.
• $pri{parse=syslogPriority}$ can cause silent event drops on some tenants. If you suspect priority parsing is the culprit, capture the priority as a plain field ($pri{regex=\\d+}$) and skip the parse directive.
• Indexed positional access attr[N] only works in mappings.version: 1. v0 mappings reject attr[0]/attr[1] syntax. If the source is positional CSV (Palo Alto Firewall is the canonical case), use v1. Reference: parsers/sentinelone/marketplace-paloaltonetworksfirewall-latest/.
• {parse=gron} flattens nested JSON to dotted keys, but the keys themselves contain dots. A source {"Resource": {"Id": 5}} becomes the single flat field unmapped.Resource.Id (one key with two dots), not a nested object. To rename it use from: "unmapped.Resource.Id" (no escapes). To rename a whole subtree use rename_tree: { from: "unmapped.Resource", to: "resource" } and the engine walks the dotted-prefix subtree. Reference: parsers/sentinelone/marketplace-cloudflare-latest/.
• attrBlacklist={field1,field2} only filters subfields produced by a {parse=...} directive. It does NOT filter top-level captured fields you named explicitly in the format string. To drop a top-level field use discardAttributes: ["fieldname"] at the parser root. Reference: parsers/sentinelone/marketplace-fortinetfortigate-latest/.
• {parse=...} subfield naming is <prefix><CapitalizedFirstLetterOfKey><restOfKey> (per SDL KB 000006743). Field details parsing ?apple=w&banana=x yields detailsApple and detailsBanana, NOT details_apple / detailsQuery_apple / details.apple. Same convention for {parse=json} (camelCase): details parsing {"foo":"hello"} produces detailsFoo, not details_foo. Confirm the exact field name with a sample query if unsure — every "obvious" alternative spelling fails silently.
• {parse=uri} and {parse=uriAttributes} consume the parent capture instead of preserving it. After parsing, the original captured value is gone from the event; only the generated subfields remain (validated against current tenant April 2026). If you also need the raw URL/query string, capture it twice: once with the parse directive, once into a sibling scratch field, and rename the scratch one in mappings. Or split path/query directly in the format: $path{regex=[^? ]+}$\??$qs{regex=[^ ]*}{parse=uriAttributes}$.
• {parse=uri} on relative URLs (no scheme/host) silently drops the query subfields on tested tenants. It still emits <prefix>Path, but <prefix><Key> siblings for query parameters never materialize. Use {parse=uriAttributes} on the query-string portion (split manually with regex) instead — it works regardless of whether the URL is absolute.
• skipNumericConversion: true is required when columns contain numeric strings you want preserved as strings. Without it, commaSeparatedvalues and pipeSeparatedValues parse will coerce "00123" → 123 (loses leading zeros), "0xDEADBEEF" → 3735928559, and "1.0" → 1. Set skipNumericConversion: true on the format whenever any column is an opaque ID or a hex/octal string. Canonical: Palo Alto Firewall.
• intermittentTimestamps: true is the only fix for sources where only the first record on each second has a timestamp. Common with MySQL general query log, AWS RDS, sometimes VPC flow logs. Without it, every line that lacks a timestamp gets the ingest timestamp, not the inferred one. Reference: parsers/sentinelone/marketplace-awsvpcflowlogs-latest/ and parsers/community/sql_database_logs-latest/.
• enum cast in v1 mappings without enum_default leaves unmapped values UNCHANGED, not null. If you want unknowns to map to a known sentinel, set enum_default: <value> on the cast. Cloudflare uses enum_default: 99 ("Unknown") consistently — copy that pattern.
• severity_id is an integer 0–6 reserved by ingest, just like severity. Setting severity_id: 999 silently fails. Use severity for the normalized integer (preferred), severity_name for the OCSF label string. Do not emit BOTH severity and severity_id from the same parser.
• Rewrites run during format parsing; mappings run after all formats. A field renamed in mappings.rename cannot be referenced by a rewrites: block on a later format on the same line because rewrites: runs per-format during capture. If you need a derived field from a renamed one, do the derivation inside mappings (use copy then cast, or replace).
• Mixing mappings.version: 0 and mappings.version: 1 syntax in the same block is a hard error. The engine rejects with "expected list got object" or "unsupported event mapper version -1". Pick one version per parser and stick to it.
• Windows Event Log XML payloads ship with double-escaped tabs/newlines. A parser for Windows EventLog in the catalog (parsers/community/microsoft_windows_eventlog-latest/) deals with \\t and \\n literals embedded inside XML strings; capturing them needs \\\\t / \\\\n (four backslashes) at the augmented-JSON layer.
• Some catalog parsers ship without dataSource.category at all (~33% of community). When you copy from Sentinel-One/ai-siem, audit the top-level attributes: block. If dataSource.category is missing, add it as "security". If present with another value, change it to "security". Same audit for dataSource.name and dataSource.vendor.
• $_=identifier$=$_=quoteOrSpace$ repeating idiom REQUIRES a leading .* AND repeat: true. This is the canonical built-in keyValue parser shape and it is documented in the public KB as ".*$_=identifier$=$_=quoteOrSpace$". Without the leading .*, the format anchors at position 0 of the message and fires zero times on any line that doesn't start with an identifier — including every syslog log (<189>date=...), CEF (CEF:0|...), Windows Event (<Event xmlns=...>), or anything with a leading timestamp/priority. The format extracts nothing in that case, silently. The .* lets the engine scan forward to the first key= and then repeat: true walks the remaining pairs from there. Tenant-validated 2026-05-26 on FortiGate syslog (<189>date=2026-05-25 time=18:07:18 devname="..." ...): without .* zero fields captured; with .* all 30+ key=value pairs captured. The keyValue built-in parser uses exactly this shape (see examples/03-key-value.json). If you also need a separate static-prefix capture (e.g. to extract a syslog priority), put it in a SIBLING format that runs before the catch-all — each format runs against the full original message independently, so the prefix format and the .*-anchored KV format don't compete.
• Conditional class assignment (one parser, multiple OCSF classes) is best done by stamping class_uid/class_name on each format's attributes: block, not on the parser root. The root attributes are baseline; per-format attributes override. Reference for fan-out via format-id sentinel + mapping predicates: parsers/sentinelone/marketplace-awsrdslogs-latest/.
• Same-name rename in v1 mappings silently NULLS the field. Writing { rename: { from: "action", to: "action" } } to "keep" a captured raw field under its own name actually deletes the value. The KV catch-all extracts action (or any other raw vendor field) directly into the event under its native name — if you want to keep it, OMIT the rename. Only write a rename when the target name is different from the source. Tenant-validated 2026-05-26: identical FortiGate action="perf-stats" event returned action=null with the self-rename present, action="perf-stats" after removing the self-rename. This bites you when "promoting" raw fields that the catch-all already captures.
• raw_data JSON blob is duplicate storage. Many upstream Observo / collector extractions leave the entire raw event in a raw_data field alongside the typed-OCSF fields they extract. If your parser is also extracting those same fields via KV catch-all or named captures, the raw_data field is redundant and storage-wasteful. Either discardAttributes: ["raw_data"] at parser root or strip upstream. Don't store the same key=value pair three times (typed + unmapped + raw_data) — this was the Netskope anti-pattern. Audit on copy.
• Unquoted numerics captured by quoteOrSpace are auto-typed as NUMBER columns. No explicit cast required. A FortiGate srcport=44321 dstport=443 sentbyte=8421 line going through .*$_=identifier$=$_=quoteOrSpace$ produces srcport: 44321 (number), dstport: 443 (number), sentbyte: 8421 (number) — all queryable with avg(), sum(), > 100, etc. without number() wrapping. The auto-inference fires when the captured token is purely numeric AND unquoted in the source. Quoted values (level="notice") stay as strings even if they look numeric. Tenant-validated 2026-05-26 on FortiGate. Caveat: SDL's column type-locking still applies — if a column was ingested as string earlier, new numeric writes coerce back to string and avg() returns NaN. The CLAUDE.md project rule "wrap with number() before arithmetic" is still the bulletproof pattern for dashboard panels and STAR rule bodies, especially for counter columns (traffic.bytes_*, traffic.packets_*, duration).
• message field is reserved and preserved on every event regardless of parser activity. SDL never lets a parser overwrite message; the raw event body always survives. Practical consequence: dashboards and PowerQuery hunts that use parse "fieldname=$x$ " from message continue to work even after a parser renames the typed columns (srcip → src_endpoint.ip). This is the backward-compat escape hatch during parser rollout — old events still parse via message-text extraction, new events parse via typed columns. To migrate a dashboard from parse-from-message to typed columns without losing coverage on legacy events, use (typed_field == * OR raw_field == *) or coalesce(typed_field, raw_field) patterns. Tenant-validated 2026-05-26 on FortiGate dashboard rewrite: panels that switched to src_endpoint.ip continued returning the same row counts because the upstream Observo path was already producing typed src_endpoint.ip too; panels using tls.sni only show fortigateParser-tagged events because Observo didn't extract SNI.
• When two ingest paths produce OCSF from the same source, they may flatten event.type / event.subtype differently. Common pattern: upstream Observo collector flattens FortiGate type=traffic subtype=forward into a single event.type='forward' (drops the parent type). The downstream parser keeps both: event.type='traffic', event.subtype='forward'. Same source, two schema shapes coexisting on different events. Dashboards and hunts that filter on subtype must use OR: (event.subtype='forward' OR event.type='forward') — this matches both legacy and parser-tagged events. The rule: every time you migrate a downstream consumer to typed OCSF columns, audit each subtype filter for backward compat with the upstream shape during the rollover window.
• parser='<name>' field may only be tagged for very recent events. Some tenants are configured to stamp the parser field only on events ingested in the last N minutes (e.g., last 30 minutes). Queries filtering on parser='myparser' against a 24h window return only that recent slice, not 24h of data. For broad coverage validation, filter on dataSource.name + metadata.version (which IS retained per-event) instead, and use parser= only for isolating a specific recent ingest test. Tenant-validated 2026-05-26.

Ingest-path gotchas (discovered empirically via live validation)

These are quirks of the ingest pipeline itself, not the parser DSL. They bite you at validation time even when the parser JSON is syntactically perfect.

• HEC ingest splits on \n BEFORE the parser runs. Multi-line events (stack traces, MySQL slow queries) get chopped up and lineGroupers cannot re-fold them. Workarounds:
  1. Fold multi-line events client-side into a single line with a sentinel character (e.g. \x1e), then have your parser split on that sentinel.
  2. Send each logical event as a single HEC event so newlines inside the event body are preserved.
• The server-host upload header is unreliable for isolating a test. SDL sometimes overrides the header to the literal source name, and if the parser extracts a host field from the log itself that wins too. Do NOT filter your validation query by host='parser-test-<uuid>' alone. Safer: filter by parser='claude_test_<name>' and _bytes > 0, and use a unique nonce in the payload to double-check isolation.
• {parse=dottedJson} prefixes subfields when the field has a non-empty name. $payload{parse=dottedJson}$ on {"user":"alice"} yields payload.user = "alice". If you want top-level fields, capture into an empty name: ${parse=dottedJson}$. Same applies to json, escapedJson, urlEncodedJson, base64EncodedJson.
• getFile on a missing config path raises HTTP 404, it does not return success=false with noSuchFile. Wrap existence checks in a try/except when scripting cleanup.

mappings block gotchas

Two syntaxes coexist on live tenants and both work. You must commit to one per parser — mixing them in the same mappings block produces confusing "expected list got object" errors.

v1 (singular, preferred for new parsers, tenant-validated April 2026):

• mappings.version: 1 is mandatory. Without it the tenant returns unsupported event mapper version -1.
• Each transformation is {<op>: {...body}}, where the op name is the outer key (e.g. {rename: {from, to}}). The public-doc form {op: "rename", from: ..., to: ...} is rejected.
• rename.from is a single string, not a list. Use copy (which does accept a list) if you need "first of N".
• cast uses type:, not to:. Example: {cast: {field: "x", type: "int"}}.
• cast overwrites the source field. There is no output: parameter — copy first, then cast the copy.
• Predicates use == (double equals).

v0 (plural-grouped, used by most S1 Marketplace parsers — AWS RDS, Corelight, Cloudflare, FortiGate):

• mappings.version: 0. Same block, different body shape.
• Ops are plural arrays on the mapping entry: renames: [...], copies: [...], constants: [...]. No transformations: wrapper.
• Every op entry uses inputs: [list] + output: + type: — even rename. type: "string" is the safe default.
• Predicates use = (single equals) in v0, both on the mapping entry and inside constants.
• WARNING: v0 constants: block is rejected on the current S1 tenant. putFile returns 400: Syntax error at position -1: Assertion failed: Expected a field when a constants: array is present, even with the documented body shape [{output, value, type}]. v0 renames: and copies: work fine; only constants: fails. Tenant-validated 2026-05-26: identical mapping block with constants removed deploys cleanly, with constants present is rejected. If you need conditional class_uid / activity_id fan-out from one parser to multiple OCSF classes, use v1 — the v1 { constant: { field, value } } op works reliably. This means: prefer v1 for any new parser that needs OCSF class fan-out. Stay on v0 only if you're extending an existing v0 parser that doesn't add new constants.

If you inherit a marketplace parser, keep it on v0 rather than rewriting (unless you need to add constants — then migrate to v1). For new parsers, pick v1. See references/mappers.md for the full side-by-side.

constant / constants op (both versions) is the workhorse for conditional OCSF class/activity assignment — give it a predicate and it only fires when matched, so you can fan one vendor-native state code (conn_state, action, etc.) out into activity_id + activity_name + type_uid triples.

Inline predicate: on individual constant ops is the cleanest escape hatch from first-match-wins in v1. When you want a long list of unconditional renames/copies AND a handful of conditional constant writes (severity bucketing, OCSF enum derivation from a vendor string, action/status/disposition fan-out), DO NOT split them across multiple top-level mapping entries — only the first matching entry fires. Instead, put EVERYTHING in one predicate: "true" entry and attach a predicate: field directly to each conditional constant op:

mappings: {
  version: 1,
  mappings: [{
    predicate: "true",
    transformations: [
      // ... unconditional renames / copies / casts ...
      { copy: { from: "raw_severity", to: "vendor.severity_raw" } },

      // Conditional constants — predicate lives ON THE OP, not the mapping entry.
      { constant: { field: "severity_id", value: 1, predicate: "vendor.severity_raw >= 0 && vendor.severity_raw <= 3"  } },
      { constant: { field: "severity_id", value: 3, predicate: "vendor.severity_raw >= 4 && vendor.severity_raw <= 6"  } },
      { constant: { field: "severity_id", value: 4, predicate: "vendor.severity_raw >= 7 && vendor.severity_raw <= 8"  } },
      { constant: { field: "severity_id", value: 5, predicate: "vendor.severity_raw >= 9 && vendor.severity_raw <= 10" } },

      { constant: { field: "action",    value: "Allowed", predicate: "raw_action == 'alert' || raw_action == 'allow'" } },
      { constant: { field: "action_id", value: 1,         predicate: "raw_action == 'alert' || raw_action == 'allow'" } },
      // ... etc ...
    ]
  }]
}

This pattern preserves first-match-wins semantics at the mapping-entry level (which can't be worked around) while still letting you express N independent conditional writes without duplicating the rename list across N per-subtype blocks. Use it whenever you have a flat schema (one event shape, many conditional derivations) — severity bucketing, action/status/disposition fan-out from a vendor enum, HTTP-status → OCSF status_id, and so on. The alternative — splitting into per-subtype top-level mapping entries — only makes sense when the per-subtype rename lists are genuinely different (different OCSF classes per event subtype). Tenant-validated May 2026 on a CEF source with ~20 conditional ops: all fired from a single predicate: "true" mapping entry; the same logic split across multiple top-level entries produced zero hits on every conditional write because only the first entry ran. See examples/11-*-cef-ocsf.json for the runnable reference.

Predicate type-mismatch silently fails on captured numerics. The KV catch-all auto-types unquoted values as NUMBER (pitfall #5 in Critical pitfalls). That means a predicate like raw_severity == '5' — quoted string — never matches a raw_severity captured as integer 5. Use bare numerics in predicates for numeric fields (raw_severity >= 4, http_response.code < 400) and quoted strings only for actually-string fields (raw_action == 'alert', format-id sentinels mySqlErrorLog == 'true'). When in doubt, run a quick query on a recent parsed event and check cellType in the response — STRING vs NUMBER — before writing the predicate. A string predicate against a numeric field silently drops every dependent op; the only symptom is the target field being null on every event, which can take several deploy/iterate cycles to localize. Tenant-validated May 2026.

OCSF severity_id is an enum {0,1,2,3,4,5,6,99}, NOT a vendor passthrough. Many sources (CEF 0-10, syslog 0-7, custom 0-100) ship a wider-range numeric severity. Renaming the raw value directly into severity_id produces invalid OCSF (e.g. severity_id=10 when a vendor uses 0-10) AND semantically false detections (vendor "5 out of 10" means Medium for that vendor, but severity_id=5 is OCSF "Critical" — every routine event suddenly looks Critical to downstream consumers). Always bucket via per-op predicates and preserve the raw value under a vendor namespace (vendor.severity_raw, cef.severity) for source fidelity. Reference bucketing for a 0-10 scale: 0-3 → 1 Informational, 4-6 → 3 Medium, 7-8 → 4 High, 9-10 → 5 Critical. For 0-7 syslog: 0-1 → 5 Critical, 2-3 → 4 High, 4 → 3 Medium, 5-7 → 1 Informational. Same severity (no suffix) is RESERVED and coerced to 0-6 by ingest, so emit severity_id (the integer) and optionally severity_name (a label string) — never both severity and severity_id from the same parser. Tenant-validated May 2026: a parser that passed CEF 0-10 directly into severity_id emitted invalid OCSF values (severity_id=10) AND tagged the bulk of routine traffic as Critical; the bucketed replacement produced semantically correct severity_ids for the same input stream.

Format-id-as-sentinel predicate: format: { id: "mySqlErrorLog", ... } auto-sets a boolean-string field mySqlErrorLog='true' on events that matched that format. Use predicate: "mySqlErrorLog='true'" (v0) or "mySqlErrorLog == 'true'" (v1) to fan mapping entries out to sub-shapes. Quote the 'true' — it's a string, not a bare boolean.

mappings.version: 1 uses FIRST-MATCH-WINS, not all-match. The first mapping block whose predicate matches consumes the event — no subsequent blocks run. This is the single most important structural constraint when using v1. Consequences:

• A predicate: "true" unconditional block placed FIRST will fire on every event and prevent all per-app conditional blocks from ever executing. Always place the catch-all at the END.
• drop / drop_tree transformations cannot be factored into a standalone unconditional block at the start and shared across per-app blocks. They must be duplicated into each per-app block. Verbose but correct.
• rename ops have the SAME problem as drop. If you have a catch-all rename block at the end (typed-OCSF migration: srcip → src_endpoint.ip, devname → device.name, etc.) AND per-subtype constant blocks earlier (for activity_id fan-out), the catch-all rename NEVER fires for events that matched a per-subtype block — only the catch-all events get renamed. The per-subtype events keep raw vendor field names. To make the parser produce typed OCSF on ALL events, you MUST duplicate the rename list into every per-subtype block alongside its constants. Tenant-validated 2026-05-26 on FortiGate v1.1.0: with renames only in catch-all, event.subtype='forward' events had activity_id=6 from the constant block but src_endpoint.ip=null because the rename never ran; with renames duplicated into the per-subtype block, all fields populated. Pattern: each per-subtype block = its own constants + the full rename list. Verbose but correct.
• v0 mappings DO run all matching blocks (marketplace parsers like Cloudflare and FortiGate rely on this: an unconditional first block handles common renames, then conditional blocks handle sub-types). If you need multiple pass-through effects, consider whether v0 is a better fit — but note v0 lacks drop/drop_tree AND v0's constants: is currently rejected on this tenant (see WARNING above), so v0 is only viable if you don't need either.

discardAttributes MUST be placed per-format, not at parser root, for scratch fields created by format captures. The parser-root discardAttributes: [...] block does NOT reliably drop fields that named-capture markers $_xxx{...}$ produced inside a format string. Scratch fields like _dw_pre, _dw_tail, _lc_pre, _lc_suf, etc. leak through to the final event despite being listed at the root. The reliable form is to put discardAttributes: ["_dw_pre", "_dw_tail", ...] ON THE FORMAT OBJECT ITSELF, as a sibling key to format:. This matches the built-in json parser's shape: formats: [ { format: "$json{parse=json}$", repeat: true, discardAttributes: ["message"] } ]. Tenant-validated 2026-05-13. Parser-root discardAttributes can still be used for noise fields produced by {parse=json} / {parse=dottedJson} flat-expansion of vendor JSON keys (e.g. flipkart_not_secure, upstream_addr) — those are not format captures and do get dropped from the root.

discardAttributes also does NOT remove fields produced by {parse=gron} at the top level. If {parse=gron} extracts host, facility, sndr, etc. directly to the event root, adding them to discardAttributes (root OR per-format) has zero effect. Use mappings.drop / mappings.drop_tree for those instead (see the next two gotchas). Tenant-validated May 2026.

attrBlacklist on a bare {parse=gron} format cannot drop top-level fields. The attrBlacklist directive only filters subfields when a NAMED PREFIX is in use — e.g. $payload{parse=gron}{attrBlacklist=debug}$ drops payload.debug. Without a named prefix (${parse=gron}{attrBlacklist=host,sndr}$), the blacklist is ignored for top-level keys because there is no prefix namespace to filter within. This means {parse=gron} with attrBlacklist cannot be used to suppress noise fields at the root level. Tenant-validated May 2026.

The correct mechanism for dropping top-level gron-produced fields is mappings.drop / mappings.drop_tree. Add { drop: { field: "host" } } (scalars) and { drop_tree: { field: "object" } } (subtrees) in the transformations of every applicable mapping block. Because v1 is first-match-wins, these drop ops must be duplicated into each per-app block AND in the catch-all block — there is no shortcut.

computeFields gotchas

• Use PowerQuery ternary a ? b : c, not if(a, b, c). The latter returns Unknown function 'if'.
• Use == for equality, not =.
• Cannot reference dashed field names. protocol-id is parsed as subtraction. Rename via mappings first, or do the translation entirely in the mappings block.
• Place the rewrite on the format that captures the source field. If a repeating key/value sub-format is what produces the field, the rewrite belongs on the sub-format — not on the frame format above it.
• The replace MAPPER op is a runtime no-op on this tenant; use a computeFields + PowerQuery replace() rewrite instead. Two traps with { replace: {...} } in a mappings block: (1) the regex key is regexp, NOT pattern — pattern returns 400: Missing required key 'regexp' on putFile (the mappers.md example showing pattern is wrong); (2) even with the correct regexp key the op silently does NOT transform the value at ingest (validated on both a dotted target process.cmd_line and a flat scratch field, 2026-06-01). The working substitute is a computeFields rewrite calling the PowerQuery replace(field, regex, replacement) string function. PowerQuery string literals are SINGLE-quoted, so a regex containing double-quotes needs no escaping inside the expression: expression: "| let _cmdline = replace(_cmdline, 'a[0-9]+=\"([^\"]*)\"', '$1')". Capture into a FLAT scratch field (computeFields and replace address flat names cleanly; dotted paths are unreliable), then rename the scratch field to the dotted OCSF target in mappings. This is how the auditd parser turns an EXECVE arg vector a0="cp" a1="/x" a2="/y" into a readable process.cmd_line = cp /x /y.
• There is NO hex / base16 decode primitive anywhere in the DSL or in computeFields. The {parse=...} directives offer base64EncodedJson / urlEncodedJson but no hex; the PowerQuery subset available to computeFields has no position-aware hex decoder either. auditd proctitle (and individual hex-encoded EXECVE args) are NUL-separated hex and cannot be turned into ASCII in-parser. Source the readable command line from the EXECVE arg vector instead (auditd stores those as plain ASCII), and leave the hex proctitle as-is. Do not burn iterations trying replace/regex to hex-decode.

Interaction between halt: true and repeat formats

• halt: true stops ALL further format matching on that line — including repeating key/value catch-alls. If you have a two-format pattern (frame + repeating extractor), do NOT put halt: true on the frame format; the extractor will never fire.

Validation (mandatory)

Always validate against the live tenant via the sdl-api skill. Do not rely solely on the syntactic plausibility of the JSON — the only authoritative test is "did the field actually appear in a query after ingest." This is what the in-console Test Parser button approximates client-side; doing it through the API exercises the real ingest pipeline.

import sys, os, time, uuid, json
_sdl_scripts = os.environ.get("SDL_API_SCRIPTS") or os.path.normpath(
    os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "sdl-api", "scripts")
)
sys.path.insert(0, _sdl_scripts)
from sdl_client import SDLClient

c = SDLClient()
PARSER_NAME = "claude_test_<descriptive_name>"   # use a claude_test_ prefix so cleanup is easy
parser_body = open("/path/to/draft.json").read()
sample = open("/path/to/sample.log").read()

# 1. Deploy
c.put_file(f"/logParsers/{PARSER_NAME}", content=parser_body)

# 2. Ingest a sample with a unique nonce + host so we can isolate it
host_tag = f"parser-test-{uuid.uuid4().hex[:8]}"
hec_ingest(content=sample, parser=PARSER_NAME, server_host=host_tag)  # HEC ingest applies the parser; replaces the removed uploadLogs

# 3. Wait briefly, then query back
time.sleep(8)  # ingest-to-search latency
res = c.power_query(
    query=f"host='{host_tag}' | columns timestamp, message, <expected_fields>",
    start_time="5m",
)
print(json.dumps(res, indent=2))

A successful validation means: (a) putFile returned success, (b) HEC ingest returned success, (c) the power_query returned at least as many rows as you ingested, (d) the expected fields are populated and not null.

If a field is missing, do NOT just retry — diagnose. Common causes: wrong escape level on a regex, a delimiter that didn't match (the format silently fails to apply), halt: true on an earlier format catching the line first, or discardAttributes dropping the field by mistake.

Cleanup

After validation, decide with the user whether to keep, rename, or delete the parser. For throwaway tests:

c.put_file(f"/logParsers/{PARSER_NAME}", delete=True)

For something the user wants to keep, rename to a non-claude_test_ name (get_file → put_file new name → delete old).

Bundled references

When a question goes deeper than this file, read the relevant reference. Each is a focused deep-dive — load only what you need.

• references/ai-siem-catalog.md — Check this first before writing a new parser. Map of the public S1 parser repo (~150 parsers across community + marketplace), per-shape template recommendations, style-variance cheat sheet, and the fetch recipe.
• references/ocsf-mapping.md — Then start here for any new parser. OCSF class quick-pick (4001/4002/3002/1001/etc.), field-to-dotted-path mapping tables for Network Activity, Authentication, and File Activity, and the two authoring idioms: capture directly into dotted names vs capture vendor-native then rename via a mappings block. Also covers boolean/enum normalization (YES/NO/UNKNOWN → tls.is_encrypted). Points at ocsf-schema-documentation.md for authoritative field names.
• references/ocsf-schema-documentation.md — Authoritative OCSF field catalog — the full SentinelOne community-documented schema: 7 categories × 96 articles × ~25,759 field entries across every OCSF event class (System Activity, Findings, IAM, Network Activity, Discovery, Application Activity, Unified Alert Management). Grep this file for any OCSF field name you're about to emit — don't invent names. Always check this before choosing a dotted path, especially for classes beyond Network/Auth/File.
• references/syntax.md — Full augmented-JSON parser syntax: formats, patterns, attributes, lineGroupers, rewrites (including computeFields and timeDelta), special fields (timestamp/severity/message), discard, halt, repeat, association, intermittentTimestamps, skipNumericConversion. Read when authoring anything beyond a simple line format.
• references/parse-directives.md — Every {parse=...} sub-parser (json/dottedJson/escaped/urlEncoded/base64Encoded variants, strict variants for arrays, uri/uriMultivalue/uriAttributes, commaKeyValues, commaSeparated/pipeSeparated, sqlToSignature, syslogPriority, dateTime{Seconds,Ms,Ns}, hoursMinutesSeconds, seconds/milliseconds, bytes/kb/mb/gb, plus per-directive attrWhitelist/attrBlacklist rules). Read whenever the body of a field is itself structured.
• references/builtin-parsers.md — Catalog of all 16 built-in parsers (accessLog, cloudfront, json, dottedJson, dottedEscapedJson, elb-access, heroku-logplex, keyValue, leveldbLog, mysqlGeneralQueryLog, mysqlSlowQueryLog, postgresLog, redshift, s3_bucket_access, spot_instance_data, systemLog) and when to alias vs override. Read first when sizing up a new log source — you may not need to write a parser at all.
• references/mappers.md — mappings block (gron-style transformations: cast, copy, copy_tree, drop, drop_tree, hash, reduce_array, rename, rename_tree, replace, zip), array index syntax, predicate semantics. Read when restructuring events to OCSF or another target schema.
• references/testing-workflow.md — Detailed validation recipe with the sdl-api skill, including how to scope queries with a unique host tag, how to interpret common error responses, and how to clean up.

Bundled examples

Annotated, runnable parser definitions. Copy and adapt rather than starting from scratch.

• examples/01-cef-over-syslog.json — Syslog-framed CEF (<PRI>timestamp host CEF:0|...|...|key=value). Demonstrates: syslogPriority parse, named timestamp pattern, positional pipe-delimited CEF header, key/value catch-all on the extension.
• examples/02-json-with-envelope.json — <timestamp> <host> {"...json..."}. Demonstrates: timestamp + host extraction, {parse=dottedJson} on the body, discardAttributes: ["message"].
• examples/03-key-value.json — Pure key=value lines. Demonstrates: the $_=identifier$=$_=quoteOrSpace$ repeating idiom and a leading static prefix.
• examples/04-multiline-stack.json — Java/Python stack traces. Demonstrates: lineGroupers with continueThrough, attribute tagging.
• examples/05-rewrite-and-mask.json — Single line format plus rewrites to mask password=... values and compute a derived field via computeFields.
• examples/06-alias.json — One-line alias parser ({ aliasTo: "json" }) for the case where a built-in already does the job.
• examples/07-juniper-srx-rtflow-ocsf.json — Juniper SRX RT_FLOW_SESSION_CREATE (structured-data syslog, RFC 5424-ish). Demonstrates: the vendor-native capture + mappings.rename to OCSF idiom end-to-end (source-address → src_endpoint.ip, nat-source-address → connection_info.src_translated.ip, etc.), repeating key/value catch-all on the structured-data body, computeFields for protocol-id → connection_info.protocol_name and encrypted YES/NO → tls.is_encrypted, Network Activity 4001 class tagging.
• examples/08-gron-capture-template.json — The PARSER_TEMPLATE shape lifted from ai-siem: capture the entire event with $unmapped.{parse=gron}$, then do every rename/copy/cast/constant in a single v1 mappings block. Ideal scaffold for JSON-shaped sources when you want all OCSF work in one place.
• examples/12-linux-auditd-ocsf.json — Linux auditd over syslog (<pri> ts audispd: type=<REC> msg=audit(<epoch>:<id>): k=v ...). Demonstrates: a type=-anchored record-type + audit-event-id capture (with the leading space that avoids the nametype= collision), the KV catch-all, v1 first-match-wins mappings fanning the record type out to OCSF classes (SYSCALL/EXECVE/PROCTITLE → Process 1007, PATH/CWD → File 1001, USER_/CRED_ → Authentication 3002, ANOM_*/AVC/SECCOMP → Detection Finding 2004, SOCKADDR → Network 4001), per-op predicate status/severity derivation, and cleaning the EXECVE arg vector into a readable process.cmd_line via a computeFields PowerQuery replace() (the replace mapper op is a no-op and there is no hex decode for proctitle).
• examples/README.md — When to pick which template.

Summary of the contract you're holding

When a user pastes a log and asks you to parse it, you owe them: (1) a parser file written against the SDL augmented-JSON DSL, (2) proof it actually extracts the right fields when ingested through the live SDL pipeline (not just the editor preview), (3) a clean handoff (final path, fields, sample query). The bundled references and examples exist so you don't have to hold the entire DSL in your head — load them as needed and reach for the templates first.

Parser deployment via s1-secops-mcp

Parser deployment and validation use the s1-secops-mcp MCP tools, which bypass the Cowork sandbox proxy entirely. Use sdl_put_file, sdl_get_file, sdl_list_files, and hec_ingest directly instead of falling back to the sdl-api skill scripts. The MCP tools run locally on your machine and make direct HTTPS calls to *.sentinelone.net without proxy interference.

Per-app sentinel pattern (multi-tenant / multi-service parsers)

Use this pattern when a single parser handles events from many distinct services or applications, each needing its own OCSF class assignment.

Pattern overview

1. Extract a discriminator field (e.g. app_name from a raw JSON app_id key) using a two-segment capture format.
2. Create one format-id sentinel per service: { id: "my_app", format: "$_scratch{regex=.*\"app_id\":\"my-app-id\"}$" }. This sets my_app='true' on matching events.
3. List all sentinel field names in discardAttributes so they don't appear in the output event.
4. Add one v1 mappings block per sentinel with the OCSF constants and any drop ops. Because v1 is first-match-wins, drops cannot be factored into a shared block — duplicate them in every block including the catch-all.