grammar-sync-update

# Grammar Sync Update ## When to use After the automated grammar sync bot merges a PR, this skill covers two scenarios: **Scenario A — New command** (7 files to change): A brand-new `xxxCommand` rule appeared in the grammar with no handler anywhere in the TypeScript layer. **Scenario B — Existing command update** (1–3 files to change): An existing command's grammar rule gained a new option, new labeled field, or new sub-rule (e.g. `CHANGE_POINT` gained `BY`, `WHERE IN` gained subquery support). **You do NOT need this skill when:** - The grammar sync PR only updated ANTLR-generated files with no new grammar rules (`.ts`, `.interp`, `.tokens` only) — those are handled automatically. **You DO need this skill when:** - A new `xxxCommand` rule appears in `packages/esql/src/parser/antlr/esql_parser.g4` with no corresponding `fromXxxCommand()` in `cst_to_ast_converter.ts`. → **Use Scenario A below.** - The `esql_parser_listener.ts` has new `enterXxxCommand` / `exitXxxCommand` entries with no converter handler. → **Use Scenario A below.** - An existing command's grammar rule changed (new context method, new labeled alternative, new keyword option) and the converter no longer covers it. → **Use Scenario B below.** ## How to detect what changed ```bash # 1. See the full grammar diff git diff HEAD~1 -- packages/esql/src/parser/antlr/esql_parser.g4 # 2. Identify new command rules (Scenario A) git diff HEAD~1 -- packages/esql/src/parser/antlr/esql_parser.g4 | grep "^+.*[Cc]ommand\b" # 3. Cross-check: listener entries without a converter handler (Scenario A) grep "enter.*Command" packages/esql/src/parser/antlr/esql_parser_listener.ts | \ sed 's/.*enter\([A-Za-z]*\)Command.*/\1/' | \ while read cmd; do grep -q "from${cmd}Command" packages/esql/src/parser/core/cst_to_ast_converter.ts || echo "MISSING: $cmd" done # 4. Identify changed rules for existing commands (Scenario B) # Look for new labeled alternatives (_newField), new keyword tokens (BY, WITH, AS), # or new sub-rule methods (ctx.newSubRule()) added to an existing command rule. git diff HEAD~1 -- packages/esql/src/parser/antlr/esql_parser.g4 | grep "^[+-]" | grep -v "^---\|^+++" ``` If step 2/3 shows an entirely new command → **Scenario A**. If step 4 shows additions inside an existing command rule → **Scenario B**. --- ## Files to change (all 7, in order) ### 1. `packages/esql/src/types.ts` **Add a typed interface** if the command has named fields (target field, inference ID, optional config, etc.): ```ts export interface ESQLAstXxxCommand extends ESQLCommand<'xxx'> { targetField: ESQLColumn; // if there's a <qualifiedName> = <expr> pattern expression?: ESQLAstExpression; // the RHS namedParameters?: ESQLMap; // if there's a WITH { map } clause } ``` Add it to the `ESQLAstCommand` union (search for `ESQLAstCommand` type definition): ```ts export type ESQLAstCommand = | ESQLCommand | ... | ESQLAstXxxCommand; // ← add here ``` If the command is trivial (only generic `args`, no named fields), use `ESQLCommand<'xxx'>` directly and skip the interface. --- ### 2. `packages/esql/src/ast/visitor/contexts.ts` Add a visitor context class before the `// Expressions` comment (search for that comment to find the spot): ```ts // XXX <qualifiedName> = <primaryExpression> [WITH <map>] export class XxxCommandVisitorContext< Methods extends VisitorMethods = VisitorMethods, Data extends SharedData = SharedData, > extends CommandVisitorContext<Methods, Data, ESQLAstXxxCommand> {} ``` Also add the import for `ESQLAstXxxCommand` at the top of the file. --- ### 3. `packages/esql/src/ast/visitor/types.ts` Three additions. Search for `agent-marker` comments to find the exact insertion points: **A — `CommandVisitorInput` union** (search for `agent-marker: append new VisitorInput entries here`): ```ts VisitorInput<Methods, 'visitXxxCommand'> & ``` **B — `CommandVisitorOutput` union** (search for `agent-marker: append new VisitorOutput entries here`): ```ts | VisitorOutput<Methods, 'visitXxxCommand'> ``` **C — `VisitorMethods` interface** (search for `visitUserAgentCommand?` — the last method — and append after it): ```ts visitXxxCommand?: Visitor<contexts.XxxCommandVisitorContext<Visitors, Data>, any, any>; ``` --- ### 4. `packages/esql/src/ast/visitor/global_visitor_context.ts` Two additions: **A — dispatcher `switch` case** (search for `visitCommandSpecific` — add a new `case` inside the `switch`): ```ts case 'xxx': { if (!this.methods.visitXxxCommand) break; return this.visitXxxCommand( parent, commandNode as ESQLAstXxxCommand, input as any ); } ``` **B — public visitor method** (search for `visitUserAgentCommand` — the last public visitor method — and append after it): ```ts public visitXxxCommand( parent: contexts.VisitorContext | null, node: ESQLAstXxxCommand, input: types.VisitorInput<Methods, 'visitXxxCommand'> ): types.VisitorOutput<Methods, 'visitXxxCommand'> { const context = new contexts.XxxCommandVisitorContext(this, node, parent); return this.visitWithSpecificContext('visitXxxCommand', context, input); } ``` Add the import for `ESQLAstXxxCommand` at the top of the file. --- ### 5. `packages/esql/src/parser/core/cst_to_ast_converter.ts` Before writing `fromXxxCommand`, grep `cst_to_ast_converter.ts` for an existing private helper. If the grammar shape matches, extend the helper and use a one-liner — don't copy another command's method body. | Grammar shape | Existing helper | |---|---| | `qualifiedName = expr` [+ `commandNamedParameters`] | `fromQualifiedNameAssignmentCommand` (`uri_parts`, `registered_domain`, `user_agent`, `ip_location`) | | `STATS` / `INLINE STATS` | `fromStatsLikeCommand` | | no args | `createCommand` one-liner | Two additions: **A — dispatcher** (search for `agent-marker: append new command dispatcher branches here` and add before it): ```ts const xxxCommandCtx = ctx.xxxCommand(); if (xxxCommandCtx) { return this.fromXxxCommand(xxxCommandCtx); } ``` **B — converter method**: ```ts private fromXxxCommand(ctx: cst.XxxCommandContext): ast.ESQLAstXxxCommand { const command = this.createCommand<'xxx', ast.ESQLAstXxxCommand>('xxx', ctx); // Assignment pattern: targetField = expression if (ctx._targetField && ctx.ASSIGN()) { const targetField = this.toColumn(ctx._targetField); const expression = this.fromPrimaryExpression(ctx.primaryExpression()); const assignment = this.toFunction('=', ctx, undefined, 'binary-expression') as ast.ESQLBinaryExpression; assignment.args.push(targetField, expression); assignment.location = this.extendLocationToArgs(assignment); command.targetField = targetField; command.expression = expression; command.args.push(assignment); } // Optional WITH { map } clause const withOption = this.fromOptionalNamedParametersWithOption(ctx.namedParametersWithOption()); if (withOption) { command.namedParameters = withOption.args[0] as ast.ESQLMap; command.args.push(withOption); } if (ctx.exception) { command.incomplete = true; } return command; } ``` **Patterns by grammar shape** — pick the one that matches: | Grammar shape | Converter call | |---|---| | Single column / qualified name | `this.toColumn(ctx._field)` | | Primary expression | `this.fromPrimaryExpression(ctx.primaryExpression())` | | Constant literal | `this.fromConstantToArray(ctx.constant())` | | String token | `this.fromStringToken(ctx.STRING().symbol)` | | Repeated sub-rule list | `ctx.xxxConfiguration_list()` loop | | Boolean expression | `this.fromBooleanExpression(ctx.booleanExpression())` | | Named option (`KEYWORD field`) | `Builder.option({ name: 'keyword', args: [...] }, { location })` | | Assignment `target = expr` | `this.toFunction('=', ctx, undefined, 'binary-expression')` | | WITH `{ map }` | `this.fromOptionalNamedParametersWithOption(ctx.namedParametersWithOption())` | --- ### 6. `packages/esql/src/pretty_print/constants.ts` The pretty printer is visitor-based and **requires no changes** for most new commands. Only update when: - Command args have **no commas** between them → add to `commandsWithNoCommaArgSeparator` - A command option uses `=` instead of a space before its value → add to `commandOptionsWithEqualsSeparator` ```ts export const commandsWithNoCommaArgSeparator = new Set([ 'dissect', 'sample', 'fork', 'promql', 'xxx', // ← add only if needed ]); ``` --- ### 7. `packages/esql/src/parser/__tests__/xxx.test.ts` (new file) Minimum coverage: 1. Parses basic syntax without errors 2. AST shape matches expected structure (`toMatchObject`) 3. Verifies `incomplete` flag for partial input 4. Round-trips through the pretty-printer ```ts import { parse } from '..'; import { BasicPrettyPrinter } from '../../pretty_print'; describe('XXX command', () => { it('parses basic syntax', () => { const { ast, errors } = parse('FROM a | XXX target = field'); expect(errors).toHaveLength(0); expect(ast[1]).toMatchObject({ type: 'command', name: 'xxx', targetField: { type: 'column', name: 'target' }, }); }); it('round-trips through the printer', () => { const src = 'FROM a | XXX target = field'; const { ast } = parse(src); expect(BasicPrettyPrinter.query(ast)).toBe(src.toUpperCase()); }); }); ``` Reference test files: `packages/esql/src/parser/__tests__/user_agent.test.ts`, `packages/esql/src/parser/__tests__/change_point.test.ts` --- ## Checklist — Scenario A (new command) - [ ] `packages/esql/src/types.ts` — new interface + union updated - [ ] `packages/esql/src/ast/visitor/contexts.ts` — `XxxCommandVisitorContext` added - [ ] `packages/esql/src/ast/visitor/types.ts` — input union, output union, `VisitorMethods` updated - [ ] `packages/esql/src/ast/visitor/global_visitor_context.ts` — switch case + public method added - [ ] `packages/esql/src/parser/core/cst_to_ast_converter.ts` — dispatcher + `fromXxxCommand()` added - [ ] `packages/esql/src/pretty_print/constants.ts` — updated only if needed - [ ] `packages/esql/src/parser/__tests__/xxx.test.ts` — tests added - [ ] `pnpm test` passes - [ ] `pnpm build` passes (no TypeScript errors) - [ ] `pnpm lint` passes --- ## Scenario B — Existing command update The grammar changed an **existing** command's rule: it gained a new option keyword (`BY`, `WITH`, `AS`), a new labeled alternative (`_newField`), or a new sub-rule. The converter method already exists — you only need to extend it. ### Files to change #### `packages/esql/src/parser/core/cst_to_ast_converter.ts` Find the existing `fromXxxCommand()` method and add handling for what's new. **New keyword option** (e.g. `CHANGE_POINT … BY field1, field2`): ```ts if (ctx.BY()) { const args = (ctx._groupings ?? []) .filter((e) => !e.exception) .map((e) => this.fromBooleanExpressionToExpressionOrUnknown(e)); const byOption = this.toByOption(ctx, args); if (byOption) { command.args.push(byOption); command.incomplete ||= byOption.incomplete; } } ``` **New labeled field** (e.g. `_targetField` added to an existing rule): ```ts if (ctx._targetField) { const col = this.toColumn(ctx._targetField); command.targetField = col; // only if types.ts interface has this field command.args.push(col); } ``` **New sub-rule variant** (e.g. `WHERE IN` gained a `LogicalInSubquery` alternative): ```ts // In the existing expression handler, add a branch before the current logic: if (ctx instanceof cst.LogicalInSubqueryContext) { return this.fromLogicalInSubquery(ctx); } // then add the new private method: private fromLogicalInSubquery(ctx: cst.LogicalInSubqueryContext): ast.ESQLAstExpression { const left = this.fromLogicalInLeft(ctx.valueExpression()); const right = this.fromSubquery(ctx.subquery());