jira

name	jira
description	Write Apache Spark JIRA tickets for optimizer rule proposals, execution improvements, or config additions. Produces structured descriptions with SQL examples and impact metrics.
license	MIT
metadata	{"version":"1.1","author":"James Xu"}

Spark JIRA Writer

Write JIRA tickets for proposing new optimizer rules, execution-layer improvements, or configuration additions to Apache Spark.

---

When to Use

Use when the user wants to:

• Submit a JIRA ticket to the Spark project
• Propose a new Catalyst optimizer rule
• Report a performance bottleneck with a concrete optimization plan
• Document a correctness issue or enhancement request

Don't use for:

• Bug reports without a proposed fix → user should file directly
• Non-Spark projects → use general writing
• Discussion-only topics without a technical proposal

---

Output Structure

Always produce a structured JIRA description in plain text (not HTML). The user can copy-paste it into the JIRA form.

1. Title

A single line describing the capability being added or improved. Do not mention concrete config key names or class names — those are implementation details that belong in the body.

Pattern:

[Component] Action + what it does + why it helps

Examples:

• [SQL] Add pre-pass rule to canonicalize COUNT(DISTINCT IF(...)) into COUNT(DISTINCT) FILTER
• [SQL] Push PartialAggregate through Expand for GROUPING SETS
• [SQL] Add config to allow bypassing pre-shuffle partial aggregation

2. Problem

Describe the real-world query patterns that perform poorly. If the feature benefits more than one scenario, give each its own named sub-section. For each scenario:

• Explain concisely why the current behavior is suboptimal
• Include a concrete, realistic SQL example that triggers the issue

Example sub-section structure:

**Scenario 1: High group cardinality.**
<explanation> <SQL example>

**Scenario 2: Skewed input data.**
<explanation> <SQL example>

If the example is from a real production workload, mention that. If not, keep it generic but realistic.

3. Root Cause (optional)

Include this section only for optimizer rule proposals where the specific rule or code path responsible for the inefficiency is non-obvious and worth calling out.

Omit for config additions or execution-layer changes where the problem is self-evident from the Problem section.

When included, point to:

• The specific optimizer rule or code path involved
• The specific logic that creates the bottleneck
• The data amplification factor (N×, 2^D, etc.)

4. Solution

For optimizer rule proposals: describe in detail —

• What the new rule does (or what the existing rule should do differently)
• The mathematical identity or invariant that makes it correct
• Safety constraints — when the rule applies and when it must not
• Where the rule fits in the optimizer batch ordering
• A brief pseudocode or Scala snippet if helpful

For config additions or execution-layer changes: describe at a high level only —

• What the config/change enables
• When it should be used (the scenarios from the Problem section)
• Any cases where it does not apply or is suppressed (correctness constraints)

Do not include concrete config key names, class names, or implementation details in this section.

5. Expected Impact

Quantify the improvement with concrete metrics where possible:

• Shuffle write before/after
• Disk spill before/after
• Wall-clock time before/after
• Expand factor before/after (for rule proposals)

Structure the impact to mirror the Problem scenarios — one sub-section per scenario if multiple were described.

If metrics are from a real production query, note that. If projected, say "projected" or "estimated."

---

Tone and Style

• Write for Spark committers and PMC members — technical, precise, no marketing language
• No hyperbole ("massive", "huge", "incredible") — use numbers
• No first person ("I propose", "we found") — use passive or direct statements
• Keep paragraphs short (2–3 sentences)
• Use code formatting for identifiers, SQL, and rule names
• Use bullet lists for multi-point explanations

---

Example Output

Title: [SQL] Add pre-pass rule to canonicalize COUNT(DISTINCT IF(cond, col, NULL))
       into COUNT(DISTINCT col) FILTER

Problem:
When a query contains many COUNT(DISTINCT IF(cond_i, col, NULL)) expressions
over the same base column, RewriteDistinctAggregates treats each unique IF(...)
expression as a distinct group. N conditions → N distinct groups → N× Expand
amplification. In production workloads with 25–60 such expressions, this produces
multi-terabyte shuffles and hour-long runtimes.

Example query pattern:

SELECT
  user_id,
  COUNT(DISTINCT IF(dt >= '2026-04-16', order_id, NULL)) AS orders_30d,
  COUNT(DISTINCT IF(dt >= '2026-02-15', order_id, NULL)) AS orders_90d,
  COUNT(DISTINCT IF(pay_status = 'paid', order_id, NULL)) AS orders_paid,
  -- ... 50 more expressions
FROM transactions
GROUP BY user_id

Root Cause:
RewriteDistinctAggregates groups distinct aggregates by their unfoldable child
sets. For COUNT(DISTINCT IF(cond1, col, NULL)), the child set is
{IF(cond1, col, NULL)} — each unique condition forms its own group, even though
all conditions apply to the same base column.

Solution:
Add a pre-pass optimizer rule (before RewriteDistinctAggregates) that
canonicalizes:

  COUNT(DISTINCT IF(cond, col, NULL))  →  COUNT(DISTINCT col) FILTER (WHERE cond)

This identity is exact: COUNT DISTINCT ignores NULLs, so nulling rows where
!cond is semantically identical to filtering them. After the rewrite, all
expressions share the same unfoldable child set {col}, collapsing to 1 distinct
group and 1× Expand regardless of condition count.

The rule is conservative:
- Only rewrites COUNT (the most common pattern in production)
- Only when the inner expression is IF(cond, base, NULL) or CASE WHEN
- Skips non-NULL fallbacks and expressions that already have a FILTER clause
- Must run before RewriteDistinctAggregates in the optimizer's Aggregate batch

Expected Impact:
In a production query with 60 conditional distinct counts, this reduces Expand
factor from 25× to 1×, shuffle write from ~2.8 TB to ~110 GB, and wall-clock
time from ~3 hours to ~15–20 minutes.