devex-survey

name	devex-survey
description	Create or analyze Developer Experience surveys. Creation mode outputs a 9-question HaTS-aligned survey; Analysis mode outputs sentiment metrics with theme clustering.

DevEx Survey

This skill defines how AI assists with Developer Experience surveys in two modes:

1. Creation Mode — Generate a statistically credible, longitudinally stable DevEx survey
2. Analysis Mode — Analyze survey results into actionable, segmented insights

Both modes enforce strict constraints on question design, statistical reporting, and guardrails against misuse.

See references/runbook.md for the full operational playbook covering distribution, cadence, and communication strategy.

---

Reasoning Framework

This skill exists because developer experience measurement requires survey science rigor (HaTS alignment, Likert scales, longitudinal consistency). Ad-hoc surveys produce unreliable data that can't be compared across quarters.

Output Contract

Artifact	Format	Handed to
Survey (Creation mode)	9-question survey with scales and rationale	Survey platform / PM
Analysis report (Analysis mode)	Markdown with sentiment metrics, themes, leverage opportunities	PM / leadership

---

How to Help

When the user invokes this skill, determine which mode applies:

• If the user asks to create, draft, or generate a survey → enter Creation Mode
• If the user provides survey data, results, or responses → enter Analysis Mode
• If unclear, ask the user which mode they need

---

CREATION MODE

Objective

Generate a quarterly Developer Experience survey that:

• Measures workflow sentiment across key developer touchpoints
• Preserves longitudinal integrity (core questions stable quarter-over-quarter)
• Aligns to HaTS (Happiness and Tracking Survey) principles found on https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43221.pdf
• Remains statistically credible with proper confidence intervals
• Minimizes survey fatigue (3-5 minute completion)
• Supports metadata-based post-processing (non-anonymous)

Research Alignment

Incorporate principles from:

• HaTS (Happiness and Tracking Survey) — Google's validated approach to tracking developer happiness over time. Key principles: tie to workflows, use consistent scales, keep surveys short, track longitudinally.
• JetBrains Developer Ecosystem Report — Industry benchmark for developer workflow patterns, tool usage, and ecosystem trends.

Survey Constraints (MANDATORY)

These constraints are non-negotiable. Every generated survey must comply:

Constraint	Requirement
Max questions	9 total (5 core + 1 pain point + 1 consolidated why + 2 floaters)
Scale	5-point Likert (see below for allowed scales)
Phrasing	Non-biased, conversational (see below)
Identity	Non-anonymous
Completion time	3-5 minutes
Layout	Single page
Floater questions	1-2 allowed
Core questions	Must remain stable quarter-over-quarter

5-Point Likert Scales

Satisfaction scale (default for core questions):

1. Very Satisfied
2. Satisfied
3. Neutral
4. Dissatisfied
5. Very Dissatisfied

Ease/difficulty scale (allowed for floater questions where it fits better):

1. Very Easy
2. Easy
3. Neutral
4. Difficult
5. Very Difficult

Floater questions may use either scale depending on what is being measured. Core questions always use the satisfaction scale.

Likert Scoring Rules

Satisfaction % = (Very Satisfied + Satisfied) / Total Responses

Dissatisfaction % = (Very Dissatisfied + Dissatisfied) / Total Responses

Neutrals remain in the denominator. Do NOT generate vanity CSAT scores.

Core Questions (Stable)

Every survey must include these 5 sentiment questions. They must remain identical across quarters to preserve longitudinal integrity.

Each question uses the phrasing pattern:

"How satisfied or dissatisfied are you with..."

Write questions the way a developer would talk — use "your code", "merging your code", not abstract process language.

The 5 Core Workflow Questions

1. Overall Developer Experience — "How satisfied or dissatisfied are you with your overall developer experience at [Company]?"
2. Inner Loop (write, build, test) — "How satisfied or dissatisfied are you with writing, building, and testing your code?"
3. Merge Process — "How satisfied or dissatisfied are you with merging your code?"
4. Release Process — "How satisfied or dissatisfied are you with releasing your code to production?"
5. Documentation Experience — "How satisfied or dissatisfied are you with the quality and availability of documentation?"

Pain Point Question

Include exactly one pain point selection question:

"What are your top 3 pain points?"

• Maximum 10 options listed
• Respondent must select exactly 3
• Options should cover the major workflow friction areas (e.g., build times, test flakiness, CI/CD reliability, environment setup, code review turnaround, documentation gaps, dependency management, deployment complexity, tooling fragmentation, on-call burden)

Consolidated Why Question

After the core questions and pain point selection, include a single open-ended question:

"Why did you respond the way you did to the above questions?"

This replaces per-question follow-ups. One consolidated "why" reduces survey fatigue while still capturing the reasoning behind sentiment responses.

Floater Questions (1-2 per quarter)

Floater questions are contextual and may rotate each quarter. Use them to assess:

• New tools recently introduced
• Recent platform investments
• Experimental workflows or process changes
• Process friction (e.g., stewardship approval, on-call experience)

Floater questions do NOT replace core questions. Core questions are always present and unchanged.

Floaters may use the satisfaction scale or the ease/difficulty scale depending on what is being measured. For example, a question about stewardship approval friction fits the ease/difficulty scale better than satisfaction.

Questions to AVOID

Do NOT ask respondents for:

• Repository or codebase
• Geographic location or country
• Job title or seniority
• Programming language
• Org or team name

These dimensions are available via metadata and will be joined during analysis. Asking for them wastes question slots and increases survey fatigue.

Optional Elements

• Open feedback field — "Is there anything else you'd like to share?"
• Follow-up consent — "Would you be open to a follow-up conversation? (Yes / No)"

Survey Structure

The full survey follows this order:

1. Q1-Q5: Core workflow questions (satisfaction scale)
2. Q6: Pain point selection (select exactly 3)
3. Q7: Consolidated "why" open-text (required)
4. Q8-Q9: Floater questions (satisfaction or ease/difficulty scale)
5. Optional: open feedback + follow-up consent

Output Format (Creation Mode)

When generating a survey, return:

1. Survey intro text (short — purpose, time estimate, non-anonymous notice, close date)
2. All questions in order with scale and format specified
3. Pain point options (up to 10)
4. Floater questions with rationale for why they're included this quarter

---

ANALYSIS MODE

Objective

Analyze survey results to produce:

• Sentiment metrics with statistical rigor
• Theme clustering from verbatim responses
• Leverage opportunities (where small moves yield big gains)
• Segmented insights using metadata
• Executive-ready summary

Analysis Requirements

1. Categorize Verbatim Responses

• Read every piece of open-text feedback
• AI-assisted theme categorization is expected
• Count theme frequency across all responses
• Surface representative quotes for each theme

2. Detect Sentiment Patterns

• Calculate satisfaction and dissatisfaction % per question
• Detect large sentiment deltas vs. prior quarter
• Identify dissatisfied to neutral opportunities (quick wins)
• Identify neutral to satisfied leverage (investment opportunities)

3. Segment Using Metadata

Join survey responses with organizational metadata to segment by:

• Programming language / tech stack
• Tenure at company
• Geographic location
• Discipline (frontend, backend, infra, mobile, etc.)
• Tool usage patterns
• Team / org distribution

Identify:

• Hotspots — teams or segments with unusually low satisfaction
• Outliers — individuals or groups that deviate sharply from the mean
• Sharp deltas — segments where satisfaction changed significantly quarter-over-quarter

4. Champion Identification

Flag respondents who:

• Provided thoughtful, detailed feedback
• Showed deep understanding of workflow friction
• Could serve as DevEx champions or design partners

5. Statistical Reporting

Always calculate and report:

Metric	Required
Total invited	Yes
Total responded	Yes
Response rate	Yes
Confidence interval	Yes
Satisfaction % per question	Yes
Dissatisfaction % per question	Yes

Target CI: 1-3%. Never publish sentiment without confidence interval.

Output Format (Analysis Mode)

Return a structured report with these sections:

1. Executive Summary — 3-5 sentence overview of key findings
2. Statistical Overview — Response rate, CI, satisfaction/dissatisfaction % per question
3. Trend Delta Summary — Quarter-over-quarter changes with direction and magnitude
4. Top 3 Friction Themes — Most frequently cited pain points with representative quotes
5. Segmented Hotspots — Metadata-driven breakdowns showing where friction concentrates
6. Leverage Recommendations — Specific, actionable opportunities ranked by impact
7. Champion Candidates — Respondents flagged for follow-up (if applicable)

---

Guardrails

These rules apply to both modes and must never be violated:

• Do NOT tie sentiment to performance evaluation. Survey data is a steering signal, not a performance metric.
• Do NOT recommend gaming response rates. Quality over quantity.
• Do NOT suggest bot-driven or automated outreach. All follow-ups must be personal and human.
• Do NOT generate vanity metrics. Report real satisfaction and dissatisfaction with CI.
• Preserve trend integrity. Never modify core questions in ways that break longitudinal comparability.
• Close the feedback loop. Always recommend sharing results and connecting improvements back to survey data.

The goal is trust, not vanity metrics.

---

Related Skills

• /analyzing-user-feedback — General user feedback analysis frameworks
• /designing-surveys — Broader survey design principles
• /storytelling-for-impact — Craft compelling narratives from survey data for shareouts
• /giving-presentations — Present survey findings at all-hands and leadership reviews