streaming-testing

name	streaming-testing
description	Compare A2A streaming behaviour across supervisor versions. Captures SSE events, analyzes metadata flags (is_narration, is_final_answer), and produces side-by-side comparison reports.

Streaming Testing

Compare A2A streaming behaviour across supervisor versions (e.g. 0.3.0 vs 0.2.41). Captures SSE events, analyzes metadata correctness, and generates side-by-side markdown reports.

Prerequisites

• Docker Compose dev stack running (docker-compose.dev.yaml)
• 0.3.0 supervisor on port 8000 (caipe-supervisor)
• 0.2.41 supervisor on port 8041 (caipe-supervisor-041)
• Python 3.10+ (scripts use stdlib only — no pip install needed)

Quick Start

# One command — captures from both supervisors, analyzes, compares
./skills/streaming-testing/run_comparison.sh "what can you do?"

# Output lands in /tmp/streaming-comparison-<timestamp>.md

Individual Scripts

All scripts live in scripts/ and use Python stdlib only.

Script	Purpose	Usage
trace_a2a_streaming.py	SSE timeline with summary stats	python3 scripts/trace_a2a_streaming.py 8000 "query"
capture_a2a_events.py	Save raw A2A events to JSON	python3 scripts/capture_a2a_events.py http://localhost:8000/ "query" /tmp/out.json
analyze_a2a_metadata.py	Validate metadata flags in a capture	python3 scripts/analyze_a2a_metadata.py /tmp/out.json
compare_a2a_events.py	Side-by-side comparison of two captures	python3 scripts/compare_a2a_events.py /tmp/a.json /tmp/b.json
analyze_accumulation_flow.py	Content accumulation + duplicate detection	python3 scripts/analyze_accumulation_flow.py "query" --port 8000
validate_artifacts.py	Parse/validate artifact reports	python3 scripts/validate_artifacts.py --query "query" --port 8000

Workflow

Step 1: Capture events from both supervisors

python3 scripts/capture_a2a_events.py http://localhost:8000/ "what can you do?" /tmp/cap-030.json
python3 scripts/capture_a2a_events.py http://localhost:8041/ "what can you do?" /tmp/cap-041.json

Step 2: Analyze metadata on each capture

python3 scripts/analyze_a2a_metadata.py /tmp/cap-030.json
python3 scripts/analyze_a2a_metadata.py /tmp/cap-041.json

This validates:

• is_final_answer is present on streaming_result artifacts
• No [FINAL ANSWER] marker leaked into text
• No internal metadata leaked (is_task_complete=, Returning structured response)
• is_narration and is_final_answer are mutually exclusive
• final_result or partial_result artifact exists
• Streaming content is non-empty

Step 3: Compare side-by-side

python3 scripts/compare_a2a_events.py /tmp/cap-030.json /tmp/cap-041.json --output /tmp/comparison.md

Generates a markdown report with:

• Summary table (total time, time to first content, artifact counts, char counts)
• Artifact name breakdown
• Safety checks (marker leaks, metadata leaks)
• Per-artifact detail table with metadata flags

Step 4 (optional): Deep-dive traces

For a live timeline view of each SSE event as it arrives:

python3 scripts/trace_a2a_streaming.py 8000 "what can you do?"
python3 scripts/trace_a2a_streaming.py 8041 "what can you do?"

Interpreting Results

Key Metrics

Metric	What it means	Good value
Time to first content	How long until the user sees text	< 5s
Total time	End-to-end response time	< 20s (simple), < 120s (RAG)
has is_final_answer	Supervisor correctly tags the final answer	True
Narration events	Narration chunks for typing-status display	0 for simple queries, > 0 for RAG
Marker leaks	[FINAL ANSWER] leaked to client	Must be 0
Metadata leaks	Internal state leaked to client	Must be 0

What to look for

• 0.3.0 has a "dead gap" before content: The generate_structured_response node builds JSON preamble before the content key. Time to first content will be higher than 0.2.41.
• 0.2.41 may leak narration text: Pre-marker buffer flush sends LLM thinking text (e.g. "Excellent! I now have comprehensive information...") to the client.
• Narration events should NOT appear in 0.2.41: The is_narration flag was added in 0.3.0.
• Both versions should have 0 marker leaks: The Slack bot and agent_executor both strip markers.

Troubleshooting

Supervisor not reachable

# Check containers
docker ps | grep supervisor

# Restart
docker restart caipe-supervisor
docker restart caipe-supervisor-041

Script hangs

The scripts have a default 120s (trace) or 300s (capture) timeout. For RAG queries, increase the timeout:

# capture_a2a_events.py reads timeout from the URL (not a CLI arg).
# For trace, it's hardcoded at 120s in http.client timeout.
# If queries consistently time out, check supervisor logs:
docker logs caipe-supervisor 2>&1 | tail -20

Empty captures

If capture_a2a_events.py produces 0 artifacts:

1. Verify the supervisor is healthy: curl http://localhost:8000/.well-known/agent.json
2. Check if the supervisor uses SSE or JSON-RPC (the script handles both)
3. Look at supervisor logs for errors: docker logs caipe-supervisor 2>&1 | tail -50

Examples

# Simple query comparison
./skills/streaming-testing/run_comparison.sh "what can you do?"

# RAG query comparison (takes longer)
./skills/streaming-testing/run_comparison.sh "what is caipe?"

# Just analyze a single capture
python3 scripts/analyze_a2a_metadata.py /tmp/cap-030.json --verbose

# Custom labels in comparison report
python3 scripts/compare_a2a_events.py /tmp/cap-030.json /tmp/cap-041.json \
    --label-a "0.3.0 (post-fix)" --label-b "0.2.41 (golden)"