services-control-plane-dev

name	services-control-plane-dev
title	Services Control Plane Development Skill Set
description	Learn how to develop a Kubernetes-based control plane for managing deployments with gRPC API, real-time state tracking, and a Terminal UI.

Services Control Plane Development

Services Control Plane Development Skill Set

This skill set documents the patterns, practices, and workflows established during the development of the services-control-plane project.

CRITICAL: After every code change, run go test ./... -v -count 1 and go build ./... to verify all tests pass and the project compiles. Do not consider a task complete until all tests pass.

CRITICAL: NEVER run tests or manual testing commands against production clusters. All testing MUST be done against the local k3d cluster only.

CRITICAL: When adding or modifying gRPC RPCs, you MUST add integration tests in k8s/tests/integration_test.go. Integration tests validate the full stack (gRPC → server → Kubernetes → PostgreSQL). Unit tests alone are NOT sufficient for gRPC changes.

Project Overview

Goal: Build a Kubernetes-based control plane for managing deployments with gRPC API, real-time state tracking, and a Terminal UI.

Stack:

• Go (gRPC server, Kubernetes operator/controllers)
• Protocol Buffers (API definitions)
• Kubernetes (Custom Resource Definitions, StatefulSets, Pods, RBAC)
• PostgreSQL (Deployment data, events, registries)
• Hashicorp Vault (Secret management)
• TypeScript/Bun (CLI with OpenTUI)
• k3d/k3s (Local testing)

Architecture:

• CRD-based: Deployments are managed via HostedServiceDeployment Custom Resources
• Operator pattern: A controller reconciles CRDs into StatefulSets and updates status
• PostgreSQL persistence: Deployment data stored in deployments table, events in deployment_events table, registries in dedicated tables
• Vault integration: Secrets stored in Hashicorp Vault, injected via Vault Agent annotations

---

Development Workflow

0. CRITICAL: Merge Workflow

NEVER auto-merge feature branches. Always follow this workflow:

1. Complete implementation in feature branch
2. Run all tests (unit + integration)
3. Report results to user with summary of changes
4. WAIT for explicit user approval before merging
5. Only merge when user says "merge", "looks good", "go ahead", etc.

This applies to:

• Sub-agent feature implementations
• Parallel feature development with worktrees
• Any branch-based development

Why? The user must have the opportunity to:

• Review implementation approach
• Inspect code changes
• Request modifications before merge
• Control timing and merge strategy

1. Feature Implementation Pattern

When implementing a new feature, follow this sequence:

1. Design the API - Update proto definitions
2. Regenerate code - Run buf generate
3. Implement server - Add gRPC method implementation
4. Write Go unit tests - Test with fake k8s client before deploying
5. Add CLI support - Create UI screens and gRPC client methods
6. Test in k3d cluster - Run integration tests
7. Update documentation - FEATURES.md with checkboxes

Example: SetReplica feature implementation

// 1. After implementing SetReplica in grpc/services/hosted.go
// 2. Write unit test in grpc/services/hosted_test.go

func TestSetReplica_ScaleUp(t *testing.T) {
    env := SetupTestEnv(t)
    defer env.Cleanup()

    hosted := createHostedService(t, env.K8sClient, nil)
    resp, err := hosted.SetReplica(ctx, &pbcommon.ReplicaRequest{
        DeploymentId: "test-deployment-3",
        Count:        3,
    })
    require.NoError(t, err)
    require.True(t, resp.Success)
}

3. Run tests: go test ./... -v
4. Test in k3d: go test ./k8s/tests/... -v
5. Mark feature complete in FEATURES.md

2. Proto-First Development

Always start with protocol buffer definitions:

service HostedService {
  rpc Logs(LogsRequest) returns (LogsResponse);
}

message LogsRequest {
  string deployment_id = 1;
  string pod_name = 2;       // optional
  int32 tail_lines = 3;       // default: 100
}

message LogsResponse {
  bool success = 1;
  string message = 2;
  repeated PodLogs pod_logs = 3;
}

Benefits:

• Contract-first design
• Auto-generated client/server stubs
• Type safety across languages
• Clear API documentation

3. Local Testing Environment

Setup: Use k3d for local Kubernetes clusters

Go CLI Commands:

# Initial setup
go run ./cmd/control-freak local setup

# After code changes - rebuild and restart
go run ./cmd/control-freak local restart

# Start port forwards (runs in foreground, Ctrl+C to stop)
go run ./cmd/control-freak local port-forward

# Teardown
go run ./cmd/control-freak local teardown

Key Files:

• k8s/setup/devenv.go - Programmatic setup/teardown using client-go (no YAML manifests)
• k8s/setup/assets/k3d-cluster.yaml - k3d cluster configuration with port mappings

Port Mappings:

• gRPC: localhost:30051 → control-freak:9000

In-cluster services (not port-forwarded by default):

• PostgreSQL: postgres-db:5432 (internal, use kubectl port-forward when needed)
• Vault: vault:8200 (internal)

4. Test Strategy

CRITICAL: Keep control-freak Image Up to Date

Before running integration tests, ALWAYS rebuild and redeploy the control-freak image to ensure the cluster has your latest code changes:

# Rebuild, import to k3d, and restart control-freak
go run ./cmd/control-freak local restart

Or manually:

docker build -t control-freak:local -f Dockerfile .
k3d image import control-freak:local -c control-plane-local
kubectl --context k3d-control-plane-local rollout restart deployment control-freak -n control-plane-local
kubectl --context k3d-control-plane-local rollout status deployment control-freak -n control-plane-local --timeout=60s

Why this matters: Integration tests call the control-freak gRPC server running in the cluster. If you've made code changes but haven't rebuilt the image, the tests will run against stale code and may pass incorrectly (or fail unexpectedly).

Go Integration Tests Against k3d Cluster (PRIMARY)

All tests run against a real k3d cluster. No fake clients - tests validate actual Kubernetes and gRPC behavior.

Location: k8s/tests/integration_test.go Framework: github.com/stretchr/testify/require Dependencies: k8s.io/client-go/kubernetes

IMPORTANT: Integration Test Patterns:

1. Always create pods via gRPC Deploy - Never use direct k8s client to create test pods
2. Cleanup at start, not end - Delete any leftover pods at the beginning of each test
3. Keep CRDs for inspection - By default, CRDs remain after tests for debugging
4. Set KEEP_CRD=false to clean up CRDs after tests

Automatic Setup/Teardown:

• Tests automatically set up k3d cluster if not running (via setup.Setup())
• Tests automatically start port forwards for gRPC
• Tests automatically tear down cluster after completion
• Set KEEP_CLUSTER=true to keep cluster running after tests

Run Tests:

go test ./k8s/tests/... -v                          # Run all integration tests
KEEP_CLUSTER=true go test ./k8s/tests/... -v         # Keep cluster after tests
go test ./k8s/tests/... -run TestSetReplica -v        # Run specific test
go test ./... -v -count 1                             # Run all tests (unit + integration)

SAFETY: Tests will REFUSE TO RUN against non-k3d clusters:

• API server host must be localhost or 127.0.0.1
• Node names must contain "k3d", "k3s", or the cluster name

Test Environment Setup:

type TestEnv struct {
    K8sClient     kubernetes.Interface   // Real k8s client
    DynamicClient dynamic.Interface      // Dynamic client for CRDs
    GRPCConn      *grpc.ClientConn       // Real gRPC connection
    GRPCClient    pbprivateservice.HostedServiceClient
    Namespace     string
}

func SetupTestEnv(t *testing.T) *TestEnv {
    // SAFETY: Refuses to run if not connected to a k3d cluster
    // Checks that API server is localhost/127.0.0.1
    // Checks that node names contain "k3d", "k3s", or the cluster name
    // Connects to gRPC on localhost:30051
}

Test Helper Functions:

// CleanupDeployment - Delete CRD and StatefulSet for a deployment
func (e *TestEnv) CleanupDeployment(deploymentID string)

// DeploySinkSql - Create a sink-sql deployment via gRPC Deploy
// Handles cleanup of existing deployment, deploys via gRPC, waits for StatefulSet
func (e *TestEnv) DeploySinkSql(deploymentID, organizationID, spkgURL string) string

// ShouldKeepCRD - Returns true unless KEEP_CRD=false
func ShouldKeepCRD() bool

// WaitFor - Poll until condition is met or timeout
func (e *TestEnv) WaitFor(timeout time.Duration, description string, condition func() bool)

// ScaleDownDeployment - Scale a deployment to 0 replicas (for cleanup without deleting CRD)
func (e *TestEnv) ScaleDownDeployment(deploymentID string)

// getPostgresDB - Start port-forward to in-cluster postgres, return *sql.DB + cleanup func
func getPostgresDB(t *testing.T, env *TestEnv) (*sql.DB, func())

Example Test (Current Pattern):

func TestDeploymentStore_DeployPersistsData(t *testing.T) {
    env := SetupTestEnv(t)
    defer env.Cleanup()

    ctx := context.Background()
    deploymentID := "depstore-test-deploy"
    organizationID := "test-org-depstore"

    // Deploy via gRPC
    _ = env.DeploySinkSql(deploymentID, organizationID, testSpkgURL)

    // Connect to postgres and verify data persisted
    db, cleanup := getPostgresDB(t, env)
    defer cleanup()

    var depID, status string
    err := db.QueryRowContext(ctx,
        `SELECT deployment_id, status FROM deployments WHERE deployment_id = $1`,
        deploymentID).Scan(&depID, &status)
    require.NoError(t, err)
    require.Equal(t, "active", status)

    // Cleanup
    if !ShouldKeepCRD() {
        env.CleanupDeployment(deploymentID)
    }
}

When Implementing New Features:

1. Write Go test in k8s/tests/integration_test.go
2. Use env.DeploySinkSql() to create test deployments via gRPC
3. Test uses real gRPC client to call the service
4. Verify results via k8s client and PostgreSQL
5. Test both success and error cases
6. Use env.WaitFor() for async operations
7. Keep CRDs for inspection by default (check with ShouldKeepCRD())
8. CRITICAL: Rebuild control-freak before testing - Run go run ./cmd/control-freak local restart to ensure cluster has latest code
9. Run go test ./k8s/tests/... -v before committing

MANDATORY Integration Tests for gRPC Changes: When adding or modifying any gRPC RPC, you MUST add integration tests that cover:

1. Success case - Test the happy path with valid inputs
2. Not found case - Test with non-existent deployment IDs
3. Validation case - Test with empty/invalid required fields
4. Update verification - If the RPC modifies state, verify the change persists

Example integration test pattern:

// ============================================================================
// Test Suite N: <FeatureName> Tests
// ============================================================================

func Test<Feature>_Success(t *testing.T) {
    env := SetupTestEnv(t)
    defer env.Cleanup()

    ctx := context.Background()
    deploymentID := "feature-test"
    organizationID := "test-org"

    // Deploy via gRPC
    stsName := env.DeploySinkSql(deploymentID, organizationID, testSpkgURL)

    // Call the RPC being tested
    resp, err := env.GRPCClient.<Method>(ctx, &pbprivateservice.<Request>{
        DeploymentId: deploymentID,
    })
    require.NoError(t, err)
    require.True(t, resp.Success, "should succeed: %s", resp.Message)

    // Verify expected state
    // ...

    // Cleanup
    if !ShouldKeepCRD() {
        env.CleanupDeployment(deploymentID)
    }
}

func Test<Feature>_EmptyDeploymentID(t *testing.T) {
    env := SetupTestEnv(t)
    defer env.Cleanup()

    ctx := context.Background()

    resp, err := env.GRPCClient.<Method>(ctx, &pbprivateservice.<Request>{
        DeploymentId: "",
    })
    require.NoError(t, err)
    require.False(t, resp.Success, "should fail with empty deployment_id")
    require.Contains(t, resp.Message, "deployment_id required")
}

Test Suites Covered:

• Suite 1: gRPC Service (server running, deployments exist)
• Suite 3: Pod State Monitoring
• Suite 6: SetReplica (scale up/down, validation)
• Suite 7: Undeploy (delete, cleanup)
• Suite 8: Logs (specific pod, all pods, previous container)
• Suite 9: Deployment State (retrieval, pod info)
• Suite 10: Deploy Hosted Sink (Hivemapper to Postgres)
• Suite 11: UpdateDeploymentConfig (spkg update, execution config, output config)
• Suite 12: GetDeployment (success, not found, empty ID, after update) + Operator Tests
• Suite 13: DeleteDeploymentConfig + End-to-End Vault + Substreams API
• Suite 14: Deployment Events
• Suite 15: AdminService
• Suite 16: Deployment Store (PostgreSQL deployments table persistence)

5. CLI Development with OpenTUI

Screen Architecture:

• Each feature gets its own screen file (screens/*.ts)
• Main index.ts coordinates screen navigation
• Screens are composable: containers, text, inputs

Example Screen:

export function createLogsScreen(renderer: CliRenderer) {
  const logsScreen = new BoxRenderable(renderer, {
    flexDirection: "column",
    width: "100%",
    height: "100%",
  })
  
  const logsContainer = new BoxRenderable(renderer, {
    flexGrow: 1,
    border: true,
  })
  
  function update(deploymentId: string, logs: string) {
    // Update screen content
  }
  
  return { logsScreen, logsContainer, update }
}

Key Binding Pattern:

renderer.keyInput.on("keypress", (key: KeyEvent) => {
  if (keyName === "l") {
    showLogsScreen(selectedPod)
  }
})

6. Kubernetes Integration

CRD-Based Deployment Management: Deployments are managed via HostedServiceDeployment Custom Resources. The operator watches CRDs and reconciles them into StatefulSets.

// CRD spec contains all deployment configuration
type HostedServiceDeploymentSpec struct {
    DeploymentID          string            `json:"deploymentId"`
    DeploymentName        string            `json:"deploymentName,omitempty"`
    OrganizationID        string            `json:"organizationId"`
    DeploymentType        string            `json:"deploymentType"`
    Replicas              int32             `json:"replicas"`
    SpkgURL               string            `json:"spkgUrl,omitempty"`
    OutputModule          string            `json:"outputModule,omitempty"`
    StartBlock            int64             `json:"startBlock,omitempty"`
    StopBlock             int64             `json:"stopBlock,omitempty"`
    Parameters            string            `json:"parameters,omitempty"`
    DSN                   string            `json:"dsn,omitempty"`
    APIKey                string            `json:"apiKey,omitempty"`
    DeploymentRequestJSON string            `json:"deploymentRequestJson,omitempty"`
    VaultSecretPath       string            `json:"vaultSecretPath,omitempty"`
    VaultSecretKeys       map[string]string `json:"vaultSecretKeys,omitempty"`
    // ... more fields
}

// CRD status is updated by the operator with live state
type HostedServiceDeploymentStatus struct {
    State            string              `json:"state,omitempty"`
    Replicas         int32               `json:"replicas,omitempty"`
    ReadyReplicas    int32               `json:"readyReplicas,omitempty"`
    StatefulSetName  string              `json:"statefulSetName,omitempty"`
    PodStates        []PodExecutionState `json:"podStates,omitempty"`
    CrashLoopBackOff bool                `json:"crashLoopBackOff,omitempty"`
    MaxRestartCount  int32               `json:"maxRestartCount,omitempty"`
}

RBAC Pattern: Always create ServiceAccount + Role + RoleBinding using client-go:

// ServiceAccount
sa := &corev1.ServiceAccount{
    ObjectMeta: metav1.ObjectMeta{
        Name:      "control-freak",
        Namespace: namespace,
    },
}
client.CoreV1().ServiceAccounts(namespace).Create(ctx, sa, metav1.CreateOptions{})

// Role
role := &rbacv1.Role{
    ObjectMeta: metav1.ObjectMeta{
        Name:      "control-freak",
        Namespace: namespace,
    },
    Rules: []rbacv1.PolicyRule{
        {
            APIGroups: []string{"apps"},
            Resources: []string{"statefulsets"},
            Verbs:     []string{"get", "list", "watch", "create", "update", "patch", "delete"},
        },
        {
            APIGroups: []string{""},
            Resources: []string{"pods", "pods/log"},
            Verbs:     []string{"get", "list"},
        },
        {
            APIGroups: []string{""},
            Resources: []string{"services", "secrets", "persistentvolumeclaims"},
            Verbs:     []string{"get", "list", "watch", "create", "update", "patch", "delete"},
        },
        {
            APIGroups: []string{"hosted.streamingfast.io"},
            Resources: []string{"hostedservicedeployments", "hostedservicedeployments/status"},
            Verbs:     []string{"get", "list", "watch", "create", "update", "patch", "delete"},
        },
    },
}

Label Selectors: Use consistent labels for querying:

labels := map[string]string{
    "deployment-id":   deploymentID,
    "organization-id": organizationID,
    "managed-by":      "services-control-plane",
    "deployment-type": deploymentType,
}

7. Data Persistence with PostgreSQL

Architecture: PostgreSQL is the durable store for all deployment data. The CRDs are the live state, but PostgreSQL persists the data across cluster rebuilds.

CRITICAL: Schema Management

All database schema changes MUST be done via SQL migrations:

• Migration files: store/psql/migrations/
• Migration script: ./script/migrate.sh
• No Go-side schema creation: schema lives solely in the migrations above. Do not add InitSchema()-style helpers that create tables from Go code.

Migration Commands:

# Check current version
./script/migrate.sh version

# Apply all pending migrations
./script/migrate.sh up

# Apply up to a specific version
./script/migrate.sh up 4

# Roll back one migration
./script/migrate.sh down

# Create a new migration
./script/migrate.sh new my_new_table

# Force version (for dirty state recovery)
./script/migrate.sh force 3

Creating New Migrations:

1. Run ./script/migrate.sh new descriptive_name
2. Edit the generated .up.sql and .down.sql files
3. Test with ./script/migrate.sh up
4. Verify rollback with ./script/migrate.sh down

Tables:

Table	Purpose	Package
deployments	All deployment configuration and metadata	store/deployment
deployment_events	Lifecycle events (created, deleted, scaled, etc.)	store/event
foundation_store_registry	Foundational store deployment registry	registry
network_endpoints	Network endpoint registry	registry
network_aliases	Network name → endpoint mappings	registry

Deployment Store (store/deployment/):

// Store manages deployment records in PostgreSQL
// NOTE: Schema is created by migrations, not Go code
store := deployment.NewStore(db)

// CRUD operations
store.Create(ctx, &Deployment{...})           // Upsert on deploy
store.Get(ctx, deploymentID)                   // Get by ID
store.ListByOrganization(ctx, orgID)           // List by org
store.UpdateStatus(ctx, deploymentID, "deleted") // Mark deleted on undeploy
store.UpdateReplicas(ctx, deploymentID, 3)       // Update on SetReplica
store.UpdateConfig(ctx, deploymentID, updates)   // Update on config change

Deployments table schema:

CREATE TABLE IF NOT EXISTS deployments (
    deployment_id TEXT PRIMARY KEY,
    organization_id TEXT NOT NULL,
    deployment_name TEXT NOT NULL DEFAULT '',
    deployment_type TEXT NOT NULL DEFAULT '',
    spkg_url TEXT NOT NULL DEFAULT '',
    output_module TEXT NOT NULL DEFAULT '',
    start_block BIGINT NOT NULL DEFAULT 0,
    stop_block BIGINT NOT NULL DEFAULT 0,
    parameters TEXT NOT NULL DEFAULT '',
    dsn TEXT NOT NULL DEFAULT '',
    api_key TEXT NOT NULL DEFAULT '',
    replicas INT NOT NULL DEFAULT 1,
    deployment_request_json JSONB DEFAULT '{}',
    status TEXT NOT NULL DEFAULT 'active',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Event Store (store/event/):

// Store records deployment lifecycle events
// NOTE: Schema is created by migrations, not Go code
es := event.NewStore(postgresDSN)
es.RecordEvent(ctx, &DeploymentEvent{...})
es.GetEvents(ctx, deploymentID, limit)
es.GetEventsByOrg(ctx, orgID, limit)

Fire-and-forget pattern: Database writes from gRPC handlers are fire-and-forget — failures are logged but never cause RPCs to fail:

// In SetReplica / Undeploy / UpdateDeploymentConfig handlers:
if s.deploymentStore != nil {
    if err := s.deploymentStore.UpdateReplicas(ctx, deploymentID, n); err != nil {
        s.logger.Warn("failed to update replicas in database", zap.Error(err))
    }
}

Exception — Hosted.Deploy is atomic. Deploy is the one handler where persistence is part of the success contract: the rows it writes to foundation_store_registry and deployments are read by every downstream handler (GetDeployment, SetReplica, UpdateDeploymentConfig, Undeploy) and there is no reconciler that backfills them from the CR. A Success response from Deploy means all of these are true:

1. The HostedServiceDeployment CR exists.
2. The foundation_store_registry row exists (if the deploy is a foundational store).
3. The deployments row exists.
4. Vault secrets, if vaultClient is configured, have been stored — there is no silent fallback to plaintext credentials in the CR spec.

Any failure between steps 1 and 3 triggers cleanupAfterFailedDeploy (grpc/services/hosted.go), which best-effort undoes prior side effects in reverse order using a fresh context with a 10s timeout. The compensation makes Deploy idempotent under retry — a failed deploy leaves no orphaned CR to trip AlreadyExists on the next attempt.

Connection: PostgreSQL DSN is passed via --postgres-dsn flag or POSTGRES_DSN env var. The event.Store opens the connection, and es.DB() is shared across all stores (deployment store, event store, registries).

8. Vault Integration

Architecture: Hashicorp Vault stores sensitive deployment secrets (database passwords, API keys). Secrets are injected into pods via Vault Agent sidecar.

Packages:

• vault/ - Vault client for storing/retrieving secrets
• vault/agent/ - Vault Agent annotation builder for pod templates
• vault/secretext/ - Proto field extension for marking secret fields
• k8s/deployer/secret_manager.go - Extracts secrets from deployment config, stores in Vault

Flow:

1. Deploy request contains output config with password
2. SecretManager extracts secret fields (marked with proto extension)
3. Secrets stored in Vault at secret/data/{orgID}/{deploymentID}
4. DSN rewritten with $ENV_VAR placeholders
5. Operator adds Vault Agent annotations to pod template
6. Vault Agent sidecar injects secrets as env vars at pod startup

9. Docker Build Pattern

Multi-stage Dockerfile:

FROM golang:1.26-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o control-freak ./cmd/control-freak

FROM alpine:latest
COPY --from=builder /app/control-freak .
CMD ["./control-freak"]

Build & Deploy to k3d:

docker build -t control-freak:latest .
docker tag control-freak:latest control-freak:local
k3d image import control-freak:local --cluster control-plane-local
kubectl delete pod -l app=control-freak -n control-plane-local

Important: Use correct image tag (control-freak:local for k3d)

10. Documentation Standards

FEATURES.md Structure:

- [x] Feature name
  - Implementation Tasks
    - [x] Task 1
    - [x] Task 2
  - Tests
    - [x] Test requirement 1
    - [x] Test requirement 2

Update after completing feature:

• Mark all tasks as [x]
• Add test count (e.g., "Verify all 44 tests pass")
• Keep implementation tasks as checklist for traceability

ISSUES.md: Keep concise

- [x] Issue description
  - Fixed: Brief solution description

TASKS.md: High-level task tracking

- [ ] Task description
  - Detailed subtasks or implementation notes
  
- [x] Completed task
  - Summary of what was accomplished

11. CI/CD Pipeline

Location: .github/workflows/ci.yml

CRITICAL: The CI workflow must stay in sync with the k8s/setup package. When modifying setup logic:

1. Always use the Go CLI commands (go run ./cmd/control-freak local setup) in CI
2. Never hardcode file paths to k8s/setup/assets/ - assets are embedded in the binary
3. The setup command handles everything: cluster creation, image build, manifest application

CI Workflow Structure:

- name: Setup k3d cluster and deploy
  run: go run ./cmd/control-freak local setup -v

This single command replaces manual steps for:

• Creating k3d cluster (uses embedded k3d-cluster.yaml)
• Building Docker image
• Loading image into k3d
• Creating all K8s resources via client-go (namespace, PostgreSQL, Vault, RBAC, CRDs, control-freak operator)
• Waiting for deployments
• Restarting control-freak

Testing CI Changes Locally:

# Simulate full CI setup
go run ./cmd/control-freak local teardown  # Clean slate
go run ./cmd/control-freak local setup -v  # Run setup with verbose output
go test ./k8s/tests/... -v                 # Run integration tests

12. Copilot Cloud Agent Environment

Location: .github/workflows/copilot-setup-steps.yml

GitHub Copilot cloud agent uses the same environment as CI. The copilot-setup-steps.yml workflow runs before Copilot starts working on any assigned task.

What the Setup Provides:

• Go 1.26 installed
• k3d cluster running with control-freak deployed
• Port forwards active (gRPC on 30051)
• All dependencies installed (kubectl, grpcurl)

CRITICAL: The copilot-setup-steps.yml must stay in sync with ci.yml. Both use:

- name: Setup k3d cluster and deploy
  run: go run ./cmd/control-freak local setup -v

13. Troubleshooting Patterns

Integration Tests Pass But Feature Doesn't Work (Stale Image):

Tests pass but the actual behavior in cluster is wrong
StatefulSet missing expected annotations or env vars
gRPC methods return old behavior

Cause: The control-freak image in k3d is outdated - tests ran against stale code. Solution: ALWAYS rebuild before running integration tests:

go run ./cmd/control-freak local restart
# Then run tests
go test ./k8s/tests/... -v

RBAC Permission Errors:

Error: pods "X" is forbidden: ... cannot get resource "pods/log"

Solution: Update the Role rules in createLocalControlFreak() (devenv.go) and redeploy.

Image Not Updating: Solution: Use the restart command which handles everything:

go run ./cmd/control-freak local restart

PostgreSQL Connection Issues: PostgreSQL runs in-cluster as postgres-db service. To connect directly:

kubectl port-forward -n control-plane-local service/postgres-db 5432:5432
psql postgres://testuser:testpass@localhost:5432/testdb?sslmode=disable

---

Setup Package

The k8s/setup package provides programmatic setup and teardown of the local k3d development environment using client-go. All Kubernetes resources (namespace, PostgreSQL, Vault, RBAC, CRDs, control-freak deployment) are created via Go code — no YAML manifests except k3d-cluster.yaml for the k3d CLI.

Embedded Assets: Only k3d-cluster.yaml is embedded (required by k3d CLI):

//go:embed assets/*
var assetsFS embed.FS

Key Functions:

import "github.com/streamingfast/services-control-plane/k8s/setup"

// Setup the k3d cluster with all components
err := setup.Setup(setup.Options{
    SkipBuild:   false,
    Verbose:     true,
    ProjectRoot: "/path/to",
    Stdout:      os.Stdout,
    Stderr:      os.Stderr,
})

// Teardown the k3d cluster
err := setup.Teardown(setup.Options{...})

// Rebuild and restart control-freak
err := setup.Restart(setup.Options{...})

// Start port forwards (returns handle to stop them)
result, err := setup.PortForward(setup.Options{...})
defer result.Stop()

// Helper functions
exists := setup.ClusterExists()
root, err := setup.FindProjectRoot()
client, err := setup.GetDevK8sClient()
err := setup.VerifyK3dCluster(client)
err := setup.WaitForDeployment(ctx, client, namespace, name)

CLI Commands (all under control-freak local):

Command	Description
setup	Create k3d cluster, build image, deploy all components
teardown	Delete the k3d cluster
restart	Rebuild image and restart control-freak deployment
port-forward	Start port forwards (runs in foreground)

---

Code Patterns

gRPC Service Method

func (s *Hosted) MethodName(ctx context.Context, req *pbprivateservice.Request) (*pbprivateservice.Response, error) {
    // 1. Log
    s.logger.Info("method request received", zap.String("id", req.GetDeploymentId()))
    
    // 2. Validate
    if req.GetDeploymentId() == "" {
        return &pbprivateservice.Response{Success: false, Message: "deployment_id required"}, nil
    }
    
    // 3. Find CRD
    cr, err := s.getCRByDeploymentID(ctx, req.GetDeploymentId())
    if err != nil {
        return &pbprivateservice.Response{Success: false, Message: "deployment not found"}, nil
    }
    
    // 4. Check org_id
    if !s.checkCROrgID(cr, req.GetOrganizationId()) {
        return &pbprivateservice.Response{Success: false, Message: "deployment not found"}, nil
    }
    
    // 5. Process (modify CRD, query k8s, etc.)
    // ...
    
    // 6. Persist to database (fire-and-forget)
    if s.deploymentStore != nil {
        if err := s.deploymentStore.UpdateSomething(ctx, ...); err != nil {
            s.logger.Warn("failed to update database", zap.Error(err))
        }
    }
    
    // 7. Record event (fire-and-forget)
    s.recordEvent(ctx, deploymentID, orgID, event.Type, details)
    
    // 8. Return
    return &pbprivateservice.Response{Success: true}, nil
}

Hosted Service Options Pattern

// Options are applied via functional options pattern
hosted := services.NewHosted(k8sClient, crdClient, namespace, db, logger,
    services.WithEventRecorder(eventRecorder),
    services.WithDeploymentStore(deploymentStore),
)

// Or with Vault
hosted := services.NewHostedWithVault(k8sClient, crdClient, namespace, db, vaultClient, logger,
    services.WithEventRecorder(eventRecorder),
    services.WithDeploymentStore(deploymentStore),
)

Context Cancellation for Goroutines

func (m *Manager) Start(id string) {
    ctx, cancel := context.WithCancel(context.Background())
    m.running[id] = cancel
    
    go func() {
        ticker := time.NewTicker(interval)
        defer ticker.Stop()
        
        for {
            select {
            case <-ctx.Done():
                return  // Clean shutdown
            case <-ticker.C:
                // Do work
            }
        }
    }()
}

func (m *Manager) Stop(id string) {
    if cancel, ok := m.running[id]; ok {
        cancel()
        delete(m.running, id)
    }
}

---

Best Practices

1. Always validate inputs in gRPC methods before processing
2. Use structured logging with zap for better observability
3. Return success/failure in response, don't use gRPC errors for business logic failures
4. Mock external dependencies in unit tests (Kubernetes client, CRD client)
5. Test happy path AND error cases in both unit and integration tests
6. Use table-driven tests in Go for multiple scenarios
7. Keep CLI responsive - fetch data async, show loading states
8. Clean up resources - cancel contexts, close connections
9. Document as you go - update FEATURES.md immediately after implementing
10. Test in k3d before production - catch issues early in realistic environment
11. Commit after every feature, ISSUES and TASKS completed and tested
12. NEVER auto-merge feature branches - always wait for user approval before merging
13. Fire-and-forget for database writes - never fail RPCs on persistence errors, except in Hosted.Deploy where persistence is part of the success contract and the handler compensates by deleting the CR / registry row / vault secrets on failure
14. Use CRDs as the source of truth for live deployment state
15. Use PostgreSQL as durable store for deployment data that survives cluster rebuilds

---

Project Structure

services-control-plane/
├── .github/
│   ├── copilot-instructions.md
│   └── workflows/
│       ├── ci.yml
│       └── copilot-setup-steps.yml
├── api/
│   └── v1alpha1/               # CRD type definitions
│       ├── types.go            # HostedServiceDeployment CRD spec/status
│       ├── groupversion_info.go
│       └── zz_generated.deepcopy.go
├── cmd/
│   └── control-freak/          # Main server entry point
│       └── cmd/
│           ├── serve.go        # gRPC server startup, wiring
│           ├── local.go        # local parent command
│           ├── setup.go        # local setup command
│           ├── teardown.go     # local teardown command
│           ├── restart.go      # local restart command
│           ├── port_forward.go # local port-forward command
├── store/                      # Data persistence (PostgreSQL)
│   ├── psql/
│   │   └── migrations/         # SQL migration files (ALL schema changes go here)
│   │       ├── 000001_init.up.sql
│   │       ├── 000002_deployments.up.sql
│   │       ├── 000003_deployment_events.up.sql
│   │       └── 000004_registry.up.sql
│   ├── deployment/             # Deployment data persistence
│   │   ├── store.go            # CRUD operations
│   │   ├── types.go            # Deployment struct, status constants
│   │   └── store_test.go       # Unit tests
│   └── event/                  # Deployment event tracking
│       ├── store.go            # Event recording and querying
│       ├── types.go            # Type, DeploymentEvent, Recorder interface
│       └── store_test.go
├── dsn/                        # DSN builder for database connections
│   └── dsn.go
├── grpc/
│   ├── server/                 # gRPC server setup and registration
│   └── services/
│       ├── hosted.go           # HostedService: Deploy, Undeploy, SetReplica, Logs, etc.
│       ├── hosted_test.go      # Unit tests with fake k8s/CRD clients
│       ├── admin.go            # AdminService: delegates to Hosted with org_id enforcement
│       ├── network_registry.go # NetworkRegistry gRPC service
│       └── foundation_store_registry.go
├── k8s/
│   ├── setup/                  # Local dev environment (client-go, no YAML manifests)
│   │   ├── devenv.go           # Setup, Teardown, Restart, PortForward
│   │   ├── isolated_ns.go      # Isolated namespace for parallel testing
│   │   └── assets/
│   │       └── k3d-cluster.yaml
│   ├── tests/                  # Go integration tests
│   │   └── integration_test.go # All integration test suites (16 suites)
│   ├── deployer/               # Deployment builders
│   │   ├── sink_sql_from_proto.go
│   │   ├── foundational_store.go
│   │   ├── secret_manager.go   # Vault secret extraction
│   │   └── common.go
│   └── operator/               # CRD operator/controller
│       ├── controller.go       # Reconcile loop
│       └── status.go           # Status update logic
├── registry/                   # Network & foundation store registries (PostgreSQL)
│   ├── schema.go               # Table definitions (for tests only)
│   ├── network.go
│   ├── foundation_store.go
│   └── *_test.go
├── vault/                      # Hashicorp Vault integration
│   ├── client.go               # Vault client (store/get/delete secrets)
│   ├── agent/                  # Vault Agent annotation builder
│   │   └── annotations.go
│   └── secretext/              # Proto secret field extension
│       └── secretext.go
├── cli/                        # TypeScript CLI
│   ├── screens/                # UI screens
│   ├── grpc.ts                 # gRPC client
│   └── index.ts                # Main CLI logic
├── proto-internal/             # Proto definitions
│   └── sf/hosted/
│       ├── common/v1/          # Shared message types
│       └── private/service/v1/ # Service definitions
├── pb/                         # Generated proto code
├── script/                     # Shell scripts
│   ├── migrate.sh              # Database migration runner (uses golang-migrate)
│   ├── wipe-db.sh              # Wipe database (development only)
│   └── motd.sh
├── Dockerfile                  # Multi-stage build
├── FEATURES.md                 # Feature tracking
├── ISSUES.md                   # Issue tracking
└── TASKS.md                    # Task list

---

Quick Reference Commands

# Development
buf generate                    # Regenerate protos
NODE_ENV=local bun run cli/index.ts  # Run CLI locally

# Cluster Management (Go CLI)
go run ./cmd/control-freak local setup              # Setup k3d cluster
go run ./cmd/control-freak local setup --skip-build # Skip Docker build
go run ./cmd/control-freak local setup -v           # Verbose output
go run ./cmd/control-freak local teardown           # Teardown cluster
go run ./cmd/control-freak local restart            # Rebuild & restart server (RUN THIS BEFORE INTEGRATION TESTS!)
go run ./cmd/control-freak local port-forward       # Start port forwards

# Testing (requires k3d cluster running)
# IMPORTANT: Always run 'local restart' first to ensure cluster has latest code!
go run ./cmd/control-freak local restart && go test ./k8s/tests/... -v  # Rebuild + test (RECOMMENDED)
go test ./k8s/tests/... -v                    # Run all integration tests (auto-setup)
KEEP_CLUSTER=true go test ./k8s/tests/... -v  # Keep cluster after tests
go test ./k8s/tests/... -run TestSetReplica -v  # Run specific test
go test ./... -v -count 1                     # Run all tests (unit + integration)

# Kubernetes
kubectl get hostedservicedeployments -n control-plane-local  # List CRDs
kubectl get pods -n control-plane-local
kubectl logs -f deployment/control-freak -n control-plane-local
kubectl describe hostedservicedeployment <name> -n control-plane-local

# PostgreSQL (requires port-forward)
kubectl port-forward -n control-plane-local service/postgres-db 5432:5432
psql "postgres://testuser:testpass@localhost:5432/testdb?sslmode=disable"
# Useful queries:
# SELECT * FROM deployments;
# SELECT * FROM deployment_events ORDER BY created_at DESC LIMIT 10;

# Database Migrations (ALL schema changes must use these)
./script/migrate.sh version              # Check current migration version
./script/migrate.sh up                   # Apply all pending migrations
./script/migrate.sh up 4                 # Migrate up to version 4
./script/migrate.sh down                 # Roll back one migration
./script/migrate.sh new my_feature       # Create new migration files
./script/migrate.sh force 3              # Force version (dirty state recovery)

# Docker (manual rebuild - prefer 'local restart' command)
docker build -t control-freak:local -f Dockerfile .
k3d image import control-freak:local -c control-plane-local
kubectl --context k3d-control-plane-local rollout restart deployment control-freak -n control-plane-local