Menu
Learn how to develop a Kubernetes-based control plane for managing deployments with gRPC API, real-time state tracking, and a Terminal UI.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
| name | services-control-plane-dev |
| title | Services Control Plane Development Skill Set |
| description | Learn how to develop a Kubernetes-based control plane for managing deployments with gRPC API, real-time state tracking, and a Terminal UI. |
This skill set documents the patterns, practices, and workflows established during the development of the services-control-plane project.
CRITICAL: After every code change, run go test ./... -v -count 1 and go build ./... to verify all tests pass and the project compiles. Do not consider a task complete until all tests pass.
CRITICAL: NEVER run tests or manual testing commands against production clusters. All testing MUST be done against the local k3d cluster only.
CRITICAL: When adding or modifying gRPC RPCs, you MUST add integration tests in k8s/tests/integration_test.go. Integration tests validate the full stack (gRPC → server → Kubernetes → PostgreSQL). Unit tests alone are NOT sufficient for gRPC changes.
Goal: Build a Kubernetes-based control plane for managing deployments with gRPC API, real-time state tracking, and a Terminal UI.
Stack:
Architecture:
HostedServiceDeployment Custom Resourcesdeployments table, events in deployment_events table, registries in dedicated tablesNEVER auto-merge feature branches. Always follow this workflow:
This applies to:
Why? The user must have the opportunity to:
When implementing a new feature, follow this sequence:
buf generateExample: SetReplica feature implementation
// 1. After implementing SetReplica in grpc/services/hosted.go
// 2. Write unit test in grpc/services/hosted_test.go
func TestSetReplica_ScaleUp(t *testing.T) {
env := SetupTestEnv(t)
defer env.Cleanup()
hosted := createHostedService(t, env.K8sClient, nil)
resp, err := hosted.SetReplica(ctx, &pbcommon.ReplicaRequest{
DeploymentId: "test-deployment-3",
Count: 3,
})
require.NoError(t, err)
require.True(t, resp.Success)
}
go test ./... -vgo test ./k8s/tests/... -vAlways start with protocol buffer definitions:
service HostedService {
rpc Logs(LogsRequest) returns (LogsResponse);
}
message LogsRequest {
string deployment_id = 1;
string pod_name = 2; // optional
int32 tail_lines = 3; // default: 100
}
message LogsResponse {
bool success = 1;
string message = 2;
repeated PodLogs pod_logs = 3;
}
Benefits:
Setup: Use k3d for local Kubernetes clusters
Go CLI Commands:
# Initial setup
go run ./cmd/control-freak local setup
# After code changes - rebuild and restart
go run ./cmd/control-freak local restart
# Start port forwards (runs in foreground, Ctrl+C to stop)
go run ./cmd/control-freak local port-forward
# Teardown
go run ./cmd/control-freak local teardown
Key Files:
k8s/setup/devenv.go - Programmatic setup/teardown using client-go (no YAML manifests)k8s/setup/assets/k3d-cluster.yaml - k3d cluster configuration with port mappingsPort Mappings:
localhost:30051 → control-freak:9000In-cluster services (not port-forwarded by default):
postgres-db:5432 (internal, use kubectl port-forward when needed)vault:8200 (internal)CRITICAL: Keep control-freak Image Up to Date
Before running integration tests, ALWAYS rebuild and redeploy the control-freak image to ensure the cluster has your latest code changes:
# Rebuild, import to k3d, and restart control-freak
go run ./cmd/control-freak local restart
Or manually:
docker build -t control-freak:local -f Dockerfile .
k3d image import control-freak:local -c control-plane-local
kubectl --context k3d-control-plane-local rollout restart deployment control-freak -n control-plane-local
kubectl --context k3d-control-plane-local rollout status deployment control-freak -n control-plane-local --timeout=60s
Why this matters: Integration tests call the control-freak gRPC server running in the cluster. If you've made code changes but haven't rebuilt the image, the tests will run against stale code and may pass incorrectly (or fail unexpectedly).
Go Integration Tests Against k3d Cluster (PRIMARY)
All tests run against a real k3d cluster. No fake clients - tests validate actual Kubernetes and gRPC behavior.
Location: k8s/tests/integration_test.go
Framework: github.com/stretchr/testify/require
Dependencies: k8s.io/client-go/kubernetes
IMPORTANT: Integration Test Patterns:
KEEP_CRD=false to clean up CRDs after testsAutomatic Setup/Teardown:
setup.Setup())KEEP_CLUSTER=true to keep cluster running after testsRun Tests:
go test ./k8s/tests/... -v # Run all integration tests
KEEP_CLUSTER=true go test ./k8s/tests/... -v # Keep cluster after tests
go test ./k8s/tests/... -run TestSetReplica -v # Run specific test
go test ./... -v -count 1 # Run all tests (unit + integration)
SAFETY: Tests will REFUSE TO RUN against non-k3d clusters:
localhost or 127.0.0.1Test Environment Setup:
type TestEnv struct {
K8sClient kubernetes.Interface // Real k8s client
DynamicClient dynamic.Interface // Dynamic client for CRDs
GRPCConn *grpc.ClientConn // Real gRPC connection
GRPCClient pbprivateservice.HostedServiceClient
Namespace string
}
func SetupTestEnv(t *testing.T) *TestEnv {
// SAFETY: Refuses to run if not connected to a k3d cluster
// Checks that API server is localhost/127.0.0.1
// Checks that node names contain "k3d", "k3s", or the cluster name
// Connects to gRPC on localhost:30051
}
Test Helper Functions:
// CleanupDeployment - Delete CRD and StatefulSet for a deployment
func (e *TestEnv) CleanupDeployment(deploymentID string)
// DeploySinkSql - Create a sink-sql deployment via gRPC Deploy
// Handles cleanup of existing deployment, deploys via gRPC, waits for StatefulSet
func (e *TestEnv) DeploySinkSql(deploymentID, organizationID, spkgURL string) string
// ShouldKeepCRD - Returns true unless KEEP_CRD=false
func ShouldKeepCRD() bool
// WaitFor - Poll until condition is met or timeout
func (e *TestEnv) WaitFor(timeout time.Duration, description string, condition func() bool)
// ScaleDownDeployment - Scale a deployment to 0 replicas (for cleanup without deleting CRD)
func (e *TestEnv) ScaleDownDeployment(deploymentID string)
// getPostgresDB - Start port-forward to in-cluster postgres, return *sql.DB + cleanup func
func getPostgresDB(t *testing.T, env *TestEnv) (*sql.DB, func())
Example Test (Current Pattern):
func TestDeploymentStore_DeployPersistsData(t *testing.T) {
env := SetupTestEnv(t)
defer env.Cleanup()
ctx := context.Background()
deploymentID := "depstore-test-deploy"
organizationID := "test-org-depstore"
// Deploy via gRPC
_ = env.DeploySinkSql(deploymentID, organizationID, testSpkgURL)
// Connect to postgres and verify data persisted
db, cleanup := getPostgresDB(t, env)
defer cleanup()
var depID, status string
err := db.QueryRowContext(ctx,
`SELECT deployment_id, status FROM deployments WHERE deployment_id = $1`,
deploymentID).Scan(&depID, &status)
require.NoError(t, err)
require.Equal(t, "active", status)
// Cleanup
if !ShouldKeepCRD() {
env.CleanupDeployment(deploymentID)
}
}
When Implementing New Features:
k8s/tests/integration_test.goenv.DeploySinkSql() to create test deployments via gRPCenv.WaitFor() for async operationsShouldKeepCRD())go run ./cmd/control-freak local restart to ensure cluster has latest codego test ./k8s/tests/... -v before committingMANDATORY Integration Tests for gRPC Changes: When adding or modifying any gRPC RPC, you MUST add integration tests that cover:
Example integration test pattern:
// ============================================================================
// Test Suite N: <FeatureName> Tests
// ============================================================================
func Test<Feature>_Success(t *testing.T) {
env := SetupTestEnv(t)
defer env.Cleanup()
ctx := context.Background()
deploymentID := "feature-test"
organizationID := "test-org"
// Deploy via gRPC
stsName := env.DeploySinkSql(deploymentID, organizationID, testSpkgURL)
// Call the RPC being tested
resp, err := env.GRPCClient.<Method>(ctx, &pbprivateservice.<Request>{
DeploymentId: deploymentID,
})
require.NoError(t, err)
require.True(t, resp.Success, "should succeed: %s", resp.Message)
// Verify expected state
// ...
// Cleanup
if !ShouldKeepCRD() {
env.CleanupDeployment(deploymentID)
}
}
func Test<Feature>_EmptyDeploymentID(t *testing.T) {
env := SetupTestEnv(t)
defer env.Cleanup()
ctx := context.Background()
resp, err := env.GRPCClient.<Method>(ctx, &pbprivateservice.<Request>{
DeploymentId: "",
})
require.NoError(t, err)
require.False(t, resp.Success, "should fail with empty deployment_id")
require.Contains(t, resp.Message, "deployment_id required")
}
Test Suites Covered:
Screen Architecture:
screens/*.ts)index.ts coordinates screen navigationExample Screen:
export function createLogsScreen(renderer: CliRenderer) {
const logsScreen = new BoxRenderable(renderer, {
flexDirection: "column",
width: "100%",
height: "100%",
})
const logsContainer = new BoxRenderable(renderer, {
flexGrow: 1,
border: true,
})
function update(deploymentId: string, logs: string) {
// Update screen content
}
return { logsScreen, logsContainer, update }
}
Key Binding Pattern:
renderer.keyInput.on("keypress", (key: KeyEvent) => {
if (keyName === "l") {
showLogsScreen(selectedPod)
}
})
CRD-Based Deployment Management:
Deployments are managed via HostedServiceDeployment Custom Resources. The operator watches CRDs and reconciles them into StatefulSets.
// CRD spec contains all deployment configuration
type HostedServiceDeploymentSpec struct {
DeploymentID string `json:"deploymentId"`
DeploymentName string `json:"deploymentName,omitempty"`
OrganizationID string `json:"organizationId"`
DeploymentType string `json:"deploymentType"`
Replicas int32 `json:"replicas"`
SpkgURL string `json:"spkgUrl,omitempty"`
OutputModule string `json:"outputModule,omitempty"`
StartBlock int64 `json:"startBlock,omitempty"`
StopBlock int64 `json:"stopBlock,omitempty"`
Parameters string `json:"parameters,omitempty"`
DSN string `json:"dsn,omitempty"`
APIKey string `json:"apiKey,omitempty"`
DeploymentRequestJSON string `json:"deploymentRequestJson,omitempty"`
VaultSecretPath string `json:"vaultSecretPath,omitempty"`
VaultSecretKeys map[string]string `json:"vaultSecretKeys,omitempty"`
// ... more fields
}
// CRD status is updated by the operator with live state
type HostedServiceDeploymentStatus struct {
State string `json:"state,omitempty"`
Replicas int32 `json:"replicas,omitempty"`
ReadyReplicas int32 `json:"readyReplicas,omitempty"`
StatefulSetName string `json:"statefulSetName,omitempty"`
PodStates []PodExecutionState `json:"podStates,omitempty"`
CrashLoopBackOff bool `json:"crashLoopBackOff,omitempty"`
MaxRestartCount int32 `json:"maxRestartCount,omitempty"`
}
RBAC Pattern: Always create ServiceAccount + Role + RoleBinding using client-go:
// ServiceAccount
sa := &corev1.ServiceAccount{
ObjectMeta: metav1.ObjectMeta{
Name: "control-freak",
Namespace: namespace,
},
}
client.CoreV1().ServiceAccounts(namespace).Create(ctx, sa, metav1.CreateOptions{})
// Role
role := &rbacv1.Role{
ObjectMeta: metav1.ObjectMeta{
Name: "control-freak",
Namespace: namespace,
},
Rules: []rbacv1.PolicyRule{
{
APIGroups: []string{"apps"},
Resources: []string{"statefulsets"},
Verbs: []string{"get", "list", "watch", "create", "update", "patch", "delete"},
},
{
APIGroups: []string{""},
Resources: []string{"pods", "pods/log"},
Verbs: []string{"get", "list"},
},
{
APIGroups: []string{""},
Resources: []string{"services", "secrets", "persistentvolumeclaims"},
Verbs: []string{"get", "list", "watch", "create", "update", "patch", "delete"},
},
{
APIGroups: []string{"hosted.streamingfast.io"},
Resources: []string{"hostedservicedeployments", "hostedservicedeployments/status"},
Verbs: []string{"get", "list", "watch", "create", "update", "patch", "delete"},
},
},
}
Label Selectors: Use consistent labels for querying:
labels := map[string]string{
"deployment-id": deploymentID,
"organization-id": organizationID,
"managed-by": "services-control-plane",
"deployment-type": deploymentType,
}
Architecture: PostgreSQL is the durable store for all deployment data. The CRDs are the live state, but PostgreSQL persists the data across cluster rebuilds.
CRITICAL: Schema Management
All database schema changes MUST be done via SQL migrations:
store/psql/migrations/./script/migrate.shInitSchema()-style helpers that create tables from Go code.Migration Commands:
# Check current version
./script/migrate.sh version
# Apply all pending migrations
./script/migrate.sh up
# Apply up to a specific version
./script/migrate.sh up 4
# Roll back one migration
./script/migrate.sh down
# Create a new migration
./script/migrate.sh new my_new_table
# Force version (for dirty state recovery)
./script/migrate.sh force 3
Creating New Migrations:
./script/migrate.sh new descriptive_name.up.sql and .down.sql files./script/migrate.sh up./script/migrate.sh downTables:
| Table | Purpose | Package |
|---|---|---|
deployments | All deployment configuration and metadata | store/deployment |
deployment_events | Lifecycle events (created, deleted, scaled, etc.) | store/event |
foundation_store_registry | Foundational store deployment registry | registry |
network_endpoints | Network endpoint registry | registry |
network_aliases | Network name → endpoint mappings | registry |
Deployment Store (store/deployment/):
// Store manages deployment records in PostgreSQL
// NOTE: Schema is created by migrations, not Go code
store := deployment.NewStore(db)
// CRUD operations
store.Create(ctx, &Deployment{...}) // Upsert on deploy
store.Get(ctx, deploymentID) // Get by ID
store.ListByOrganization(ctx, orgID) // List by org
store.UpdateStatus(ctx, deploymentID, "deleted") // Mark deleted on undeploy
store.UpdateReplicas(ctx, deploymentID, 3) // Update on SetReplica
store.UpdateConfig(ctx, deploymentID, updates) // Update on config change
Deployments table schema:
CREATE TABLE IF NOT EXISTS deployments (
deployment_id TEXT PRIMARY KEY,
organization_id TEXT NOT NULL,
deployment_name TEXT NOT NULL DEFAULT '',
deployment_type TEXT NOT NULL DEFAULT '',
spkg_url TEXT NOT NULL DEFAULT '',
output_module TEXT NOT NULL DEFAULT '',
start_block BIGINT NOT NULL DEFAULT 0,
stop_block BIGINT NOT NULL DEFAULT 0,
parameters TEXT NOT NULL DEFAULT '',
dsn TEXT NOT NULL DEFAULT '',
api_key TEXT NOT NULL DEFAULT '',
replicas INT NOT NULL DEFAULT 1,
deployment_request_json JSONB DEFAULT '{}',
status TEXT NOT NULL DEFAULT 'active',
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
Event Store (store/event/):
// Store records deployment lifecycle events
// NOTE: Schema is created by migrations, not Go code
es := event.NewStore(postgresDSN)
es.RecordEvent(ctx, &DeploymentEvent{...})
es.GetEvents(ctx, deploymentID, limit)
es.GetEventsByOrg(ctx, orgID, limit)
Fire-and-forget pattern: Database writes from gRPC handlers are fire-and-forget — failures are logged but never cause RPCs to fail:
// In SetReplica / Undeploy / UpdateDeploymentConfig handlers:
if s.deploymentStore != nil {
if err := s.deploymentStore.UpdateReplicas(ctx, deploymentID, n); err != nil {
s.logger.Warn("failed to update replicas in database", zap.Error(err))
}
}
Exception — Hosted.Deploy is atomic. Deploy is the one handler where persistence is part of the success contract: the rows it writes to foundation_store_registry and deployments are read by every downstream handler (GetDeployment, SetReplica, UpdateDeploymentConfig, Undeploy) and there is no reconciler that backfills them from the CR. A Success response from Deploy means all of these are true:
HostedServiceDeployment CR exists.foundation_store_registry row exists (if the deploy is a foundational store).deployments row exists.vaultClient is configured, have been stored — there is no silent fallback to plaintext credentials in the CR spec.Any failure between steps 1 and 3 triggers cleanupAfterFailedDeploy (grpc/services/hosted.go), which best-effort undoes prior side effects in reverse order using a fresh context with a 10s timeout. The compensation makes Deploy idempotent under retry — a failed deploy leaves no orphaned CR to trip AlreadyExists on the next attempt.
Connection: PostgreSQL DSN is passed via --postgres-dsn flag or POSTGRES_DSN env var. The event.Store opens the connection, and es.DB() is shared across all stores (deployment store, event store, registries).
Architecture: Hashicorp Vault stores sensitive deployment secrets (database passwords, API keys). Secrets are injected into pods via Vault Agent sidecar.
Packages:
vault/ - Vault client for storing/retrieving secretsvault/agent/ - Vault Agent annotation builder for pod templatesvault/secretext/ - Proto field extension for marking secret fieldsk8s/deployer/secret_manager.go - Extracts secrets from deployment config, stores in VaultFlow:
SecretManager extracts secret fields (marked with proto extension)secret/data/{orgID}/{deploymentID}$ENV_VAR placeholdersMulti-stage Dockerfile:
FROM golang:1.26-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o control-freak ./cmd/control-freak
FROM alpine:latest
COPY --from=builder /app/control-freak .
CMD ["./control-freak"]
Build & Deploy to k3d:
docker build -t control-freak:latest .
docker tag control-freak:latest control-freak:local
k3d image import control-freak:local --cluster control-plane-local
kubectl delete pod -l app=control-freak -n control-plane-local
Important: Use correct image tag (control-freak:local for k3d)
FEATURES.md Structure:
- [x] Feature name
- Implementation Tasks
- [x] Task 1
- [x] Task 2
- Tests
- [x] Test requirement 1
- [x] Test requirement 2
Update after completing feature:
[x]ISSUES.md: Keep concise
- [x] Issue description
- Fixed: Brief solution description
TASKS.md: High-level task tracking
- [ ] Task description
- Detailed subtasks or implementation notes
- [x] Completed task
- Summary of what was accomplished
Location: .github/workflows/ci.yml
CRITICAL: The CI workflow must stay in sync with the k8s/setup package. When modifying setup logic:
go run ./cmd/control-freak local setup) in CIk8s/setup/assets/ - assets are embedded in the binaryCI Workflow Structure:
- name: Setup k3d cluster and deploy
run: go run ./cmd/control-freak local setup -v
This single command replaces manual steps for:
k3d-cluster.yaml)Testing CI Changes Locally:
# Simulate full CI setup
go run ./cmd/control-freak local teardown # Clean slate
go run ./cmd/control-freak local setup -v # Run setup with verbose output
go test ./k8s/tests/... -v # Run integration tests
Location: .github/workflows/copilot-setup-steps.yml
GitHub Copilot cloud agent uses the same environment as CI. The copilot-setup-steps.yml workflow runs before Copilot starts working on any assigned task.
What the Setup Provides:
CRITICAL: The copilot-setup-steps.yml must stay in sync with ci.yml. Both use:
- name: Setup k3d cluster and deploy
run: go run ./cmd/control-freak local setup -v
Integration Tests Pass But Feature Doesn't Work (Stale Image):
Tests pass but the actual behavior in cluster is wrong
StatefulSet missing expected annotations or env vars
gRPC methods return old behavior
Cause: The control-freak image in k3d is outdated - tests ran against stale code. Solution: ALWAYS rebuild before running integration tests:
go run ./cmd/control-freak local restart
# Then run tests
go test ./k8s/tests/... -v
RBAC Permission Errors:
Error: pods "X" is forbidden: ... cannot get resource "pods/log"
Solution: Update the Role rules in createLocalControlFreak() (devenv.go) and redeploy.
Image Not Updating: Solution: Use the restart command which handles everything:
go run ./cmd/control-freak local restart
PostgreSQL Connection Issues:
PostgreSQL runs in-cluster as postgres-db service. To connect directly:
kubectl port-forward -n control-plane-local service/postgres-db 5432:5432
psql postgres://testuser:testpass@localhost:5432/testdb?sslmode=disable
The k8s/setup package provides programmatic setup and teardown of the local k3d development environment using client-go. All Kubernetes resources (namespace, PostgreSQL, Vault, RBAC, CRDs, control-freak deployment) are created via Go code — no YAML manifests except k3d-cluster.yaml for the k3d CLI.
Embedded Assets: Only k3d-cluster.yaml is embedded (required by k3d CLI):
//go:embed assets/*
var assetsFS embed.FS
Key Functions:
import "github.com/streamingfast/services-control-plane/k8s/setup"
// Setup the k3d cluster with all components
err := setup.Setup(setup.Options{
SkipBuild: false,
Verbose: true,
ProjectRoot: "/path/to",
Stdout: os.Stdout,
Stderr: os.Stderr,
})
// Teardown the k3d cluster
err := setup.Teardown(setup.Options{...})
// Rebuild and restart control-freak
err := setup.Restart(setup.Options{...})
// Start port forwards (returns handle to stop them)
result, err := setup.PortForward(setup.Options{...})
defer result.Stop()
// Helper functions
exists := setup.ClusterExists()
root, err := setup.FindProjectRoot()
client, err := setup.GetDevK8sClient()
err := setup.VerifyK3dCluster(client)
err := setup.WaitForDeployment(ctx, client, namespace, name)
CLI Commands (all under control-freak local):
| Command | Description |
|---|---|
setup | Create k3d cluster, build image, deploy all components |
teardown | Delete the k3d cluster |
restart | Rebuild image and restart control-freak deployment |
port-forward | Start port forwards (runs in foreground) |
func (s *Hosted) MethodName(ctx context.Context, req *pbprivateservice.Request) (*pbprivateservice.Response, error) {
// 1. Log
s.logger.Info("method request received", zap.String("id", req.GetDeploymentId()))
// 2. Validate
if req.GetDeploymentId() == "" {
return &pbprivateservice.Response{Success: false, Message: "deployment_id required"}, nil
}
// 3. Find CRD
cr, err := s.getCRByDeploymentID(ctx, req.GetDeploymentId())
if err != nil {
return &pbprivateservice.Response{Success: false, Message: "deployment not found"}, nil
}
// 4. Check org_id
if !s.checkCROrgID(cr, req.GetOrganizationId()) {
return &pbprivateservice.Response{Success: false, Message: "deployment not found"}, nil
}
// 5. Process (modify CRD, query k8s, etc.)
// ...
// 6. Persist to database (fire-and-forget)
if s.deploymentStore != nil {
if err := s.deploymentStore.UpdateSomething(ctx, ...); err != nil {
s.logger.Warn("failed to update database", zap.Error(err))
}
}
// 7. Record event (fire-and-forget)
s.recordEvent(ctx, deploymentID, orgID, event.Type, details)
// 8. Return
return &pbprivateservice.Response{Success: true}, nil
}
// Options are applied via functional options pattern
hosted := services.NewHosted(k8sClient, crdClient, namespace, db, logger,
services.WithEventRecorder(eventRecorder),
services.WithDeploymentStore(deploymentStore),
)
// Or with Vault
hosted := services.NewHostedWithVault(k8sClient, crdClient, namespace, db, vaultClient, logger,
services.WithEventRecorder(eventRecorder),
services.WithDeploymentStore(deploymentStore),
)
func (m *Manager) Start(id string) {
ctx, cancel := context.WithCancel(context.Background())
m.running[id] = cancel
go func() {
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return // Clean shutdown
case <-ticker.C:
// Do work
}
}
}()
}
func (m *Manager) Stop(id string) {
if cancel, ok := m.running[id]; ok {
cancel()
delete(m.running, id)
}
}
Hosted.Deploy where persistence is part of the success contract and the handler compensates by deleting the CR / registry row / vault secrets on failureservices-control-plane/
├── .github/
│ ├── copilot-instructions.md
│ └── workflows/
│ ├── ci.yml
│ └── copilot-setup-steps.yml
├── api/
│ └── v1alpha1/ # CRD type definitions
│ ├── types.go # HostedServiceDeployment CRD spec/status
│ ├── groupversion_info.go
│ └── zz_generated.deepcopy.go
├── cmd/
│ └── control-freak/ # Main server entry point
│ └── cmd/
│ ├── serve.go # gRPC server startup, wiring
│ ├── local.go # local parent command
│ ├── setup.go # local setup command
│ ├── teardown.go # local teardown command
│ ├── restart.go # local restart command
│ ├── port_forward.go # local port-forward command
├── store/ # Data persistence (PostgreSQL)
│ ├── psql/
│ │ └── migrations/ # SQL migration files (ALL schema changes go here)
│ │ ├── 000001_init.up.sql
│ │ ├── 000002_deployments.up.sql
│ │ ├── 000003_deployment_events.up.sql
│ │ └── 000004_registry.up.sql
│ ├── deployment/ # Deployment data persistence
│ │ ├── store.go # CRUD operations
│ │ ├── types.go # Deployment struct, status constants
│ │ └── store_test.go # Unit tests
│ └── event/ # Deployment event tracking
│ ├── store.go # Event recording and querying
│ ├── types.go # Type, DeploymentEvent, Recorder interface
│ └── store_test.go
├── dsn/ # DSN builder for database connections
│ └── dsn.go
├── grpc/
│ ├── server/ # gRPC server setup and registration
│ └── services/
│ ├── hosted.go # HostedService: Deploy, Undeploy, SetReplica, Logs, etc.
│ ├── hosted_test.go # Unit tests with fake k8s/CRD clients
│ ├── admin.go # AdminService: delegates to Hosted with org_id enforcement
│ ├── network_registry.go # NetworkRegistry gRPC service
│ └── foundation_store_registry.go
├── k8s/
│ ├── setup/ # Local dev environment (client-go, no YAML manifests)
│ │ ├── devenv.go # Setup, Teardown, Restart, PortForward
│ │ ├── isolated_ns.go # Isolated namespace for parallel testing
│ │ └── assets/
│ │ └── k3d-cluster.yaml
│ ├── tests/ # Go integration tests
│ │ └── integration_test.go # All integration test suites (16 suites)
│ ├── deployer/ # Deployment builders
│ │ ├── sink_sql_from_proto.go
│ │ ├── foundational_store.go
│ │ ├── secret_manager.go # Vault secret extraction
│ │ └── common.go
│ └── operator/ # CRD operator/controller
│ ├── controller.go # Reconcile loop
│ └── status.go # Status update logic
├── registry/ # Network & foundation store registries (PostgreSQL)
│ ├── schema.go # Table definitions (for tests only)
│ ├── network.go
│ ├── foundation_store.go
│ └── *_test.go
├── vault/ # Hashicorp Vault integration
│ ├── client.go # Vault client (store/get/delete secrets)
│ ├── agent/ # Vault Agent annotation builder
│ │ └── annotations.go
│ └── secretext/ # Proto secret field extension
│ └── secretext.go
├── cli/ # TypeScript CLI
│ ├── screens/ # UI screens
│ ├── grpc.ts # gRPC client
│ └── index.ts # Main CLI logic
├── proto-internal/ # Proto definitions
│ └── sf/hosted/
│ ├── common/v1/ # Shared message types
│ └── private/service/v1/ # Service definitions
├── pb/ # Generated proto code
├── script/ # Shell scripts
│ ├── migrate.sh # Database migration runner (uses golang-migrate)
│ ├── wipe-db.sh # Wipe database (development only)
│ └── motd.sh
├── Dockerfile # Multi-stage build
├── FEATURES.md # Feature tracking
├── ISSUES.md # Issue tracking
└── TASKS.md # Task list
# Development
buf generate # Regenerate protos
NODE_ENV=local bun run cli/index.ts # Run CLI locally
# Cluster Management (Go CLI)
go run ./cmd/control-freak local setup # Setup k3d cluster
go run ./cmd/control-freak local setup --skip-build # Skip Docker build
go run ./cmd/control-freak local setup -v # Verbose output
go run ./cmd/control-freak local teardown # Teardown cluster
go run ./cmd/control-freak local restart # Rebuild & restart server (RUN THIS BEFORE INTEGRATION TESTS!)
go run ./cmd/control-freak local port-forward # Start port forwards
# Testing (requires k3d cluster running)
# IMPORTANT: Always run 'local restart' first to ensure cluster has latest code!
go run ./cmd/control-freak local restart && go test ./k8s/tests/... -v # Rebuild + test (RECOMMENDED)
go test ./k8s/tests/... -v # Run all integration tests (auto-setup)
KEEP_CLUSTER=true go test ./k8s/tests/... -v # Keep cluster after tests
go test ./k8s/tests/... -run TestSetReplica -v # Run specific test
go test ./... -v -count 1 # Run all tests (unit + integration)
# Kubernetes
kubectl get hostedservicedeployments -n control-plane-local # List CRDs
kubectl get pods -n control-plane-local
kubectl logs -f deployment/control-freak -n control-plane-local
kubectl describe hostedservicedeployment <name> -n control-plane-local
# PostgreSQL (requires port-forward)
kubectl port-forward -n control-plane-local service/postgres-db 5432:5432
psql "postgres://testuser:testpass@localhost:5432/testdb?sslmode=disable"
# Useful queries:
# SELECT * FROM deployments;
# SELECT * FROM deployment_events ORDER BY created_at DESC LIMIT 10;
# Database Migrations (ALL schema changes must use these)
./script/migrate.sh version # Check current migration version
./script/migrate.sh up # Apply all pending migrations
./script/migrate.sh up 4 # Migrate up to version 4
./script/migrate.sh down # Roll back one migration
./script/migrate.sh new my_feature # Create new migration files
./script/migrate.sh force 3 # Force version (dirty state recovery)
# Docker (manual rebuild - prefer 'local restart' command)
docker build -t control-freak:local -f Dockerfile .
k3d image import control-freak:local -c control-plane-local
kubectl --context k3d-control-plane-local rollout restart deployment control-freak -n control-plane-local