container

name	container
description	Container engine abstraction, Docker/Podman patterns, persistent container lifecycle, provisioning, containerfile validation, path handling, transient retry behavior, and Linux-only policy. Use when working on internal/container/, internal/containerplan/, internal/provision/, internal/runtime/container*.go, pkg/invowkfile container runtime fields, or container-related tests/docs.

Container Engine Skill

This skill covers the container runtime implementation in Invowk, including the engine abstraction layer, Docker/Podman support, and sandbox-aware execution.

Use this skill when working on:

• internal/container/ - Container engine abstraction
• internal/containerplan/ - Persistent container planning and deterministic target resolution
• internal/runtime/container*.go - Container runtime implementation
• internal/provision/ - Container provisioning logic
• pkg/invowkfile/containerfile_path.go and container runtime validation - schema-facing containerfile contracts
• Container-related tests

---

Normative Quick Rules

• .agents/rules/version-pinning.md defines the canonical image and pinning policy.
• .agents/rules/windows.md defines host vs container path semantics.
• .agents/rules/testing.md defines cross-platform/container testing policy.
• This skill focuses on container-runtime implementation details; rules remain authoritative on policy conflicts.

---

Linux-Only Container Support

CRITICAL: The container runtime ONLY supports Linux containers.

Supported	NOT Supported
Debian-based images (debian:stable-slim)	Alpine-based images (alpine:*)
Standard Linux containers	Windows container images

Why no Alpine: musl-based environments have many subtle gotchas; we prioritize reliability over image size.

Why no Windows containers: They're rarely used and would introduce too much extra complexity to Invowk's auto-provisioning logic.

In tests, docs, and examples: Always use debian:stable-slim as the reference image.

Image Validation (Early Rejection)

ValidateSupportedRuntimeImage() (image_policy.go) enforces the image policy before provisioning to fail fast:

ValidateSupportedRuntimeImage(image) error
  ├── isWindowsContainerImage(image) — pattern matching (mcr.microsoft.com/windows/*, etc.)
  └── isAlpineContainerImage(image)  — segment-aware matching (last path segment only)

Segment-aware Alpine detection: isAlpineContainerImage() strips tag/digest suffixes, then checks if the bare name equals "alpine" or has /alpine as the last path segment. This avoids false positives on images like "go-alpine-builder:v1" or "myorg/alpine-tools". Matches: alpine, alpine:3.20, docker.io/library/alpine:latest.

---

Engine Interface

The Engine interface (engine.go) defines the unified contract for all container operations:

type Engine interface {
    Build(ctx context.Context, opts BuildOptions) error
    Run(ctx context.Context, opts RunOptions) (*RunResult, error)
    InspectContainer(ctx context.Context, name ContainerName) (*ContainerInfo, error)
    Create(ctx context.Context, opts CreateOptions) (*CreateResult, error)
    Start(ctx context.Context, containerID ContainerID) error
    Exec(ctx context.Context, containerID ContainerID, command []string, opts RunOptions) (*RunResult, error)
    Remove(ctx context.Context, containerID ContainerID, force bool) error
    ImageExists(ctx context.Context, image ImageTag) (bool, error)
    RemoveImage(ctx context.Context, image ImageTag, force bool) error
    Name() string
    Version(ctx context.Context) (string, error)
    Available() bool
}

Interactive PTY support is exposed by the smaller CommandPreparer adapter contract, not the main Engine interface:

type CommandPreparer interface {
    BinaryPath() string
    BuildRunArgs(opts RunOptions) []string
    PrepareRunCommand(ctx context.Context, opts RunOptions) (*exec.Cmd, func(), error)
}

Key Pattern: The interface exposes portable runtime operations only. Vendor-specific or helper-only methods such as image inspection internals stay on BaseCLIEngine or concrete types.

---

BaseCLIEngine Embedding Pattern

Both Docker and Podman engines embed BaseCLIEngine (engine_base.go) for shared CLI command construction:

type BaseCLIEngine struct {
    binaryPath         string
    execCommand        ExecCommandFunc       // For mocking in tests
    volumeFormatter    VolumeFormatFunc      // SELinux label injection
    runArgsTransformer RunArgsTransformer    // Podman --userns=keep-id
}

Responsibilities

Method	Purpose
BuildArgs(), RunArgs()	Construct CLI arguments
RunCommand(), RunCommandCombined()	Execute commands
FormatVolumeMount(), ParseVolumeMount()	Volume mount handling
ResolveDockerfilePath()	Path resolution with traversal protection

Functional Options

// For testing - inject mock command executor
eng := NewDockerEngine(WithExecCommand(mockExec))

// For Podman - SELinux label injection
eng := NewPodmanEngine(WithVolumeFormatter(selinuxFormatter))

// For Podman - rootless mode
eng := NewPodmanEngine(WithRunArgsTransformer(usernsKeepID))

---

Docker vs Podman Implementation

Docker (docker.go)

Minimal implementation—mostly delegates to BaseCLIEngine:

type DockerEngine struct {
    *BaseCLIEngine
}

func NewDockerEngine(opts ...BaseCLIEngineOption) *DockerEngine {
    path, _ := exec.LookPath("docker")
    allOpts := []BaseCLIEngineOption{WithName(string(EngineTypeDocker)), WithImageExistsSubCmd("inspect")}
    allOpts = append(allOpts, opts...)
    return &DockerEngine{BaseCLIEngine: NewBaseCLIEngine(HostFilesystemPath(path), allOpts...)}
}

Missing binaries are represented by an empty binary path and reported through Available()/factory selection, not by constructor errors.

Podman (podman.go)

More complex due to Linux-specific features:

Binary Discovery:

path := findPodmanBinary() // tries podman, then podman-remote

Important: discovery is based on executable lookup (exec.LookPath), not shell parsing. Interactive shell aliases/functions (for example alias podman=podman-remote) are not visible to Invowk's non-interactive process execution. Ensure podman or podman-remote exists as a real executable in PATH.

Automatic Enhancements:

1. SELinux Volume Labels: Automatically adds :z labels to volumes on SELinux systems

   // Checks /sys/fs/selinux existence (more reliable than checking enforce status)
   func isSELinuxPresent() bool {
       _, err := os.Stat("/sys/fs/selinux")
       return err == nil
   }
   

2. Rootless Compatibility: Injects --userns=keep-id to preserve host UID/GID

   // Transforms 'run' and 'create' commands, inserted before image name
   func makeUsernsKeepIDAdder() RunArgsTransformer { ... }
   

---

Persistent Container Planning

internal/containerplan/ owns the pure planning step for persistent container targets: ResolvePersistentTarget selects CLI, config, or derived names deterministically and applies create_if_missing policy before runtime execution mutates engine state.

Keep planning rules in internal/containerplan/; keep engine lifecycle operations in internal/container/; keep orchestration and user-facing execution flow in internal/runtime/.

Persistent container start/create/exec/remove paths must coordinate with engines through container.LifecycleCoordinator when the selected engine implements it. This keeps persistent-container lifecycle operations aligned with the same serialization rules used for transient runs.

---

Sysctl Override (ping_group_range Prevention)

Rootless Podman's default_sysctls configuration causes crun to write net.ipv4.ping_group_range=0 0 in each new network namespace. When multiple containers start concurrently, these writes race and produce EINVAL (exit code 126).

Prevention Layer

On Linux with local Podman, NewPodmanEngine() calls sysctlOverrideOpts(binaryPath) which:

1. Checks if the binary is podman-remote (via name + symlink resolution) — skips if remote
2. Creates a temp file via createSysctlOverrideTempFile() containing [containers]\ndefault_sysctls = []\n
3. Returns WithCmdEnvOverride("CONTAINERS_CONF_OVERRIDE", tempPath) + WithSysctlOverridePath(tempPath) + WithSysctlOverrideActive(true)
4. Every Podman subprocess opens the path independently and reads the override config
5. The temp file is cleaned up by BaseCLIEngine.Close() when the engine is released

On non-Linux platforms, sysctlOverrideOpts() (podman_sysctl_other.go) returns nil — Podman runs inside a VM where host-side env vars don't reach crun. Instead, run serialization falls back to an in-process mutex since flock can't reach the VM.

On podman-remote (Fedora Silverblue/toolbox), sysctlOverrideOpts() returns nil — the env var only affects the remote client, not the Podman service that calls crun. Detected via isRemotePodman() which checks binary name + follows symlinks. On Linux, Podman run paths use flock (acquireRunLock()) for cross-process serialization instead.

Cross-Process Serialization (flock)

When the sysctl override is not active, BaseCLIEngine.Run(), SandboxAwareEngine.Run(), and non-persistent interactive PrepareRunCommand() paths serialize Podman runs to prevent the ping_group_range race. On Linux, acquireRunLock() (run_lock_linux.go) acquires a blocking flock(2) on $XDG_RUNTIME_DIR/invowk-podman.lock (fallback: os.TempDir()). This provides cross-process serialization — all invowk processes on the same machine share the flock. On non-Linux, acquireRunLock() returns an error, causing fallback to sync.Mutex for intra-process protection only. Prepared interactive commands return a cleanup function; release the serialization cleanup only after the PTY command has exited.

Stderr Buffering

runWithRetry() buffers stderr per-attempt so that transient error messages from crun (written directly to the inherited stderr fd before Go can decide to retry) never leak to the user's terminal. On success, non-transient failure, or retry exhaustion, the final attempt's buffer is flushed to the caller's original writer. On transient failure with retries remaining, the buffer is discarded and retried. Interactive mode uses a PTY and bypasses runWithRetry() retries, but non-persistent Podman runs still acquire the serialization cleanup through PrepareRunCommand().

SysctlOverrideChecker Interface

The SysctlOverrideChecker and LifecycleCoordinator interfaces (engine.go and engine_base.go) let runtime and persistent-container paths query whether the temp file override is active and coordinate lifecycle operations:

type SysctlOverrideChecker interface {
    SysctlOverrideActive() bool
}

type LifecycleCoordinator interface {
    AcquireLifecycle(ctx context.Context, operation string) (func(), error)
}

Implemented by: PodmanEngine, SandboxAwareEngine (forwards to wrapped engine)

Used in: Podman run preparation and persistent lifecycle paths — when the checker returns false, run/lifecycle paths acquire flock (Linux) or mutex (non-Linux); when not implemented (Docker), serialization is skipped entirely.

CmdCustomizer Interface

The CmdCustomizer interface (engine_base.go) propagates overrides through engines that create exec.Cmd outside CreateCommand():

type CmdCustomizer interface {
    CustomizeCmd(cmd *exec.Cmd)
}

Implemented by: BaseCLIEngine, SandboxAwareEngine

Used in:

• SandboxAwareEngine.Build/Run/Remove/ImageExists/RemoveImage — sandbox commands bypass CreateCommand
• SandboxAwareEngine.PrepareRunCommand() — interactive host-spawn commands are built outside the wrapped engine

Key Files

File	Purpose
podman_sysctl_linux.go	createSysctlOverrideTempFile(), isRemotePodman(), sysctlOverrideOpts() (Linux temp file)
podman_sysctl_other.go	No-op sysctlOverrideOpts() (macOS/Windows stub)
engine_base.go	CmdCustomizer, SysctlOverrideChecker, EngineCloser, WithCmdEnvOverride(), WithSysctlOverridePath(), WithSysctlOverrideActive(), Close()
podman.go	SysctlOverrideActive(), Close() methods on PodmanEngine
run_serialization.go	Podman run serialization helper, fallback mutex, prepared-command cleanup lease
internal/runtime/container_exec.go	runWithRetry() retry logic and stderr buffering
internal/runtime/container_prepare.go	CommandPreparer type assertion and prepared cleanup composition

---

Path Handling (Host vs Container)

CRITICAL: Container paths always use forward slashes (/), regardless of host platform.

Two Path Domains

Domain	Separator	Example
Host paths	Platform-native (\ on Windows)	C:\app\config.json
Container paths	Always /	/workspace/script.sh

Conversion Pattern

// Converting host path to container path
containerPath := "/workspace/" + filepath.ToSlash(relPath)

// WRONG: filepath.Join uses backslashes on Windows
containerPath := filepath.Join("/workspace", relPath)  // Broken on Windows!

Path Security

Containerfile validation is layered:

• pkg/invowkfile.ContainerfilePath.Validate() and ValidateContainerfilePath() enforce the user-facing relative path contract.
• pkg/invowkfile/sync_runtime_behavioral_test.go keeps Go validation aligned with the CUE runtime schema.
• internal/container.ResolveDockerfilePath() protects engine build paths when converting a context directory plus Dockerfile path into CLI arguments.

See .agents/rules/windows.md for comprehensive path handling guidance.

---

SandboxAwareEngine Wrapper

The SandboxAwareEngine (sandbox_engine.go) is a decorator for Flatpak/Snap execution:

Problem: Container engines run on the host, not inside the sandbox. Paths don't match.

Solution: Execute commands via flatpak-spawn --host or snap run --shell.

type SandboxAwareEngine struct {
    wrapped     Engine
    sandboxType platform.SandboxType
}

// Factory function wraps engine if sandbox detected
func NewEngine(preferredType EngineType) (Engine, error) {
    engine := createEngine(preferredType)
    return NewSandboxAwareEngine(engine), nil  // Auto-wraps if needed
}

---

Engine Factory Functions

Preference with Fallback

// Tries preferred engine first, falls back to alternative
engine, err := container.NewEngine(container.EngineTypePodman)  // or EngineTypeDocker

Auto-Detection

// Tries Podman first (better for rootless), then Docker
engine, err := container.AutoDetectEngine()

Both return wrapped SandboxAwareEngine.

---

Exit Code Handling

Container engines absorb exec.ExitError into result.ExitCode and return (result, nil). This means the error return is always nil for process exit failures — callers must check result.ExitCode:

result := &RunResult{}
if err != nil {
    if exitErr, ok := errors.AsType[*exec.ExitError](err); ok {
        result.ExitCode = exitErr.ExitCode()  // Absorbed into result, err return is nil
    } else {
        result.ExitCode = 1
        result.Error = err  // Actual error (network, etc.)
    }
}
return result, nil  // Always nil error for exit code failures

Important for retry logic: Since engine.Run() returns (result, nil) for transient exit codes (125, 126), retry code must check both the error return AND result.ExitCode. See runWithRetry() in container_exec.go.

---

Testing Patterns

Unit Tests with Per-Test Mock Recorders

All container unit tests use per-test MockCommandRecorder instances for parallel safety:

func TestDockerBuild(t *testing.T) {
    t.Parallel()

    t.Run("with no-cache", func(t *testing.T) {
        t.Parallel()
        recorder := NewMockCommandRecorder()
        eng := newTestDockerEngine(t, recorder)  // Injects via WithExecCommand()

        eng.Build(ctx, opts)

        // Verify expected arguments
        if !slices.Contains(recorder.LastArgs, "--no-cache") {
            t.Error("expected --no-cache flag")
        }
    })
}

Never use package-level global mutation (execCommand = mockFn) for mock injection. The execCommand var is test-scoped in engine_mock_test.go and only used by 3 mock infrastructure self-tests.

Integration Tests

func TestDockerBuild_Integration(t *testing.T) {
    t.Parallel()
    if testing.Short() {
        t.Skip("skipping integration test in short mode")
    }
    testutil.AcquireContainerSemaphore(t)
    ctx := testutil.ContainerTestContext(t, testutil.DefaultContainerTestTimeout)
    // Test with real container engine using ctx — transient errors handled by runWithRetry()
}

testscript Container Setup

Container tests using testscript use containerSetup in tests/cli/cmd_container_test.go. It layers common setup, creates a dedicated temporary HOME, writes the test-scoped container config, probes engine health, and registers orphaned-container cleanup with env.Defer().

Setup: func(env *testscript.Env) error {
    return containerSetup(env)
},

Container Test Timeout Strategy

Multi-layer timeout strategy prevents indefinite hangs:

1. Per-test deadline (3 minutes): testscript.Params{Deadline: deadline}
2. Cleanup via env.Defer(): Removes orphaned containers
3. CI explicit timeout (15 minutes): Safety net for catastrophic failures

---

Transient Error Classification

The IsTransientError() function (transient.go) is a shared classifier for transient container engine errors that may succeed on retry. It is used by both production retry logic (ensureImage() in internal/runtime/container_provision.go) and the test smoke test (tests/cli/cmd_test.go).

Classified as transient:

• Exit code 125 (generic engine error — storage/cgroup issues)
• ping_group_range (rootless Podman user namespace race)
• OCI runtime error (generic OCI failures)
• Network errors: Temporary failure resolving, Could not resolve host, connection timed out, connection refused
• Storage errors: error creating overlay mount, error mounting layer

Explicitly NOT transient:

• nil errors
• context.Canceled / context.DeadlineExceeded (retrying cancelled operations is never useful)

Build Retry in ensureImage()

Container image builds (engine.Build()) are retried up to 3 times with exponential backoff (2s, 4s) on transient errors. Non-transient errors fail immediately. The caller's context deadline naturally bounds total retry time.

Run Retry in runWithRetry()

Container runs (engine.Run()) are retried up to 5 times with exponential backoff (1s, 2s, 4s, 8s) on transient errors. This is critical because engine.Run() absorbs exec.ExitError into result.ExitCode and always returns (result, nil) — so the retry logic must check both the error return AND result.ExitCode via container.IsTransientEngineExitCode() (exit codes 125 and 126). Run retries are more aggressive than build retries (5 vs 3 attempts) because Podman ping_group_range races are more frequent under heavy parallelism and runs are fast.

Container validation pattern: Container dependency validation lives in internal/app/commandadapters/dependency_runtime.go and must guard against transient exit codes after result.Error handling. Use the local checkTransientExitCode helper:

if err := checkTransientExitCode(result, label); err != nil {
    return err
}

Without this guard, transient engine failures (125/126) after retry exhaustion get misreported as domain-specific errors ("not found", "not set", etc.). The helper centralizes the pattern through runtime.IsTransientContainerEngineExitCode.

---

File Organization

File	Purpose
engine.go	Engine interfaces, LifecycleCoordinator, options, factory helpers
engine_base.go	Shared CLI implementation, CmdCustomizer interface
docker.go	Docker concrete implementation
podman.go	Podman + SELinux/rootless logic
podman_sysctl_linux.go	Temp file-based sysctl override (Linux only)
podman_sysctl_other.go	No-op sysctl override stub (non-Linux)
run_lock_linux.go	flock-based cross-process lock (acquireRunLock(), runLock)
run_lock_other.go	No-op stub, forces fallback to sync.Mutex
run_serialization.go	Podman run serialization and prepared-command cleanup lease
image_policy.go	Image validation (ValidateSupportedRuntimeImage, Alpine/Windows rejection)
sandbox_engine.go	Flatpak/Snap wrapper decorator
transient.go	Shared transient error classifier
doc.go	Package documentation
../containerplan/	Pure persistent-container target planning (ResolvePersistentTarget)
../../pkg/invowkfile/containerfile_path.go	User-facing containerfile path value object and validation

Runtime files (in internal/runtime/):

File	Purpose
container_exec.go	Container execution, runWithRetry(), IsTransientExitCode() (exported), flushStderr()
container_provision.go	Image preparation and provisioning
container_persistent.go	Persistent container create/start/exec/remove flow and lifecycle coordination
container_prepare.go	CommandPreparer type assertion and prepared cleanup composition
container_exec_test.go	Unit tests for runWithRetry(): stderr buffering, exit codes, context cancellation
container_test.go	Unit tests for isAlpineContainerImage(), isWindowsContainerImage(), ValidateSupportedRuntimeImage()

---

Common Pitfalls

Pitfall	Symptom	Fix
Using filepath.Join() for container paths	Backslashes on Windows	Use string concat with / or filepath.ToSlash()
Forgetting HOME in testscript	"mkdir /no-home: permission denied"	Set HOME to env.WorkDir in Setup
Testing with Alpine images	Unexpected musl behavior	Always use debian:stable-slim
Missing SELinux labels	Permission denied in Podman	Use Podman's auto-labeling or explicit :z
Container tests hanging	CI timeout	Use per-test deadline + cleanup in env.Defer()
Flaky container builds in CI	Exit code 125, DNS failures	IsTransientError() + build retry in ensureImage() handles this; CI pre-pulls debian:stable-slim
Flaky container runs under parallelism	Exit code 125/126, ping_group_range	runWithRetry() in container_exec.go retries runs with exponential backoff; checks both err and result.ExitCode
Mock tests share recorder across parallel subtests	Race condition on recorder state	Use per-subtest NewMockCommandRecorder() + engine instances; never share a recorder with Reset() across parallel subtests