| name | go |
| description | Go CLI development: building the charly binary, running tests, understanding
the source code structure.
MUST be invoked before reading or modifying any Go source file in charly/.
|
Go - CLI Development
Overview
The charly CLI is a Go program in the charly/ directory. It uses the Kong CLI framework, go-containerregistry for OCI operations, and YAML parsing for configuration. All computation, validation, and building logic lives in Go. Taskfiles are used only for bootstrapping (building charly itself).
Unified YAML loader (LoadUnified)
The unified format's entry point is LoadUnified(dir) at charly/unified.go. It reads <dir>/charly.yml, recursively resolves the import: statement (max depth 8, cycle-safe via visited set), and parses every file as a YAML multi-document stream (so bundle files with --- separators work). Every document is unified node-form (name-first): mergeUnifiedDocs runs each document through the shared routing core — classifyDoc (top-level-key inspection) → the closed #NodeDoc CUE gate → normalizeNodeInto (the reserved-word-driven node decomposer in reserved_registry.go). #NodeDoc (schema/node.cue) is the SOLE load-time gate for every loaded document, the root charly.yml and discovered manifests alike. classifyDoc does NOT route a legacy kind-keyed / root-shape document — it HARD-REJECTS it with a charly migrate hint (the legacy mergeKindDoc / firstKindKey / kindKeyedDoc / VmDoc routing was deleted in the #NodeDoc-sole-gate cutover). The legacy-shape detector is rootShapeKeySet (CUE-derived: spec.DocDirectives + every spec.KindWords entry, plus the legacyKindAliases deploy/check) — a DETECTOR only, no routing reads it. F9 BOOTSTRAP PHASE: before the early gateSchemaVersion schema gate, LoadUnified runs runBootstrapPhase(rootData) (bootstrap_phase.go) — it enumerates providerRegistry.providersInPhase(sdk.PhaseBootstrap) and invokes each one's Invoke(OpBootstrap, {config}), threading the returned (possibly transformed) bytes, so a bootstrap-phase plugin can rewrite the raw root bytes BEFORE validation rejects them. LoadUnified seeds the transformed root into loadUnifiedInto via the fileOverrides map (keyed on the root's abs path), so the rewrite reaches the actual PARSE + the post-merge gate — not just the early version gate. Bootstrap plugins are compiled-in (in-proc), so this never re-enters the validated-config load. Today only the no-op candy/plugin-example-bootstrap registers in this phase — migrate is NOT a bootstrap-phase transform (it runs in CORE, invoked explicitly by charly migrate — never as an in-loader byte-rewrite; the load gate keeps the Run: charly migrate reject for a stale config, because migration is whole-project file-based + host-coupled and cannot run on root bytes inside LoadUnified). Phases are an sdk.Phase* set declared per-capability via ProvidedCapability.phase (proto field 9, lifted in buildUnit/buildUnitInProc onto the phaseCarrier; phaseOfProvider defaults to runtime). A no-op bootstrap plugin (candy/plugin-example-bootstrap) returns the bytes unchanged.
UnifiedFile.ApplyDiscover(rootDir) walks the flat generic discover: list after initial merge. discover: is DiscoverConfig = []ScanSpec ({path, recursive, manifest}) — no kind dimension. For each spec, findEntityDirs finds directories containing the spec's manifest (default UnifiedFileName — charly.yml, the ONE filename the code knows; a missing discover path is a no-op, not an error), and applyDiscoveredManifest validates each discovered document through the SAME classifyDoc → #NodeDoc gate: a candy: node registers a lazy From: directory reference (scanCandy parses + validates it later), every other node decomposes + merges via normalizeNodeInto. Explicit map entries always win over discovered entries. ApplyDiscover runs in the loader's main path (loadUnifiedInto depth-0 boundary, for the root AND every namespace), so discovered image nodes (candy: nodes carrying base:/from:, the former box:) reach ProjectConfig — not just the layer-loading path. The authoring kind vocabulary is CUE-derived — spec.KindWords projected to the kindWordSet membership set in reserved_registry.go; the former hand kindKeys/kindKeysSet/entityKind lists were deleted.
Projections to today's concrete types: ProjectConfig() → *Config, ProjectDistroConfig() → *DistroConfig, etc. Existing LoadConfig / LoadBuildConfigForBox / LoadBundleConfig continue to work unchanged — migration to the unified entry point is incremental.
Binary-embedded default config (charly/embed_defaults.go). The loader has ONE document-interpretation path (mergeUnifiedDocs). The binary-embedded default config is plain node-form YAML at charly/charly.yml (//go:embed charly.yml, embed_defaults.go), parsed by the SAME unified loader as any project charly.yml — there is no CUE-source front-end and no compile step: embeddedDefaults feeds the embedded bytes straight through the UNCHANGED mergeUnifiedDocs, then applyEmbeddedDefaults merges the vocabulary in as the lowest-priority base (project-wins). The embedded vocabulary is schema-validated against charly/schema (#Distro/#Builder/#Init/#Resource/#Sidecar) through the shared validateVocabularyCollections helper (validate.go, also used by charly box validate for project files) — guarded by TestEmbeddedDefaults_SchemaConformance, with TestEmbeddedDefaults_SameLoaderPath proving the embed flows through the identical loader core.
CUE is the single source of truth — the charly/spec package
The charly.yml ingress schema has ONE author-of-record: the CUE definitions in charly/schema/*.cue. The Go param structs, the reserved-word vocabulary, and the kind/verb wiring are GENERATED or DERIVED from that source — there are no hand-maintained parallel copies. The pieces:
charly/spec — the generated param structs. task cue:gen regenerates this package from charly/schema/*.cue. Its members:
spec/cue_types_gen.go — generated by cue exp gengotypes (then yaml-tag retagged). Carries the Code generated … DO NOT EDIT banner; NEVER hand-edit. Every authored param type lives here (Box, Candy, Vm, Op, Deploy, …).spec/vocab_gen.go — generated by the companion charly/internal/schemagen (-mode=vocab). The CUE-derived reserved-word slices: KindWords, ResourceKinds, DocDirectives, StepKeywords, ContextWords, DataKeys, OpFields, OpVerbs, AuthoringVerbs.spec/union_types.go — hand-written faithful union / shorthand types. cue exp gengotypes degrades every CUE disjunction to any/map[string]any/an empty struct, so the matching CUE def is annotated @go(-) (suppressing the lossy generated type) and the precise Go type is hand-written here in the SAME package, referenced by the generated structs by name.spec/charly_names.go — hand-written charly-name aliases (type BoxConfig = Box, type VmSpec = Vm, …). Def-level @go(CharlyName) is BROKEN in cue v0.16.1 (it dangles the fields that reference the renamed def, producing uncompilable Go — RDD-verified on a live spike), so the charly NAME is exposed as a Go type alias here instead of via a def-level attribute. The per-FIELD @go(GoName,…) attributes (which DO work) carry the name/pointer/type overrides in charly/schema/*.cue.spec/gen_repro_test.go — TestGenReproducible, the reproducibility gate: re-runs the same task cue:gen tools into a temp dir and diffs against the committed cue_types_gen.go + vocab_gen.go, failing on any drift (skips gracefully when the pinned cue CLI is absent).spec/scalar_aliases.go, spec/hand_state_types.go, spec/charly_methods.go — hand-written supporting scalar aliases, runtime state types, and the pure methods (Op.Kind(), …) that moved into package spec alongside the types they operate on.
charly/internal/schemagen (main.go) — the companion generator. Three modes: -mode=concat (concatenate schema/*.cue into one schema_spec.cue compilation unit headed package spec), -mode=vocab (compile that and emit spec/vocab_gen.go), -mode=retag (the principled Go yaml-tag transform on the gengotypes output → spec/cue_types_gen.go). Compiled and documented — never sed on generated Go.charly/internal/schemaconcat (schemaconcat.go) — the shared concat contract (R3). The ONE ConcatSchema both the RUNTIME (cue_schema.go's sharedCueSchema) and the dev-time generator call to fold every package-less schema/*.cue file into one compilation unit, so the schema the runtime validates against and the Go types gengotypes produces can never drift. A leaf package depending only on the stdlib io/fs abstraction (runtime passes its //go:embed FS, the generator passes os.DirFS).charly/spec_aliases.go (package main) — the zero-churn repoint + the compile-time parity gate. Every package-main param type is now a type X = spec.X alias (BoxConfig, Op, CandyYAML, VmSpec, ServiceEntry, …); the hand struct DEFINITIONS were deleted. Because every BoxConfig{…} / []ServiceEntry reference throughout package main compiles against the spec types, the Go compiler IS the field-parity check: a renamed or removed spec field (or wire-key/type change) fails the build at this surface. Name collisions where the package-main type is a different concept stay hand-written (CalVer, CandyRef, the runtime Candy — the param is aliased as CandyYAML).charly/reserved_registry.go (package main) — the reserved-word ⇄ handler registry + the startup bijection gate. Hosts the CUE-derived membership sets that REPLACED the former eight hand vocab lists (kindKeys/kindKeysSet/rootShapeKeys/docDirectiveKeys from unified.go + nodeEntityKinds/nodeResourceKinds/nodeStepVerbs/nodeDataKeys/nodeDocDirectives from node_parse.go): kindWordSet, resourceKindSet, stepKeywordSet, dataKeySet, docDirectiveSet (each a set view of a spec.* slice). reservedKindHandlers binds each kind word to its CUE def ("vm" → "#Vm"), VerbCatalog dispatches verbs. The init() runs the kind + verb bijection checks fail-fast at process start (mirrored as TestReservedWordRegistry_*): checkKindBijection(reservedKindHandlers, spec.KindWords) and checkVerbBijection(VerbCatalog, spec.OpVerbs, spec.AuthoringVerbs) — so a kind or verb can never be added to the schema without a handler, nor a handler kept after its word is dropped. normalizeNodeInto (the reserved-word-driven node decomposer) also lives here.charly/uniform_api_gate_test.go (package main) — the F11 uniform-API gate (TestNoSinglePluginAPISurface), the externalization capstone. Asserts the "generic over ad-hoc" invariant STRUCTURALLY: no provider WORD appears in the plugin↔kernel API SURFACE — not as an sdk.Op* selector value, a sdk.ProvidedCapability/StepContract field name, a reverse-channel RPC method name (ExecutorService/CheckContextService/Provider/PluginMeta), or a hostBuilders key. The forbidden-word universe is the union of the CUE-derived spec.OpVerbs ∪ spec.KindWords ∪ spec.OpFields (which carries the externalized check verbs cdp/vnc/mcp/kube/libvirt/spice/adb/appium) ∪ spec.AuthoringVerbs plus every compiled-in non-command provider word, minus genericConceptCollisions (venue — the generic ExecutorService.Venue RPC coincides with the #Op venue field). The invariant is structural, NOT a user-count (each capability flag has exactly ONE candy/plugin-example-* user — "≥1 by construction"). Per-plugin data rides the opaque Substrate json.RawMessage; a host MAY call the generic connectPluginByWord(class, word) with a specific word ARGUMENT (the word is data, never API shape — those call-sites are not scanned). Sibling of the startup bijection gates; has a teeth arm (a re-introduced provider word must trip it) + fixed RPC-method allowlists (a new reverse RPC is a conscious, reviewed addition).
import-namespace loader (UnifiedFile.Import + Namespaces)
UnifiedFile.Import (type ImportList, YAML tag import) is the single composition statement. Its custom UnmarshalYAML accepts a mixed-shape sequence: a scalar item → ImportEntry{Ref: …} (flat, Namespace == ""); a single-key mapping item → ImportEntry{Namespace: alias, Ref: …} (namespaced). A matching MarshalYAML round-trips both shapes (migrators rely on it). validateNamespaceAlias enforces a bare lowercase-hyphenated alias (no dots).
loadUnifiedInto processes the queue:
- Flat entries are loaded and root-merged into the importing
UnifiedFile (same root-wins merge that drives same-repo file splits + an imported build-vocabulary override).
- Namespaced entries call
loadNamespaceCached(ref, base, nsCache, loadingRepos), which loads the target as a fully-resolved, isolated UnifiedFile (its own flat imports + its own namespaced imports, with a FRESH visited set for its file-cycle detection) and mounts it under merged.Namespaces[alias]. These entries are NOT flat-merged into the root maps — they are referenced qualified.
UnifiedFile.Namespaces (map[string]*UnifiedFile, YAML tag -, never authored directly) holds the mounted children; projectConfigCached projects it to Config.Namespaces (map[string]*Config, pointer-keyed cache → self-references project safely).
Cycle-break by REPO IDENTITY (ns_identity.go), not pinned version. Two maps cooperate in loadNamespaceCached: nsCache is the version-keyed (canonicalRef: repo@version/subpath) diamond memo — it dedups identical refs across a load; loadingRepos is the ancestor/cycle set, keyed by REPO IDENTITY (nsRepoIdentity: a remote ref's RepoPath, or a local path's git remote origin). BEFORE any fetch, if the ref's repo identity is already in loadingRepos (an ancestor still on the load stack), the loader returns that in-progress node — so an import cycle between two projects that import each other (or a transitive back-import of an ancestor still being loaded) terminates even when the loop's pins diverge: a back-reference to a DIFFERENT pinned version of an in-progress repo resolves to the in-progress node instead of fetching (and recursing into) a divergent — possibly stale-schema — snapshot. LoadUnified seeds loadingRepos[rootIdentity] = merged (the root's identity comes from its optional repo: field, else git remote origin), so any transitive import of the root's OWN repo resolves to the local working tree — the importing project's namespace pins win. loadingRepos entries are pushed before recursing and popped after (stack-scoped — two SIBLING imports of the same repo at different versions still each load); the root seed is never popped. A whole-repo ref with an empty sub-path resolves to that repo's charly.yml. Covered by TestImportNamespace_DivergentVersionMutualCycle + TestNsRepoIdentity in ns_identity_test.go.
Namespace resolver (charly/namespace.go)
The resolver implements Go-package-member semantics over Config.Namespaces:
splitNamespaceRef(ref) — splits a qualified ref on its FIRST . into (ns, rest); a bare ref returns ok=false; the remainder may itself be qualified (a.b.c → "a", "b.c").resolveBoxRef(ref) / resolveLocalRef(ref) — bare names resolve in the current Config; ns.name descends into c.Namespaces[ns] recursively, returning the entry plus the Config (namespace context) it lives in.resolveNamespacedBases(out, …) — after the local image set resolves, pulls every namespace-qualified base: (and qualified builder: ref, but only for images that actually have layers to build) into out, keyed by fully-qualified name, iterating to a fixpoint (a pulled-in image may reference a deeper namespaced base).pullNamespacedBox(from, ref, keyPrefix, …) — descends the namespace chain to the leaf, re-keys the entry's own internal base to the fully-qualified ancestor so the build graph references it correctly, and recurses to pull that ancestor.
The inheritance rule lives here: distro:/build: are VALUES → inherited across a namespace boundary; builder: is a map of namespace-relative REFS → NOT inherited (the consumer declares its own). See the file header comment for the rationale (avoid leaking a base-namespace-relative ref into a consumer where that namespace doesn't exist). leafName(ref) strips every namespace prefix to the final member name (arch.arch-builder → arch-builder); paired with resolveBoxRef's returned namespace Config it keys the resolved entity in that Config.Box map (used by the reachability walk below).
Remote-layer resolver (charly/refs.go + charly/layers.go) — per-entity version + reachability-scoped collection
@github layer refs resolve in TWO phases: the :vTAG git tag is only the FETCH coordinate (which commit to clone); the layer's own version: field — read AFTER fetch — is the authoritative identity that drives dedup + warn-and-newest-wins.
CandyRef (charly/refs.go) — the single representation of a require: / candy: ref. It stores the ORIGINAL ref string (Raw, with any @repo prefix and :version suffix); .Bare() (the map-key form), .Version() (the pinned git tag — the FETCH coordinate, NOT the identity), and .IsRemote() are DERIVED. A resolved slot carries the qualified sibling key set by qualifyRemoteSiblingDeps after a remote layer is fetched, so ONE list serves both the graph (keys on .Bare()) and the transitive fetch (keys on the immutable .Raw). Candy.Require / Candy.IncludedCandy are []CandyRef — there are no parallel bare/raw arrays.- Two-phase per-entity-version resolution —
CollectRemoteRefsOpts (charly/refs.go) collects EVERY distinct (repo, git-tag) a bare ref is referenced at — it does NOT collapse to one winning tag and does NOT warn (the git tag is just where to clone from). The ScanAllCandyWithConfigOpts fix-point (charly/layers.go) fetches each (repo, git-tag) (tracking scanned (repo,git-tag,ref) triples), reads each materialization's per-entity version:, accumulates candidates per bare ref, then pickCandyVersion arbitrates: same per-entity version across different git tags → NO warning, the newest git tag wins for freshness (compareSemver); different per-entity versions → warn once (naming both per-entity versions + sources) and the newest per-entity version wins (compareCalVer). Exactly one materialization per bare ref reaches the layer map, so the graph + intermediates are unchanged. A fetched layer with NO version: is a HARD ERROR (no fallback — first-party remotes are backfilled by remote-cache auto-migration, EnsureRepoDownloaded → RunProjectMigrations). pickCandyVersion is the SOLE arbiter for direct AND transitive refs, so a transitive dep can never silently pull a different version of an already-resolved layer. This is why a repo re-tag of an UNCHANGED layer no longer warns — the old resolver compared the repo git tag, which advances on every push.
- Reachability-scoped collection (
CollectRemoteRefsOpts.collectBox) — collection walks ONLY the enabled root images + the namespaced images reachable via their base:/builder: edges (resolveBoxRef + leafName), plus local layers' transitive deps. It does NOT scan every image and kind:local template of every imported namespace (that over-collection pulled unrelated layers pinned at a different tag — e.g. a namespace's charly-cachyos workstation template's chrome — and tripped the version policy). Builder edges ARE followed when an image builds (a namespaced fedora.fedora-builder is built as an intermediate and needs its rpmfusion/yay layers); dropping them under-collects ("unknown layer").
- One unified populator (
populateCandyFromYAML, charly/unified.go) — both scanCandy (discovered-layer-dir path) and synthesizeInlineCandy (charly.yml inline path) call it, so they can't drift. The Has* predicates (HasEnv/HasPorts/HasVolumes/…) are derived methods; only the filesystem-probe caches (HasPixiToml/HasSrcDir/…) stay fields.
charly box reconcile (see /charly-build:reconcile) is the operator tool that aligns the on-disk git-tag pins so every reference of a repo fetches one commit, clearing any residual per-entity-version warning.
Capabilities — BoxMetadata alias + label completeness check
Capabilities = BoxMetadata (type alias in charly/capabilities.go). CapabilityLabelMap lists every field with its OCI label home; TestCapabilityLabelCompleteness fails the build if an BoxMetadata field lacks a mapping. This invariant keeps charly bundle from-box reliable: every field deploy code might consult is readable from a pushed image's labels alone, independent of charly.yml.
Kubernetes substrate (EXTERNAL — deploy:k8s, candy/plugin-kube)
target: k8s is an EXTERNAL deploy substrate (F1): there is no in-proc k8s DeployTarget — it resolves to externalDeployTarget over the reverse channel, served out-of-process by candy/plugin-kube's deploy:k8s provider (beside its kube: verb). The Kustomize GENERATOR is the COMPILED-IN candy/plugin-k8sgen (M13, verb:k8sgen serving OpEmit; the workload-kind heuristic selectWorkloadKind maps the generic kind: enum to Deployment/StatefulSet/DaemonSet/Job/CronJob/Pod) — kept SEPARATE from the heavy external plugin-kube because it has no client-go dependency and must resolve in the project-less from-box path. charly/k8s_generate.go is now a thin in-core SHIM: GenerateK8sKustomize lifts the 3 caps scalars (Port/UID/GID) + spec.Deploy + spec.K8s into a spec.K8sGenInput, Invokes the candy (OpEmit → spec.K8sGenReply manifest docs), validates each doc HOST-SIDE via the M16 egress shim (ValidateEgressValue), then writes the base/+overlays/ tree. The host-side deploy:k8s preresolver (charly/k8s_deploy_preresolve.go) + charly bundle from-box --target k8s both call this shim (unchanged signature). The plugin runs kubectl --context <ctx> apply -k. See /charly-internals:install-plan + /charly-kubernetes:kubernetes.
VM target (external substrate)
target: vm is an EXTERNAL deploy substrate, exactly like local/android/k8s: there is no in-proc VM DeployTarget. It resolves to externalDeployTarget over the reverse channel, served out-of-process by candy/plugin-deploy-vm's deploy:vm provider. UNLIKE k8s, the vm substrate DOES consume the InstallPlan IR — the plugin walks the plan via the SAME shared charly/plugin/kit.WalkPlans the local deploy uses, but the executor the reverse channel serves is the guest SSHExecutor (charly/deploy_executor_ssh.go), so the same walk runs INSIDE the guest (bash bodies via ssh guest 'sudo bash -s'). The DeployExecutor interface (charly/deploy_executor.go) decouples "how shell commands run" from the walk — ShellExecutor + SSHExecutor are the two implementations.
The host-side VM venue lifecycle (boot the domain, build the guest SSH executor, nested pod-in-guest, teardown, the charly vm Start/Stop/Status/Logs/Shell/Rebuild) lives in the registered vmSubstrateLifecycle hook (charly/vm_deploy_lifecycle.go), implementing the substrateLifecycle interface (charly/deploy_substrate_lifecycle.go) — the one external substrate that owns a real venue lifecycle.
charly bundle add vm:<name> dispatches through bundle_add_cmd.go::dispatchNode → ResolveTarget → externalDeployTarget (no per-kind dispatch function); bundle_add_cmd_vm.go carries the host-side VM-only helpers that REMAIN (deployNestedPodsInGuest, vmNameFromDeployName, sshReverseRunner, saveVmDeployState, removeVmDeployEntry). Full architecture + preflight flow lives in /charly-internals:vm-deploy-target.
YAML surface ↔ Go identifier convention
The codebase keeps wire format (YAML keys) and internal names (Go fields/types) in strict symmetry — plural YAML keys get plural Go identifiers, singular get singular. The singular builder: / distro: / init: top-level keys in the embedded build vocabulary (charly/charly.yml) and project charly.yml carry singular Go identifiers: BuilderMap, BoxConfig.Builder, BuilderConfig.Builder, DistroConfig.Distro, InitConfig.Init. The rule: if you change a YAML tag, also rename the Go identifier. Tests enforce this indirectly — struct literals won't compile if they disagree. Note: the OCI label key is grouped under platform.* / builder.* sub-namespaces — see LabelPlatformDistro, LabelPlatformFormat, LabelBuilderUse, LabelBuilderProvide in labels.go. Label wire-names are decoupled from YAML/Go identifiers by design.
Kong default:"withargs" for parent+leaf commands
Kong normally treats a struct as either a branch (has child cmd:"" subcommands) OR a leaf (accepts arg:"" positionals and has a Run() method) — not both. When you want both shapes on the same parent command (e.g., charly config <image> runs setup AND charly config mount|status|… dispatch to subcommands), tag the default child with default:"withargs". Kong then dispatches to that child when the first token doesn't match a subcommand name, passing positional args/flags through.
One use in the codebase:
charly/config_image.go — BoxConfigCmd.Setup is the default (default:"withargs"); charly config <image> routes through BoxConfigSetupCmd while charly config mount|status|… dispatch explicitly.
charly check no longer needs this pattern: every live-container verb (wl/cdp/vnc/dbus/… and libvirt) is now an out-of-process declarative verb, NOT a charly check subcommand, so no subcommand name can shadow the charly check live <image> positional (CheckLiveCmd.Box is a plain arg:"").
Mode purity: LoadConfig must NOT read charly.yml
OCI labels are written exclusively from charly.yml at charly box build / charly box generate time. charly.yml is deploy-mode state and must never bleed into the baked image. The key guarantee lives in charly/config.go:LoadConfig — it calls LoadConfigRaw only, with no MergeDeployOverlay.
The rule: every build-mode command (anything under charly box …) calls LoadConfig. If you ever re-introduce MergeDeployOverlay inside LoadConfig, you will silently contaminate OCI labels with whatever is in the user's local charly.yml — exactly the bug that made images bake ports: ["5900:5900","9250:9222"] from a stale charly.yml entry instead of the charly.yml-declared ["5900:5900","9222:9222","9224:9224"].
Deploy-mode commands (charly config, charly start, charly stop, charly update, charly bundle add, charly bundle del, charly shell, charly cmd, charly service, charly vm create, …) read labels via ExtractMetadata and then apply the deploy overlay explicitly via MergeDeployOntoMetadata(meta, dc, instance). This split is load-bearing — never collapse it.
Host-deploy specifics: charly bundle add host is deploy mode (reads both charly.yml and charly.yml), not build mode — despite looking like "install on host, not into an image". The compiler (BuildDeployPlan in install_build.go) is pure and shared with build mode, but the invocation path reads charly.yml for add_candy: and install_opts: like every other deploy-mode command.
InstallPlan IR — the shared intermediate representation
The DEPLOY paths (pod/vm + external [local/k8s/android]) route through a shared IR; build-mode Containerfile emission is a SEPARATE generator (writeCandySteps → emitTasks, reading each layer's ops directly), NOT the IR. The k8s substrate is EXTERNAL and does NOT consume the IR — it generates a Kustomize tree host-side (see "Kubernetes substrate" above). Flow:
Layer + ResolvedBox + HostContext
→ BuildDeployPlan (install_build.go) [pure; deploy-path only, NOT charly box build]
→ InstallPlan (install_plan.go)
→ DeployTarget.Emit (OCITarget/PodDeployTarget) / UnifiedDeployTarget lifecycle
├── OCITarget (build_target_oci.go) → Containerfile text (pod-overlay add_candy: synthesis)
├── PodDeployTarget (deploy_target_pod.go) → overlay + quadlet
└── externalDeployTarget (deploy_target_external.go) → out-of-process plugin over the OpExecute reverse channel
(deploy:local — candy/plugin-deploy-local walks the IR
via kit.WalkPlans, host-engine steps via RunHostStep;
deploy:vm — candy/plugin-deploy-vm runs the SAME walk
INSIDE the guest over the guest SSHExecutor, with the
host-side vmSubstrateLifecycle hook owning the venue;
deploy:k8s — host preresolver generates the Kustomize
tree, plugin runs kubectl apply -k; deploy:android)
OCITarget is constructed only by PodDeployTarget; charly box build/generate emit via the writeCandySteps → emitTasks generator (generate.go + tasks.go), sharing the package-cascade / shell-snippet / localpkg compiler helpers with the IR. Full reference lives in /charly-internals:install-plan — go there before touching any of those files. Supporting Go files (ledger, builder_run, shell_profile, reverse_ops, service_render, deploy_ref, hostdistro, migrate_services_tool) are covered in /charly-internals:local-infra.
VM-path architecture
The VM path spans the following module topology:
| File | Role |
|---|
charly/spec/cue_types_gen.go (generated) | VmSpec (= Vm) + VmSource discriminated union (cloud_image / bootc) + VmChecksum + VmNetwork + VmSSH + VmKeyInjection |
charly/spec/cue_types_gen.go (generated) | VmCloudInit + VmCloudInitUser/File/Network/Mirrors + VmCharlyInstall (auto/scp/url/skip state machine) |
charly/libvirt_yaml.go | LibvirtDomain + 30+ sub-types (features, CPU, clock, memory backing, numatune, cputune, devices, seclabel, launch security, resource, sysinfo) — the opencharly YAML-facing shape of the libvirt: stanza |
charly/libvirt_yaml_bridge.go | RenderDomainXML/BuildLibvirtDomainXML pure functions (build a libvirtxml.Domain tree, marshal to XML) + buildDomainDevices device emission (passt backend, portForward attribute order, virtio-gpu default, SMBIOS credentials, XMLPassthrough merge) |
charly/qemu_render.go | RenderQemuArgv for direct-QEMU backend |
charly/cloud_init_render.go + cloud_init_iso.go | RenderCloudInit + ResolveKeyInjectionChannels + composeUsers (adopt-merge) + WriteSeedISO via xorriso/genisoimage/mkisofs |
charly/vm_cloud_image.go + http_fetch.go | BuildCloudImage pipeline: fetch URL + sha256 sidecar + resize + seed ISO render |
charly/charly_install.go | EnsureCharlyInVenue — the GENERIC "copy charly into a running venue" mechanism (container podman cp / VM-SSH scp / host install, all via DeployExecutor.PutFile): returns the charly invocation command, copying the host os.Executable() to a non-$PATH /tmp/charly-<calver> on absence/older (idempotent, never shadows a packaged charly). Used by nested from-image delegation, so an image need not bake the charly layer. EnsureCharlyInGuest is the VM-deploy strategy wrapper (auto/scp/url/skip) layered on top |
charly/ovmf_paths.go | ResolveOvmfPaths (per-distro OVMF_CODE/VARS paths) + EnsurePerVmNvram + ResolveOvmfForSpec (bios-sentinel returning empty strings) |
charly/schema/vm.cue + cue_kind_vm.go | #Vm — the closed CUE schema validating VmSpec + the #LibvirtDomain/#VmCloudInit subtrees (the Go VM/libvirt validators were deleted; CUE owns it via the per-kind registry) |
charly/deploy_executor*.go | DeployExecutor interface + ShellExecutor + SSHExecutor with WaitForSSH + WaitForCloudInit |
charly/deploy_target_external.go | externalDeployTarget — the adapter the external vm substrate (and local/android/k8s) routes through |
charly/deploy_substrate_lifecycle.go + charly/vm_deploy_lifecycle.go | the substrateLifecycle interface + vmSubstrateLifecycle host-side VM venue lifecycle hook (boot/executor/preflight/nested-pods/teardown + charly vm Start/Stop/Status/Logs/Shell/Rebuild) |
candy/plugin-deploy-vm/ | the out-of-process deploy:vm plugin — kit.WalkPlans over the guest SSHExecutor |
charly/bundle_add_cmd_vm.go | host-side VM-only deploy helpers that REMAIN (deployNestedPodsInGuest, vmNameFromDeployName, sshReverseRunner, saveVmDeployState, removeVmDeployEntry); charly bundle add vm:<name> itself dispatches through dispatchNode → ResolveTarget → externalDeployTarget |
charly/vm_create_spec.go + vm_build.go | CLI command wiring for charly vm build/create reading kind: vm entities |
charly/libvirt_helpers.go + libvirt_yaml_listen.go | helpers shared by the libvirt YAML bridge + qemu_render argv emitter (VmRuntimeParams); structured <listen> support for LibvirtGraphics |
unified.go VM support (C2-substrate): "vm" is NO LONGER a spec.KindWords kind — the 5 substrate kinds (pod/vm/k8s/local/android) were externalized to the compiled-in candy/plugin-substrate (kind:pod/vm/k8s/local/android, Structural:true), so vm LEFT spec.KindWords + the #Node disjunction (no #VmArm) but STAYS in spec.ResourceKinds (member nesting) and its #VmValue def is KEPT for the host-side value gate. A vm: node resolves via recognizedKind (the compiled-in provider) → runPluginKind → foldSubstrateKind, which host-decodes the CANONICAL node via the core loader (buildBundleNode for a deploy shape → uf.Bundle, decodeNodeValue for a bare template → uf.VM map[string]*VmSpec — the C2-substrate TEMPLATE fold arm), validates its value against the KEPT #VmValue def (validateKindValueCUE), threads it to plugin-substrate's OpLoad via op.Env (spec.StructuralKindLoadEnv.Standalone), and folds the plugin's ECHO into the typed map. VmSpec = spec.Vm is the generated param alias. #NodeDoc is the sole STRUCTURE gate; a residual legacy vm:-keyed (or vms:-plural) document is hard-rejected by classifyDoc with a charly migrate hint (rootShapeKeySet unions spec.ResourceKinds so the substrate words stay legacy-detectable).
Full subsystem references: /charly-internals:vm-spec, /charly-internals:libvirt-renderer, /charly-internals:cloud-init-renderer, /charly-internals:vm-deploy-target, /charly-internals:ovmf, /charly-internals:cutover-policy.
Self-exec coordination: host → container AND host → host
The charly binary self-execs in two distinct directions.
Host → container — the host charly delegates to a container-baked (or copied-in) charly via exec … charly <subcommand>. The surviving site is nested from-image delegation: a from-image plan re-invokes charly inside the venue, and EnsureCharlyInVenue (charly/charly_install.go) copies the host binary in on demand when the venue lacks it. The best-effort desktop notification in charly/notify.go (sendVenueNotification) is NOT a self-exec site — it drives the venue's session bus with gdbus directly, no in-container charly.
Host → host (none) — there is NO host→host self-exec for check verbs. Every live-container verb (wl/cdp/vnc/dbus/mcp/record/kube/adb/appium/spice/libvirt) dispatches OUT-OF-PROCESS through the provider registry to its plugin candy (EXEC-based verbs drive the venue over the DeployExecutor reverse channel; endpoint verbs dial a host-pre-resolved address) — never by spawning a charly subprocess. charly's core carries no in-proc live-verb dispatch machinery.
The rule: whenever you rename a subcommand path crossed by the surviving host→container self-exec site (nested from-image delegation), edit the host-side invocation strings AND plan a coordinated rebuild of every image that bakes the charly layer (affected images: grep charly.yml for - charly$).
Quick Reference
| Action | Command | Description |
|---|
| Build | task build:charly | Compile to bin/charly and install as Arch package |
| Install | task build:install | Install charly as Arch package (uses pre-built binary) |
| Run tests | cd charly && go test ./... | Run all tests |
| Run specific test | cd charly && go test -run TestName ./... | Run single test |
| Vet | cd charly && go vet ./... | Static analysis |
| Format | cd charly && gofmt -w . | Format code |
Project Directory Structure
project/
├── bin/charly # Built by `task build:charly` (gitignored)
├── charly/ # Go module (go 1.25.3, kong CLI, go-containerregistry)
│ └── charly.yml # The binary's embedded default config (//go:embed,
│ # embed_defaults.go): distro/builder/init/resource
│ # build vocabulary + the sidecar: template library.
│ # Parsed by the SAME unified loader as any project
│ # charly.yml; a project ships none of it.
├── .build/ # Generated Containerfiles (gitignored)
├── charly.yml # Image definitions
├── Taskfile.yml # Bootstrap tasks only
├── taskfiles/ # Build.yml, Setup.yml
├── candy/<name>/ # Layer directories (160 layers)
├── plugins/ # Git submodule (overthink-plugins, 5 plugins, 244 skills)
└── templates/ # supervisord.header.conf (referenced by init.supervisord.header_file)
Submodule convention: plugins/ is a submodule rooted at the
overthink-plugins repo. Clone with --recurse-submodules or run
git submodule update --init after a plain clone. See
/charly-internals:skills for the skill-authoring and sync conventions.
Source Code Map
Core Generation
| File | Purpose |
|---|
main.go | CLI entry point (Kong framework). CLI struct carries two global path fields: Dir (-C / --dir / env CHARLY_PROJECT_DIR) and Repo (--repo / env CHARLY_PROJECT_REPO). When Repo is set, main() resolves it via ResolveProjectRepo and assigns the cache path back into Dir; when Dir is non-empty (after that resolution), main() calls os.Chdir(Dir) before ctx.Run() — one-line intervention that propagates to every os.Getwd() call site throughout build-mode commands without requiring per-command plumbing. --repo and --dir are mutually exclusive (fast-fail). Covered by TestCharlyDir_FlagChdir, TestCharlyDir_Errors, TestCharlyRepo_FlagChdir, TestCharlyRepo_DirConflict, TestCharlyRepo_DefaultExpansion in main_dir_test.go + main_repo_test.go. Load-bearing for charly mcp serve inside a container where cwd resolves to /workspace (the charly-mcp layer default) — either bind-mounted with the project, or empty in which case bootstrapProject() auto-falls back to the upstream repo. |
main_repo.go | --repo resolver. DefaultProjectRepo = "github.com/overthinkos/overthink". normalizeRepoSpec(spec) handles four spec shapes: "default" literal, bare owner/repo (auto-prefix github.com/ when first segment has no dot), bare owner/repo@ref, host-qualified host.tld/owner/repo[@ref]. ResolveProjectRepo(spec) reuses EnsureRepoDownloaded from refs.go so the project-repo cache shares ~/.cache/charly/repos/ (override CHARLY_REPO_CACHE) with the existing remote-layer cache. Empty version triggers GitDefaultBranch resolution. |
config.go | charly.yml parsing, inheritance resolution. BuildFormats type. Distro field. ResolvedBox.Tags (union). SupportsTag(), SupportsBuild() methods |
format_config.go | DistroConfig (with per-distro Formats), BuilderConfig types. BuildFile loader struct matches the three top-level build-vocabulary sections (distro:, builder:, init:). LoadBuildConfigForBox reads the project charly.yml (via LoadUnified, with the embedded build vocabulary merged in as the project-wins base) and splits it into DistroConfig / BuilderConfig / InitConfig views. Per-image config resolution with remote ref support |
format_template.go | Go text/template rendering engine. Template helpers: cacheMounts, cacheMountsOwned, quote, default, splitFirst, replace, join. InstallContext, BuildStageContext types |
layers.go | Layer scanning, file detection. CandyYAML (the = spec.Candy generated param alias — no hand struct; see spec_aliases.go), CUE-decoded via cue_loader.go; load-time top-level typo-detection via the rejectUnknownCandyTopLevelKeys guard — no custom UnmarshalYAML. Task struct + Kind() method (exactly-one-verb). derivePackageSectionsFromCalamares is the SOLE package-surface populator: every distro: key (bare / versioned / compound) → a per-distro tagSections entry (NOT a shared format section — that collapse caused the non-deterministic deb-repo bug); top-level package: → topPackages (folded at resolve time); arch aur: keeps its aur format section. compileSystemPackageSteps (install_build.go) cascades these — see /charly-internals:install-plan. The runtime Candy.ExternalBuilder field (the reserved word of an EXTERNAL builder plugin a candy selects; from the candy manifest external_builder:) lives here, populated by populateCandyFromYAML (unified.go) and resolved at build via OpResolve (generate.go emitExternalBuilderStages). |
tasks.go | All task emission logic — per-verb emitters (emitMkdirBatch, emitCopy, emitWrite, emitLinkBatch, emitDownload, emitSetcapBatch, emitCmd, emitBuild), emitTasks orchestrator, stageInlineContent (content-addressed), resolveUserSpec, taskSubstPath, taskUnresolvedRefs. Adjacent-coalescing (taskCoalescesWith). Shell-quoting helpers: shellSingleQuote(s) for standard '...' escaping (used by LABEL values + emitDownload env entries) and shellAnsiQuote(s) for bash ANSI-C $'...' quoting (used by emitCmd so multi-line script bodies survive podman's line-oriented Dockerfile parser). emitDownload env rule: uses export VAR=val; (semicolon-terminated) not VAR=val cmd, because bash expands ${VAR} in URL arguments before the cmd-prefix environment is assembled. ~430 lines, single home for install-task codegen. plugin: verb case + emitPluginFragment: a run: plugin step emits placement-agnostically — a builtin ProvisionActor renders an act shell RUN in-proc, any other resolved provider renders via emitPluginFragment → Invoke(OpEmit) → spec.EmitReply.Fragment spliced verbatim (in-proc for a builtin, go-plugin gRPC for an external). See /charly-internals:plugin + /charly-build:generate. |
generate.go | Containerfile generation — the build-mode emitter (the IR/OCITarget is deploy-only). NewGenerator connects the project's external plugin candies (loadProjectPlugins) so a run: plugin verb/builder executes at build time (the build-time plugin connect seam). BUILDER legs (all via OpResolve, C10): emitBuilderStages/emitBuilderArtifacts render the four DETECTION-builders (pixi/npm/aur; cargo inline via writeCandySteps) by Invoke(OpResolve)ing their plugins (resolveDetectionBuilder → the shared resolveBuilderStage → the plugins' kit.BuilderResolve — NOT an in-core vocabulary); emitExternalBuilderStages/emitExternalBuilderArtifacts do the same for external_builder:-selected out-of-tree candies (resolveExternalBuilder, minimal input). Both splice the cached spec.BuilderResolveReply (Stage pre-main-FROM, CopyArtifacts+CopyBinary post-main-FROM; cargo's InlineFragment in-candy) — an empty stage / unresolvable word fails loudly. writeCandySteps orchestrates per-layer: packages → emitTasks (from tasks.go) → builders → USER reset. Package resolution goes through the SAME resolveCascadePackages (install_build.go) the deploy compiler uses — ONE distro-specificity cascade for build AND deploy (folds the top-level package: base + unions distro tag sections most-specific-first), then renders the primary format's install template; non-primary build formats (aur) emit from their own format section. There is no separate build-path Phase1/Phase2 resolution anymore. Config-driven format install and bootstrap from the build vocabulary (distro: + builder: sections — the embedded default lives in charly/charly.yml); builder STAGE templates moved out to the plugins' kit.BuilderResolve (C10 — the builder: section retains detection + cache mounts + the deploy host phase + pixi's context inputs). writeLabels is called at the END of the final stage (after the final USER directive) — the volatile LabelDescription value would otherwise invalidate every downstream RUN/COPY on a baked-plan edit; with LABELs-at-end, only the LABEL steps themselves re-emit (cache preserves all install work). writeJSONLabel routes every JSON label value through shellSingleQuote so embedded ' chars in test commands (awk '{print $1}') don't break podman's key=value LABEL parser. |
validate.go | All validation rules. validateCandyTasks enforces exactly-one-verb, per-verb required modifiers, var key rules, path/mode/caps format, ${VAR} resolution checks. validateCandyContents accepts a candy whose only content is an external_builder: selection (or a plugin: block / candy: composition / data) as legitimately shipping no install files. Format/builder validation against config definitions (not hardcoded maps) |
version.go | CalVer computation |
scaffold.go | new layer scaffolding (single-layer dir creation with stub charly.yml) |
scaffold_project.go | new project scaffolding + charly.yml mutation helpers (ScaffoldProject, AddBox, AddCandyToBox, RemoveCandyFromBox). All YAML round-trips go through the yaml.v3 Node API so comments + key order are preserved. Tested in scaffold_project_test.go. |
scaffold_cmds.go | All Kong command structs for the MCP-first authoring surface: NewProjectCmd, NewBoxCmd, BoxSetCmd, BoxAddCandyCmd, BoxRmCandyCmd, BoxFetchCmd, BoxRefreshCmd, BoxWriteCmd, BoxCatCmd, CandyCmd (with Set + four add-{rpm,deb,pac,aur} aliases), CandyAddPkgCmd. Houses resolveProjectFile() — the path-traversal guard for box write / box cat. Houses detectPkgSection(os.Args) — the workaround for Kong not exposing "which alias triggered me" to a shared struct. Houses appendCandyPackages() with the scaffold's null-package: → sequence upgrade. |
yaml_setter.go | SetByDotPath(path, dotpath, valueYAML) — generic comment-preserving YAML setter used by charly box set and charly candy set. Walks *yaml.Node trees; creates intermediate mappings on demand; rejects descent into scalars. Tested in yaml_setter_test.go (comment preservation, list values, intermediate-mapping creation, scalar-descent error path). |
Plugins, external deploy & build-time emit
Provider/registry/SDK internals are owned by /charly-internals:plugin; the external-deploy lifecycle + wire types by /charly-internals:install-plan. The file map:
| File | Purpose |
|---|
deploy_target_external.go | externalDeployTarget — Add/Test/Update/Del for an OUT-OF-PROCESS deploy provider over the executor reverse channel (OpExecute); records teardown ops to the ledger keyed on computeDeployID |
plugin_step_external.go | externalPluginStepProvider — the StepProvider for StepKindExternalPlugin (a run: plugin: <verb> step served by an OUT-OF-PROCESS plugin). EmitOCI→Invoke(OpEmit) fragment (reusing emitPluginFragment, R3) is the only in-proc Emit. At DEPLOY time the step is executed via executeExternalPluginStep→InvokeWithExecutor(OpExecute) on the executor reverse channel — reached as a host-engine step over RunHostStep during the external local/vm deploy walk — recording the reply's dynamic ReverseOps to the CandyRecord. The executorInvoker interface (InvokeWithExecutor, satisfied SOLELY by *grpcProvider) is the discriminator. The IR kind StepKindExternalPlugin + ExternalPluginStep struct live in install_plan.go; compileActOp (install_build.go) routes an external (executorInvoker) plugin verb to it; allStepKinds + the bijection gate (provider_step.go) register it. Owned by /charly-internals:install-plan. |
plugin_prescan.go | Byte-gated, additive parse pre-scan: prescanPluginManifest registers an external deploy SUBSTRATE word (ClassDeployTarget, consumed by unified.go's loader path before the provider connects), an external COMMAND word (ClassCommand, registered via registerDeclaredExternalCommand, snapshot via declaredExternalCommandWords, consumed by prescanProjectCommandWords in main.go before kong.Parse), AND (F4) an external KIND word (ClassKind → registerDeclaredKind; recognizedKind = connected-OR-prescanned). F4 kind connect: unlike a deploy substrate (which defers to the bundle builder) or a command (lazy-connect on invocation), a kind: <plugin-word> entity must DECODE its body during load (runPluginKind), so connectDeclaredKindPlugins (called at the depth-0 loader hook right after the prescan) host-builds + connects the declared kind plugins BEFORE mergeUnifiedDocs. It is re-entrancy-GUARDED (inKindConnectPass): the connect re-loads the project (LoadConfig/ScanAllCandyWithConfigOpts → LoadUnified → the SAME root that contains the kind node), and the nested load skips the pre-pass while normalizeNodeInto DEFERS (skips, no error) the not-yet-connected kind node — so the nested scan succeeds and the OUTER pass then has the provider registered + decodes. A declared kind whose provider never connects is a LOUD error in normalizeNodeInto (never a silent drop). Example: candy/plugin-example-kind (out-of-process-only). F5 flat-vs-structural decode (runPluginKind, provider_kind_invoke.go): a FLAT kind lands its OpLoad body opaquely in uf.PluginKinds[disc][name] (F4); a STRUCTURAL kind (capability Structural=true, carried by the structuralKindCarrier on the grpc/inproc provider) returns a spec.Deploy (BundleNode) member tree runPluginKind json-unmarshals + folds into uf.Bundle[name] — the SAME map buildBundleNodeInto populates for a builtin pod, so the folded member goes through the SAME validateDeploy. This is the channel that externalizes the structural kind decoders: group is DONE (C2-group — the COMPILED-IN candy/plugin-group serves kind:group, Structural:true) and the 5 deploy-substrate kinds pod/vm/k8s/local/android are DONE (C2-substrate — the COMPILED-IN candy/plugin-substrate serves all 5, Structural:true; the shared builtin standaloneKind + cue_kind_*.go-arm removal, all left spec.KindWords + the #Node disjunction but STAY in spec.ResourceKinds so the loader still nests their members). Unlike group (a scalar #GroupInput value decoded in the plugin from op.Params), a substrate value is RICH + core-referencing, so the host uses the F5 Standalone channel: foldSubstrateKind (provider_kind_invoke.go) host-decodes the CANONICAL node via the core loader (buildBundleNode deploy → uf.Bundle, decodeStandaloneTemplateJSON template → uf.Pod/uf.VM/… — the TEMPLATE-map fold arm extending F5's deploy-only fold), validates the value host-side against the KEPT #<Kind>Value def (validateKindValueCUE, replacing the removed #Node arm's closedness — a self-contained plugin schema can't carry the rich value), threads it via op.Env (spec.StandaloneLoad), and folds the plugin's pure ECHO. candy is DONE too (C2-candy — the LAST structural kind; candy/plugin-candy-kind, COMPILED-IN): foldCandyKind host-decodes via the bootstrap-critical core candyIsImage + buildCandy (which STAY core — the discovered-candy pre-check calls them directly, so the compiled-in plugin has no bootstrap cycle), validates against the KEPT #CandyValue (validateKindValueCUE), threads spec.Box/spec.Candy via the SAME StandaloneLoad channel (candy-image/candy-layer shapes), and folds the echo into uf.Box/uf.Candy. candy is Structural:false (it nests no deploy members) and routes via an explicit gn.disc=="candy" host branch. So the #Node disjunction now has ZERO built-in arms (#Node: {...} — a structural gate only; per-kind value closedness is host-side) and spec.KindWords is EMPTY — every authoring kind is plugin-served. Authored-member INPUT-threading: the node's AUTHORED resource-member children cannot ride op.Params (closed #<Kind>Input), so runPluginKind PRE-DECODES them host-side via the SAME core recursion the builtin path uses (buildResourceMemberChildren, node_bundle.go — the ONE member-decode source, called by buildBundleNode too, R3) and threads the decoded subtree to OpLoad via op.Env (spec.StructuralKindLoadEnv); the plugin attaches them to its reply, so the reconstructed Bundle is byte-equivalent to the former builtin group (proven by TestExternalStructKind_StructuralDecode + the check-group / check-structkind runtime beds). A FLAT kind carrying members is a hard error (no silent drop). The parser gate admits sub-entity children under a recognized external STRUCTURAL kind (externalKindMayNestMembers/recognizedStructuralKind); core non-resource kinds stay guarded. Example: candy/plugin-group (compiled-in structural kind); candy/plugin-example-structkind (out-of-process-only witness). |
plugin_command_prescan.go | The EARLY (pre-kong.Parse) external-COMMAND-word prescan: prescanProjectCommandWords resolves the project dir pre-parse (projectDirPreParse: CHARLY_PROJECT_DIR → scanDirFlag over os.Args → cwd) and registers each declared command word so charly <word> PARSES; connectCommandPlugin is the LAZY connect (LoadConfig → ScanAllCandyWithConfigOpts → loadProjectPlugins scoped to the one word → resolve(ClassCommand, word)), paid only on an actual charly <word> invocation |
provider_command_external.go | OUT-OF-PROCESS command dispatch: collectExternalCommandPlugins builds a Kong grammar holder per prescanned word with the provider UNconnected (prov nil) so the CLI parses; dispatchExternalCommand lazy-connects on invocation (connectCommandPlugin) and forwards the pass-through args via Invoke(OpRun, {"args":[…]}); NestedCommandProvider nests an external command under a parent (e.g. charly check kube). The BUILTIN command path is provider_command.go (CommandProvider.KongCommand() + Go Run; builtinCommandBase.Invoke is in-proc-only). F8 command compile-in: dispatchCommand (the dispatch entry, called from main) routes a parsed dynamic command by PLACEMENT — a COMPILED-IN command candy (registered in-proc as an inprocProvider, not a *grpcProvider) dispatches IN-PROC via dispatchInProcCommand → Invoke(OpRun, {"args":[…]}), so the candy's handler runs in charly's own process (native stdio); an out-of-process one keeps dispatchExternalCommand/syscall.Exec. The dynamic Kong grammar (externalCommandHolder) is identical for both placements — only the dispatch transport differs (the command half of placement-invisibility). Example: candy/plugin-example-command (dual-placement, compiled-in) |
check_venue.go | checkLocalTarget routes an external deploy host-side (the SAME path target: local takes) for charly check live / charly check <verb>, R3 |
spec/deploy_wire.go | Deploy IR wire types shared with the plugin SDK: Scope, ReverseOp (+ ReverseOpPluginScript), InstallPlanView, DeployVenue, DeployReply; plus the build-time BuildEnv / EmitReply for OpEmit, and BuilderResolveInput + BuilderResolveReply ({Stage, CopyArtifacts, CopyBinary, InlineFragment}) for the builder OpResolve leg |
tasks.go:emitPluginFragment | Renders a plugin verb's BUILD-context Containerfile fragment via Invoke(OpEmit) → spec.EmitReply.Fragment (placement-agnostic above the registry) |
generate.go:emitBuilderStages / emitBuilderArtifacts / resolveDetectionBuilder | The DETECTION-builder BUILDER leg (C10): for each candy a builder DETECTS, connects the plugin (ensureBuildersConnected) + Invoke(OpResolve) via the shared resolveBuilderStage → spec.BuilderResolveReply (Stage pre-main-FROM, CopyArtifacts+CopyBinary post-main-FROM; cargo's InlineFragment splices in writeCandySteps). Renders via the plugins' kit.BuilderResolve, NOT an in-core vocabulary. Detection stays host-side (candyNeedsBuilder) |
generate.go:emitExternalBuilderStages / emitExternalBuilderArtifacts / resolveExternalBuilder | The external_builder: BUILDER leg: emit an out-of-tree ClassBuilder candy's multi-stage via the SAME resolveBuilderStage/Invoke(OpResolve) (minimal input — candy name only); selected by a candy's external_builder: field, requires a non-empty Stage |
build_emit_test.go | TestEmitPluginFragment_BuildTimeOpEmit — the build-time-plugin-execution gate (a non-ProvisionActor provider's fragment is spliced via Invoke(OpEmit)) |
provider_bench_test.go | The E3 perf go/no-go gate: TestPerfGate_BuiltinVerbsSkipEnvelope, BenchmarkVerbTypedDispatchFork (0-alloc) vs BenchmarkVerbEnvelopeMarshal — builtins skip the JSON Invoke envelope; it is paid ONLY out-of-process |
Dependency & Graph
| File | Purpose |
|---|
graph.go | Topological sort (layers + images), ResolveBoxOrder() |
intermediates.go | Auto-intermediate image computation (trie analysis). createIntermediate() inherits Distro and BuildFormats from the parent image first, falling back to cfg.Defaults.* only when the parent is external or empty. Inverting this (defaults winning over the explicit parent) mis-tags every arch-rooted intermediate as build: [rpm], so every layer section keyed on pac: emits an empty RUN step (symptom: arch-ssh-client ships without direnv / gnupg / openssh). Regression guard: TestComputeIntermediates_InheritDistroFromParent uses defaults.Build=[rpm] but expects arch-rooted intermediates to come out [pac]. |
Build & Runtime
| File | Purpose |
|---|
build.go | build command (sequential image building, retry logic) |
merge.go | merge command (post-build layer merging) |
shell.go | shell command (execs engine run) |
start.go | start/stop commands |
status.go | status command (structured table/detail view, live tool probing, --json) |
commands.go | enable/disable/logs/update/remove |
service.go | service command (init system service management inside containers) |
data.go | Volume data seeding (provisionData, seedKind, SeederHelperImage) for bind-backed + named-volume targets, driven by charly config --seed/--force-seed |
hooks.go | Lifecycle hooks (post_enable, pre_remove) collection and execution |
remote_image.go | Remote image ref resolution, pull-or-build |
vm.go | VM lifecycle: create, start, stop, destroy, list, console, ssh |
vm_build.go | VM disk image builds (qcow2, raw via bootc install) |
vm_libvirt.go | Libvirt backend: VM operations via session-level libvirt |
vm_qemu.go | QEMU backend: direct VM operations via qemu-system |
smbios_credentials.go | SSH key injection via SMBIOS/systemd credentials at VM boot |
libvirt.go | Libvirt XML snippet collection and injection |
verb_preresolve.go + verb_preresolve_wrappers.go | The GENERIC per-verb host-side preresolver registry (F1) — registerVerbPreresolver(word, fn) + the opaque CheckEnv.Substrate json.RawMessage, the check-verb analogue of registerDeployPreresolver + DeployVenue.Substrate. invokeVerbProvider runs the registered preresolver for the verb word (no per-verb hardcode in the dispatch — the Uniform API Invariant); the 4 endpoint verbs (cdp/vnc/mcp/spice) register a wrapper marshalling their resolved *Env into CheckEnv.Substrate (decoded by the matching plugin into its own endpoint type), and kube registers one that rewrites the op's KubeContext. The former hardcoded CheckEnv.{Cdp,Vnc,Mcp,Spice} typed fields + the hardcoded 5-preresolver dispatch block are gone. |
cdp_preresolve.go | Host-side cdp: endpoint pre-resolver (the analogue of mcp_preresolve.go): resolves the container's published CDP port (9222) into the CheckEnv snapshot so the out-of-process candy/plugin-cdp provider dials without container inspection. The cdp: verb (open/list/close/text/html/url/screenshot/click/type/eval/wait/coords/raw + spa-*) and its CDP WebSocket client live entirely out-of-process in candy/plugin-cdp — the core's former minimal CDP client (browser_cdp.go, golang.org/x/net/websocket) was DELETED when wl externalized (the wl coordinate-translation helper was that client's last in-core user), so golang.org/x/net is now an INDIRECT dependency. The host keeps only this endpoint pre-resolution. |
vnc_preresolve.go | Host-side vnc: endpoint pre-resolver (the analogue of cdp_preresolve.go/mcp_preresolve.go): resolves the dual pod/vm RFB endpoint — a pod's published port 5900, or a VM's libvirt VNC display reached via bridge/tunnel — plus resolveVNCPassword (the VNC credential store), into the CheckEnv snapshot so the out-of-process candy/plugin-vnc provider dials and speaks RFB without container inspection. The RFB verb itself (screenshot/click/type/key/mouse/status/passwd/rfb) and the custom RFC 6143 client live out-of-process in candy/plugin-vnc (vnc.go + vnc_vm.go + vnc_client.go were deleted, the stdlib RFB client moved there); the host keeps only the endpoint pre-resolution + the credential store. The separate charly ssh tunnel vnc (ssh.go) SSH tunnel that forwards a VM's VNC endpoint stays in core, unaffected by the verb externalization. |
Infrastructure
| File | Purpose |
|---|
engine.go | Docker/Podman abstraction, ResolveBoxEngineForDeploy() |
registry.go | Remote image inspection (go-containerregistry) |
transfer.go | Cross-engine image transfer |
runtime_config.go | ~/.config/charly/config.yml, secret_backend key, credential maps |
network.go | Shared "charly" container network management |
machine.go | Podman machine management (rootful VM builds) |
Configuration
Key types — user_policy + exclude_distros architecture:
| Type / Field | File | Purpose |
|---|
DistroDef.BaseUser *BaseUserDef | format_config.go | Pointer to a declared pre-existing uid-1000 account in the upstream base image. Nil when not declared (fedora/arch/debian); set for ubuntu ({ubuntu, 1000, 1000, /home/ubuntu}). Inherited via resolveInherits so a child distro with no base_user: inherits the parent's |
BaseUserDef | format_config.go | Four required fields: Name, UID, GID, Home. Parsed from the embedded build vocabulary's distro.<name>.base_user: |
BoxConfig.UserPolicy string | config.go:130 | YAML field user_policy. Values: auto (default) / adopt / create. Drives the reconciliation switch in ResolveBox |
ResolvedBox.UserAdopted bool | config.go:194 | True when the policy reconciliation adopted a distro's BaseUser (User/UID/GID/Home overwritten). Consumed by writeBootstrap in generate.go to skip the useradd step |
Op.ExcludeDistros []string | checkspec.go | Per-test filter — test runner in checkrun.go:runOne skips the check when any of the image's distro tags intersects with this list. Reason reported as excluded on distro "<tag>" |
TagPkgConfig.Raw map[string]any | layers.go | Captures the full YAML map for a tag section (e.g. debian:13:), not just package:. Enables repos:, keys:, options: inside tag sections. Read by the generator's install-template emission path |
Policy reconciliation flow (charly/config.go:ResolveBox, after distroDef loaded):
policy := img.UserPolicy
if policy == "" { policy = c.Defaults.UserPolicy }
if policy == "" { policy = "auto" }
baseUser := (*BaseUserDef)(nil)
if resolved.DistroDef != nil { baseUser = resolved.DistroDef.BaseUser }
userExplicitlySet := img.User != "" || c.Defaults.User != ""
switch policy {
case "adopt":
if baseUser == nil { return nil, fmt.Errorf(...) }
resolved.UserAdopted = true
case "auto":
if baseUser != nil && !userExplicitlySet {
resolved.UserAdopted = true
}
case "create":
}
See /charly-image:image "user_policy" for the user-facing decision matrix, /charly-build:build "base_user:" for the declarative side, and /charly-build:generate "writeBootstrap" for the consumer side.
Existing configuration files
| File | Purpose |
|---|
env.go | ENV merging, path expansion |
envfile.go | .env file parsing (ParseEnvFile, ParseEnvBytes), runtime env var resolution/merging |
security.go | Container security config collection, CLI args generation. Merges Mounts from layer security configs |
labels.go | OCI label constants. LabelDescription (ai.opencharly.description) carries the LabelDescriptionSet — each LabeledDescription (a Description string) plus its Plan []Step list; BoxMetadata's *LabelDescriptionSet field is populated by ExtractMetadata when present |
egress.go | Egress validation SHIM (M16) — ValidateEgress/ValidateEgressValue/validateTextEgress/ValidateXMLEgress keep their signatures but resolve verb:egress + Invoke(OpValidate, {kind,label,mode,data}) (plain host→plugin dispatch). The validation logic + the egress CUE schemas (package-less defs + the vendored cloud_config) moved to the compiled-in candy/plugin-egress (egress-schemas/), which holds them internally + serves only a trivial Describe schema. The former in-core egressDef/registerVendoredEgressKind/egressKindDefs + the 7 cue_*egress*.go registrars were deleted. See /charly-internals:egress. |
volumes.go | Named volume collection/mounting |
alias.go | Command aliases (wrapper scripts) |
deploy.go | Per-deployment config overlay, DeployVolumeConfig, ResolveVolumeBacking(), saveDeployState(), cleanDeployEntry() (instance-aware provides cleanup) |
provides.go | Env/MCP provides injection, removeBySource(), removeByExactSource() (instance-specific cleanup), podAwareMCPProvides() |
enc.go | Encrypted-volume in-core SHIM + deploy-model (C16a). Keeps ResolvedBindMount, the config loader (loadEncryptedVolume), the path/probe helpers (encryptedPlainDir/isEncryptedMounted/isEncryptedInitialized/cipherPopulatedPlainEmpty — consumed synchronously by the mandatorily-core ResolveVolumeBacking + verifyBindMounts), encStatus (pure probe+print), and the credential passphrase resolution (resolveEncPassphrase*/awaitKeyringUnlockViaPlugin). encMount/encUnmount/encPasswd/ensureEncryptedMounts are thin shims that HOST-PRELIFT the per-volume plan (encPlanFor: resolved cipher/plain dirs + init/mounted flags + scope-unit) + the passphrase, then encExecViaPlugin resolves verb:enc and Invokes OpExecute. The gocryptfs / systemd-run --scope --unit=charly-enc-<dir>-<volume> / fusermount3 / extpass SHELLING lives in candy/plugin-enc now, NOT core (-allow_other for rootless keep-id, stale-scope retry, all there). The encMount all-mounted fast-path (skip passphrase when every volume is already mounted) stays in the shim |
devices.go | The KEPT core GPU/device surface after the GPU/VFIO host-DETECTION externalized to candy/plugin-gpu (C11). Holds the embedded detection DATA tables (devicePatterns/gpuRenderVendors/pciClassLabels, kept in core because charly doctor's device report reads devicePatterns — threaded to the plugin via the shims, R3) + the pure host-INDEPENDENT env/group helpers appendAutoDetectedEnv() (centralizes injection of HSA_OVERRIDE_GFX_VERSION/DRINODE/DRI_NODE), appendEnvUnique, appendGroupsForAMDGPU, LogDetectedDevices, memlockUnlimited, AutoDetectFlags. The sysfs/exec detection PRIMITIVES moved out (see gpu_shim.go) |
gpu_shim.go | The in-core SHIMS for GPU/VFIO host detection (C11): DetectGPU/DetectAMDGPU/DetectVFIO/DetectHostDevices (package vars, testability) + EnsureCDI/MemlockLimitBytes/VfioGroupAccessible/detectAMDGFXVersion resolve verb:gpu and Invoke the COMPILED-IN candy/plugin-gpu (OpRun, action-multiplexed spec.GpuProbeInput) — the k8sgen/egress resolve+Invoke pattern. The detection RESULT types alias package spec: type VFIOReport = spec.VFIOReport (+ VFIOGpu/VFIOPCIDevice/DetectedDevices), so the ~10 consumers (config_image/start/shell CDI-env sites, charly doctor, charly vm gpu, charly vm create, gpu_allocate.go) compile unchanged. In-proc placement keeps MemlockLimitBytes reading charly's OWN RLIMIT_MEMLOCK. Wire types: spec/gpu_wire.go. The DRIVER-SWITCH (C9, 1B) now ALSO resolves+Invokes verb:gpu — the gpu_shim.go shims switchGPUDriverMode/gpuSwitchModeTolerant/groupInMode/currentGPUMode/gpuDisplayDriver/gpuWedgeDetected/ensureCDIRoot/gpuSwitchPlan dispatch the OpRun DRIVER-SWITCH actions (spec.GpuSwitchInput/GpuSwitchReply); the logic moved into candy/plugin-gpu (switch.go), the mode/driver consts + wedge sentinel + pure SelectGPUByVendor/NormalizePCIVendor into spec. Auto-allocation (gpu_allocate.go) STAYS core (host-side VmSpec orchestrator consuming DetectVFIO); the config-coupled GPU-consumer helpers moved to gpu_imply.go (still core) |
preempt.go | The HOST side of the resource arbiter after cutover C9: the arbiter LOGIC (ResourceArbiter) moved into the COMPILED-IN candy/plugin-preempt (verb:arbiter). Core keeps (a) the in-core PROXY — newResourceArbiter() returns *arbiterProxy whose Status/ReleaseClaimant/reconcileStranded/clearPoison/resourcePoisoned + the Lease + acquireResourceForClaimant/acquireExclusiveForClaimant/acquireSharedForClaimant/releaseResourceClaim shims resolve+Invoke verb:arbiter (the ~6 consumers compile through the shims, R3), and (b) the 7 arbiter HOST-SEAM impls (gatherPreemptibleHolders/holderRunning/holderStop/holderStart/gatherResources/holderAddrFor/lookupVMClaimant/waitStoppedHost) the arbiter calls back over ExecutorService.HostArbiter. Persisted + seam wire types live in spec/arbiter_wire.go; the ledger I/O + poison + liveness + mode-math live IN the plugin (arbiter.go/arbiter_support.go) |
arbiter_host.go | The HOST handler for the C9 ExecutorService.HostArbiter reverse channel: arbiterHostServer.dispatch runs the 7 arbiter host-seams, projecting gatherPreemptibleHolders → []spec.HolderDescriptor + gatherResources → token→vendor, folding the stop seam's wait (holderStop+waitStoppedHost), and routing switchMode/ensureCDI to the gpu shims |
tunnel.go | The RESOLUTION half of the tunnel subsystem (C16b externalization): the wire types TunnelConfig/TunnelPort, the pure helpers schemeTarget/tailscaleFlag/isTCPFamily/ValidPublicPorts (shared with the quadlet emitter), the config-path helpers tunnelConfigDir/tunnelConfigPath (referenced by quadlet.go's generateTunnelUnit), and the resolution ResolveTunnelConfig/TunnelConfigFromMetadata/parseHostPorts/buildPortMapping/resolveProto. The EXECUTION leg (tailscale serve/funnel + the cloudflared lifecycle) externalized to candy/plugin-tunnel — see tunnel_plugin.go |
tunnel_plugin.go | The CORE adapter for the EXTERNALIZED tunnel execution leg (C16b, the welded-verb pattern mirror of credential_plugin.go): the TunnelStart/TunnelStop/cloudflareTunnelSetup seams forward a resolved TunnelConfig to verb:tunnel as a {method, config} plugin_input envelope over the Invoke registry (tunnelProvider is registry-first so the compiled-in provider resolves project-lessly, then falls back to connectPluginByWord for baked/source). The tailscale serve/funnel + cloudflared lifecycle live in candy/plugin-tunnel/ (compiled into charly via compiled_plugins:, or out-of-process); verb:tunnel also serves a creds-free plan dry-run returning the argv it WOULD run (box/fedora's check-tunnel-pod bed R10) |
quadlet.go | Quadlet .container file generation, Secret= directives |
credential_plugin.go | The CORE adapter for the EXTERNALIZED credential store (C2 dep-shed removed go-keyring; the godbus dep-shed removed godbus too — charly/go.mod links neither): CredentialStore interface, ResolveCredential(), DefaultCredentialStore() (→ pluginCredentialStore), resolveSecretBackend(), credentialHealth(), pluginCredentialStore.awaitUnlock + the credentialAwaiter seam (enc.go's source=locked keyring wait — RPCs verb:credential await-unlock, the godbus PropertiesChanged subscription running IN the plugin), the setDefaultCredentialStoreForTest seam. Every method forwards to verb:credential (served out-of-process by candy/plugin-secrets, or the baked /usr/lib/charly/plugins binary). The store backends + the keyring-unlock waiter + the charly secrets CLI + GPG .secrets surface live in candy/plugin-secrets/ now, NOT core. Generic host-adapter seam (F7/C7): callCtx connects via connectPluginByWord(ClassVerb, "credential") — the ONE on-demand connect for a verb word that appears in NO plan step. vm_plugin_client.go (invokeVmPluginEnv → verb:libvirt) and k8s_plugin.go (invokeKubePlugin → verb:kube) now route through the SAME seam (connectPluginByWordRef adds an optional canonical-ref fallback for a project whose closure references the plugin candy nowhere, e.g. a box/<distro> VM bed) — the bespoke ensureVmPluginConnected sync.Once + kube's bare ResolveVerb were deleted (R3) |
secrets.go | Container secret collection from labels, Podman secret provisioning, SecretArgs(), generateAndStoreSecret, the interactive promptPassword (a deploy-time operator prompt) |
Remote Layer Refs
| File | Purpose |
|---|
refs.go | Remote ref types, parsing, cache management. CHARLY_REPO_OVERRIDE (RepoOverrideEnv) Go-replace-style local-tree override (repoOverrideDir). selfSuperprojectOverridePair(dir) derives the bed project's OWN superproject override (git rev-parse --show-superproject-working-tree → rootRepoIdentity); mergeRepoOverrides appends it after operator entries (operator wins). runCheckBed auto-applies it so a box/<distro> bed tests LOCAL parent-repo candies, never the pinned remote — the candy-ref analogue of auto --dev-local-pkg. Tests: repo_override_test.go. |
refs_git.go | Git operations: clone, resolve ref, tag resolution |
Declarative Testing
Implements the charly check live / charly check box commands and the
ai.opencharly.description OCI label. User-facing authoring, verb catalog,
runtime variables, and charly.yml overlay rules live in /charly-check:check — this
section is the Go-implementation map.
| File | Purpose |
|---|
checkspec.go | Op (the = spec.Op generated param alias — no hand struct; Op.Kind() is a package-spec method) — the unified verb vocabulary, the former Task + Check merged into one; Kind() enforces exactly-one verb. Built-in verbs (file/port/command/http/package/service/process/dns/user/group/interface/kernel-param/mount/addr/matching) plus the live-container verbs — ALL out-of-process now (wl/cdp/vnc/dbus/kube/adb/appium/spice/mcp/record/libvirt), dispatched via invokeVerbProvider to their plugin candies (all still verb words on core #Op; wl was the LAST compiled-in live verb). Status on the http verb is a plain int — not a MatcherList. One expected code per test; no [200, 302] list shorthand. Matcher + MatcherList with custom YAML and JSON unmarshalers for scalar/list/map shorthand — symmetry between charly.yml authoring and hand-crafted OCI labels. The plan step types travel in LabelDescriptionSet (the ai.opencharly.description label carries a Plan []Step field per LabeledDescription). Extended ${NAME[:arg]} matcher (in ExpandTestVars) — backward-compatible widening of taskVarRefPattern in tasks.go. No bash-style defaults: ${VAR:-fallback} is unsupported; only ${IDENT}. ExpandTestVars, TestVarRefs, IsRuntimeOnlyVar, Op.ExpandVars. |
checkvars.go | ResolveCheckVarsBuild / ResolveCheckVarsRuntime. InspectContainer is a swappable package-level var (test-friendly pattern matching InspectLabels in labels.go). Maps podman inspect output into HOST_PORT:<N>, VOLUME_PATH:<name>, VOLUME_CONTAINER_PATH:<name>, CONTAINER_IP, CONTAINER_NAME, ENV_<NAME>. |
checkrun.go | Runner, Executor interface, ContainerExecutor (via podman exec), ImageExecutor (via podman run --rm). CheckStatus/CheckResult types (named to avoid collision with doctor.go). Per-verb dispatch for file/port/command/http. Matcher evaluation: matchOne + matchNumeric (lt/le/gt/ge). validMatcherOps allowlist kept in lockstep with the runner switch by TestMatcher_AllowlistRunnerSync. Output formatters: text, JSON, TAP. |
checkrun_verbs.go | Dispatch for the remaining verbs: package (rpm/dpkg/pacman), service (supervisorctl + systemctl), process (pgrep), dns (host-side net.LookupIP or in-container getent), user/group (getent passwd/group), interface (ip -o addr show + MTU), kernel-param (sysctl -n), mount (findmnt), addr (host-side net.DialTimeout or in-container nc -z), matching (pure in-process value matching). resolvePackageName(c, distros) implements the distro-aware package-map: when Check.PackageMap is non-empty, the first entry in Runner.Distros that matches a key wins; otherwise Check.Package is used as-is. Covered by TestResolvePackageName (6 sub-cases including empty-map fallback, first-matching-tag-wins priority, and empty-string-map-value fall-through). Runner.Distros is populated from meta.Distro at both entry points in check_cmd.go. |
check_members.go | Cross-deployment probing — a DRIVER deployment probing a SEPARATE SUBJECT. liveTargetResolver is the venue-from-position TargetResolver for charly check live + beds (resolves a driver/member via resolveCheckVenue + ResolveCheckVarsRuntime); wired into CheckLiveCmd.Run (pod path) AND runVm (VM path). The unified ${HOST:member} / ${HOST:member:port} address var (applyHostVars → collectHostRefs → resolveHostVars) is pre-resolved into Runner.HostVars, overlaid by effectiveEnv onto whatever resolver is active (primary / venue-swapped / harness — one injection point). ${HOST:member} (no :port) = the subject's charly-<member> container DNS (via resolveContainer, which also verifies running); ${HOST:member:port} (with :port) = host-reachable resolveCheckEndpoint. Registered runtime-only in checkspec.go runtimeOnlyVarPrefixes. |
