productize-chart-env

name

productize-chart-env

description

Productize a Lerian Helm chart's configuration from the app's config/.env.example onto lerian-common. Typed KNOBS are reserved for DEPENDENCY CONNECTIONS (db/cache/broker/ streaming/service-discovery/tenant-manager/auth/object-storage) via domain helpers/masks; ALL other app config stays an escape-hatch passthrough with the default in the template; credentials become fail-fast secrets. Use when productizing a chart, or when the app added/renamed env vars and the chart must re-sync to the .env contract.

Productize a chart's config from the app .env (the #1741 pattern)

Premise (non-negotiable)

The app's config is the authority — NOT the chart's prior render. Use the config/.env.example as the working surface, BUT the true authority is what the app actually READS — the config struct (config.go / config.ts / LoadConfig). The .env.example is the documented surface and can LAG the struct (real example: STREAMING_SASL_MECHANISM is read by the app + emitted by the streaming helper but absent from br-ccs's .env.example). So: cross-check the .env.example against the config struct, and never let a missing .env line hide a key the app reads — this is exactly why the schema allowlist unions the .env keys with the chart's RENDERED keys (Step 8) and why coverage.py FAILS on a struct/emitted key that the surface misses. (Aligns with ring-dev-team:creating-helm-charts, which measures coverage against the config struct, not the example file.)

Every ACTIVE env var must be COVERED (settable + defaulted), but coverage is TIERED — a typed knob is NOT the goal for every key. Reserve typed knobs for DEPENDENCY CONNECTIONS; everything else is an escape-hatch passthrough with the default in the template.

The dividing question is by NATURE, not by helper: "is this how the app reaches a DEPENDENCY?" Databases, cache, broker, streaming, service-discovery, tenant-manager/auth, object storage — the things a non-expert MUST wire to get running. Those get a masked, typed, validated knob. All the rest — rate-limit, outbox, swagger, cors, pagination, probes, timeouts, retention, pool tuning — has a sensible default and is reached only by the rare expert, via the escape hatch.

Every ACTIVE var is therefore exactly one of:

• dependency knob → lerian-common.datastore.value mask (db/cache/broker: host/port/user/ssl) OR a domain helper (multiTenant.env, streaming.env, serviceDiscovery.env, otel.env, globalValue auth) + global.* block OR a small typed mask for a dependency the lib doesn't cover yet (object storage: endpoint/region/bucket). This is the DOCUMENTED, schema-guarded surface.
• passthrough config → {{ $cm.KEY | default "<.env value>" | quote }} — default in the template, overridable via configmap.<KEY>. NOT a grouped param, NOT a <comp>.<group> block.
• secret → secrets.yaml + a fail-fast (multiTenant.secret, or a required guard)
• gap → recorded backlog (app declares it, chart can't cover it yet)
• omitted → with a written reason

Why this is not overengineering: the old model routed all ~120 keys through cfgValue grouped params. That MASKS THE NAME but never REDUCES THE COUNT — a non-expert facing 120 redis.connMaxIdleTimeMins is no better off than facing 120 env vars, and each grouped param is a hand-maintained PARALLEL contract that can silently diverge from the env key (the postgresName vs postgres.name bug) with the schema unable to catch it. Tiering cuts the typed surface ~90→~15 knobs and the schema ~50% while staying byte-identical (the grouped params were freshly invented, not the app's contract — dropping them changes nothing that shipped).

Two escape hatches stay open (every mature chart keeps one — do NOT remove them):

• <comp>.configmap.<KEY> — the PRIMARY override surface now: sets/overrides any passthrough key ($cm.KEY reads it first, before the template default).
• <comp>.extraEnvVars — injects ANY var, incl. one the app added before the chart re-synced. The typed dependency knobs are the documented, must-set surface; the hatches carry the long tail. See "Market alignment" at the end for why we still enumerate the .env (for the schema allowlist) without turning every key into a knob.

Values structure (STANDARD)

Top-level tiers:

1. global: — config shared across MORE THAN ONE component (env-wide). Helm-special (propagates to subcharts). Holds: observability, multiTenant (infra), datastores (SHARED mask), serviceDiscovery, streaming, plus k8s-infra imageRegistry / imagePullSecrets / commonLabels.
2. One <componentName>: block PER INDEPENDENT DEPLOYMENT. componentName = the deployment's name in camelCase, NO hyphen (brCcs, brSta, worker) — access via .Values.brSta, NEVER index .Values "br-sta". Holds that deployment's DEPENDENCY masks/toggles only — datastores (DEDICATED mask), multiTenant/serviceDiscovery/streaming toggles, externalXDefinitions (bootstrap-for-external) — plus configmap: {} (the escape hatch where ALL non-dependency app config is overridden), extraEnvVars, image/service/resources/ replicaCount/ingress (k8s), secrets. NO <comp>.<group> cfgValue blocks (rateLimit/ outbox/swagger/…): under the tiered model those are passthrough with the default in the template, reached via configmap.<KEY>. Example: br-ccs → brCcs:; br-sta → brSta: + worker:.
3. Subchart dependencies (postgresql/valkey/rabbitmq/…): Helm REQUIRES subchart values at the ROOT under the subchart name — they CANNOT be nested (Helm passes .Values.postgresql to the postgresql subchart by name). Group them visually with a # --- dependencies --- comment. Our ABSTRACTION over them (the datastores mask + externalXDefinitions bootstrap) lives inside the <componentName>, not at root.

EXCEPTION — single-deployment chart with in-pod sidecars stays FLAT. A chart that is ONE Deployment whose extra containers are SIDECARS in the same pod (br-slc: app + signer + xsd-validator

• mqbridge + a migrations Job) does NOT wrap under a component key. Its sidecars/jobs share pod-level config (securityContext, serviceAccount, nodeSelector, resources, the app image) — nesting the app under <componentName> while the sidecars reference .Values.securityContext breaks them. Keep it FLAT: config groups + k8s at root, sidecars/migrations as their own top-level blocks. The chart IS the component; the umbrella's subchart-name namespacing already scopes it (br-slc.redis).

Still non-conformant (fix on sight): a hyphenated component key needing index .Values "br-sta" (early br-sta) — always camelCase. Note the umbrella cost: a wrapped chart is br-ccs.brCcs.redis (double level) vs a flat one br-slc.redis (single) — the wrapper is the accepted price of one block per deployment when a chart genuinely has more than one.

Inputs

• The chart dir (e.g. charts/br-ccs).
• The app's config/.env.example. Fetch it (repos are private but reachable): gh api repos/LerianStudio/<app>/contents/config/.env.example --jq '.content' | base64 -d
• The component root: single-service uses .Values.<comp> (br-ccs brCcs), or a flat chart uses .Values directly (br-slc: config under .Values.configmap.data). Multi-component (br-sta manager+worker): productize the manager only — the worker inherits via envFrom.

Step 0 — Introspect lerian-common FIRST (do not assume the domain map)

Before classifying anything, enumerate what the vendored lerian-common actually offers, so the skill wires whatever exists TODAY (and picks up new domain helpers as the lib evolves):

# every helper the chart's lerian-common exposes
grep -rhoE 'define "lerian-common\.[a-zA-Z.]+"' charts/lerian-common/templates/ | sort -u
# read a helper's contract (inputs + Usage) before wiring it
sed -n '/define "lerian-common.serviceDiscovery.env"/,/^{{- end/p' \
    charts/lerian-common/templates/_service_discovery.tpl
# authoritative reference (versioned): the lib's doc.go / chart docs

Match each .env domain prefix to a helper that EXISTS. If a cross-cutting prefix has no helper (e.g. a brand-new subsystem), record it as a gap — do NOT hand-roll the domain env; raise it so lerian-common gains the helper first. Confirm the lib version in the chart's Chart.yaml dependency and check its docs/UPGRADE-*.md for contract changes.

Domain map (cross-cutting → lerian-common helper + global block)

The closed dependency vocabulary — the STANDARD — lives in references/dependency-contract.md: the full table (per domain: helper, global.* block, per-component knob, secrets), the object-storage gap, and the rule for adding a new domain (extend lerian-common, never a per-chart knob). Read it — it is what makes "knob IFF it matches a declared dependency domain" mechanical instead of per-app judgment. The quick table below is a summary; the contract is canonical.

Current at time of writing — verify against Step 0, the introspection is the source of truth. These ALWAYS go to a helper (never a passthrough) — that is where global.* env-wide config lives. The helpers already exist; you are only WIRING them.

.env prefix	helper	global block	notes
MULTI_TENANT_*	multiTenant.env + multiTenant.secret	global.multiTenant	toggle via <comp>.multiTenant.enabled; URL/REDIS_HOST required when on; API_KEY fail-fast
ENABLE_TELEMETRY, OTEL_*	otel.env (+ otel.envFlat for identity)	global.observability	shared 3 keys derive; name/library/version per-component
POSTGRES_*/REDIS_*/RABBITMQ_* host+conn	datastore.value (type postgres/redis/broker)	global.datastores (+ <comp>.datastores)	host/port/user/ssl/replicaHost only; TUNING (pool/timeouts) stays cfgValue
SD_*	serviceDiscovery.env or .envFlat	global.serviceDiscovery	envFlat (flat passthrough) is the simplest for a consumer chart
STREAMING_*	streaming.env + streaming.secret	global.streaming	full SASL/broker contract; STREAMING_SASL_PASSWORD is a secret
PLUGIN_AUTH_*	globalValue (block auth)	global.auth	ENABLED always; HOST emitted only when non-empty
OBJECT_STORAGE_* (endpoint/region/bucket)	objectStorage.value mask (lerian-common ≥1.5.0)	global.objectStorage.<name> / <comp>.objectStorage.<name>	S3/SeaweedFS dependency; credentials (*_ACCESS_KEY_ID/*_SECRET_ACCESS_KEY) → secrets.yaml (fail-fast). On charts pinned to 1.4.0 the mask helper is unavailable → keep only the fields (endpoint/region/bucket/disableSSL/usePathStyle) as escape-hatch passthrough with a note; the credentials STILL go to , the ConfigMap. Wire the mask when the pin bumps.

Everything NOT in this map is passthrough config (rate-limit, outbox, swagger, cors, pagination, probes, timeouts, retention, pool/client tuning, service-specific knobs) → {{ $cm.KEY | default "X" }}, NOT a grouped param. Classify by NATURE, not by which helper exists: a dependency connection with no helper yet (object storage) still gets a knob (add a mask); a non-dependency key never gets one even though cfgValue could route it. cfgValue is now used ONLY for the rare case where a dependency knob needs the configmap.<KEY> › knob › default precedence (e.g. a toggle the render-gate drives); prefer the plain passthrough otherwise.

Two global.* tiers (P1 #4): the domain blocks above (global.observability, global.multiTenant, global.datastores, global.serviceDiscovery, global.streaming, global.auth) are the env-wide CONFIG tier. Additionally ensure the k8s-INFRA tier exists for the umbrella: global.imageRegistry, global.imagePullSecrets, global.storageClass, commonLabels — shared once at the umbrella and read by every component subchart.

Classification (config vs secret)

Per Lerian values.md: a var is a secret if it carries credential material (*PASSWORD, *_SECRET, *API_KEY value, *CLIENT_SECRET, *CRYPTO*, LICENSE_KEY, *ACCESS_KEY/*SECRET_KEY, ORGANIZATION_IDS, *_TOKEN value). Secrets go to secrets.yaml, emitted only when provided, with a fail-fast when the owning feature is enabled. BEWARE false positives: a credential-named var that is actually a PATH or an empty-default config (WEBHOOK_API_KEY_FILE, AWS_ACCESS_KEY_ID="") stays config — see Rule 1 below.

Procedure

1. Fetch + split the .env.example into ACTIVE (^KEY=) and COMMENTED (^# KEY=) vars. Both are part of the contract (commented = optional).
2. Source of truth for defaults = the PRIOR MANAGER render, not values.yaml. Render the pre-change chart (helm template ... -s templates/configmap.yaml) and read KEY: value. (Reading values.yaml directly mixes in the worker's configmap — Rule 4.)
3. Classify every var via the domain map + secret rules above.
4. Emit the domain helpers (MT/OTEL/datastore/SD/streaming/auth) exactly as their Usage docs show, gated + global.*, adding the matching global.<block> + <comp>.<group> values.
5. Emit each key by its tier (declare {{- $cm := .Values.<comp>.configmap | default dict -}} at the top):
   • Passthrough (the default — every non-dependency key): {{ $cm.KEY | default "<.env value>" | quote }}. No group var, no <comp>.<group> block. Keep any {{- if ... }} / range conditional exactly. This is 80–90% of the keys.
   • Dependency knob: wire the domain helper/mask (Step 4) for db/cache/broker/streaming/SD/ MT/auth, and — on lerian-common ≥1.5.0 — objectStorage.value / kms.value for object storage / KMS. On a chart still pinned to 1.4.0 those two helpers do not exist: leave the non-secret OBJECT_STORAGE_* / KMS_* fields as escape-hatch passthrough with a # TODO 1.5.0 note and wire the mask when the pin bumps. The credentials (*_ACCESS_KEY_ID / *_SECRET_ACCESS_KEY / KMS_VAULT_SECRET_ID) ALWAYS go to secrets.yaml with a fail-fast — never the escape-hatch ConfigMap, on any lerian-common version (a ConfigMap is cleartext; routing a credential there is a CWE-312 exposure). Use cfgValue ONLY when a dependency knob genuinely needs the configmap.<KEY> › knob › default precedence (rare — e.g. a toggle the fixture drives).
   • Carry any value that DIFFERED from the template default into the passthrough | default "X" (Rule 3) — e.g. br-ccs shipped real bucket names, so , not .

RULES (the gotchas — every one cost real debugging; bake them in)

1. Credential-like NAMED config keys trip the secret-in-configmap gate, which only recognises | default "X" (not cfgValue's "default" "X"). For such a key, append a literal | default "<val>" after the cfgValue (… ) | default "" | quote }}) so the gate sees a non-credential default and skips it — or keep it a plain $cm.KEY | default "" | quote passthrough for *_FILE paths.
2. Comments between rendered lines must be YAML #, never {{- /* */ -}} — the {{-/-}} trim markers eat the surrounding newlines and glue two keys onto one line.
3. Non-default shipped values: any key whose values.yaml value differed from the template's default (e.g. br-ccs OBJECT_STORAGE_*_BUCKET) MUST be carried into the grouped param default, or emptying configmap breaks byte-identity. The .env.example is the authority for the default's VALUE — mirror it verbatim even when a value looks wrong (br-sta ships REDIS_MIN_RETRY_BACKOFF=8 > REDIS_MAX_RETRY_BACKOFF=1; the chart must match, not "fix" it — an inverted default is an APP bug to report upstream, never silently corrected in the chart, which would drift the chart from the app's own default).
4. Source of truth = MANAGER render, not awk over values.yaml — a naive ^ configmap: match also captures worker.configmap, injecting worker-only keys into the manager.
5. rateLimit.env / auth.env emit a FIXED key set that may not match the chart (rateLimit.env adds ALLOW_RATELIMIT_DISABLED + RATE_LIMIT_REDIS_TIMEOUT_MS; auth.env always emits HOST). If the chart's surface differs, use per-key cfgValue/globalValue instead.
6. If you generate lines with a Python f-string, }} becomes } — use a normal string or a post-regex to restore | quote }}.
7. Optional/opt-in keys ({{- if $cm.X }}, if REPORTER_ENABLED, range over lists) stay conditional — keep the guard when demoting to a passthrough ().

Self-check (the skill refuses to finish unless all pass)

Run scripts/coverage.py --chart-dir <chart> --env <app>.env --fixture <enable-all> — it renders the WHOLE chart (one helm template, so worker / signer / mqbridge / sub-component ConfigMaps are all included) and reconciles emitted keys vs the .env:

• Coverage: every ACTIVE .env var is emitted (as a dependency knob or a passthrough), an emit-when-set secret, or belongs to a WIRED domain — else it is a gap. A passthrough key still renders its default, so it counts as covered; it must also appear in the propertyNames.enum allowlist.
• Helper actually wired (not just prefix-matched): a domain (SD/streaming/MT/OTEL/auth) counts as covered only when its ANCHOR key (SD_ENABLED, STREAMING_ENABLED, …) renders. An SD_* var in the .env with the helper UNWIRED is a hard FAIL ("domain helper not wired") — this is what catches a whole domain the chart forgot (byte-identical to the prior chart would hide it).
• No raw drift: an emitted key not in the .env is flagged chart-only (dead config or .env gap).
• Byte-identical (regression guard, NOT completeness): default render == prior productized render, except intentionally-gated keys (MT/streaming infra now only render when enabled).
• Configured-path check (byte-identity of the DEFAULT render is necessary but NOT sufficient). A mask can feed the ConfigMap yet be IGNORED by the migration Job, the bootstrap Job, the init SQL, or a container/Service port — a silent break that a default-only diff never sees. Render a CONFIGURED scenario: set each dependency mask/knob to a DISTINCT value (custom host, custom db name, custom port) and assert it flows through EVERY consumer of that value, not just the ConfigMap. Bake that scenario into the render-gate fixture so CI keeps covering it. (Also: guard against a default argument that fails eagerly — | default (include "…that can fail…") runs even when the primary value is present; resolve first, then fail on the empty result.)
• Schema: helm lint/template validates against the shipped values.schema.json — a typo'd key under a closed block is rejected (the user-facing half of the coverage check). Verify with a NEGATIVE test on BOTH closed surfaces: a configmap.<TYPO> (allowlist) AND a dependency-mask typo (datastores.postgres.buket, objectStorage.<name>.reigon, kms.vaultMont) must each fail helm template.
• Docs: the README parameter table regenerates from the # -- annotations with no diff.
• helm lint + the repo render-gate + strict standard all green.

Emit a coverage report at the end: N config / N secret / N helper / N datastore / gaps list. For any judgment call the heuristics were unsure about, print REVIEW: <key> classified as <kind> so the human confirms in the PR — the skill proposes, the reviewer decides.

Verification behavior of the checks

• coverage.py PROVES emission with a configured render — name/allowlist matches are never enough. For every active .env key the DEFAULT render does not emit, coverage.py runs a second render that SETS the key (<comp>.configmap.<KEY> for config, <comp>.secrets.<KEY> for secrets, paths auto-detected from values.yaml) and checks the sentinel value actually surfaces in the rendered ConfigMap/Secret. Outcomes: a not-emitted key absent from the schema allowlist → FAIL (operator cannot set it); an allowlisted key whose probe emits NOTHING → FAIL (a dead allowlist entry — configmap.<KEY> has no template line, so propertyNames.enum accepting the name is misleading); a secret-classified key that produces no Secret entry when set → FAIL (silently-dropped credential); a helper-not-wired domain gap → FAIL. A key that DOES surface under the probe is genuinely settable → covered. If the probe cannot run (no PyYAML / no override path / render error) it degrades to a REVIEW line, never a silent pass.
• Consequence to expect: this can surface previously-hidden gaps on already-productized charts — an allowlist entry the template never renders, or a .env key not applicable to a given binary that should be dropped from the allowlist rather than left as a phantom knob. Triage in the PR.
• gen-schema.py closes the dependency-mask blocks (datastores.<type>, objectStorage.<name>, kms) with the contract's finite field sets (MASK_FIELDS), under BOTH <comp>.* and global.*. A mask-field typo (datastores.postgres.buket, kms.vaultMont) is now REJECTED at helm install instead of silently falling to the default. Because the masks ship EMPTY, the field set is sourced from references/dependency-contract.md (the lerian-common helper contract), not the chart values — keep MASK_FIELDS in sync with the contract whenever the library adds a mask field.

Output

Modified templates/configmap.yaml (dependency keys via helpers/masks; everything else a $cm.KEY | default passthrough), templates/secrets.yaml, values.yaml (only the tiered surface — no <comp>.<group> blocks for non-dependency config), values.schema.json (closed dependency blocks + propertyNames.enum allowlist on configmap), values-quickstart.yaml (the layperson layer), a render fixture (.github/configs/helm-render-values/<chart>.yaml enabling the dependency paths), and the coverage report. One chart per PR.

Market alignment (why we diverge, and where we adopt)

Studied against consolidated OSS charts (Bitnami common/postgresql/redis, Grafana/Loki/ kube-prometheus-stack, cert-manager, ingress-nginx) + the Helm Chart Best Practices.

Where we land (a deliberate MIDDLE): the market splits into "type every knob" (Bitnami) and "pass a structured config blob" (Grafana grafana.ini, ingress-nginx controller.config). We reject both extremes. Full typing (our earlier model) masks the NAME but not the COUNT — 120 grouped params overwhelm a non-expert exactly as 120 env vars do, and each is a hand-maintained parallel contract that silently drifts. The opaque blob throws away typing where it matters. Our finite, app-owned .env lets us do better: type only the dependency connections (the must-set surface a non-expert wires) and leave the long tail as a passthrough with template defaults — Bitnami-style typing where it earns its keep, blob-style passthrough where it doesn't. We still ENUMERATE the whole .env, but for the SCHEMA ALLOWLIST (propertyNames.enum on the escape hatch → typo protection) and the coverage check, NOT to mint a knob per key. coverage.py is what keeps the allowlist in sync with the app.

Where we were WRONG / adopt from the market:

• Every mature chart — even 100%-typed Bitnami and 100%-blob Grafana — keeps an env escape hatch. Never remove extraEnvVars / the configmap.<KEY> override. configmap: {} = "no defaults", not "locked". (Premise section.)
• Ship a strict values.schema.json — additionalProperties:false on the typed dependency blocks
  • propertyNames.enum on the configmap escape hatch (typo protection without per-key typing). The user-facing guardrail. (Step 8.)
• Annotation-driven docs (# -- / @param) + auto-generated README — never hand-write tables. (Step 6.)
• Secrets: lookup+manage (reuse-on-upgrade), not only fail-fast; + a leaked-secret guard. (Step 7.)
• A k8s-infra global.* tier for the umbrella (registry/pullSecrets/storageClass/commonLabels). (Domain map.)

Exception: if an app ever ships a genuinely NESTED config file (not flat env), model it as a structured passthrough map rendered to a ConfigMap (grafana.ini pattern) — do NOT flatten every leaf into a grouped param. Full enumeration is for the flat 12-factor .env surface only.