design-patterns-horizontal-versioning

# Horizontal Versioning ## Overview Dodder's persistent data structures evolve through horizontal versioning: new versions are added alongside old ones, never replacing them. Every version is a separate concrete struct that implements a shared interface. Old versions remain as valid deserialization targets so that legacy data can always be read and then upgraded forward. ## Core Principles 1. **One interface, many versions.** All versions of a data structure implement the same interface. Consumers depend only on the interface. 2. **Concrete implementations are separate.** Each version is its own struct in its own file (e.g., `toml_v0.go`, `toml_v1.go`). Versions must not embed or compose each other. They may share imported utility types (e.g., `compression_type.CompressionType`) but each struct stands alone. 3. **No redundant encoding of config data.** A concrete implementation encodes exactly the fields for its version. It does not carry compatibility shims, optional fields for other versions, or union-style "one of these is set" patterns. Each version can assume that its data is exclusively its own. 4. **Upgrade paths transform data between versions.** When data needs to move from one version to another, the `Upgrade()` method on the source version constructs the target version's struct, mapping and transforming fields as needed. This is the only mechanism for cross-version data flow. 5. **Serialization reflects the concrete version.** After an upgrade, the data is serialized using the new version's type string and coder. There is no concept of a "compatible" or "polymorphic" on-disk format. ## Architecture ### Package Naming Convention | Suffix | Purpose | Examples | |--------|---------|---------| | `*_blobs` | Data schemas (metadata, content structure) | `type_blobs`, `tag_blobs`, `repo_blobs`, `workspace_config_blobs` | | `*_configs` | Backend/infrastructure configuration | `blob_store_configs`, `repo_configs`, `genesis_configs` | | `*_stores` | Runtime implementations consuming configs | `blob_stores` | ### Four Components of a Versioned Type #### 1. Stable Interface Defined once per data structure. All versions implement it. ```go // go/internal/lima/type_blobs/main.go type Blob interface { GetFileExtension() string GetBinary() bool GetMimeType() string GetVimSyntaxType() string WithFormatters WithFormatterUTIGroups WithStringLuaHooks } ``` #### 2. Versioned Concrete Structs Each version is a separate struct with its own fields. No embedding between versions. Shared utility types are fine; shared struct definitions are not. ```go // go/internal/golf/blob_store_configs/toml_v0.go type TomlV0 struct { BasePath string `toml:"base-path,omitempty"` AgeEncryption markl_age_id.Id `toml:"age-encryption,omitempty"` CompressionType compression_type.CompressionType `toml:"compression-type"` LockInternalFiles bool `toml:"lock-internal-files"` } // go/internal/golf/blob_store_configs/toml_v2.go type TomlV2 struct { HashBuckets values.IntSlice `toml:"hash_buckets"` BasePath string `toml:"base_path,omitempty"` HashTypeId string `toml:"hash_type-id"` Encryption markl.Id `toml:"encryption"` CompressionType compression_type.CompressionType `toml:"compression-type"` LockInternalFiles bool `toml:"lock-internal-files"` } ``` Note: both use `compression_type.CompressionType` (shared utility type), but neither embeds the other. V2 has `HashBuckets` and `HashTypeId` that V0 lacks. V0 has `AgeEncryption` which V2 replaced with the more general `Encryption`. #### 3. Type String Registration Each version gets a unique type string registered in `echo/ids/types_builtin.go`. A `VCurrent` alias tracks the latest version. ```go // go/internal/echo/ids/types_builtin.go const ( TypeTomlBlobStoreConfigV0 = "!toml-blob_store_config-v0" TypeTomlBlobStoreConfigV1 = "!toml-blob_store_config-v1" TypeTomlBlobStoreConfigV2 = "!toml-blob_store_config-v2" TypeTomlBlobStoreConfigVCurrent = TypeTomlBlobStoreConfigV2 ) func init() { registerBuiltinTypeString(TypeTomlBlobStoreConfigV0, genres.Unknown, false) registerBuiltinTypeString(TypeTomlBlobStoreConfigV1, genres.Unknown, false) registerBuiltinTypeString(TypeTomlBlobStoreConfigV2, genres.Unknown, false) } ``` #### 4. Coder Registration A coder map associates each type string with a deserializer that produces the correct concrete struct behind the shared interface. ```go // go/internal/golf/blob_store_configs/coding.go var Coder = hyphence.CoderToTypedBlob[Config]{ Blob: hyphence.CoderTypeMapWithoutType[Config]( map[string]interfaces.CoderBufferedReadWriter[*Config]{ ids.TypeTomlBlobStoreConfigV0: hyphence.CoderToml[Config, *Config]{ Progenitor: func() Config { return &TomlV0{} }, }, ids.TypeTomlBlobStoreConfigV1: hyphence.CoderToml[Config, *Config]{ Progenitor: func() Config { return &TomlV1{} }, }, ids.TypeTomlBlobStoreConfigV2: hyphence.CoderToml[Config, *Config]{ Progenitor: func() Config { return &TomlV2{} }, }, }, ), } ``` ### Type-Dispatched Serialization The `hyphence` package provides the generic serialization layer. Files on disk use `---` delimiters to separate a type header from the body. On read, the type string from the header selects the correct coder: ```go // go/internal/foxtrot/hyphence/coder_type_map.go type TypedBlob[BLOB any] struct { Type ids.TypeStruct Blob BLOB } type CoderTypeMapWithoutType[BLOB any] map[string]interfaces.CoderBufferedReadWriter[*BLOB] func (coderTypeMap CoderTypeMapWithoutType[BLOB]) DecodeFrom( typedBlob *TypedBlob[BLOB], bufferedReader *bufio.Reader, ) (n int64, err error) { tipe := typedBlob.Type coder, ok := coderTypeMap[tipe.String()] if !ok { err = errors.ErrorWithStackf("no coders available for type: %q", tipe) return } n, err = coder.DecodeFrom(&typedBlob.Blob, bufferedReader) return } ``` ### Upgrade Paths Non-current versions implement `ConfigUpgradeable` to provide forward migration. The `Upgrade()` method constructs the next version's struct directly, mapping and transforming fields: ```go // go/internal/golf/blob_store_configs/main.go type ConfigUpgradeable interface { Config Upgrade() (Config, ids.TypeStruct) } // go/internal/golf/blob_store_configs/toml_v1.go func (blobStoreConfig TomlV1) Upgrade() (Config, ids.TypeStruct) { upgraded := &TomlV2{ HashBuckets: blobStoreConfig.HashBuckets, BasePath: blobStoreConfig.BasePath, HashTypeId: markl.FormatIdHashSha256, CompressionType: blobStoreConfig.CompressionType, LockInternalFiles: blobStoreConfig.LockInternalFiles, } upgraded.Encryption.ResetWithMarklId(blobStoreConfig.Encryption) return upgraded, ids.GetOrPanic(ids.TypeTomlBlobStoreConfigV2).TypeStruct } ``` Callers run a loop that upgrades iteratively until reaching a version that does not implement `ConfigUpgradeable` (the current version): ```go for { configUpgraded, ok := typedConfig.Blob.(blob_store_configs.ConfigUpgradeable) if !ok { break } typedConfig.Blob, typedConfig.Type = configUpgraded.Upgrade() } ``` After this loop, the data is at the current version and will be serialized with the current version's type string. ## Adding a New Version 1. **Create the struct** in a new file (e.g., `toml_v3.go`). Define only the fields this version needs. Do not carry forward deprecated fields or compatibility shims. 2. **Implement the interface** that all versions of this type share. 3. **Register the type string** in `echo/ids/types_builtin.go`. Add a new constant and update `VCurrent` to point to the new version. 4. **Add an `Upgrade()` method** to the previous version that constructs the new version, mapping and transforming fields. The previous version now implements `ConfigUpgradeable`. 5. **Register the coder** in the package's `coding.go`, adding a new entry to the type-to-coder map with a `Progenitor` that returns a zero-value of the new struct. 6. **Regenerate test fixtures** with `just test-bats-update-fixtures` if the change affects stored data formats. Review and commit the updated fixtures. ## Common Mistakes | Mistake | Correct Approach | |---------|-----------------| | Embedding one version struct inside another | Each version is independent. Share imported utility types, not struct composition. | | Adding optional fields to an existing version for new features | Create a new version. Existing versions are frozen. | | Carrying forward deprecated fields "for compatibility" | New versions define only their own fields. The upgrade path handles transformation. | | Forgetting to update `VCurrent` | Always point `VCurrent` at the newest version after adding it. | | Modifying `Upgrade()` on the current version | Only non-current versions implement `ConfigUpgradeable`. The current version is the upgrade chain's terminus. | ## Migration Status The codebase has three Coder architectures at different stages of migration toward the canonical `CoderToTypedBlob` pattern. This section tracks each versioned type family, its current architecture, and what remains to reach full compliance with the pattern described above. ### Target Architecture All versioned types should use **Architecture A: `CoderToTypedBlob`** with `CoderTypeMapWithoutType` maps and `Progenitor` functions. All non-current versions should implement `Upgrade()` to form a complete chain from the oldest version to `VCurrent`. ### Architecture Summary | Architecture | Description | Target? | |---|---|---| | **A: `CoderToTypedBlob`** | Declarative type-string→progenitor map in `coding.go`. Uses `hyphence.CoderToml` with `Progenitor` functions. | Yes | | **B: `TypedStore` + switch** | Hand-written struct with one `domain_interfaces.TypedStore` field per version. Dispatch via `switch` on type string. | No — migrate to A | | **C: Custom constructor map** | Package-local `coderConstructors` map producing custom `coder` structs. `Closet` bridges to `CoderTypeMapWithoutType` at runtime. | No — migrate to A | ### Per-Type Family Status #### `blob_store_configs` (Architecture A) — `golf/blob_store_configs/coding.go` | Version | Type String | Struct | Coder | Upgrade | |---|---|---|---|---| | V0 | `!toml-blob_store_config-v0` | `TomlV0` | registered | **missing** — no `Upgrade()` to V1 | | V1 | `!toml-blob_store_config-v1` | `TomlV1` | registered | V1→V2 implemented | | V2 (current) | `!toml-blob_store_config-v2` | `TomlV2` | registered | terminus | | SFTP Explicit V0 | `!toml-blob_store_config_sftp-explicit-v0` | `TomlSFTPV0` | registered | N/A (single version) | | SFTP SSH Config V0 | `!toml-blob_store_config_sftp-ssh_config-v0` | `TomlSFTPViaSSHConfigV0` | registered | N/A (single version) | | Pointer V0 | `!toml-blob_store_config-pointer-v0` | **no struct** | **no coder** | orphan type string | | Inventory Archive V0 | `!toml-blob_store_config-inventory_archive-v0` | **no struct** | **no coder** | orphan type string | Remaining work: - [ ] Add `TomlV0.Upgrade()` → V1 - [ ] Remove or implement `TypeTomlBlobStoreConfigPointerV0` (orphan registration) - [ ] Remove or implement `TypeTomlBlobStoreConfigInventoryArchiveV0` (orphan registration) - [ ] Adopt `registerToml` helper (defined in `coding.go` but unused — dead code) #### `repo_configs` (Architecture A) — `golf/repo_configs/coding.go` | Version | Type String | Struct | Coder | Upgrade | |---|---|---|---|---| | V0 (deprecated) | `!toml-config-v0` | `V0` | registered | **missing** — no `Upgrade()` to V1 | | V1 | `!toml-config-v1` | `V1` | registered | **missing** — no `Upgrade()` to V2 | | V2 (current) | `!toml-config-v2` | `V2` | registered | terminus | Remaining work: - [ ] Add `V0.Upgrade()` → V1 - [ ] Add `V1.Upgrade()` → V2 - [ ] Define `ConfigUpgradeable` interface (or reuse from `blob_store_configs`) #### `genesis_configs` (Architecture A) — `hotel/genesis_configs/coder.go` | Version | Type String | Struct (Private) | Struct (Public) | Coder | Upgrade | |---|---|---|---|---|---| | V1 | `!toml-config-immutable-v1` | `TomlV1Private` | `TomlV1Public` | registered (both coders) | **missing** — no `Upgrade()` to V2 | | V2 (current) | `!toml-config-immutable-v2` | `TomlV2Private` | `TomlV2Public` | registered (both coders) | terminus | Remaining work: - [ ] Add `TomlV1Private.Upgrade()` → V2 - [ ] Add `TomlV1Public.Upgrade()` → V2 #### `repo_blobs` (Architecture A) — `golf/repo_blobs/coding.go` | Version | Type String | Struct | Coder | Upgrade | |---|---|---|---|---| | Local Override V0 | `!toml-repo-local_override_path-v0` | `TomlLocalOverridePathV0` | registered | N/A | | XDG V0 | `!toml-repo-dotenv_xdg-v0` | `TomlXDGV0` | registered | N/A | | URI V0 (default) | `!toml-repo-uri-v0` | `TomlUriV0` | registered | N/A | No version progression — all at V0. No work needed unless new versions are added. #### `workspace_config_blobs` (Architecture A) — `hotel/workspace_config_blobs/io.go` | Version | Type String | Struct | Coder | Upgrade | |---|---|---|---|---| | V0 (current) | `!toml-workspace_config-v0` | `V0` | registered | terminus | Single version. No work needed unless a new version is added. #### `type_blobs` (Architecture B) — `lima/type_blobs/coder.go` | Version | Type String | Struct | Coder | Upgrade |