design-patterns-horizontal-versioning

name	design_patterns-horizontal_versioning
description	Use when adding a new version of a persistent data structure, creating a new blob or config type, implementing an upgrade path between versions, or registering a coder for a versioned type. Also applies when working with type_blobs, blob_store_configs, blob_stores, hyphence, or any package following the *_blobs / *_configs / *_stores naming convention.
triggers	["add a new version","add a blob type","add a config type","upgrade path","version migration","register a coder","type_blobs","blob_store_configs","hyphence","horizontal versioning"]

Horizontal Versioning

Overview

Dodder's persistent data structures evolve through horizontal versioning: new versions are added alongside old ones, never replacing them. Every version is a separate concrete struct that implements a shared interface. Old versions remain as valid deserialization targets so that legacy data can always be read and then upgraded forward.

Core Principles

1. One interface, many versions. All versions of a data structure implement the same interface. Consumers depend only on the interface.

2. Concrete implementations are separate. Each version is its own struct in its own file (e.g., toml_v0.go, toml_v1.go). Versions must not embed or compose each other. They may share imported utility types (e.g., compression_type.CompressionType) but each struct stands alone.

3. No redundant encoding of config data. A concrete implementation encodes exactly the fields for its version. It does not carry compatibility shims, optional fields for other versions, or union-style "one of these is set" patterns. Each version can assume that its data is exclusively its own.

4. Upgrade paths transform data between versions. When data needs to move from one version to another, the Upgrade() method on the source version constructs the target version's struct, mapping and transforming fields as needed. This is the only mechanism for cross-version data flow.

5. Serialization reflects the concrete version. After an upgrade, the data is serialized using the new version's type string and coder. There is no concept of a "compatible" or "polymorphic" on-disk format.

Architecture

Package Naming Convention

Suffix	Purpose	Examples
*_blobs	Data schemas (metadata, content structure)	type_blobs, tag_blobs, repo_blobs, workspace_config_blobs
*_configs	Backend/infrastructure configuration	blob_store_configs, repo_configs, genesis_configs
*_stores	Runtime implementations consuming configs	blob_stores

Four Components of a Versioned Type

1. Stable Interface

Defined once per data structure. All versions implement it.

// go/internal/lima/type_blobs/main.go
type Blob interface {
    GetFileExtension() string
    GetBinary() bool
    GetMimeType() string
    GetVimSyntaxType() string
    WithFormatters
    WithFormatterUTIGroups
    WithStringLuaHooks
}

2. Versioned Concrete Structs

Each version is a separate struct with its own fields. No embedding between versions. Shared utility types are fine; shared struct definitions are not.

// go/internal/golf/blob_store_configs/toml_v0.go
type TomlV0 struct {
    BasePath          string                           `toml:"base-path,omitempty"`
    AgeEncryption     markl_age_id.Id                  `toml:"age-encryption,omitempty"`
    CompressionType   compression_type.CompressionType `toml:"compression-type"`
    LockInternalFiles bool                             `toml:"lock-internal-files"`
}

// go/internal/golf/blob_store_configs/toml_v2.go
type TomlV2 struct {
    HashBuckets       values.IntSlice                  `toml:"hash_buckets"`
    BasePath          string                           `toml:"base_path,omitempty"`
    HashTypeId        string                           `toml:"hash_type-id"`
    Encryption        markl.Id                         `toml:"encryption"`
    CompressionType   compression_type.CompressionType `toml:"compression-type"`
    LockInternalFiles bool                             `toml:"lock-internal-files"`
}

Note: both use compression_type.CompressionType (shared utility type), but neither embeds the other. V2 has HashBuckets and HashTypeId that V0 lacks. V0 has AgeEncryption which V2 replaced with the more general Encryption.

3. Type String Registration

Each version gets a unique type string registered in echo/ids/types_builtin.go. A VCurrent alias tracks the latest version.

// go/internal/echo/ids/types_builtin.go
const (
    TypeTomlBlobStoreConfigV0       = "!toml-blob_store_config-v0"
    TypeTomlBlobStoreConfigV1       = "!toml-blob_store_config-v1"
    TypeTomlBlobStoreConfigV2       = "!toml-blob_store_config-v2"
    TypeTomlBlobStoreConfigVCurrent = TypeTomlBlobStoreConfigV2
)

func init() {
    registerBuiltinTypeString(TypeTomlBlobStoreConfigV0, genres.Unknown, false)
    registerBuiltinTypeString(TypeTomlBlobStoreConfigV1, genres.Unknown, false)
    registerBuiltinTypeString(TypeTomlBlobStoreConfigV2, genres.Unknown, false)
}

4. Coder Registration

A coder map associates each type string with a deserializer that produces the correct concrete struct behind the shared interface.

// go/internal/golf/blob_store_configs/coding.go
var Coder = hyphence.CoderToTypedBlob[Config]{
    Blob: hyphence.CoderTypeMapWithoutType[Config](
        map[string]interfaces.CoderBufferedReadWriter[*Config]{
            ids.TypeTomlBlobStoreConfigV0: hyphence.CoderToml[Config, *Config]{
                Progenitor: func() Config { return &TomlV0{} },
            },
            ids.TypeTomlBlobStoreConfigV1: hyphence.CoderToml[Config, *Config]{
                Progenitor: func() Config { return &TomlV1{} },
            },
            ids.TypeTomlBlobStoreConfigV2: hyphence.CoderToml[Config, *Config]{
                Progenitor: func() Config { return &TomlV2{} },
            },
        },
    ),
}

Type-Dispatched Serialization

The hyphence package provides the generic serialization layer. Files on disk use --- delimiters to separate a type header from the body. On read, the type string from the header selects the correct coder:

// go/internal/foxtrot/hyphence/coder_type_map.go
type TypedBlob[BLOB any] struct {
    Type ids.TypeStruct
    Blob BLOB
}

type CoderTypeMapWithoutType[BLOB any] map[string]interfaces.CoderBufferedReadWriter[*BLOB]

func (coderTypeMap CoderTypeMapWithoutType[BLOB]) DecodeFrom(
    typedBlob *TypedBlob[BLOB],
    bufferedReader *bufio.Reader,
) (n int64, err error) {
    tipe := typedBlob.Type
    coder, ok := coderTypeMap[tipe.String()]
    if !ok {
        err = errors.ErrorWithStackf("no coders available for type: %q", tipe)
        return
    }
    n, err = coder.DecodeFrom(&typedBlob.Blob, bufferedReader)
    return
}

Upgrade Paths

Non-current versions implement ConfigUpgradeable to provide forward migration. The Upgrade() method constructs the next version's struct directly, mapping and transforming fields:

// go/internal/golf/blob_store_configs/main.go
type ConfigUpgradeable interface {
    Config
    Upgrade() (Config, ids.TypeStruct)
}

// go/internal/golf/blob_store_configs/toml_v1.go
func (blobStoreConfig TomlV1) Upgrade() (Config, ids.TypeStruct) {
    upgraded := &TomlV2{
        HashBuckets:       blobStoreConfig.HashBuckets,
        BasePath:          blobStoreConfig.BasePath,
        HashTypeId:        markl.FormatIdHashSha256,
        CompressionType:   blobStoreConfig.CompressionType,
        LockInternalFiles: blobStoreConfig.LockInternalFiles,
    }
    upgraded.Encryption.ResetWithMarklId(blobStoreConfig.Encryption)
    return upgraded, ids.GetOrPanic(ids.TypeTomlBlobStoreConfigV2).TypeStruct
}

Callers run a loop that upgrades iteratively until reaching a version that does not implement ConfigUpgradeable (the current version):

for {
    configUpgraded, ok := typedConfig.Blob.(blob_store_configs.ConfigUpgradeable)
    if !ok {
        break
    }
    typedConfig.Blob, typedConfig.Type = configUpgraded.Upgrade()
}

After this loop, the data is at the current version and will be serialized with the current version's type string.

Adding a New Version

1. Create the struct in a new file (e.g., toml_v3.go). Define only the fields this version needs. Do not carry forward deprecated fields or compatibility shims.

2. Implement the interface that all versions of this type share.

3. Register the type string in echo/ids/types_builtin.go. Add a new constant and update VCurrent to point to the new version.

4. Add an Upgrade() method to the previous version that constructs the new version, mapping and transforming fields. The previous version now implements ConfigUpgradeable.

5. Register the coder in the package's coding.go, adding a new entry to the type-to-coder map with a Progenitor that returns a zero-value of the new struct.

6. Regenerate test fixtures with just test-bats-update-fixtures if the change affects stored data formats. Review and commit the updated fixtures.

Common Mistakes

Mistake	Correct Approach
Embedding one version struct inside another	Each version is independent. Share imported utility types, not struct composition.
Adding optional fields to an existing version for new features	Create a new version. Existing versions are frozen.
Carrying forward deprecated fields "for compatibility"	New versions define only their own fields. The upgrade path handles transformation.
Forgetting to update VCurrent	Always point VCurrent at the newest version after adding it.
Modifying Upgrade() on the current version	Only non-current versions implement ConfigUpgradeable. The current version is the upgrade chain's terminus.

Migration Status

The codebase has three Coder architectures at different stages of migration toward the canonical CoderToTypedBlob pattern. This section tracks each versioned type family, its current architecture, and what remains to reach full compliance with the pattern described above.

Target Architecture

All versioned types should use Architecture A: CoderToTypedBlob with CoderTypeMapWithoutType maps and Progenitor functions. All non-current versions should implement Upgrade() to form a complete chain from the oldest version to VCurrent.

Architecture Summary

Architecture	Description	Target?
A: CoderToTypedBlob	Declarative type-string→progenitor map in coding.go. Uses hyphence.CoderToml with Progenitor functions.	Yes
B: TypedStore + switch	Hand-written struct with one domain_interfaces.TypedStore field per version. Dispatch via switch on type string.	No — migrate to A
C: Custom constructor map	Package-local coderConstructors map producing custom coder structs. Closet bridges to CoderTypeMapWithoutType at runtime.	No — migrate to A

Per-Type Family Status

blob_store_configs (Architecture A) — golf/blob_store_configs/coding.go

Version	Type String	Struct	Coder	Upgrade
V0	!toml-blob_store_config-v0	TomlV0	registered	missing — no Upgrade() to V1
V1	!toml-blob_store_config-v1	TomlV1	registered	V1→V2 implemented
V2 (current)	!toml-blob_store_config-v2	TomlV2	registered	terminus
SFTP Explicit V0	!toml-blob_store_config_sftp-explicit-v0	TomlSFTPV0	registered	N/A (single version)
SFTP SSH Config V0	!toml-blob_store_config_sftp-ssh_config-v0	TomlSFTPViaSSHConfigV0	registered	N/A (single version)
Pointer V0	!toml-blob_store_config-pointer-v0	no struct	no coder	orphan type string
Inventory Archive V0	!toml-blob_store_config-inventory_archive-v0	no struct	no coder	orphan type string

Remaining work:

• Add TomlV0.Upgrade() → V1
• Remove or implement TypeTomlBlobStoreConfigPointerV0 (orphan registration)
• Remove or implement TypeTomlBlobStoreConfigInventoryArchiveV0 (orphan registration)
• Adopt registerToml helper (defined in coding.go but unused — dead code)

repo_configs (Architecture A) — golf/repo_configs/coding.go

Version	Type String	Struct	Coder	Upgrade
V0 (deprecated)	!toml-config-v0	V0	registered	missing — no Upgrade() to V1
V1	!toml-config-v1	V1	registered	missing — no Upgrade() to V2
V2 (current)	!toml-config-v2	V2	registered	terminus

Remaining work:

• Add V0.Upgrade() → V1
• Add V1.Upgrade() → V2
• Define ConfigUpgradeable interface (or reuse from blob_store_configs)

genesis_configs (Architecture A) — hotel/genesis_configs/coder.go

Version	Type String	Struct (Private)	Struct (Public)	Coder	Upgrade
V1	!toml-config-immutable-v1	TomlV1Private	TomlV1Public	registered (both coders)	missing — no Upgrade() to V2
V2 (current)	!toml-config-immutable-v2	TomlV2Private	TomlV2Public	registered (both coders)	terminus

Remaining work:

• Add TomlV1Private.Upgrade() → V2
• Add TomlV1Public.Upgrade() → V2

repo_blobs (Architecture A) — golf/repo_blobs/coding.go

Version	Type String	Struct	Coder	Upgrade
Local Override V0	!toml-repo-local_override_path-v0	TomlLocalOverridePathV0	registered	N/A
XDG V0	!toml-repo-dotenv_xdg-v0	TomlXDGV0	registered	N/A
URI V0 (default)	!toml-repo-uri-v0	TomlUriV0	registered	N/A

No version progression — all at V0. No work needed unless new versions are added.

workspace_config_blobs (Architecture A) — hotel/workspace_config_blobs/io.go

Version	Type String	Struct	Coder	Upgrade
V0 (current)	!toml-workspace_config-v0	V0	registered	terminus

Single version. No work needed unless a new version is added.

type_blobs (Architecture B) — lima/type_blobs/coder.go

Version	Type String	Struct	Coder	Upgrade
V0 (deprecated)	!toml-type-v0	TomlV0	CoderToTypedBlob map	missing — no Upgrade() to V1
V1 (current)	!toml-type-v1	TomlV1	CoderToTypedBlob map	terminus

Remaining work:

• Migrate to CoderToTypedBlob pattern (replace struct + switch with coding.go map)
• Add TomlV0.Upgrade() → V1

tag_blobs (Architecture B) — mike/typed_blob_store/tag.go

Version	Type String	Struct	Coder	Upgrade
V0	!toml-tag-v0	V0	TypedStore field + switch	missing
Toml V1	!toml-tag-v1	TomlV1	TypedStore field + switch	missing
Lua V1	!lua-tag-v1	LuaV1	TypedStore field + switch	missing
Lua V2 (current)	!lua-tag-v2	LuaV2	TypedStore field + switch	terminus

Remaining work:

• Migrate to CoderToTypedBlob pattern
• Decide upgrade chain across format boundaries (Toml→Lua or separate chains)
• Rename V0 to TomlV0 for naming consistency with other versions

inventory_list_coders (Architecture C) — lima/inventory_list_coders/main.go

Version	Type String	Coder	Upgrade
V0 (deprecated)	!inventory_list-v0	missing — registered in types_builtin.go but no entry in coderConstructors	N/A
V1	!inventory_list-v1	custom coder via coderConstructors	missing — no Upgrade() to V2
V2 (current)	!inventory_list-v2	custom coder via coderConstructors	terminus
JSON V0	!inventory_list-json-v0	custom coder via coderConstructors	N/A (alternate format)

Remaining work:

• Remove or implement coder for TypeInventoryListV0 (orphan type string)
• Migrate to CoderToTypedBlob pattern (partially bridged via Closet)
• Add V1 upgrade to V2

Orphan Type Strings

Type strings registered in echo/ids/types_builtin.go with no struct or coder:

Type String	Constant	Status
!toml-blob_store_config-pointer-v0	TypeTomlBlobStoreConfigPointerV0	No struct, no coder
!toml-blob_store_config-inventory_archive-v0	TypeTomlBlobStoreConfigInventoryArchiveV0	No struct, no coder
!inventory_list-v0	TypeInventoryListV0	Deprecated, no coder
!zettel_id_list-v0	TypeZettelIdListV0	Marked "not used yet", no coder