# @adobe/data
Adobe Data Oriented Programming Library
## Documentation
[API Reference](https://adobe.github.io/data/)
[ECS Performance Test](https://adobe.github.io/data/perftest.html)
[P2P Tic-Tac-Toe Demo](https://adobe.github.io/data/p2p-tictactoe/) — serverless browser-to-browser multiplayer via WebRTC + @adobe/data-sync
## Breaking API Changes
Until we reach 1.0.0, minor version changes may be API breaking.
### 0.9.77
**`ephemeral` fully removed — use `nonPersistent`**
The `ephemeral` name (deprecated in 0.9.70) is gone entirely: `Schema.ephemeral`, the `"ephemeral"` component alias, and `Entity.isEphemeral`. Use `nonPersistent` / `Entity.isNonPersistent`.
```ts
// before
{ ..., ephemeral: true }
ensureArchetype(["id", "cursor", "ephemeral"])
// after
{ ..., nonPersistent: true }
ensureArchetype(["id", "cursor", "nonPersistent"])
```
**Store snapshot format changed and is now versioned**
`db.toData()` / `store.toData()` output is now stamped with `version: ECS_SNAPSHOT_VERSION` (currently `1`), and `fromData` ignores any snapshot whose version does not match. Two consequences:
- nonPersistent resources/entities are correctly excluded from snapshots (previously a nonPersistent resource could deserialize to `undefined`).
- Snapshots produced by earlier versions (which carried no `version` field) are no longer loaded — `fromData` logs a `console.warn` and silently keeps the current (freshly-constructed) state instead of reconstructing incorrectly. Discard and regenerate persisted snapshots.
This only affects the `toData`/`fromData` snapshot path (e.g. `createStoragePersistenceService`). The `@adobe/data-persistence` incremental journal format is unaffected.
### 0.9.70
**`TransactionResult.ephemeral` → `TransactionResult.persistent` (inverted)**
The boolean is now `true` when at least one changed entity is persistent (id ≥ 0), rather than `true` when all changed entities are non-persistent.
```ts
// before
if (!result.ephemeral) save(result);
// after
if (result.persistent) save(result);
```
**`TransactionResult.transient` → `TransactionResult.intermediate`**
Renamed to remove the false antonym relationship with `persistent`. Both are orthogonal: `intermediate` = preview step (will be rolled back), `persistent` = touches saved entities.
```ts
// before
if (!result.transient && !result.ephemeral) save(result);
// after
if (!result.intermediate && result.persistent) save(result);
```
**`TransactionOptions.transient` → `TransactionOptions.intermediate`**
```ts
// before
db.execute(fn, { transient: true });
// after
db.execute(fn, { intermediate: true });
```
**`TransactionIntent "transient"` → `"intermediate"` (sync wire protocol)**
Affects `observe.envelopes` intent values and `ClientMessage` kind strings in `@adobe/data-sync`.
```ts
// before
db.observe.envelopes(({ intent }) => {
if (intent === "transient") { ... }
});
transport.send({ kind: "transient", envelope });
// after
db.observe.envelopes(({ intent }) => {
if (intent === "intermediate") { ... }
});
transport.send({ kind: "intermediate", envelope });
```
**`SyncServiceOptions.maxTransientsPerSecond` → `maxIntermediatesPerSecond`**
**`Schema.ephemeral` deprecated → use `Schema.nonPersistent`**
The `"ephemeral"` component string still works at runtime with a `console.warn`; update to `"nonPersistent"`.
```ts
// before
{ ..., ephemeral: true }
ensureArchetype(["id", "cursor", "ephemeral"])
// after
{ ..., nonPersistent: true }
ensureArchetype(["id", "cursor", "nonPersistent"])
```
## Data Oriented Programming
This library is built using a data oriented and functional programming paradigm.
We prefer composition over inheritance, avoid classes when possible and emphasize separation of concerns.
## Data
This library uses data oriented design paradigm and prefers pure functional interfaces whenever practical.
For our purposes, `Data` is immutable `JSON` (de)serializable objects and primitives.
### Why immutable Data?
We prefer Data because it is easy to:
- serialize
- deserialize
- inspect
- compare
- hash
- validate
We prefer immutable Data because it is easy to:
- reason about
- avoid side effects
- avoid defensive copying
- use for pure function arguments and return values
- use with concurrency
- memoize results
- use as cache key (stringified)
- use as cache value
### What is Data Oriented Design?
Sanders Mertens covers this thoroughly in his ECS FAQ:
[https://github.com/SanderMertens/ecs-faq?tab=readme-ov-file#what-is-data-oriented-design](https://github.com/SanderMertens/ecs-faq?tab=readme-ov-file#what-is-data-oriented-design)
### Data Schemas
When runtime type information is required we use `JSON Schema`. In order to avoid redundancy, we can use `FromSchema` to derive a typescript type at compile time.
Example:
```typescript
const Float32Schema = { type: "number", precision: 1 } as const satisfies Schema;
const Float32 = FromSchema<typeof Float32Schema>; // number
```
It is important that we use `as const` for the schema definition. Without that, the typeof Float32Schema would be `{ type: string, precision: number }` instead of `{ type: "number", precision: 1 }` and we would not be able to derive a correct type using `FromSchema`.
### Normalized Data
Data is considered "Normalized" when all object keys contained anywhere within it are lexigraphically sorted. We include a `normalize` function which performs this conversion. This is particularly useful when you want to use data as a cache key.
```typescript
const notNormalized = { b: 2, a: 1 };
const normalized = normalize(notNormalized); // { a: 1, b: 2 }
```
## Observables
An `Observable<T>` is a subscription function that you can pass a callback function to. Your callback function can accept a single argument of type `T`. The subscription function returns a dispose function that accepts no parameters and can be called at any point in the future to cancel your subscription.
Your callback function *may* be called back synchronously (before the initial call returns) zero or one times and asynchronously later any number of times.
For more information see the [Observable API documentation](https://adobe.github.io/data/api/modules/observe.html)
### Observable Types
```typescript
/**
* Called to add a new observer to an Observable.
* Returns an Unobserve function that can be called to stop observing the Observable.
*/
export type Observable<T> = (observer: Callback<T>) => Unobserve;
/**
* Callback function called zero or more times with a sequence of values.
* *may* be called back synchronously, immediately when selector function is called.
* *may* also be called back asynchronously later any number of times.
* *may* be called multiple sequential times with the same value.
*/
export type Callback<T> = (value: T) => void;
/**
* Function called to stop observing an Observable.
*/
export type Unobserve = () => void;
```
An Observable can be thought of sort of like a Promise but with a few important differences.
- A Promise only yields a single value, an Observable yields a sequence of values.
- A Promise can reject with an error, an Observable can not. (It could yield type `MyResult | MyError` though.)
- A Promise can only resolve asynchronously, an Observable *may* yield an initial result synchronously.
- An Observable is allowed to call the Callback callback function immediately upon observation if it has a value.
- A Promise begins executing immediately, an Observable may lazily wait for a first observer before taking any action.
- An Observable can be unobserved.
### Observable Creation
This simple definition of an Observable makes it very easy to create them from various sources.
- `fromConstant<T>(value: T) => Observable<T>`
- `fromPromise<T>(promise: Promise<T>): Observable<T>`
- `fromPromiseWithError<T, E extends Error>(promise: Promise<T>): Observable<T | E>`
- `fromObservableProperties` converts an object with named Observable values into a single Observable of an object with corresponding types.
For instance:
```typescript
fromObservables({ a: fromConstant(1), b: fromConstant("s")})
// yields type Observable<{ a: number, b: string }>
```
`fromObservableProperties` will only yield a result once every contained observable has provided a result. If some properties are optional then use the `withOptional` higher order function described below.
### Observable Modification
They are also very easy to modify in order to build higher order Observables.
For the purposes of discussion let us use an array to indicate the sequential values yielded by an Observable with the first element being any synchronous callback and `void` meaning no synchronous callback.
So our `fromConstant(12)` would yield `[12]` and `fromPromise(Promise.resolve(12))` would yield `[void, 12]`
These examples will use the form of: Sequence Before => Higher Order Function => Sequence After.
withDeduplicate removes any sequentially equivalent values:
`[1, 1, 2, 2, 2, 3]` => `withDeduplicate` => `[1, 2, 3]`
withDefault replaces any undefined values, including initial synchronous value with a default value:
`[void, 1, 2, undefined, 4]` => `withDefault(-1)` => `[-1, 1, 2, -1, 4]`
withMap applies a mapping function to each value at notification time:
`[void, 1, 2, 3]` => `withMap(x => x * 2)` => `[void, 2, 4, 6]`
withOptional converts an `Observable<T>` to `Observable<T | undefined>` and will always respond synchronously.
`[void, 1, 2, 3, 4]` => `withOptional` => `[undefined, 1, 2, 3, 4]`
## Cache
The cache module contains functions and interfaces related to caching, hashing, memoization and storage and retrieval of `Blob`s.
The `memoize` function and `BlobStore` are the most important public interfaces in the cache submodule. They are described below. See the API reference for hashing and other lower level functions.
The `memoize` function can be used to cache expensive, deterministic, asynchronous functions that take data arguments and return data results. In addition to caching the results, it also blocks multiple simultaneous calls with the same arguments and makes them wait for the first result.
### BlobStore
It is important to handle `Blob`s correctly within an application.
The `BlobStore` interface and corresponding `blobStore` exported instance provide a convenient way to convert `Blob`s into small JSON handles to them called `BlobRef`s. Those `BlobRef`s can later be converted back into a `Blob`.
`Blob`s within the `blobStore` are generally persisted across sessions using the browsers buid in `Cache` storage.
`BlobRef`s have a number of advantages over directly using blobs.
`BlobRef`s are:
- small json objects
- deterministic for each `Blob` based on mime type and content.
- suitable for persistence to locations with limited size.
- suitable for use as a cache key
- suitable for use as a cache value
## Schemas
Contains some standard data type schemas in JSON Schema format for convenience. These types are most frequently used when describing components within the Entity Component System. Knowing the precise size and signedness of primitive numeric types allows the ECS to store these values within tightly packed typed arrays internally. This makes memory usage and processing time more efficient.
## Entity Component System (ECS)
A high-performance, strongly typed ECS database for TypeScript, inspired by [Flecs](https://www.flecs.dev/flecs/md_docs_2Docs.html). All application state is modeled as composable plugins, and all mutations flow through observable, undoable transactions.
For a complete guide covering plugins, transactions, observability, composition, and transient/ephemeral semantics, see the **[ECS README](./packages/data/src/ecs/README.md)**.
```typescript
import { Database } from "@adobe/data/ecs";
const myPlugin = Database.Plugin.create({
resources: {
score: { default: 0 as number },
},
transactions: {
addPoints: (t, points: number) => {
t.resources.score += points;
},
},
});
const db = Database.create(myPlugin);
db.observe.resources.score((score) => console.log("Score:", score));
db.transactions.addPoints(10);
```
### What is an ECS?
Sanders Mertens covers this thoroughly in his ECS FAQ:
[https://github.com/SanderMertens/ecs-faq?tab=readme-ov-file#what-is-ecs](https://github.com/SanderMertens/ecs-faq?tab=readme-ov-file#what-is-ecs)
In addition to the standard Entity, Component, and System definitions, we also use the term **Resource** — a global singleton value defined on the ECS itself, not attached to any specific entity.
## Corporate contributors
If your primary GitHub account is a corporate / Enterprise Managed User (EMU) account, it may be unable to fork or push to public repositories like this one. The fix is to contribute from a separate personal GitHub identity on the same machine, using SSH host aliases so each identity uses its own key.
**Problem:** one machine, multiple GitHub identities (e.g. corporate vs. personal), needing a different SSH key per identity — but `git@github.com` only supports one key per config entry.
**Solution:** define extra `Host` aliases in `~/.ssh/config` that all point at the real `github.com`, each pinned to a different key with `IdentitiesOnly yes`:
```sshconfig
# Default identity (e.g. your corporate account)
Host github.com
HostName github.com
AddKeysToAgent yes
UseKeychain yes
IdentityFile ~/.ssh/id_ed25519
IdentitiesOnly yes
# Alternate identity (e.g. your personal account)
Host github.com-personal
HostName github.com
AddKeysToAgent yes
UseKeychain yes
IdentityFile ~/.ssh/id_ed25519_personal
IdentitiesOnly yes
```
Then pick the identity by swapping the host in the remote URL:
```sh
git@github.com-personal:adobe/data.git # uses the personal-account key
git@github.com:adobe/data.git # uses the default key
```
`IdentitiesOnly yes` is required — without it, `ssh-agent` may offer the wrong key first and authenticate against the wrong account.
**Setup:**
1. Generate a separate SSH key per GitHub identity and add each public key to the corresponding GitHub account.
2. Add one host-alias block per non-default identity to `~/.ssh/config`, as above.
3. For any repo that should use a non-default identity, set its remote to the aliased host (`git remote set-url origin git@github.com-personal:adobe/data.git`), and set that repo's local `user.email` to the matching account.
The `-personal` suffix is just an arbitrary alias name — any label works as long as `HostName` maps it back to `github.com`.