edg-expression

name	edg-expression
description	Help compose edg expressions. Explain functions, debug syntax, and generate the right incantation for a given use case.
user-invocable	true

edg Expression Helper

You help users compose, debug, and understand edg expressions. edg uses expr-lang/expr as its expression engine, extended with ~60 built-in functions for data generation, distribution sampling, and reference data access.

What you do

• Explain what a specific function does and when to use it
• Compose expressions for a described use case
• Debug expression syntax errors
• Suggest the REPL for interactive testing: edg repl

Expressions in YAML vs DSL

All expressions are identical in both formats - same functions, same syntax. Only the surrounding config structure differs:

YAML:

args:
  - ref_rand('fetch_users').id
  - gen('email')

DSL (positional):

query_name `SELECT ...` (ref_rand('fetch_users').id, gen('email'))

DSL (named):

query_name `SELECT ...` (user_id: ref_rand('fetch_users').id, email: gen('email'))

When helping with expressions, ask which format the user is working with if it affects the answer (e.g., wrapping syntax). The expression itself is always the same.

Quick Reference

Identifiers

Function	Description
uuid_v7()	Sortable UUID (preferred for PKs)
uuid_v4()	Random UUID
seq(start, step)	Auto-incrementing counter per worker
seq_alpha(length)	Auto-incrementing alpha sequence per worker (aaa, aab, aac, ...)
seq_global(name)	Shared auto-incrementing counter across all workers (requires seq: config section)
seq_alpha_global(name)	Shared auto-incrementing alpha sequence across all workers (requires seq: config with length)
seq_rand(name)	Uniform random from generated sequence values
seq_zipf(name, s, v)	Zipfian-distributed pick from sequence (hot early values)
seq_norm(name, mean, stddev)	Normal-distributed pick from sequence
seq_pareto(name, alpha)	Pareto-distributed pick from sequence (continuous power-law)
seq_exp(name, rate)	Exponential-distributed pick from sequence
seq_lognorm(name, mu, sigma)	Log-normal-distributed pick from sequence
objectid()	MongoDB ObjectID (24-char hex string)

Random Data (gofakeit)

Function	Description
gen('email')	Random email
gen('firstname')	Random first name
gen('lastname')	Random last name
gen('number:1,100')	Random int in range
gen('sentence:5')	Random sentence
regex('[A-Z]{3}-[0-9]{4}')	String matching regex
bool()	Random true/false

Numeric Distributions

Function	Description
uniform(min, max)	Flat/even distribution
uniform_f(min, max, precision)	Uniform float with decimal places
norm(mean, stddev, min, max)	Bell curve
norm_f(mean, stddev, min, max, precision)	Bell curve float
exp(rate, min, max)	Exponential decay
lognorm(mu, sigma, min, max)	Right-skewed long tail
pareto(alpha, max)	Continuous power-law, lower values dominate
zipf(s, v, max)	Power-law hot keys
nurand(A, x, y)	TPC-C non-uniform random

Set Distributions

Function	Description
set_rand(values, weights)	Uniform or weighted pick from a set
set_norm(values, mean, stddev)	Normal distribution over set indices
set_exp(values, rate)	Exponential over set indices
set_pareto(values, alpha)	Pareto over set indices (continuous power-law)
set_zipf(values, s, v)	Zipfian over set indices
set_lognorm(values, mu, sigma)	Log-normal over set indices

Dates & Times

Function	Description
timestamp(min, max)	Random RFC3339 timestamp
timestamp_step()	Next monotonic timestamp (requires timestamp_steps in count:)
timestamp_steps(min, max, interval_or_count)	Count of steps between min and max; third arg is interval string ('5m') or integer count (10000). Sets up timestamp_step()
date(format, min, max)	Random formatted date
date_offset(duration)	Now +/- duration
duration(min, max)	Random Go duration string
time(min, max)	Random HH:MM:SS

Strings & Formatting

Function	Description
template(format, args...)	Go fmt.Sprintf
json_obj(k1, v1, ...)	Build JSON object
json_arr(minN, maxN, pattern)	Build JSON array
array(minN, maxN, pattern)	PostgreSQL array literal
vector(dims, clusters, spread)	vector literal with clustered unit vectors (uniform)
vector_pareto(dims, clusters, spread, alpha)	vector literal with Pareto centroid selection
vector_zipf(dims, clusters, spread, s, v)	vector literal with Zipfian centroid selection
vector_norm(dims, clusters, spread, mean, stddev)	vector literal with normal centroid selection
embed(text...)	Real vector embedding via external API (OpenAI-compatible). Variadic, joins args with space. Requires a license and --embed-api-key

LLM / Structured Generation

Function	Description
complete(tool_name, prompt)	Generate structured data via LLM tool calling. Returns a map; access fields with dot notation. Retries up to 3 times on missing/invalid tool calls. Validates response types against tool schema. 120s per-request timeout
complete_array(tool_name, prompt, count)	Generate N structured items in a single LLM call. Returns []map; use with ref_each() to iterate. Schema auto-wrapped in array. Memoized by (tool, prompt, count). Same retry/validation as complete()

Use locals to call once per row and access multiple fields:

# Single item
locals:
  review: 'complete("review", "Review: " + name)'
args:
  - local("review").review_text
  - local("review").sentiment
  - local("review").rating

# Batch (N items in one call)
locals:
  reviews: 'complete_array("review", "Generate 5 reviews", 5)'
args:
  - ref_each(local("reviews")).review_text
  - ref_each(local("reviews")).sentiment
  - ref_each(local("reviews")).rating

Requires a license and --complete-api-key (or EDG_COMPLETE_API_KEY). Configure endpoint with --complete-url, model with --complete-model. Any OpenAI-compatible API works (Ollama, vLLM, etc.). Tool schema defined in complete: YAML section.

Reference Data

Function	Scope	Description
ref_rand(name)	None	Fresh random row every call
ref_same(name)	Per-query	Same row within one query execution
ref_diff(name)	Per-query	Unique row per call within one query
ref_perm(name)	Per-worker	Fixed row for worker lifetime
ref_each(query_or_dataset)	Expansion / Per-query	SQL string: one execution per returned row. Named dataset (unquoted): sequential iteration with same-row caching
ref_n(name, field, min, max)	None	N unique values as CSV string
ref_norm(name, mean, stddev)	None	Row picked via normal distribution
ref_exp(name, rate)	None	Row picked via exponential distribution
ref_lognorm(name, mu, sigma)	None	Row picked via log-normal distribution
ref_pareto(name, alpha)	None	Row picked via Pareto distribution
ref_zipf(name, s, v)	None	Row picked via Zipfian distribution

Aggregation (over named datasets)

Function	Description
count(name)	Row count
sum(name, field)	Sum of field
avg(name, field)	Average of field
min(name, field)	Minimum of field
max(name, field)	Maximum of field
distinct(name, field)	Count of distinct values

Conditionals & Dependencies

Function	Description
arg(index)	Reference earlier arg by zero-based index
arg('name')	Reference earlier arg by name (named args only)
result()	First row of current query's SELECT result (post_print only)
cond(pred, trueVal, falseVal)	Ternary conditional (inline, for args)
nullable(expr, probability)	NULL with given probability
coalesce(v1, v2, ...)	First non-nil value
const(value)	Literal constant
fail(message)	Stop current worker gracefully with error
fatal(message)	Terminate entire process immediately

Control Flow (config-level, not expression functions)

Conditional branching uses the same expr-lang expressions but at the config level, not inside args:

• if: condition must evaluate to boolean (e.g., "ref_same('x').balance >= local('amount')")
• match: expression can be any type; compared to each eq: value as strings
• eq: expressions can be any type (string literals need inner quotes: "'savings'")
• All conditional expressions have access to ref_same(), ref_rand(), local(), global(), etc.

Correlated Totals

Function	Description
distribute_sum(total, minN, maxN, precision)	N random parts summing exactly to total (comma-separated)
distribute_weighted(total, weights, noise, precision)	Split total by proportional weights with controlled noise (0=exact, 1=random)

Multi-Value

Function	Description
nurand_n(A, x, y, minN, maxN)	N unique NURand values (comma-separated)
norm_n(mean, stddev, min, max, minN, maxN)	N unique normally-distributed values (comma-separated)
weighted_sample_n(name, field, weightField, minN, maxN)	N weighted unique rows from a dataset (comma-separated)

Batch

Function	Description
batch(n)	Sequential indices [0, n)
gen_batch(total, size, pattern)	Batched gofakeit values
iter()	1-based row counter for batch queries (resets per query)

Uniqueness

Function	Description
uniq(expression)	Retry expression until unique value found (100 attempts default)
uniq(expression, maxRetries)	Same, with custom retry limit
uniq(expr1, expr2 [, ...])	Composite uniqueness - retries until the tuple is unique. Returns []any; index to pick column. Same-row calls with identical expressions return cached tuple
uniq(expr1, expr2, maxRetries)	Composite with custom retry limit

PII & Masking

Function	Description
gen_locale('first_name', 'ja_JP')	Locale-aware name/address/phone generation
gen_locale('name', 'de_DE')	Full name in locale order (eastern = last+first, western = first last)
mask(value)	Deterministic 16-char hex token (same input -> same output)
mask(value, length)	Hex token truncated to length chars
mask(value, 'base64')	Base64-encoded token
mask(value, 'base32')	Base32-encoded token
mask(value, 'asterisk')	Repeated * characters (default 16)
mask(value, 'asterisk', 4)	4 asterisks
mask(value, 'redact')	Fixed string [REDACTED], length ignored
mask(value, 'email')	Preserves @domain, masks local part with *
mask(value, 'email', 4)	****@domain.com

Supported locales: en_US, ja_JP, de_DE, fr_FR, es_ES, pt_BR, zh_CN, ko_KR (aliases: ja, de, fr, etc.)

Iteration Counter

Function	Description
global_iter()	Monotonic iteration counter shared across all workers. Increments with every query execution regardless of which statement runs. Use with math functions to make data change shape over the life of a workload

Math

Function	Description
abs(x)	Absolute value
acos(x)	Arc cosine (radians)
asin(x)	Arc sine (radians)
atan(x)	Arc tangent (radians)
atan2(y, x)	Two-argument arc tangent (radians)
ceil(x)	Smallest integer >= x
cos(x)	Cosine (radians)
floor(x)	Largest integer <= x
log(x)	Natural logarithm
log10(x)	Base-10 logarithm
mod(x, y)	Floating-point remainder
pi	Pi constant (3.14159...)
pow(x, y)	x raised to the power y
sin(x)	Sine (radians)
sqrt(x)	Square root
tan(x)	Tangent (radians)

Other

Function	Description
blob(n)	Random n bytes as raw binary (cross-database)
bytes(n)	Hex-encoded random bytes (pgx only)
bit(n)	Fixed-length bit string
varbit(n)	Variable-length bit string
inet(cidr)	Random IP in CIDR block
ltree(parts...)	PostgreSQL ltree path from dot-joined parts
point(lat, lon, radiusKM)	Random geographic point (map)
point_wkt(lat, lon, radiusKM)	Random point as WKT string
polygon(lat, lon, minKM, maxKM, points)	Jagged polygon vertices ([]map with .lat/.lon)
polygon_wkt(lat, lon, minKM, maxKM, points)	Jagged polygon as WKT POLYGON string

Common Patterns

Mutually exclusive columns

args:
  - bool()                            # coin flip
  - cond(arg(0), gen('email'), nil)   # email if true
  - cond(!arg(0), gen('phone'), nil)  # phone if false

Computed total from earlier args

# Positional
args:
  - uniform_f(1.00, 99.99, 2)   # price
  - gen('number:1,10')           # quantity
  - arg(0) * float(arg(1))      # total = price * qty

# Named (equivalent)
args:
  price: uniform_f(1.00, 99.99, 2)
  quantity: gen('number:1,10')
  total: arg('price') * float(arg('quantity'))

Full name from parts

# Positional
args:
  - gen('firstname')
  - gen('lastname')
  - arg(0) + ' ' + arg(1)       # "Alice Smith"

# Named (equivalent)
args:
  first: gen('firstname')
  last: gen('lastname')
  full: arg('first') + ' ' + arg('last')

Weighted category selection

args:
  - set_rand(['electronics', 'clothing', 'books', 'food'], [40, 30, 20, 10])

Hot-key access (Zipfian)

args:
  - zipf(2.0, 1.0, 999)  # value 0 is most frequent

Hotspot rows (distribution-based ref)

args:
  - ref_zipf('fetch_ids', 2.0, 1.0).id        # heavy skew toward early rows
  - ref_pareto('fetch_ids', 2.0).id            # continuous power-law, first rows dominate
  - ref_norm('fetch_ids', 0.5, 0.2).id         # bell curve centered at midpoint
  - ref_exp('fetch_ids', 1.5).id               # exponential decay from first row
  - ref_lognorm('fetch_ids', 0.0, 0.5).id      # right-skewed access pattern

Worker-pinned partition

args:
  - ref_perm('fetch_warehouses').w_id  # same warehouse for entire worker lifetime

Invoice line items (distribute_sum)

args:
  - uuid_v4()                           # invoice id
  - uniform_f(100, 10000, 2)            # total
  - distribute_sum(arg(1), 3, 7, 2)     # 3-7 amounts summing to total
# SQL: unnest(string_to_array('$3', ',')::NUMERIC[])

Subtotal/tax/shipping breakdown (distribute_weighted)

args:
  - uniform_f(100, 10000, 2)                           # total
  - distribute_weighted(arg(0), [85, 10, 5], 0.1, 2)   # ~85/10/5 split with 10% noise
# SQL: split_part('$2', ',', 1)::NUMERIC  -- subtotal
#      split_part('$2', ',', 2)::NUMERIC  -- tax
#      split_part('$2', ',', 3)::NUMERIC  -- shipping

Masking PII

args:
  first_name: gen_locale('first_name', 'ja_JP')
  last_name: gen_locale('last_name', 'ja_JP')
  full_name: arg('last_name') + arg('first_name')
  email: gen('email')
  masked_name: mask(arg('full_name'))                # hex token
  masked_email: mask(arg('email'), 'email', 4)       # ****@example.com
  redacted_phone: mask(arg('phone'), 'redact')       # [REDACTED]

Unique codes in batch inserts

# 200 airports each with a unique 3-char IATA code
- name: populate_airport
  type: exec_batch
  count: 200
  size: 100
  args:
    - iter()
    - uniq("gen('airlineairportiata')")
  query: |-
    INSERT INTO airport (id, code) VALUES ($1::INT, $2)

Composite uniqueness across columns

# 500 people with unique (first_name, last_name) pairs
- name: populate_people
  type: exec_batch
  count: 500
  size: 100
  args:
    - uniq("gen('first_name')", "gen('last_name')")[0]
    - uniq("gen('first_name')", "gen('last_name')")[1]
    - gen('email')
  query: |-
    INSERT INTO people (first_name, last_name, email) VALUES ($1, $2, $3)

Real embeddings

# With generated data
args:
  - gen('productname')
  - gen('sentence:3')
  - embed(arg(0), arg(1))            # embed name + description
query: |-
  INSERT INTO product (name, description, embedding)
  VALUES ($1, $2, $3::VECTOR)

# With reference dataset (each product exactly once)
args:
  - ref_each(product_catalog).name
  - ref_each(product_catalog).description
  - embed(ref_each(product_catalog).name, ref_each(product_catalog).description)
query: |-
  INSERT INTO product (name, description, embedding)
  VALUES ($1, $2, $3::VECTOR)

Requires a license and --embed-api-key (or EDG_EMBED_API_KEY). Configure endpoint with --embed-url, model with --embed-model, dimensions with --embed-dimensions, max batch size with --embed-max-batch. Any OpenAI-compatible API works (Ollama, vLLM, etc.).

Error handling with fail/fatal

args:
  # Stop worker if map lookup misses
  - {'us': 'us-east-1', 'eu': 'eu-west-1'}[env('REGION')] ?? fail('unknown REGION')

  # Kill entire process on critical misconfiguration
  - fatal('missing required config')

fail() stops only the current worker; fatal() terminates the entire process.

Temporal patterns (global_iter + math)

Combine global_iter() with math functions to make generated data change shape over the life of a workload. Define total_iters as a global and estimate it from: workers * duration_seconds / avg_latency_seconds.

# Zipf skew drift: popularity concentrates over time
zipf(initial_skew + (final_skew - initial_skew) * global_iter() / total_iters, 1, products - 1)

# Logarithmic growth: steep rise then plateau (inflation, adoption)
floor(base_price * (1.0 + log(1.0 + global_iter() / 1000.0)) * 100.0) / 100.0

# Sine wave seasonality: oscillation on a growing baseline
floor(abs(base_traffic + 0.5 * sqrt(global_iter()) + amplitude * sin(2.0 * pi * global_iter() / period)))

# Periodic spikes: cos² pulse at regular intervals
pow(cos(pi * mod(global_iter(), interval) / interval), 2.0) * 10.0

# Bounded drift: arctangent saturation (sensor calibration)
100.0 + (2.0 * atan(sqrt(global_iter()) / 100.0) / pi) * 15.0 + norm_f(0, 0.5, -2, 2, 2)

Debugging Tips

• Use edg repl to test any expression interactively without a database
• Expressions are compiled at startup; syntax errors will be caught immediately
• arg(index) is zero-based and only works within the same query's args list; arg('name') works with named args (map-style args:)
• Named and positional arg forms are mutually exclusive per query
• ref_same resets between queries; ref_perm never resets
• Set distributions accept expr-lang array literals: ['a', 'b', 'c']
• Weights in set_rand are relative, not percentages ([40, 30, 20, 10] and [4, 3, 2, 1] behave identically)