creating-timeplus-apps

# Creating Timeplus Apps ## Overview A Timeplus app (`.tpapp`) is a zip archive that bundles a streaming data pipeline — DDL resources (streams, views, materialized views) plus dashboards — into a single installable unit. The installer provisions everything in order and rolls back on failure. ## Directory Structure ``` my-app/ ├── manifest.yaml # required ├── ddl/ │ ├── 001_first.sql # executed in filename order │ ├── 002_second.sql │ └── ... └── dashboards/ └── main.json # array of panel objects ``` Package it: ```bash cd my-app && zip -r ../my-app.tpapp manifest.yaml ddl/ dashboards/ ``` Install via API: ```bash curl -X POST http://localhost:8000/default/api/v1beta2/apps/install \ -F "file=@my-app.tpapp" ``` Override `config:` values at install time with `config[<key>]=<value>` form fields (multipart) — the neutron handler parses any form field matching `config[*]` into the rendered config map: ```bash curl -X POST http://localhost:8000/default/api/v1beta2/apps/install \ -F "file=@my-app.tpapp" \ -F "config[strategy]=sign" \ -F "config[num_stocks]=5" ``` For JSON-body installs (URL fetch), use `{"url": "...", "config": {"strategy": "sign"}}`. ## manifest.yaml ```yaml package_format_version: 1 # must be 1 id: io.example.my-app # reverse-domain, unique name: My App version: 1.0.0 author: Acme description: What this app does. icon: "data:image/png;base64,..." # optional — base64 data URI; frontend shows default when absent categories: # optional — free-form tags for discovery/filtering - security - observability db_name: my_app # ^[a-z][a-z0-9_]{0,31}$, used as-is config: # optional — user-supplied parameters - key: websocket_url type: string required: true description: WebSocket feed URL - key: api_key type: string required: true secret: true # mask value in UI; stored as IsSecret description: API key - key: timeout type: integer required: false default: "30" description: Connection timeout in seconds - key: tls_enabled type: bool required: false default: "false" description: Enable TLS - key: topics type: list required: true description: Kafka topics (JSON array of strings, e.g. '["a","b"]') - key: broker_type type: choice required: true description: Message broker options: - kafka - pulsar - redpanda - key: features type: multi_choice required: false default: '["metrics"]' description: Features to enable options: - metrics - tracing - alerting python_packages: # optional — installed before any DDL runs - json5>=0.9.6 - websocket-client>=1.4.0 resources: # executed in listed order - file: ddl/001_source.sql type: external_stream name: raw_feed - file: ddl/002_events.sql type: stream name: events - file: ddl/003_mv.sql type: materialized_view name: mv_events dashboards: - file: dashboards/main.json name: My Dashboard description: Real-time view ``` ### Resource types | type | DDL verb | idempotent form | rolled back with | |---|---|---|---| | `stream` | `CREATE STREAM` | `CREATE STREAM IF NOT EXISTS` | `DROP STREAM` | | `external_stream` | `CREATE EXTERNAL STREAM` | `CREATE EXTERNAL STREAM IF NOT EXISTS` | `DROP STREAM` | | `mutable_stream` | `CREATE MUTABLE STREAM` | `CREATE MUTABLE STREAM IF NOT EXISTS` | `DROP STREAM` | | `materialized_view` | `CREATE MATERIALIZED VIEW` | `CREATE MATERIALIZED VIEW IF NOT EXISTS` | `DROP VIEW` | | `view` | `CREATE VIEW` | `CREATE VIEW IF NOT EXISTS` | `DROP VIEW` | | `external_table` | `CREATE TABLE` | `CREATE TABLE IF NOT EXISTS` | `DROP TABLE` | | `udf` | `CREATE FUNCTION` | `CREATE OR REPLACE FUNCTION` (see below) | `DROP FUNCTION` | | `task` | `CREATE TASK` | `CREATE TASK IF NOT EXISTS` | `DROP TASK` | | `alert` | `CREATE ALERT` | `CREATE ALERT IF NOT EXISTS` | `DROP ALERT` | | `input` | `CREATE INPUT` | `CREATE INPUT IF NOT EXISTS` | `DROP INPUT` | | `dictionary` | `CREATE DICTIONARY` | `CREATE DICTIONARY IF NOT EXISTS` | `DROP DICTIONARY` | | `format_schema` | `CREATE FORMAT SCHEMA` | `CREATE FORMAT SCHEMA IF NOT EXISTS` | `DROP FORMAT SCHEMA` | | `named_collection` | `CREATE NAMED COLLECTION` | `CREATE NAMED COLLECTION IF NOT EXISTS` | `DROP NAMED COLLECTION` | ## DDL Template Variables DDL files are rendered with Go `text/template` using `{{ }}` delimiters. | Expression | Expands to | |---|---| | `{{ .DB }}` | The resolved database name (value of `db_name`) | | `{{ .Config.key_name }}` | Value of config key (after defaults applied) | **Always use dot notation for config values** — `{{ .Config.my_key }}`, never `{{ index .Config "my_key" }}`. ```sql -- ddl/002_events.sql CREATE STREAM IF NOT EXISTS {{ .DB }}.events ( id string, payload string ) TTL to_datetime(_tp_time) + INTERVAL 24 HOUR; ``` ```sql -- ddl/001_source.sql CREATE EXTERNAL STREAM IF NOT EXISTS {{ .DB }}.raw_feed (msg string) SETTINGS url='{{ .Config.websocket_url }}', type='websocket'; ``` ## Idempotency: every `CREATE` must be re-runnable App upgrades re-run every DDL file against the existing database — the installer does **not** drop resources first. A `CREATE` that fails on the second run breaks upgrade. Every DDL file in `apps/*/ddl/` must use one of the following forms: | Resource | Required form | Why | |---|---|---| | Everything except `udf` | `CREATE … IF NOT EXISTS <name>` | Re-running is a no-op. Existing rows, downstream consumers, and the resource UUID are preserved — critical for `view` and `materialized_view`, which are referenced by name from other resources and from dashboards. | | `udf` | `CREATE OR REPLACE FUNCTION <name>` | UDFs are global (no database qualifier — see [udf](#udf)) and their body is typically what changes in an upgrade. `OR REPLACE` hot-swaps the implementation; `IF NOT EXISTS` would silently keep the old code on upgrade. | **Do not use `CREATE OR REPLACE VIEW`.** A view is a dependency for downstream MVs, queries, and dashboards. `OR REPLACE` would drop and recreate it, breaking anything referencing the view during the brief recreation window and on schema changes. Use `CREATE VIEW IF NOT EXISTS` — if the view definition needs to change, bump the app version and treat it as a migration (drop + recreate explicitly, or version the view name). **Forms that do NOT accept `IF NOT EXISTS`:** - `SYSTEM INSTALL PYTHON PACKAGE` — parser rejects `IF NOT EXISTS`. Don't put this in DDL; declare packages in `manifest.yaml` under `python_packages` (see [Python packages](#python-packages-not-available-at-ddl-time)). - `CREATE FUNCTION IF NOT EXISTS db.fn` and `CREATE FORMAT SCHEMA IF NOT EXISTS db.fs` — both reject the `db.` prefix. UDFs and format schemas live in a global namespace; never qualify them with `{{ .DB }}.`. ## Dashboard Template Variables Dashboard JSON is rendered with `[[ ]]` delimiters (to avoid collision with the frontend's `{{filter_*}}` runtime variables). ```json { "viz_content": "SELECT * FROM [[ .DB ]].events WHERE _tp_time > now() - {{filter_time_range}}" } ``` | Expression | Expands to | |---|---| | `[[ .DB ]]` | Database name | | `[[ .Config.key ]]` | Config value | | `{{filter_*}}` | Left as-is — resolved by the frontend at query time | **Template processing runs before JSON parsing.** This means template expressions inside JSON string values may contain unescaped `"` characters — the file does not need to be valid JSON before substitution. ## Template Functions (Sprig) Both DDL (`{{ }}`) and dashboard (`[[ ]]`) templates have the full [Sprig](https://masterminds.github.io/sprig/) function library available — the same library used by Helm. Use these to manipulate config values at install time. ### Working with `list` config values Config keys of type `list` are stored as a JSON array string (e.g. `["BTC-USD","ETH-USD","SOL-USD"]`). Use `fromJson` to parse them before passing to other functions. **Render as comma-separated string** (e.g. for dashboard selector `inlineValues`): ```json "inlineValues": "[[ join "," (fromJson .Config.product_ids) ]]" ``` → `"inlineValues": "BTC-USD,ETH-USD,SOL-USD"` **Embed directly as JSON array** (e.g. in a DDL Python string): ```sql product_ids = '{{ .Config.product_ids }}' ``` → `product_ids = '["BTC-USD","ETH-USD","SOL-USD"]'` **Get the first element** (e.g. for a selector `defaultValue`): ```json "defaultValue": "[[ index (fromJson .Config.product_ids) 0 ]]" ``` → `"defaultValue": "BTC-USD"` ### Commonly used functions | Function | Example | Result | |---|---|---| | `join sep list` | `join "," (fromJson .Config.topics)` | `a,b,c` | | `fromJson s` | `fromJson .Config.product_ids` | parsed slice | | `default val s` | `default "30" .Config.timeout` | config value or fallback | | `upper s` | `upper .Config.env` | `PRODUCTION` | | `lower s` | `lower .Config.env` | `production` | | `trim s` | `trim .Config.url` | strips whitespace | | `replace old new s` | `replace "-" "_" .Config.id` | `BTC_USD` | | `splitList sep s` | `splitList "," .Config.tags` | `["a","b","c"]` | | `first list` | `first (fromJson .Config.ids)` | first element | | `last list` | `last (fromJson .Config.ids)` | last element | | `len list` | `len (fromJson .Config.ids)` | count | Full function reference: https://masterminds.github.io/sprig/ ## Dashboard JSON Reference For the full dashboard panel specification — all chart types, `viz_config` fields, control panels, position grid, update modes, and working examples — see: **`skill/references/dashboard-spec.md`** This covers: - Panel structure (`id`, `title`, `position`, `viz_type`, `viz_content`, `viz_config`) - 12-column position grid and common width/height values - Template variables (`[[ .DB ]]` vs `{{filter_*}}`) - Control panels: `selector` (dropdown) and `text_input` - Chart types: `line`, `area`, `bar`, `column`, `singleValue`, `table`, `ohlc`, `geo`, `md`, `grammar` (3.2+ — generic Vistral-grammar-driven viz: scatter, layered marks, band-axis bars, stacked area, custom transforms, etc.). For the underlying `VistralSpec` grammar (marks, transforms, scales, encode channels), see the Vistral skill: [`vistral/agentskill/SKILL.md`](https://github.com/timeplus-io/vistral/blob/main/agentskill/SKILL.md). - All `viz_config.config` fields per chart type with defaults - `updateMode` (`"all"` / `"key"` / `"time"`) — when to use each - Default color palette - Common mistakes ## Timeplus SQL Reference For writing correct Timeplus streaming SQL in DDL files, refer to the Timeplus SQL skill: https://github.com/timeplus-io/AgentSkills/tree/main/timeplus-sql-guide This covers streaming query syntax, window functions, tumble/hop aggregations, `_tp_time` semantics, and other Timeplus-specific SQL features used in streams, views, and materialized views. ## File Ordering and Dependencies Name DDL files with a numeric prefix so they execute in dependency order: ``` 001_source_stream.sql ← external streams / sources 002_target_stream.sql ← destination streams 003_mv_extract.sql ← materialized views (depend on streams) 004_v_aggregated.sql ← views (depend on streams/MVs) ``` ## Config Types Seven types are supported. Omitting `type` defaults to `string`. | Type | Stored as | Valid example | Notes | |------|-----------|---------------|-------| | `string` | plain string | `"localhost:9092"` | Default type | | `integer` | decimal string | `"30"`, `"-5"` | Must be a whole number | | `float` | decimal string | `"3.14"`, `"30"` | Decimal or whole | | `bool` | `"true"` or `"false"` | `"true"` | No other values accepted |