native-trigger

# Skill: Adding Native Trigger Services This skill provides comprehensive guidance for adding new native trigger services to Windmill. Native triggers allow external services (like Nextcloud, Google Drive, etc.) to trigger Windmill scripts/flows via webhooks or push notifications. ## Architecture Overview The native trigger system consists of: 1. **Database Layer** - PostgreSQL tables and enum types 2. **Backend Rust Implementation** - Core trait, handlers, and service modules in the `windmill-native-triggers` crate 3. **Frontend Svelte Components** - Configuration forms and UI components ### Key Files | Component | Path | |-----------|------| | Core module with `External` trait | `backend/windmill-native-triggers/src/lib.rs` | | Generic CRUD handlers | `backend/windmill-native-triggers/src/handler.rs` | | Background sync logic | `backend/windmill-native-triggers/src/sync.rs` | | OAuth/workspace integration | `backend/windmill-native-triggers/src/workspace_integrations.rs` | | Re-export shim (windmill-api) | `backend/windmill-api/src/native_triggers/mod.rs` | | TriggerKind enum | `backend/windmill-common/src/triggers.rs` | | JobTriggerKind enum | `backend/windmill-common/src/jobs.rs` | | Frontend service registry | `frontend/src/lib/components/triggers/native/utils.ts` | | Frontend trigger utilities | `frontend/src/lib/components/triggers/utils.ts` | | Trigger badges (icons + counts) | `frontend/src/lib/components/graph/renderers/triggers/TriggersBadge.svelte` | | Workspace integrations UI | `frontend/src/lib/components/workspaceSettings/WorkspaceIntegrations.svelte` | | OAuth config form component | `frontend/src/lib/components/workspaceSettings/OAuthClientConfig.svelte` | | OpenAPI spec | `backend/windmill-api/openapi.yaml` | | Reference: Nextcloud module | `backend/windmill-native-triggers/src/nextcloud/` | | Reference: Google module | `backend/windmill-native-triggers/src/google/` | ### Crate Structure The native trigger code lives in the `windmill-native-triggers` crate (`backend/windmill-native-triggers/`). The `windmill-api` crate re-exports everything via a shim: ```rust // backend/windmill-api/src/native_triggers/mod.rs pub use windmill_native_triggers::*; ``` All new service modules go in `backend/windmill-native-triggers/src/`. --- ## Core Concepts ### The `External` Trait Every native trigger service implements the `External` trait defined in `lib.rs`: ```rust #[async_trait] pub trait External: Send + Sync + 'static { // Associated types: type ServiceConfig: Debug + DeserializeOwned + Serialize + Send + Sync; type TriggerData: Debug + Serialize + Send + Sync; type OAuthData: DeserializeOwned + Serialize + Clone + Send + Sync; type CreateResponse: DeserializeOwned + Send + Sync; // Constants: const SUPPORT_WEBHOOK: bool; const SERVICE_NAME: ServiceName; const DISPLAY_NAME: &'static str; const TOKEN_ENDPOINT: &'static str; const REFRESH_ENDPOINT: &'static str; const AUTH_ENDPOINT: &'static str; // Required methods: async fn create(&self, w_id, oauth_data, webhook_token, data, db, tx) -> Result<Self::CreateResponse>; async fn update(&self, w_id, oauth_data, external_id, webhook_token, data, db, tx) -> Result<serde_json::Value>; async fn get(&self, w_id, oauth_data, external_id, db, tx) -> Result<Self::TriggerData>; async fn delete(&self, w_id, oauth_data, external_id, db, tx) -> Result<()>; async fn exists(&self, w_id, oauth_data, external_id, db, tx) -> Result<bool>; async fn maintain_triggers(&self, db, workspace_id, triggers, oauth_data, synced, errors); fn external_id_and_metadata_from_response(&self, resp) -> (String, Option<serde_json::Value>); // Methods with defaults: async fn prepare_webhook(&self, db, w_id, headers, body, script_path, is_flow) -> Result<PushArgsOwned>; fn service_config_from_create_response(&self, data, resp) -> Option<serde_json::Value>; fn additional_routes(&self) -> axum::Router; async fn http_client_request<T, B>(&self, url, method, workspace_id, tx, db, headers, body) -> Result<T>; } ``` Key design points: - **`update()` returns `serde_json::Value`** - the resolved service_config to store. Each service is responsible for building the final config. - **`maintain_triggers()`** - periodic background maintenance. Each service implements its own strategy (Nextcloud: reconcile with external state; Google: renew expiring channels). - **No `list_all()` in the trait** - services that need it (Nextcloud) implement it privately; services that don't (Google) use different maintenance strategies. - **No `get_external_id_from_trigger_data()` or `extract_service_config_from_trigger_data()`** - removed in favor of the `maintain_triggers` pattern. ### Create Lifecycle: Two Paths The `create_native_trigger` handler in `handler.rs` supports two creation flows, controlled by `service_config_from_create_response()`: **Path A: Short (Google pattern)** - `service_config_from_create_response()` returns `Some(config)`: 1. `create()` registers on external service 2. `external_id_and_metadata_from_response()` extracts the ID 3. `service_config_from_create_response()` builds the config directly from input data + response metadata 4. Stores trigger in DB -- done, no extra round-trip Use this when the external_id is known before the create call (e.g., Google generates the channel_id as a UUID upfront and includes it in the webhook URL). **Path B: Long (Nextcloud pattern)** - `service_config_from_create_response()` returns `None` (default): 1. `create()` registers on external service (webhook URL has no external_id yet) 2. `external_id_and_metadata_from_response()` extracts the ID 3. `update()` is called to fix the webhook URL with the now-known external_id 4. `update()` returns the resolved service_config 5. Stores trigger in DB Use this when the external_id is assigned by the remote service and the webhook URL needs to be corrected after creation. ### OAuth Token Storage (Three-Table Pattern) OAuth tokens are stored across three tables, NOT in `workspace_integrations.oauth_data` directly: | Table | What's Stored | |-------|---------------| | `workspace_integrations` | `oauth_data` JSON with `base_url`, `client_id`, `client_secret`, `instance_shared` flag; `resource_path` pointing to the variable | | `variable` | Encrypted `access_token` (at the path stored in `resource_path`), linked to `account` via `account` column | | `account` | `refresh_token`, keyed by `workspace_id` + `client` (service name) + `is_workspace_integration = true` | The `decrypt_oauth_data()` function in `lib.rs` assembles these into a unified struct: ```rust pub struct OAuthConfig { pub base_url: String, pub access_token: String, // decrypted from variable pub refresh_token: Option<String>, // from account table pub client_id: String, // from oauth_data or instance settings pub client_secret: String, // from oauth_data or instance settings } ``` Instance-level sharing: when `oauth_data.instance_shared == true`, `client_id` and `client_secret` are read from global settings instead of workspace_integrations. ### URL Resolution The `resolve_endpoint()` helper handles both absolute and relative OAuth URLs: ```rust pub fn resolve_endpoint(base_url: &str, endpoint: &str) -> String { if endpoint.starts_with("http://") || endpoint.starts_with("https://") { endpoint.to_string() // Google: absolute URLs } else { format!("{}{}", base_url, endpoint) // Nextcloud: relative paths } } ``` ### ServiceName Methods `ServiceName` is the central registry enum. Each variant must implement these match arms: | Method | Purpose | |--------|---------| | `as_str()` | Lowercase identifier (e.g., `"google"`) | | `as_trigger_kind()` | Maps to `TriggerKind` enum | | `as_job_trigger_kind()` | Maps to `JobTriggerKind` enum | | `token_endpoint()` | OAuth token endpoint (relative or absolute) | | `auth_endpoint()` | OAuth authorization endpoint | | `oauth_scopes()` | Space-separated OAuth scopes | | `resource_type()` | Resource type for token storage (e.g., `"gworkspace"`) | | `extra_auth_params()` | Extra OAuth params (e.g., Google needs `access_type=offline`, `prompt=consent`) | | `integration_service()` | Maps to the workspace integration service (usually `*self`) | | `TryFrom<String>` | Parse from string | | `Display` | Delegates to `as_str()` | --- ## Step-by-Step Implementation Guide ### Step 1: Database Migration Create a new migration file: `backend/migrations/YYYYMMDDHHMMSS_newservice_trigger.up.sql` ```sql -- Add the service to the native_trigger_service enum ALTER TYPE native_trigger_service ADD VALUE IF NOT EXISTS 'newservice'; -- Add to TRIGGER_KIND enum (used for trigger tracking) ALTER TYPE TRIGGER_KIND ADD VALUE IF NOT EXISTS 'newservice'; -- Add to job_trigger_kind enum (used for job tracking) ALTER TYPE job_trigger_kind ADD VALUE IF NOT EXISTS 'newservice'; ``` Also create the corresponding down migration. ### Step 2: Update windmill-common Enums #### `backend/windmill-common/src/triggers.rs` Add variant to `TriggerKind` enum, and update `to_key()` and `fmt()` implementations. #### `backend/windmill-common/src/jobs.rs` Add variant to `JobTriggerKind` enum and update the `Display` implementation. ### Step 3: Backend Service Module Create a new directory: `backend/windmill-native-triggers/src/newservice/` #### `mod.rs` - Type Definitions ```rust use serde::{Deserialize, Serialize}; pub mod external; // pub mod routes; // Only if you need additional service-specific routes /// OAuth data deserialized from the three-table pattern. /// The actual structure is built by decrypt_oauth_data() from variable + account + workspace_integrations. #[derive(Debug, Clone, Deserialize, Serialize)] pub struct NewServiceOAuthData { pub base_url: String, // from workspace_integrations.oauth_data pub access_token: String, // decrypted from variable table pub refresh_token: Option<String>, // from account table // Note: client_id and client_secret are in OAuthConfig, not here // unless the service needs them at runtime for API calls } /// Configuration provided by user when creating/updating a trigger. /// Stored as JSON in native_trigger.service_config. #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] pub struct NewServiceConfig { // Service-specific configuration fields pub folder_path: String, pub file_filter: Option<String>, } /// Data retrieved from the external service about a trigger. /// Returned by the get() method and shown in the UI. #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] pub struct NewServiceTriggerData { pub folder_path: String, pub file_filter: Option<String>, // Fields that shouldn't affect service_config comparison should use #[serde(skip_serializing)] } /// Response from external service when creating a trigger/webhook. #[derive(Debug, Deserialize)] pub struct CreateTriggerResponse { pub id: String, } /// Handler struct (stateless, used for routing) #[derive(Copy, Clone)] pub struct NewService; ``` #### `external.rs` - External Trait Implementation ```rust use async_trait::async_trait; use reqwest::Method; use sqlx::PgConnection; use std::collections::HashMap; use windmill_common::{ error::{Error, Result}, BASE_URL, DB, }; use crate::{ generate_webhook_service_url, External, NativeTrigger, NativeTriggerData, ServiceName, sync::{SyncError, TriggerSyncInfo}, }; use super::{NewService, NewServiceConfig, NewServiceOAuthData, NewServiceTriggerData, CreateTriggerResponse}; #[async_trait] impl External for NewService { type ServiceConfig = NewServiceConfig; type TriggerData = NewServiceTriggerData; type OAuthData = NewServiceOAuthData; type CreateResponse = CreateTriggerResponse; const SERVICE_NAME: ServiceName = ServiceName::NewService; const DISPLAY_NAME: &'static str = "New Service"; const SUPPORT_WEBHOOK: bool = true; const TOKEN_ENDPOINT: &'static str = "/oauth/token"; const REFRESH_ENDPOINT: &'static str = "/oauth/token"; const AUTH_ENDPOINT: &'static str = "/oauth/authorize"; async fn create( &self, w_id: &str, oauth_data: &Self::OAuthData, webhook_token: &str, data: &NativeTriggerData<Self::ServiceConfig>, db: &DB, tx: &mut PgConnection, ) -> Result<Self::CreateResponse> { let base_url = &*BASE_URL.read().await; // external_id is None during create (we get it from the response)