connector-ops

# Connector Operations Skill Manage Unstract connectors with full lifecycle support: add, remove, and modify operations for database, filesystem, and queue connector types. ## When to Use This Skill - User requests adding a new connector (e.g., "add a Redis connector", "create MongoDB database connector") - User requests removing a connector (e.g., "remove the Oracle connector", "delete Box filesystem") - User requests modifying a connector (e.g., "add SSL support to MySQL", "update BigQuery schema") ## Architecture Overview Connectors live in `/unstract/connectors/src/unstract/connectors/` with three categories: | Type | Base Class | Directory | Mode | |------|------------|-----------|------| | Database | `UnstractDB` | `databases/` | `ConnectorMode.DATABASE` | | Filesystem | `UnstractFileSystem` | `filesystems/` | `ConnectorMode.FILE_SYSTEM` | | Queue | `UnstractQueue` | `queues/` | `ConnectorMode.MANUAL_REVIEW` | ### Connector Structure Each connector follows this structure: ``` connector_name/ ├── __init__.py # Metadata dict with is_active flag ├── connector_name.py # Main connector class ├── constants.py # Optional constants └── static/ ├── json_schema.json # Configuration UI schema └── settings.yaml # Optional settings ``` ### Key Files | File | Purpose | |------|---------| | `base.py` | Root `UnstractConnector` abstract class | | `connectorkit.py` | Singleton registry for all connectors | | `databases/unstract_db.py` | Database connector base class | | `filesystems/unstract_file_system.py` | Filesystem connector base class | | `queues/unstract_queue.py` | Queue connector base class | | `databases/register.py` | Auto-discovery for database connectors | | `filesystems/register.py` | Auto-discovery for filesystem connectors | --- ## Operation: ADD Connector ### Step 1: Gather Requirements Ask the user for: 1. **Connector name** (e.g., "Redis", "MongoDB", "Wasabi") 2. **Connector type** (database, filesystem, or queue) 3. **Brief description** of use case ### Step 2: Research the Service Use web search to discover: 1. **Official Python library** for the service 2. **Existing fsspec provider** — check if a mature fsspec-compatible library exists (e.g., `s3fs` for S3, `adlfs` for Azure, `gcsfs` for GCS, `boxfs` for Box). **Always prefer wrapping an existing provider over building a custom one.** 3. **All supported authentication modes** (API key, OAuth, connection string, certificates, etc.) 4. **Required connection parameters** for each auth mode 5. **Official logo/icon** source Document findings before proceeding. If no fsspec provider exists, note that a custom `AbstractFileSystem` subclass will be needed (see Step 5b). ### Step 3: Generate Connector ID Create a unique connector ID using this pattern: ```python f"{short_name}|{uuid4()}" # Example: "redis|a1b2c3d4-e5f6-7890-abcd-ef1234567890" ``` ### Step 4: Create Directory Structure ```bash # For database connector mkdir -p unstract/connectors/src/unstract/connectors/databases/{connector_name}/static # For filesystem connector mkdir -p unstract/connectors/src/unstract/connectors/filesystems/{connector_name}/static # For queue connector mkdir -p unstract/connectors/src/unstract/connectors/queues/{connector_name}/static ``` ### Step 5: Create Connector Files #### 5a. Create `__init__.py` ```python from .{connector_name} import {ClassName} metadata = { "name": {ClassName}.__name__, "version": "1.0.0", "connector": {ClassName}, "description": "{Description of the connector}", "is_active": True, } ``` #### 5b. Create Main Connector Class Use the appropriate template from `assets/templates/`: - `database.py.template` for database connectors - `filesystem.py.template` for filesystem connectors - `queue.py.template` for queue connectors Read the template and adapt it for the specific service. Key methods to implement: **For Database Connectors:** - `get_engine()` → Return database connection - `sql_to_db_mapping()` → Map Python types to DB types - `execute()` → Execute queries (inherited, may override) **For Filesystem Connectors:** There are two approaches depending on whether an fsspec provider exists: **Approach 1: Wrapping an existing fsspec provider (PREFERRED)** When a library like `s3fs`, `adlfs`, `gcsfs`, `boxfs` exists, the connector only extends `UnstractFileSystem`: - `get_fsspec_fs()` → Return the provider's filesystem instance (e.g., `S3FileSystem(...)`) - `test_credentials()` → Verify connection works - `extract_metadata_file_hash()` → Extract file hash from metadata - `is_dir_by_metadata()` → Check if path is directory - `extract_modified_date()` → Extract last modified date - **Do NOT implement `ls`, `walk`, `info`, etc.** — the provider handles all file operations. - See `minio.py` for reference. **Approach 2: Custom AbstractFileSystem (ONLY when no provider exists)** Create a custom `AbstractFileSystem` subclass with ONLY these core methods: - `ls(path, detail=True)` — List directory contents (REQUIRED) - `info(path)` — Get single item metadata (optimization, optional) - `_open(path, mode)` — Open a file for reading - `cat_file(path)` — Read file contents - `pipe_file(path, value)` — Write bytes (NOT `write_bytes` — that's an alias) - `rm(path)` — Delete a file - `mkdir(path)` — Create directory **CRITICAL: NEVER override** `listdir()`, `stat()`, `exists()`, `isdir()`, `isfile()`, `delete()`, `read_bytes()`, or `write_bytes()`. These are provided by `AbstractFileSystem` and delegate to `ls()`/`info()`/`pipe_file()`. **`walk()` exception**: Override `walk()` ONLY if the filesystem has path normalization that differs from `_strip_protocol` (e.g., SharePoint maps `""` to `"root"`), or if the service raises exceptions that aren't `FileNotFoundError`/`OSError` (the only types the base catches). If you override `walk()`, you MUST support the `detail` kwarg (return dicts when `True`, lists when `False`) and the `on_error` callback. - See `sharepoint.py` for reference. **For Queue Connectors:** - `get_engine()` → Return queue connection - `enqueue()` → Add message to queue - `dequeue()` → Get message from queue - `peek()` → View next message without removing #### 5c. Create JSON Schema Generate `static/json_schema.json` based on researched auth modes: ```json { "title": "{Connector Display Name}", "type": "object", "allOf": [ { "required": ["connectorName"], "properties": { "connectorName": { "type": "string", "title": "Name of the connector" } } }, { "oneOf": [ // Add each auth mode as a separate option ] } ] } ``` Use `"format": "password"` for sensitive fields. ### Step 6: Fetch Logo Execute the logo fetch script: ```bash python .claude/skills/connector-ops/scripts/fetch_logo.py "{service_name}" "{output_path}" ``` The script tries these sources in order: 1. **SimpleIcons** (simpleicons.org) 2. **Devicon** (devicon.dev) 3. **Logo.dev API** 4. **Web search** for official logo 5. **Skip** if not found (log warning) Place logo at: `/frontend/public/icons/connector-icons/{ConnectorName}.png` ### Step 7: Add Dependencies Research required Python packages and add to `pyproject.toml`: ```bash # Read current dependencies cat unstract/connectors/pyproject.toml # Add new dependency in the [project.dependencies] section # Follow existing version pinning patterns (e.g., ~=, ==, >=) ``` ### Step 8: Create Tests Generate both mock-based and integration tests in `unstract/connectors/tests/`: #### Mock-based Test (always runnable): ```python import unittest from unittest.mock import patch, Mock class Test{ClassName}(unittest.TestCase): def setUp(self): self.config = { # Test configuration } @patch("{module_path}.{connection_method}") def test_connection_params(self, mock_connect): mock_connect.return_value = Mock() connector = {ClassName}(self.config) # Assertions ``` #### Integration Test (requires real service): ```python import os import unittest class Test{ClassName}Integration(unittest.TestCase): @unittest.skipUnless( os.getenv("{CONNECTOR}_HOST"), "Integration test requires {CONNECTOR}_* environment variables" ) def test_real_connection(self): config = { "host": os.getenv("{CONNECTOR}_HOST"), # ... other env vars } connector = {ClassName}(config) self.assertTrue(connector.test_credentials()) ``` ### Step 9: Verify Run verification in order: ```bash # 1. Syntax/type check cd unstract/connectors && python -m py_compile src/unstract/connectors/{type}/{name}/{name}.py # 2. Run mock tests cd unstract/connectors && python -m pytest tests/{type}/test_{name}.py -v # 3. Run full connector test suite cd unstract/connectors && python -m pytest tests/ -v --ignore=tests/{type}/test_{name}_integration.py ``` ### Step 10: Report to User Provide summary: ``` ## Connector Added: {Name} **Files created:** - `src/unstract/connectors/{type}/{name}/__init__.py` - `src/unstract/connectors/{type}/{name}/{name}.py` - `src/unstract/connectors/{type}/{name}/static/json_schema.json` - `tests/{type}/test_{name}.py` - `tests/{type}/test_{name}_integration.py` - `frontend/public/icons/connector-icons/{Name}.png` **Dependencies added:** - `{package}~={version}` **Verification:** - Syntax check: PASSED - Mock tests: PASSED (X tests) - Test suite: PASSED **To run integration tests:** ```bash export {CONNECTOR}_HOST=your_host export {CONNECTOR}_USER=your_user export {CONNECTOR}_PASSWORD=your_password cd unstract/connectors && python -m pytest tests/{type}/test_{name}_integration.py -v ``` ``` --- ## Operation: REMOVE Connector ### Step 1: Identify Connector Locate the connector by name or ID: ```bash # Search for connector grep -r "class {Name}" unstract/connectors/src/ ``` ### Step 2: Check Dependencies Search for usages across the codebase: ```bash grep -r "{connector_id}" --include="*.py" . grep -r "from.*{connector_name}" --include="*.py" . ``` Warn user if connector is referenced elsewhere. ### Step 3: Remove Files ```bash # Remove connector directory rm -rf unstract/connectors/src/unstract/connectors/{type}/{name}/ # Remove tests rm -f unstract/connectors/tests/{type}/test_{name}*.py # Remove icon rm -f frontend/public/icons/connector-icons/{Name}.png ``` ### Step 4: Clean Dependencies (Optional) If the removed connector was the only user of a dependency, offer to remove it from `pyproject.toml`. ### Step 5: Verify ```bash # Ensure no import errors cd unstract/connectors && python -c "from unstract.connectors.connectorkit import Connectorkit; Connectorkit()" # Run test suite cd unstract/connectors && python -m pytest tests/ -v ``` ### Step 6: Report to User ``` ## Connector Removed: {Name} **Files deleted:** - `src/unstract/connectors/{type}/{name}/` (directory)