# Datus Database Adapters
Independent database adapter packages for Datus Agent.
## Overview
This repository contains database adapter implementations for Datus Agent, using a plugin architecture where each database is an independent package that can be installed on demand.
## Database SQL guidance
Adapters with a `datus.skills` entry point bundle a `db-<dialect>-sql` skill. Its
`SKILL.md` contains query and existing-data semantics; linked `references/` files
provide task-specific DDL, writes, loading and procedural guidance on demand.
Use an Agent with `skill_location` and skill-bundle reference reading support for
the complete guidance. The legacy SQL-notes hook returns only the short entrypoint
body; custom hosts can resolve its reference links relative to the skill directory
under `get_skills_dir()`. Coordinate Agent support before deploying these adapter
updates to hosts that need the operational references.
## Architecture
```
datus-agent (Core package - SQLite/DuckDB only)
↓
Plugin Adapters (Independent packages, install as needed)
├── datus-sqlalchemy (SQLAlchemy base layer)
│ ├── datus-mysql
│ ├── datus-starrocks
│ ├── datus-postgresql
│ ├── datus-hologres
│ ├── datus-doris
│ ├── datus-tidb
│ ├── datus-oracle
│ ├── datus-gaussdb
│ └── datus-dws
│
└── Native SDK Adapters
├── datus-athena
├── datus-snowflake
├── datus-clickzetta
├── datus-maxcompute
├── datus-hana (official Python hdbcli; limited local SQL analysis)
```
## Implemented Adapters
### 1. datus-sqlalchemy
SQLAlchemy base connector providing shared functionality for SQLAlchemy-based adapters.
**Installation**:
```bash
pip install datus-sqlalchemy
```
**Features**:
- SQLAlchemy engine and connection management
- Unified error handling
- Multiple result format support
- Connection pool management
- Metadata retrieval
---
### 2. datus-mysql
MySQL database adapter.
**Installation**:
```bash
pip install datus-mysql
```
**Features**:
- Full MySQL support
- INFORMATION_SCHEMA metadata queries
- SHOW CREATE statement support
- Complete CRUD operations
---
### 3. datus-starrocks
StarRocks database adapter (MySQL protocol compatible).
**Installation**:
```bash
pip install datus-starrocks
```
**Features**:
- Inherits all MySQL functionality
- Multi-Catalog support
- Materialized view support
- StarRocks-specific optimizations
---
### 4. datus-doris
Apache Doris database adapter (MySQL protocol compatible).
**Installation**:
```bash
pip install datus-doris
```
**Features**:
- Inherits MySQL query and connection functionality
- Multi-Catalog support with the built-in `internal` catalog
- Asynchronous materialized view metadata and DDL support
- Doris-specific migration and table-layout guidance
---
### 5. datus-snowflake
Snowflake database adapter (native SDK).
**Installation**:
```bash
pip install datus-snowflake
```
**Features**:
- Native Snowflake SDK
- Multi-database and schema support
- Tables, views, and materialized views
- Efficient SHOW commands
- Arrow format support
---
### 6. datus-clickzetta
ClickZetta Lakehouse database adapter (native SDK).
**Installation**:
```bash
pip install datus-clickzetta
```
**Features**:
- Native ClickZetta ZettaPark SDK
- Full SQL support (DDL, DML, DQL)
- Workspace and schema management
- Volume/Stage file operations
- Metadata discovery
- Connection pooling and session management
---
### 7. datus-maxcompute
Alibaba Cloud MaxCompute adapter (PyODPS).
**Installation**:
```bash
pip install datus-maxcompute
```
**Features**:
- Automatic two-level (`project.table`) and three-level (`project.schema.table`) namespace detection
- SQL execution through MaxCompute jobs with explicit per-project namespace hints
- Instance Tunnel reads in Arrow, pandas, list, and CSV formats, with bounded CSV iteration
- Tables, views, materialized views, DDL, columns, partitions, and sample-row metadata
- Manually triggered cloud integration coverage for both namespace models
---
### 8. datus-hologres
Alibaba Cloud Hologres adapter (PostgreSQL wire protocol).
**Installation**:
```bash
pip install datus-hologres
```
**Features**:
- Database, schema, and table namespace support
- Hologres internal and foreign table discovery
- Hologres storage properties in generated DDL
- PostgreSQL-compatible SQL execution with Hologres transaction safeguards
- Tiny TPC-H cloud integration coverage on a real Hologres instance
---
### 9. datus-oracle
Oracle Database adapter using python-oracledb Thin mode.
**Installation**:
```bash
pip install datus-oracle
```
**Features**:
- Oracle 19c-compatible schema, metadata, query, and transfer behavior
- Optional `DialectOperations` for row limits, count queries, types, identifiers, and batch writes
- Adapter-packaged Oracle SQL skill with an Agent 0.3.9 notes compatibility bridge
- Docker integration coverage with an unprivileged Oracle user
---
### 10. datus-gaussdb
GaussDB / openGauss adapter (PostgreSQL wire protocol).
**Installation**:
```bash
pip install datus-gaussdb
```
**Features**:
- Official `gaussdb` driver on Linux (sha256, md5, and sm3 authentication)
- Pure-Python `pg8000` driver on macOS and any platform (sha256 and md5 authentication)
- Optional `psycopg2` driver escape hatch for md5-authenticated servers
- TLS modes through `verify-full`, with `verify-ca` recommended for production
- Database and schema namespace support with PostgreSQL-compatible SQL execution
- Ephemeral Docker TLS integration coverage against an openGauss instance
---
### 11. datus-tidb
TiDB adapter (MySQL wire protocol).
**Installation**:
```bash
pip install datus-tidb
```
**Features**:
- Inherits MySQL connection, execution, and metadata handling
- TiFlash columnar replica state through `get_tiflash_replicas()`
- Flags the two DDL clauses TiDB accepts without honouring (`CHECK`, `FULLTEXT`)
- Adapter-packaged TiDB SQL skill covering unsupported constructs and TiFlash MPP push-down
- Four-container Docker integration coverage (PD, TiKV, TiDB, TiFlash) including columnar and MPP assertions
---
### 12. datus-dws
Huawei Cloud GaussDB(DWS) adapter (PostgreSQL wire protocol, MPP warehouse).
**Installation**:
```bash
pip install datus-dws
```
**Features**:
- Inherits the psycopg2 transport from `datus-postgresql`; no dialect or driver of its own
- Native `pg_get_tabledef()` DDL preserving orientation, compression, and all three distribution forms
- Strips cluster-specific `TO GROUP` and `TABLESPACE` clauses for migration targets
- ORA/TD/MySQL compatibility-mode probing, with ORA's empty-string and division semantics surfaced in migration notes
- DWS system schema filtering (`cstore`, `sys`, `pg_recyclebin`, `dbms_*`, `utl_*`)
- Adapter-packaged DWS SQL skill covering the silently-wrong ORA-mode constructs
- Independent ORA/TD/MySQL cloud integration jobs on a real DWS cluster
---
### 13. datus-athena
Amazon Athena adapter using the official AWS boto3 SDK.
**Installation**:
```bash
pip install datus-athena
```
**Features**:
- Glue Catalog, database, table, view and partition-column metadata
- Workgroup-aware S3 and managed result handling
- Query polling, idempotent submission, native cancellation and execution diagnostics
- Bounded API pagination with list, CSV, pandas and Arrow results
- Offline SDK Stubber coverage and opt-in real AWS acceptance tests
See [configuration, permissions and testing](datus-athena/README.md).
Real AWS acceptance passed in S3 and managed-results modes; see the
[validation record](datus-athena/AWS_TESTING.md#validation-record-2026-10-16).
---
## Development Guide
### Development Environment Setup
This repository uses workspace configuration to manage multiple adapter packages:
```bash
# Clone repository
git clone https://github.com/your-org/datus-db-adapters.git
cd datus-db-adapters
# Install all adapters (development mode)
uv sync
# Or use pip
pip install -e datus-sqlalchemy
pip install -e datus-mysql
pip install -e datus-starrocks
pip install -e datus-doris
pip install -e datus-tidb
pip install -e datus-snowflake
pip install -e datus-maxcompute
pip install -e datus-hologres
pip install -e datus-oracle
```
**Note**: The root `pyproject.toml` is only for development environment management. End users should install individual packages.
### Creating a New Adapter
1. **Choose Base Layer**:
- If database is SQL standard compatible, inherit from `datus-sqlalchemy`
- If native SDK is needed, inherit directly from `BaseSqlConnector`
2. **Create Package Structure**:
```bash
datus-<database>/
├── datus_<database>/
│ ├── __init__.py
│ ├── config.py
│ └── connector.py
├── tests/
├── pyproject.toml
└── README.md
```
3. **Implement Connector**:
```python
from datus.tools.db_tools.base import BaseSqlConnector
class MyDatabaseConnector(BaseSqlConnector):
def __init__(self, ...):
super().__init__(dialect="mydatabase")
# Initialize connection
# Implement required abstract methods
def execute_query(self, sql: str) -> ExecuteSQLResult:
...
```
4. **Configure Entry Point**:
```toml
[project.entry-points."datus.adapters"]
mydatabase = "datus_mydatabase:register"
```
5. **Registration Function**:
```python
# __init__.py
from datus.tools.db_tools import connector_registry
from .connector import MyDatabaseConnector
def register():
connector_registry.register("mydatabase", MyDatabaseConnector)
register()
```
## Using with Datus Agent
Adapters are automatically discovered via entry points:
```yaml
# config.yaml
namespace:
mydatabase:
type: mysql # or snowflake, starrocks, etc.
host: localhost
port: 3306
username: root
password: password
database: mydb
```
Datus Agent will automatically load the corresponding adapter.
---
## Contributing
Contributions of new database adapters are welcome!
1. Fork this repository
2. Create a new adapter package
3. Add tests and documentation
4. Submit a Pull Request
---
## License
Apache License 2.0