| name | Dremio Python Libraries |
| description | Enables an AI agent to use dremio-simple-query (lightweight SQL/Arrow Flight) and DremioFrame (full dataframe builder with ingestion, modeling, and admin) to interact with Dremio from Python. |
Dremio Python Skill
This skill covers two Python libraries for working with Dremio. Choose the right tool for the job:
| Need | Use |
|---|
| Run SQL and get results as Arrow, Pandas, Polars, or DuckDB | dremio-simple-query |
| Data ingestion, CRUD, Iceberg management, admin, modeling, charting, orchestration | DremioFrame |
Part 1: dremio-simple-query (Lightweight SQL)
A lightweight library that uses Apache Arrow Flight for high-performance SQL queries against Dremio.
Installation
pip install dremio-simple-query
Credential Configuration
Option 1: Profiles (Recommended)
Both dremio-simple-query and DremioFrame share the same profile file at ~/.dremio/profiles.yaml. Create it with the following structure:
profiles:
my_cloud:
type: cloud
base_url: https://api.dremio.cloud
auth:
type: pat
token: MY_PAT_TOKEN
my_software_pat:
type: software
base_url: https://dremio.company.com
auth:
type: pat
token: MY_PAT_TOKEN
my_software_basic:
type: software
base_url: https://dremio.company.com
auth:
type: username_password
username: my_user
password: my_password
my_software_oauth:
type: software
base_url: https://dremio.company.com
auth:
type: oauth
client_id: MY_CLIENT_ID
client_secret: MY_CLIENT_SECRET
Then connect using a profile name:
from dremio_simple_query.connectv2 import DremioConnection
dremio = DremioConnection(profile="my_cloud")
Option 2: Environment Variables / Direct Parameters
from dremio_simple_query.connectv2 import DremioConnection
from os import getenv
from dotenv import load_dotenv
load_dotenv()
dremio = DremioConnection(
location=getenv("ARROW_ENDPOINT"),
token=getenv("DREMIO_TOKEN"),
project_id=getenv("DREMIO_PROJECT_ID")
)
dremio = DremioConnection(
location="grpc+tls://dremio.company.com:32010",
username="my_user",
password="my_password"
)
Agent prompt: "I need to connect to Dremio from Python. Are you using Dremio Cloud or Dremio Software? Do you have a PAT (Personal Access Token), or should we use username/password? Do you already have a ~/.dremio/profiles.yaml configured?"
Query & Output Methods
Always use the V2 client (dremio_simple_query.connectv2).
from dremio_simple_query.connectv2 import DremioConnection
dremio = DremioConnection(profile="my_profile")
stream = dremio.toArrow("SELECT * FROM my_table")
arrow_table = stream.read_all()
batch_reader = stream.to_reader()
df = dremio.toPandas("SELECT * FROM my_table")
df = dremio.toPolars("SELECT * FROM my_table")
duck_rel = dremio.toDuckDB("SELECT * FROM my_table")
result = duck_rel.query("my_table", "SELECT * FROM my_table").fetchall()
Querying Arrow results with DuckDB (zero-copy)
import duckdb
stream = dremio.toArrow("SELECT * FROM my_table")
my_table = stream.read_all()
con = duckdb.connection()
results = con.execute("SELECT * FROM my_table").fetchall()
Part 2: DremioFrame (Full-Featured Dataframe Builder)
A comprehensive Python library providing a dataframe builder interface for Dremio with CRUD, ingestion, admin, modeling, charting, orchestration, and AI features. Currently in alpha.
Installation
pip install dremioframe
pip install "dremioframe[image_export]"
Optional dependency groups are documented at:
https://github.com/developer-advocacy-dremio/dremio-cloud-dremioframe/blob/main/docs/getting_started/dependencies.md
Credential Configuration
Option 1: Profiles (Recommended — shared with dremio-simple-query)
Create ~/.dremio/profiles.yaml (same format as above), then:
from dremioframe.client import DremioClient
client = DremioClient()
client = DremioClient(profile="my_profile")
You can generate the profiles file using dremio-cli (from the dremio-cli-skill) or create it manually.
Option 2: Environment Variables
Create a .env file in your project:
DREMIO_PAT=your_dremio_cloud_pat_here
DREMIO_PROJECT_ID=your_dremio_project_id_here
Option 3: Direct Parameters
client = DremioClient()
client = DremioClient(
hostname="dremio.example.com",
pat="your_pat_here",
tls=True,
mode="v26"
)
client = DremioClient(
hostname="localhost",
username="admin",
password="password123",
tls=False,
mode="v25"
)
Core Usage
Querying Data (Dataframe Builder)
from dremioframe.client import DremioClient
client = DremioClient()
df = (client.table('finance.bronze.transactions')
.select("transaction_id", "amount", "customer_id")
.filter("amount > 1000")
.limit(5)
.collect())
df = client.query("SELECT * FROM finance.silver.customers")
(client.table('finance.bronze.transactions')
.group_by("customer_id")
.agg(total_spent="SUM(amount)")
.show())
customers = client.table('finance.silver.customers')
(client.table('finance.bronze.transactions')
.join(customers, on="transactions.customer_id = customers.customer_id")
.show())
df.mutate(amount_with_tax="amount * 1.08").show()
df.at_snapshot("123456789").show()
SQL Functions
from dremioframe import F
(client.table("finance.silver.sales")
.select(
F.col("dept"),
F.sum("amount").alias("total_sales"),
F.rank().over(F.Window.order_by("amount")).alias("rank")
)
.show())
Data Ingestion
client.ingest_api(
url="https://api.example.com/users",
table_name="finance.bronze.users",
mode="merge",
pk="id"
)
import pandas as pd
data = pd.DataFrame({"id": [1, 2], "name": ["A", "B"]})
client.table("finance.bronze.raw_data").insert(
"finance.bronze.raw_data", data=data, batch_size=1000
)
client.table("finance.silver.customers").merge(
target_table="finance.silver.customers",
on="customer_id",
matched_update={"name": "source.name", "updated_at": "source.updated_at"},
not_matched_insert={"customer_id": "source.customer_id", "name": "source.name"},
data=data
)
Export
df.to_csv("transactions.csv")
df.to_parquet("transactions.parquet")
Charting
(client.table('finance.gold.sales_summary')
.chart(kind="bar", x="category", y="total_sales", save_to="sales.png"))
Admin & Catalog
print(client.catalog.list_catalog())
client.admin.create_reflection(
dataset_id="...", name="my_ref", type="RAW", display_fields=["col1"]
)
print(df.explain())
Data Quality
df.quality.expect_not_null("customer_id")
df.quality.expect_row_count("amount > 10000", 5, "ge")
Documentation Reference
When you need more detail on a specific feature, read the relevant doc page from the repos below.
dremio-simple-query Docs
DremioFrame Docs — Getting Started
DremioFrame Docs — Data Engineering
DremioFrame Docs — Analysis & Visualization
DremioFrame Docs — Data Modeling
DremioFrame Docs — AI Capabilities
DremioFrame Docs — Orchestration
DremioFrame Docs — Admin & Governance
DremioFrame Docs — Reference & Performance
Agent Workflow
- Assess the use case:
- Simple SQL queries → use
dremio-simple-query
- Ingestion, CRUD, admin, modeling, charting → use
dremioframe
- Check credentials — Ask the user if they have
~/.dremio/profiles.yaml or env vars set up. Guide them through configuration.
- Install the library —
pip install dremio-simple-query or pip install dremioframe.
- Write Python code using the patterns above.
- Look up docs — If you need details on a specific feature, read the relevant documentation URL from the tables above using your URL-reading tools.
Tips
- Both libraries share the same
~/.dremio/profiles.yaml format — configure once, use everywhere.
- For
dremio-simple-query, always use the V2 client (dremio_simple_query.connectv2), not the legacy V1.
- DremioFrame is in alpha — features are rapidly evolving. Check the docs for the latest API.
- For Dremio Cloud, the Arrow Flight endpoint is typically
grpc+tls://data.dremio.cloud:443.
- For Dremio Software, the Arrow Flight port is typically
32010 (e.g., grpc+tls://dremio.company.com:32010).
- DremioFrame's
mode parameter matters: use "v26" for Software v26+ with PAT, "v25" for older username/password auth.
