pixeltable-skill

name	pixeltable-skill
description	Build multimodal AI applications with Pixeltable — declarative tables replace LangChain + pandas + vector DB with one system. Automates chunking, embedding, retrieval, tool-calling agents, and 25+ AI provider integrations (OpenAI, Anthropic, Gemini, etc.) via computed columns that run on insert. Use when building RAG pipelines, processing images/video/audio/documents, orchestrating LLM inference, or deploying agents with persistent memory. Covers incremental computation, version control, similarity search, FastAPI serving, and production patterns. Do NOT use for general Python or direct PostgreSQL administration.
license	Apache-2.0
metadata	{"author":"Pixeltable","version":"2.3.0","category":"data-infrastructure","tags":["multimodal","ai","data","tables","embeddings","rag","udf","video","audio","images","documents","agents","tools","fastapi","declarative","computed-columns","vector-search"],"documentation":"https://docs.pixeltable.com/","support":"https://github.com/pixeltable/pixeltable/discussions"}

What is Pixeltable?

Open-source Python library for declarative multimodal AI data infrastructure — tables, computed columns, embedding indexes, and 25+ AI providers in one interface.

pip install pixeltable (Python >= 3.10) | Docs | GitHub

STOP — Do NOT Use These With Pixeltable

Pixeltable replaces the entire imperative stack you're about to reach for. If you find yourself importing any of these, you are off-path:

1. Do NOT use LangChain / LlamaIndex / Haystack / LangGraph — Pixeltable has built-in document chunking (document_splitter), embedding indexes (add_embedding_index), retrieval (.similarity()), and tool-calling (pxt.tools() + invoke_tools()). There is zero reason to add a framework on top.
2. Do NOT use pandas as a working data store — Pixeltable tables ARE the store. Use .collect().to_pandas() only for final export/visualization, never as an intermediate.
3. Do NOT write for row in ...: loops calling AI models — wrap the call in a computed column. Pixeltable handles batching, rate limiting, caching, and retry automatically.
4. Do NOT install a separate vector database (Pinecone, Chroma, FAISS, Qdrant, Weaviate, pgvector) — call t.add_embedding_index(col, embedding=fn) and search with .similarity(). The index lives inside the table.
5. Do NOT write while not done: agent loops — model the agent as a table where inserting a row triggers the entire computed-column chain (LLM → tool selection → tool execution → final answer) declaratively.

See anti-patterns.md for the full 15-bias reference with wrong/right code examples.

---

Task Router

Jump to the right section based on what you're building:

If the user wants to...	Read
Create tables, insert data, query	Core Concepts (below) and core-api.md
Add AI-powered columns (summarize, classify, embed)	Computed Columns (below) and providers.md
Chunk documents, extract video frames, split audio	Views and Iterators (below) and core-api.md → Views
Build semantic search / embedding indexes	Embedding Indexes (below) and core-api.md → Embedding Indexes
Build a RAG pipeline	workflows.md → RAG Pipeline
Build a tool-calling agent	Tool-Calling Agent Pipeline (below) and workflows.md → Tool-Calling Agent
Build an agent with persistent memory	agents-memory-mcp.md — chat history, knowledge bank, user scoping
Use MCP tools with an agent	agents-memory-mcp.md → Adding MCP Tools
Use invoke_tools() with OpenAI, Groq, Gemini, Bedrock	agents-memory-mcp.md → Multi-Provider
Build a video RAG agent (video + search + agent)	video-rag-agents.md — dedicated combined recipe
Process video (frames, transcription, visual search)	workflows.md → Video Analysis Pipeline
Process images (classify, tag, search)	workflows.md → Image Classification and Search
Process audio (transcribe, summarize)	workflows.md → Audio Transcription
Wrangle data for ML training (label, version, export)	ml-data-pipeline.md — ingest, enrich, snapshot, PyTorch export
Export to PyTorch, Parquet, or pandas	ml-data-pipeline.md → Export for Training
Look up structured data with retrieval_udf	ml-data-pipeline.md → Retrieval UDFs
Retry failed computed columns	Error Handling (below) — recompute_columns()
Use agentic patterns (chaining, routing, parallelization, eval-optimize)	agentic-patterns.md — 6 patterns + 2 reasoning strategies
Run batch processing (ingest, compute, export, exit)	workflows.md → Batch Processing
Configure rate limits, media storage, API keys	core-api.md → Configuration
Export to CSV, JSON, Parquet, LanceDB	core-api.md → Export
Export to SQL databases (Postgres, Snowflake, SQLite)	core-api.md → Export to SQL
Share tables across teams (publish, replicate)	core-api.md → Data Sharing
Compare multiple AI providers	workflows.md → Multi-Provider Comparison
Build a FastAPI web app (hand-written endpoints)	workflows.md → FastAPI App Pattern
Serve tables/queries via FastAPIRouter (v0.6+)	workflows.md → FastAPIRouter and core-api.md → Serving
Serve via CLI (pxt serve + TOML config)	core-api.md → pxt serve
Store media in Pixeltable Cloud (pxtfs://)	core-api.md → Media Destinations
Write UDFs or query functions	UDFs / Query Functions (below) and core-api.md → UDFs
Use pxt.tools() and invoke_tools() for agents	Tool-Calling Agent Pipeline (below) and core-api.md → Tools and Agents
Avoid common mistakes (wrong imports, broken schemas, serialization)	Common Pitfalls (below) and core-api.md → Common Pitfalls
Understand what NOT to use with Pixeltable (LangChain, pandas, vector DBs)	anti-patterns.md — 15 training-distribution biases with wrong/right code
Look up a specific provider's import and output shape	providers.md → Quick Reference

Critical Warnings — Read Before Writing Code

1. openai.vision does not exist — use openai.chat_completions with image_url content blocks
2. Cast to pxt.String before embedding — use .text.astype(pxt.String) on AI function outputs before add_embedding_index
3. if_exists='ignore' won't fix bugs — if a computed column has wrong logic, you must drop_column() then recreate; re-running is a silent no-op
4. Import frame_iterator as a function — from pixeltable.functions.video import frame_iterator, NOT from pixeltable.iterators import FrameIterator
5. Use string= keyword in similarity — always t.col.similarity(string=query), not positional

See Common Pitfalls below for full details and code examples.

Starting a New Project

Scaffold a complete Pixeltable project from the Starter Kit in one command:

# Application templates (each builds on a structural pattern)
uvx pixeltable-new --template multimodal-rag my-kb        # serving + backend (web UI)
uvx pixeltable-new --template video-intel my-video-app    # serving (pure declarative)
uvx pixeltable-new --template agent my-agent              # serving + backend (web UI)
uvx pixeltable-new --template audio-intel my-podcast-app  # serving + backend (web UI)
uvx pixeltable-new --template content-pipeline my-pipe    # batch (pipeline.py)
uvx pixeltable-new --template data-lab my-dataset         # batch (export.py)

# Structural patterns (API/pipeline scaffolds)
uvx pixeltable-new myapp                # default: declarative serving pattern
uvx pixeltable-new myapp --backend      # FastAPI API scaffold (headless)
uvx pixeltable-new myapp --batch        # batch processing script with export_sql

# Discovery
uvx pixeltable-new --list               # show all patterns + templates

Each template builds on one of the three structural patterns (serving, backend, batch), so you already know how to run and deploy it.

Core Concepts

Tables and Column Types

import pixeltable as pxt

pxt.create_dir('my_project', if_exists='ignore')

t = pxt.create_table('my_project.documents', {
    'title': pxt.String,
    'content': pxt.String,
    'image': pxt.Image,
    'video': pxt.Video,
    'audio': pxt.Audio,
    'doc': pxt.Document,
    'metadata': pxt.Json,
    'score': pxt.Float,
    'count': pxt.Int,
    'is_active': pxt.Bool,
    'created_at': pxt.Timestamp,
}, if_exists='ignore')

Available types: String, Int, Float, Bool, Image, Video, Audio, Document, Json, Array, Timestamp, Date, UUID, Binary. Use pxt.Required[pxt.String] for non-nullable.

Tables with Auto-Generated Keys

Use uuid7() for auto-generated primary keys (recommended for production):

from pixeltable.functions.uuid import uuid7

t = pxt.create_table('my_project.items', {
    'content': pxt.String,
    'uuid': uuid7(),           # auto-generated on insert
    'timestamp': pxt.Timestamp,
}, primary_key=['uuid'], if_exists='ignore')

Inserting Data

t.insert([{'title': 'Doc 1', 'content': 'Hello world', 'score': 0.95}])  # list of dicts
t.insert(title='Doc 2', content='Single row', score=0.75)                 # keyword syntax
t.insert(source='path/to/data.csv')                                       # from file

Computed Columns

Auto-run on insert. Chain AI providers, UDFs, or expressions:

from pixeltable.functions.openai import chat_completions

t.add_computed_column(
    summary=chat_completions(
        messages=[{'role': 'user', 'content': t.content}],
        model='gpt-4o-mini'
    ).choices[0].message.content,
    if_exists='ignore'
)

t.add_computed_column(upper_title=t.title.upper(), if_exists='ignore')

Querying

results = t.select(t.title, t.score).collect()
results = t.where(t.score > 0.8).select(t.title, t.content).collect()
results = t.order_by(t.score, asc=False).limit(10).select(t.title).collect()
count = t.count()
df = t.select(t.title, t.score).collect().to_pandas()
items = list(t.select(title=t.title, score=t.score).collect().to_pydantic(MyModel))

Views and Iterators

Split rows into sub-rows (chunking, frame extraction, audio splitting):

from pixeltable.functions.document import document_splitter
from pixeltable.functions.video import frame_iterator
from pixeltable.functions.string import string_splitter
from pixeltable.functions.audio import audio_splitter

# Chunk documents into 300-token pieces (requires: pip install tiktoken)
chunks = pxt.create_view(
    'my_project.doc_chunks', t,
    iterator=document_splitter(t.doc, separators='token_limit', limit=300),
    if_exists='ignore'
)

# Extract video frames at 1 fps
frames = pxt.create_view(
    'my_project.video_frames', t,
    iterator=frame_iterator(t.video, fps=1.0),
    if_exists='ignore'
)

# Split text into sentences
sentences = pxt.create_view(
    'my_project.sentences', t,
    iterator=string_splitter(t.content, separators='sentence'),
    if_exists='ignore'
)

# Split audio into 30-second chunks
audio_chunks = pxt.create_view(
    'my_project.audio_chunks', t,
    iterator=audio_splitter(audio=t.audio, duration=30.0),
    if_exists='ignore'
)

# Filtered view (no iterator needed)
active = pxt.create_view(
    'my_project.active', t.where(t.is_active == True),
    if_exists='ignore'
)

Embedding Indexes and Similarity Search

from pixeltable.functions.huggingface import clip, sentence_transformer

embed_fn = clip.using(model_id='openai/clip-vit-base-patch32')
t.add_embedding_index('content', embedding=embed_fn, if_exists='ignore')

# Search
sim = t.content.similarity(string='search query')
results = t.order_by(sim, asc=False).limit(5).select(t.title, t.content, sim).collect()

# Image search with text (multimodal CLIP)
sim = t.image.similarity(string='a photo of a cat')