pixeltable-data

name	pixeltable-data
description	Pixeltable multimodal data workflows — table creation, computed columns, AI transformations, embedding indexes, and cross-modal queries.
version	0.1.0
author	Jero (LATTICE / MARPA Design Studios)
triggers	["create a pixeltable table","pixeltable workflow","multimodal data pipeline","pixeltable query","embedding index"]
tools	["Bash","Read","Write","Edit"]

Pixeltable Data

USE WHEN the user wants to build multimodal data pipelines with Pixeltable — creating tables for images/video/audio/text, adding computed columns with AI transformations, building embedding indexes, or querying across modalities.

What It Does

Creates and manages Pixeltable declarative data workflows that automatically handle versioning, lineage, and AI transformations for multimodal data (images, video, audio, text, documents).

Core Concepts

• Directory — Namespace for organizing tables (like a schema)
• Table — Typed multimodal data store with automatic versioning
• Computed Column — Column whose value is derived from other columns via functions
• View — Filtered/transformed view of a table (snapshot or live)
• Embedding Index — Vector similarity search on any column
• UDF — User-defined function for custom transformations

Table Creation

import pixeltable as pxt

# Create directory
pxt.create_dir("my_project")

# Create table with typed columns
t = pxt.create_table("my_project.documents", {
    "doc": pxt.Document,
    "title": pxt.String,
    "category": pxt.String,
    "timestamp": pxt.Timestamp,
})

# Image table
img_table = pxt.create_table("my_project.images", {
    "image": pxt.Image,
    "caption": pxt.String,
    "source": pxt.String,
})

# Video table with frame extraction
vid_table = pxt.create_table("my_project.videos", {
    "video": pxt.Video,
    "title": pxt.String,
})

Computed Columns (AI Transformations)

from pixeltable.functions import openai, huggingface

# Add AI-computed columns — these run automatically on insert
t.add_computed_column(
    summary=openai.chat_completions(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": t.doc.extract_text()}],
    ).choices[0].message.content
)

t.add_computed_column(
    embedding=openai.embeddings(
        model="text-embedding-3-small",
        input=t.title,
    ).data[0].embedding
)

# Image transformations
img_table.add_computed_column(
    thumbnail=img_table.image.resize((256, 256))
)

img_table.add_computed_column(
    description=openai.chat_completions(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image"},
                {"type": "image_url", "image_url": {"url": img_table.image}},
            ],
        }],
    ).choices[0].message.content
)

Embedding Index and Similarity Search

# Create embedding index
t.add_embedding_index("embedding", metric="cosine")

# Similarity search
results = t.order_by(t.embedding.similarity("search query"), asc=False).limit(10).collect()

Querying

# Filter and select
results = t.where(t.category == "research").select(t.title, t.summary).collect()

# Aggregation
counts = t.group_by(t.category).select(t.category, pxt.count()).collect()

# Cross-table joins via computed columns
# (Pixeltable handles lineage automatically)

Views

# Snapshot view (frozen at creation time)
v = pxt.create_view("my_project.recent_docs", t.where(t.timestamp > cutoff))

# Frame extraction view from video
frames = pxt.create_view(
    "my_project.video_frames",
    vid_table,
    iterator=pxt.iterators.FrameIterator(fps=1),
)

Installation

uv pip install pixeltable

Patterns

Document Processing Pipeline

t = pxt.create_table("project.docs", {"doc": pxt.Document, "source": pxt.String})
t.add_computed_column(text=t.doc.extract_text())
t.add_computed_column(chunks=t.text.chunk(chunk_size=500, overlap=50))
chunks_view = pxt.create_view("project.chunks", t, iterator=pxt.iterators.ChunkIterator("chunks"))
chunks_view.add_computed_column(embedding=openai.embeddings(model="text-embedding-3-small", input=chunks_view.chunk).data[0].embedding)
chunks_view.add_embedding_index("embedding", metric="cosine")

Multimodal RAG

# Query across text + image embeddings
text_results = text_table.order_by(text_table.embedding.similarity(query), asc=False).limit(5)
image_results = img_table.order_by(img_table.clip_embedding.similarity(query), asc=False).limit(5)

Rules

• Always create a directory before creating tables
• Computed columns run automatically on every insert — be mindful of API costs
• Use collect() to materialize query results into a DataFrame
• Embedding indexes require a computed column with vector output
• Pixeltable handles versioning automatically — every change is tracked