메뉴
Efficient storage and retrieval of genomic variant data using TileDB. Scalable VCF/BCF ingestion, incremental sample addition, compressed storage, parallel queries, and export capabilities for population genomics.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
| name | tiledbvcf |
| description | Efficient storage and retrieval of genomic variant data using TileDB. Scalable VCF/BCF ingestion, incremental sample addition, compressed storage, parallel queries, and export capabilities for population genomics. |
| license | MIT license |
| metadata | {"version":"1.0","skill-author":"Jeremy Leipzig"} |
TileDB-VCF is a high-performance C++ library with Python and CLI interfaces for efficient storage and retrieval of genomic variant-call data. Built on TileDB's sparse array technology, it enables scalable ingestion of VCF/BCF files, incremental sample addition without expensive merging operations, and efficient parallel queries of variant data stored locally or in the cloud.
This skill should be used when:
Preferred Method: Conda/Mamba
# Enter the following two lines if you are on a M1 Mac
CONDA_SUBDIR=osx-64
conda config --env --set subdir osx-64
# Create the conda environment
conda create -n tiledb-vcf "python<3.10"
conda activate tiledb-vcf
# Mamba is a faster and more reliable alternative to conda
conda install -c conda-forge mamba
# Install TileDB-Py and TileDB-VCF, align with other useful libraries
mamba install -y -c conda-forge -c bioconda -c tiledb tiledb-py tiledbvcf-py pandas pyarrow numpy
Alternative: Docker Images
docker pull tiledb/tiledbvcf-py # Python interface
docker pull tiledb/tiledbvcf-cli # Command-line interface
Create and populate a dataset:
import tiledbvcf
# Create a new dataset
ds = tiledbvcf.Dataset(uri="my_dataset", mode="w",
cfg=tiledbvcf.ReadConfig(memory_budget=1024))
# Ingest VCF files (must be single-sample with indexes)
# Requirements:
# - VCFs must be single-sample (not multi-sample)
# - Must have indexes: .csi (bcftools) or .tbi (tabix)
ds.ingest_samples(["sample1.vcf.gz", "sample2.vcf.gz"])
Query variant data:
# Open existing dataset for reading
ds = tiledbvcf.Dataset(uri="my_dataset", mode="r")
# Query specific regions and samples
df = ds.read(
attrs=["sample_name", "pos_start", "pos_end", "alleles", "fmt_GT"],
regions=["chr1:1000000-2000000", "chr2:500000-1500000"],
samples=["sample1", "sample2", "sample3"]
)
print(df.head())
Export to VCF:
import os
# Export two VCF samples
ds.export(
regions=["chr21:8220186-8405573"],
samples=["HG00101", "HG00097"],
output_format="v",
output_dir=os.path.expanduser("~"),
)
Create TileDB-VCF datasets and incrementally ingest variant data from multiple VCF/BCF files. This is appropriate for building population genomics databases and cohort studies.
Requirements:
Common operations:
Query variant data with high performance across genomic regions, samples, and variant attributes. This is appropriate for association studies, variant discovery, and population analysis.
Common operations:
Export data in various formats for downstream analysis or integration with other genomics tools. This is appropriate for sharing datasets, creating analysis subsets, or feeding other pipelines.
Common operations:
TileDB-VCF excels at large-scale population genomics analyses requiring efficient access to variant data across many samples and genomic regions.
Common workflows:
TileDB-VCF Data Model:
Schema Configuration:
# Custom schema with specific tile extents
config = tiledbvcf.ReadConfig(
memory_budget=2048, # MB
region_partition=(0, 3095677412), # Full genome
sample_partition=(0, 10000) # Up to 10k samples
)
Critical: TileDB-VCF uses 1-based genomic coordinates following VCF standard:
Region specification formats:
# Single region
regions = ["chr1:1000000-2000000"]
# Multiple regions
regions = ["chr1:1000000-2000000", "chr2:500000-1500000"]
# Whole chromosome
regions = ["chr1"]
# BED-style (0-based, half-open converted internally)
regions = ["chr1:999999-2000000"] # Equivalent to 1-based chr1:1000000-2000000
Performance considerations:
TileDB-VCF seamlessly works with cloud storage:
# S3 dataset
ds = tiledbvcf.Dataset(uri="s3://bucket/dataset", mode="r")
# Azure Blob Storage
ds = tiledbvcf.Dataset(uri="azure://container/dataset", mode="r")
# Google Cloud Storage
ds = tiledbvcf.Dataset(uri="gcs://bucket/dataset", mode="r")
TileDB-VCF provides a command-line interface with the following subcommands:
Available Subcommands:
create - Creates an empty TileDB-VCF datasetstore - Ingests samples into a TileDB-VCF datasetexport - Exports data from a TileDB-VCF datasetlist - Lists all sample names present in a TileDB-VCF datasetstat - Prints high-level statistics about a TileDB-VCF datasetutils - Utils for working with a TileDB-VCF datasetversion - Print the version information and exit# Create empty dataset
tiledbvcf create --uri my_dataset
# Ingest samples (requires single-sample VCFs with indexes)
tiledbvcf store --uri my_dataset --samples sample1.vcf.gz,sample2.vcf.gz
# Export data
tiledbvcf export --uri my_dataset \
--regions "chr1:1000000-2000000" \
--sample-names "sample1,sample2"
# List all samples
tiledbvcf list --uri my_dataset
# Show dataset statistics
tiledbvcf stat --uri my_dataset
# Calculate allele frequencies
af_df = tiledbvcf.read_allele_frequency(
uri="my_dataset",
regions=["chr1:1000000-2000000"],
samples=["sample1", "sample2", "sample3"]
)
# Perform sample QC
qc_results = tiledbvcf.sample_qc(
uri="my_dataset",
samples=["sample1", "sample2"]
)
# Advanced configuration
config = tiledbvcf.ReadConfig(
memory_budget=4096,
tiledb_config={
"sm.tile_cache_size": "1000000000",
"vfs.s3.region": "us-east-1"
}
)
Open Source Documentation:
For Large-Scale/Production Genomics:
Getting Started:
When your genomics workloads outgrow single-node processing, TileDB-Cloud provides enterprise-scale capabilities for production genomics pipelines.
Note: This section covers TileDB-Cloud capabilities based on available documentation. For complete API details and current functionality, consult the official TileDB-Cloud documentation and API reference.
1. Create Account and Get API Token
# Sign up at https://cloud.tiledb.com
# Generate API token in your account settings
2. Install TileDB-Cloud Python Client
# Base installation
pip install tiledb-cloud
# With genomics-specific functionality
pip install tiledb-cloud[life-sciences]
3. Configure Authentication
# Set environment variable with your API token
export TILEDB_REST_TOKEN="your_api_token"
import tiledb.cloud
# Authentication is automatic via TILEDB_REST_TOKEN
# No explicit login required in code
Large-Scale Ingestion
# TileDB-Cloud: Distributed VCF ingestion
import tiledb.cloud.vcf
# Use specialized VCF ingestion module
# Note: Exact API requires TileDB-Cloud documentation
# This represents the available functionality structure
tiledb.cloud.vcf.ingestion.ingest_vcf_dataset(
source="s3://my-bucket/vcf-files/",
output="tiledb://my-namespace/large-dataset",
namespace="my-namespace",
acn="my-s3-credentials",
ingest_resources={"cpu": "16", "memory": "64Gi"}
)
Distributed Query Processing
# TileDB-Cloud: VCF querying across distributed storage
import tiledb.cloud.vcf
import tiledbvcf
# Define the dataset URI
dataset_uri = "tiledb://TileDB-Inc/gvcf-1kg-dragen-v376"
# Get all samples from the dataset
ds = tiledbvcf.Dataset(dataset_uri, tiledb_config=cfg)
samples = ds.samples()
# Define attributes and ranges to query on
attrs = ["sample_name", "fmt_GT", "fmt_AD", "fmt_DP"]
regions = ["chr13:32396898-32397044", "chr13:32398162-32400268"]
# Perform the read, which is executed in a distributed fashion
df = tiledb.cloud.vcf.read(
dataset_uri=dataset_uri,
regions=regions,
samples=samples,
attrs=attrs,
namespace="my-namespace", # specifies which account to charge
)
df.to_pandas()
Data Sharing and Collaboration
# TileDB-Cloud provides enterprise data sharing capabilities
# through namespace-based permissions and group management
# Access shared datasets via TileDB-Cloud URIs
dataset_uri = "tiledb://shared-namespace/population-study"
# Collaborate through shared notebooks and compute resources
# (Specific API requires TileDB-Cloud documentation)
Cost Optimization
Security and Compliance
✅ Migrate to TileDB-Cloud if you have:
Next Steps:
Deterministically query 78 public scientific, biomedical, materials science, regulatory, finance, and demographics databases through documented REST APIs. Use for reproducible lookups of compounds, genes, proteins, pathways, variants, clinical trials, patents, economic indicators, structures, astronomy objects, environmental records, or database-backed scientific facts when endpoints, filters, pagination, and provenance need to be explicit.
Hugging Face Transformers for loading Hub models, running pipeline inference, text generation, and Trainer fine-tuning on NLP, vision, audio, and multimodal tasks. Use when working with AutoModel, pipelines, tokenizers, or TrainingArguments—not for general ML outside the Transformers library.
Autonomously improve a real artifact (code, training recipe, agent harness, data pipeline, prompt) against an objective and an evaluator, using Hypothesis Tree Refinement (HTR) from the Arbor paper. Use this whenever someone wants to iteratively optimize something over many experiments without overfitting — e.g. "get my model's eval score up", "improve this agent/harness", "tune this pipeline", "beat the baseline on this benchmark", "run a search over approaches and keep the best", "do an MLE-bench / Kaggle-style optimization", or any long-horizon "make this artifact better and don't just memorize the dev set" task. Trigger it even when the user doesn't say "Arbor" or "hypothesis tree" but describes repeated experiment-and-evaluate loops, branching exploration of competing ideas, or worries about a dev/test gap. Runs Claude itself as the coordinator with subagent executors in isolated git worktrees; for the standalone `arbor` CLI tool see references/arbor-upstream.md.
Standard single-cell RNA-seq analysis pipeline. Use for QC, normalization, dimensionality reduction (PCA/UMAP/t-SNE), clustering, differential expression, visualization, and converting R-friendly single-cell formats such as Seurat or SingleCellExperiment RDS files into h5ad for Scanpy. Best for exploratory scRNA-seq analysis with established workflows. For deep learning models use scvi-tools; for data format questions use anndata.
Complete mass spectrometry analysis platform. Use for proteomics and metabolomics workflows—feature detection, peptide/protein identification, label-free and isobaric quantification, adduct/accurate-mass annotation, and complex LC-MS/MS pipelines. Supports extensive file formats and algorithms. For simple spectral comparison and small-molecule library matching use matchms.
Design experiments and studies BEFORE data is collected — choosing a design, randomizing, blocking, and laying out treatment combinations so the results will actually be interpretable. Use whenever someone is planning a study, asks how to assign subjects/samples to groups, mentions randomization, blocking, stratification, controls, factorial or fractional-factorial designs, design of experiments (DOE), screening many factors, response-surface optimization, crossover or repeated-measures or split-plot designs, cluster/group randomization, Latin squares, plate layouts, batch/run-order effects, replication vs. pseudoreplication, or sequential/adaptive/group-sequential designs. Trigger this even for informal phrasings like "how should I set up this experiment", "how do I avoid confounding", "what's the best way to test these 6 factors", or "assign these mice to conditions". For computing the sample size or power once the design is chosen, use statistical-power; for analyzing data already collected, use statistica