// Efficient storage and retrieval of genomic variant data using TileDB. Scalable VCF/BCF ingestion, incremental sample addition, compressed storage, parallel queries, and export capabilities for population genomics.
[HINT] Baixe o diretório completo da skill incluindo SKILL.md e todos os arquivos relacionados
| name | tiledbvcf |
| description | Efficient storage and retrieval of genomic variant data using TileDB. Scalable VCF/BCF ingestion, incremental sample addition, compressed storage, parallel queries, and export capabilities for population genomics. |
| license | MIT license |
| metadata | {"skill-author":"Jeremy Leipzig"} |
TileDB-VCF is a high-performance C++ library with Python and CLI interfaces for efficient storage and retrieval of genomic variant-call data. Built on TileDB's sparse array technology, it enables scalable ingestion of VCF/BCF files, incremental sample addition without expensive merging operations, and efficient parallel queries of variant data stored locally or in the cloud.
This skill should be used when:
Preferred Method: Conda/Mamba
# Enter the following two lines if you are on a M1 Mac
CONDA_SUBDIR=osx-64
conda config --env --set subdir osx-64
# Create the conda environment
conda create -n tiledb-vcf "python<3.10"
conda activate tiledb-vcf
# Mamba is a faster and more reliable alternative to conda
conda install -c conda-forge mamba
# Install TileDB-Py and TileDB-VCF, align with other useful libraries
mamba install -y -c conda-forge -c bioconda -c tiledb tiledb-py tiledbvcf-py pandas pyarrow numpy
Alternative: Docker Images
docker pull tiledb/tiledbvcf-py # Python interface
docker pull tiledb/tiledbvcf-cli # Command-line interface
Create and populate a dataset:
import tiledbvcf
# Create a new dataset
ds = tiledbvcf.Dataset(uri="my_dataset", mode="w",
cfg=tiledbvcf.ReadConfig(memory_budget=1024))
# Ingest VCF files (must be single-sample with indexes)
# Requirements:
# - VCFs must be single-sample (not multi-sample)
# - Must have indexes: .csi (bcftools) or .tbi (tabix)
ds.ingest_samples(["sample1.vcf.gz", "sample2.vcf.gz"])
Query variant data:
# Open existing dataset for reading
ds = tiledbvcf.Dataset(uri="my_dataset", mode="r")
# Query specific regions and samples
df = ds.read(
attrs=["sample_name", "pos_start", "pos_end", "alleles", "fmt_GT"],
regions=["chr1:1000000-2000000", "chr2:500000-1500000"],
samples=["sample1", "sample2", "sample3"]
)
print(df.head())
Export to VCF:
import os
# Export two VCF samples
ds.export(
regions=["chr21:8220186-8405573"],
samples=["HG00101", "HG00097"],
output_format="v",
output_dir=os.path.expanduser("~"),
)
Create TileDB-VCF datasets and incrementally ingest variant data from multiple VCF/BCF files. This is appropriate for building population genomics databases and cohort studies.
Requirements:
Common operations:
Query variant data with high performance across genomic regions, samples, and variant attributes. This is appropriate for association studies, variant discovery, and population analysis.
Common operations:
Export data in various formats for downstream analysis or integration with other genomics tools. This is appropriate for sharing datasets, creating analysis subsets, or feeding other pipelines.
Common operations:
TileDB-VCF excels at large-scale population genomics analyses requiring efficient access to variant data across many samples and genomic regions.
Common workflows:
TileDB-VCF Data Model:
Schema Configuration:
# Custom schema with specific tile extents
config = tiledbvcf.ReadConfig(
memory_budget=2048, # MB
region_partition=(0, 3095677412), # Full genome
sample_partition=(0, 10000) # Up to 10k samples
)
Critical: TileDB-VCF uses 1-based genomic coordinates following VCF standard:
Region specification formats:
# Single region
regions = ["chr1:1000000-2000000"]
# Multiple regions
regions = ["chr1:1000000-2000000", "chr2:500000-1500000"]
# Whole chromosome
regions = ["chr1"]
# BED-style (0-based, half-open converted internally)
regions = ["chr1:999999-2000000"] # Equivalent to 1-based chr1:1000000-2000000
Performance considerations:
TileDB-VCF seamlessly works with cloud storage:
# S3 dataset
ds = tiledbvcf.Dataset(uri="s3://bucket/dataset", mode="r")
# Azure Blob Storage
ds = tiledbvcf.Dataset(uri="azure://container/dataset", mode="r")
# Google Cloud Storage
ds = tiledbvcf.Dataset(uri="gcs://bucket/dataset", mode="r")
TileDB-VCF provides a command-line interface with the following subcommands:
Available Subcommands:
create - Creates an empty TileDB-VCF datasetstore - Ingests samples into a TileDB-VCF datasetexport - Exports data from a TileDB-VCF datasetlist - Lists all sample names present in a TileDB-VCF datasetstat - Prints high-level statistics about a TileDB-VCF datasetutils - Utils for working with a TileDB-VCF datasetversion - Print the version information and exit# Create empty dataset
tiledbvcf create --uri my_dataset
# Ingest samples (requires single-sample VCFs with indexes)
tiledbvcf store --uri my_dataset --samples sample1.vcf.gz,sample2.vcf.gz
# Export data
tiledbvcf export --uri my_dataset \
--regions "chr1:1000000-2000000" \
--sample-names "sample1,sample2"
# List all samples
tiledbvcf list --uri my_dataset
# Show dataset statistics
tiledbvcf stat --uri my_dataset
# Calculate allele frequencies
af_df = tiledbvcf.read_allele_frequency(
uri="my_dataset",
regions=["chr1:1000000-2000000"],
samples=["sample1", "sample2", "sample3"]
)
# Perform sample QC
qc_results = tiledbvcf.sample_qc(
uri="my_dataset",
samples=["sample1", "sample2"]
)
# Advanced configuration
config = tiledbvcf.ReadConfig(
memory_budget=4096,
tiledb_config={
"sm.tile_cache_size": "1000000000",
"vfs.s3.region": "us-east-1"
}
)
Open Source Documentation:
For Large-Scale/Production Genomics:
Getting Started:
When your genomics workloads outgrow single-node processing, TileDB-Cloud provides enterprise-scale capabilities for production genomics pipelines.
Note: This section covers TileDB-Cloud capabilities based on available documentation. For complete API details and current functionality, consult the official TileDB-Cloud documentation and API reference.
1. Create Account and Get API Token
# Sign up at https://cloud.tiledb.com
# Generate API token in your account settings
2. Install TileDB-Cloud Python Client
# Base installation
pip install tiledb-cloud
# With genomics-specific functionality
pip install tiledb-cloud[life-sciences]
3. Configure Authentication
# Set environment variable with your API token
export TILEDB_REST_TOKEN="your_api_token"
import tiledb.cloud
# Authentication is automatic via TILEDB_REST_TOKEN
# No explicit login required in code
Large-Scale Ingestion
# TileDB-Cloud: Distributed VCF ingestion
import tiledb.cloud.vcf
# Use specialized VCF ingestion module
# Note: Exact API requires TileDB-Cloud documentation
# This represents the available functionality structure
tiledb.cloud.vcf.ingestion.ingest_vcf_dataset(
source="s3://my-bucket/vcf-files/",
output="tiledb://my-namespace/large-dataset",
namespace="my-namespace",
acn="my-s3-credentials",
ingest_resources={"cpu": "16", "memory": "64Gi"}
)
Distributed Query Processing
# TileDB-Cloud: VCF querying across distributed storage
import tiledb.cloud.vcf
import tiledbvcf
# Define the dataset URI
dataset_uri = "tiledb://TileDB-Inc/gvcf-1kg-dragen-v376"
# Get all samples from the dataset
ds = tiledbvcf.Dataset(dataset_uri, tiledb_config=cfg)
samples = ds.samples()
# Define attributes and ranges to query on
attrs = ["sample_name", "fmt_GT", "fmt_AD", "fmt_DP"]
regions = ["chr13:32396898-32397044", "chr13:32398162-32400268"]
# Perform the read, which is executed in a distributed fashion
df = tiledb.cloud.vcf.read(
dataset_uri=dataset_uri,
regions=regions,
samples=samples,
attrs=attrs,
namespace="my-namespace", # specifies which account to charge
)
df.to_pandas()
Data Sharing and Collaboration
# TileDB-Cloud provides enterprise data sharing capabilities
# through namespace-based permissions and group management
# Access shared datasets via TileDB-Cloud URIs
dataset_uri = "tiledb://shared-namespace/population-study"
# Collaborate through shared notebooks and compute resources
# (Specific API requires TileDB-Cloud documentation)
Cost Optimization
Security and Compliance
✅ Migrate to TileDB-Cloud if you have:
Next Steps: