vastai-data-handling

name	vastai-data-handling
description	Manage training data and model artifacts securely on Vast.ai GPU instances. Use when transferring data to instances, managing checkpoints, or implementing secure data lifecycle on rented hardware. Trigger with phrases like "vastai data", "vastai upload data", "vastai checkpoints", "vastai data security", "vastai artifacts".
allowed-tools	Read, Write, Edit, Bash(vastai:*), Bash(ssh:*), Bash(scp:*)
version	1.11.0
license	MIT
author	Jeremy Longshore <[e-mail]>
tags	["saas","vast-ai","compliance","data"]
compatibility	Designed for Claude Code

Vast.ai Data Handling

Overview

Manage training data and model artifacts securely on Vast.ai GPU instances. Covers data transfer, encryption, checkpoint management, and cleanup. Critical consideration: Vast.ai instances run on shared hardware operated by third-party hosts.

Prerequisites

• Vast.ai instance with SSH access
• Cloud storage (S3, GCS) for persistent artifacts
• Understanding of data sensitivity classification

Instructions

Step 1: Data Transfer Patterns

# Small datasets (<5GB): Direct SCP
scp -P $PORT -r ./data/ root@$HOST:/workspace/data/

# Large datasets (5-50GB): Compressed transfer
tar czf - ./data/ | ssh -p $PORT root@$HOST "tar xzf - -C /workspace/"

# Very large datasets (>50GB): Cloud storage staging
# Upload to S3/GCS first, then download on instance
ssh -p $PORT root@$HOST "aws s3 sync s3://bucket/dataset/ /workspace/data/"

Step 2: Encrypted Data Transfer

import subprocess, os

def encrypt_and_upload(local_path, host, port, remote_path, passphrase):
    """Encrypt data before transferring to Vast.ai instance."""
    encrypted = f"{local_path}.enc"
    # Encrypt with AES-256
    subprocess.run([
        "openssl", "enc", "-aes-256-cbc", "-salt", "-pbkdf2",
        "-in", local_path, "-out", encrypted,
        "-pass", f"pass:{passphrase}",
    ], check=True)

    # Transfer encrypted file
    subprocess.run([
        , , (port), encrypted,
        ,
    ], check=)

    
    subprocess.run([
        , , (port), ,
        
        
        
    ], check=)

    os.remove(encrypted)