large-data-storage-planning

name	large-data-storage-planning
title	Large-scale data storage strategy with cost, free-at-scale levers, and preprocessing
description	Class-level workflow for minimizing burn on PB-scale storage while preserving reliability and processing throughput.
version	1.0.0
author	Hermes
license	MIT

Large-scale data storage strategy with cost, free-at-scale levers, and preprocessing

Scope

Use this skill whenever the user asks for multi-PB/EB-class storage strategy, cheapest-at-scale architecture, or ways to reduce storage burn through preprocessing/credits.

Core rule

At 12 PB order, free tiers are not a strategy — they are a bridge. If a requirement implies “for free at scale,” only realistic paths are:

1. Make eligible data public and apply for open-data sponsorship/public hosting routes,
2. Stack startup credits/partner grants for storage and processing,
3. Reduce bytes before persistence (dedupe/compression/format transforms).

Decision framework (always use)

1. Classify first (mandatory before pricing)
   • Public vs private data.
   • Hot/active vs cold/archival.
2. Compute TCO, not headline rate
   • Storage rate ($/TB-mo)
   • retrieval + egress penalties
   • API/transaction costs
   • minimums and hidden floor charges
   • durability / replication assumptions
3. Prefer cheapest private archival for the immutable bulk, with a small hot/cache slice for active work.
4. Only keep “free” as a primary lever if it changes runway (credits/sponsorship consuming spend where possible).

Baseline architecture pattern

• Layer 1: Cold private archive
  • Archive class or equivalent with strict lifecycle policies.
  • If ops capacity is high and risk tolerance allows, tape-class/hybrid patterns may beat cloud archive.
• Layer 2: Hot/cook layer
  • Small, high-velocity object storage for preprocessing and short-term compute windows.
• Layer 3: Public route (if eligible)
  • Move open datasets into sponsorship/public channels to reduce private paid footprint.

“Cheapest realistic” sequence

1. Run classify + preprocess pass first.
2. Submit sponsorship/public-data options for eligible slices.
3. Apply all feasible startup credit/grant programs in parallel.
4. Finalize private cold tier only for restricted or sensitive data.

Proven execution template

1. Slice dataset by sensitivity and access pattern.
2. Estimate current monthly footprint for hot + cold.
3. Run credits/sponsorship timeline (who, what, deadlines, docs).
4. Build cost model: monthly_cost = TB * rate + egress + retrieval + ops overhead.
5. Pilot one shard through chosen architecture for one-week burn validation.
6. Expand only after retrieval/reduce loop confirms expected cost and latency.

Processing lever (often bigger than provider choice)

• Parse/clean before cold write.
• Deduplicate aggressively via content hash.
• Transcode to compressed archival-friendly formats.
• Drop temporary artifacts and keep only canonical artifacts.
• Enforce immutable manifests and checksums.

Frequent mistakes (pitfalls)

• Using free tiers as scale strategy (10–25 GB quotes are noise at PB scale).
• Ignoring retrieval economics (egress or rehydrate can erase apparent savings).
• Overbuilding hot layer and carrying unnecessary active storage for months.
• No credits-first execution plan, especially in startups/AI startup programs.
• Assuming managed object prices are all that matter and skipping region, minimum duration, and API charges.

Verification workflow

For each candidate stack, capture and archive:

• provider pricing excerpts (storage + retrieval + egress + minimums),
• credit program terms and cap amounts,
• 12 PB total monthly burn and credits runway in a one-page comparison.
• Decision rationale: why this option is “lowest-risk + lowest-burn” for this dataset profile.

Maintenance

• Keep all numeric evidence in references/ files.
• If new provider classes or credit programs appear, append them to:
  • references/provider-pricing-notes.md
  • references/credit-programs.md

References

• Start with references/provider-pricing-notes.md for session-specific evidence and rate sanity checks.