agentsop-selfhost-decision

# Self-host vs Managed-cloud Decision — A Project-Kickoff Rubric > **Overlay, not a deep dive.** This skill answers *where to run* (self-host vs managed), not *which engine* ([[agentsop-llm-engine-selection]]) or *how to build the app* ([[agentsop-dify]]). It fires first, at kickoff, and hands off to those once the side is chosen. --- ## 1. 何时激活 (When to Activate) ### 1.1 直接信号 (Direct triggers) - At kickoff you must decide **where an LLM or LLM platform runs**: a managed API/cloud (OpenAI / Anthropic / Bedrock / Dify Cloud) vs **your own GPUs / your own Docker** (vLLM, self-hosted Dify). - **Cost pressure**: monthly managed spend is climbing; someone says *"should we just run our own and stop paying per token?"* - **Compliance pressure**: a **data-residency / air-gap / regulated-data** requirement (finance, medical, gov, GDPR region-lock) appears and the managed path is suddenly in question. - You're comparing a self-hostable platform's tiers — e.g. **Dify Cloud Pro ($59) vs self-deployed Docker** [architjn.com/blog/dify-cloud-pricing-plans], or managed-vLLM-as-a-service vs your own H100s. ### 1.2 反向信号 (Skip this rubric when) - The decision is already *self-host*, and the open question is **which engine** → go to **[[agentsop-llm-engine-selection]]** (vLLM vs TGI vs SGLang vs TensorRT-LLM vs llama.cpp). - The decision is already *self-host Dify*, and the open question is **how to build/operate it** → go to **[[agentsop-dify]]**. - **Single user / hobby / one stream** — the answer is "just call the managed API"; no rubric needed. - **Training / fine-tuning** siting — different cost structure (burst GPU, spot, not steady-state serving). ### 1.3 心智门槛 (Mental check) > This rubric exists because the loud reflex — *"running our own is cheaper / more serious"* — is **true only above a volume crossover, and only if you have the ops capacity, and only if compliance hasn't already forced your hand.** The job is to evaluate the **gate before the slider**, and to cost the **ops burden**, not just the GPU. 判断公式: - 如果你的瓶颈是 **per-token spend at high, predictable volume** → self-host *may* win (run OP-1). - 如果你的瓶颈是 **data can't leave our boundary** → compliance gate decides (run OP-2), cost is secondary. - 如果你的瓶颈是 **we don't have anyone to run a GPU box at 3am** → managed wins regardless of the GPU math (run OP-3). --- ## 2. 核心心智模型 (Core Mental Model) ### 2.1 The trade, in one line > **Self-host trades ops burden for control + unit-cost-at-scale. Managed trades $/token for zero ops. The crossover is a function of two axes: volume and compliance.** - **Managed**: you pay **$/token**, marginal, no fixed cost, no ops. Cost scales *linearly* with usage and never sleeps. - **Self-host**: you pay a **fixed floor** (GPU + ops labor + infra) plus a tiny marginal cost. Cost is *flat-then-cheap-per-unit* — but only if utilization stays high. ### 2.2 Two axes, not one — and they are not symmetric ``` COMPLIANCE (a GATE — binary, evaluated FIRST) │ managed FORBIDDEN │ self-host (or in-region managed) MANDATORY ──────────────────┼──────────────────────────────────────────► VOLUME │ (a SLIDER — managed allowed │ below V*: managed cheaper continuous │ above V*: self-host cheaper crossover) │ (IF utilization high + ops capacity exists) ``` - **Compliance is a GATE**: binary, evaluated **before** cost. A hard data-residency / air-gap NO **fails the managed path regardless of volume** [dify Case 3]. (Caveat: a managed in-region / VPC / BAA tier can re-open the gate.) - **Volume is a SLIDER**: a continuous cost crossover `V*`. Below it managed wins; above it self-host wins — *conditionally*. ### 2.3 The crossover formula (OP-1) ``` managed_cost(V) = V × $/token_managed selfhost_cost(V) = (GPU + ops_labor + infra_fixed) + V × $/token_marginal V* = (GPU + ops_labor + infra_fixed) / ($/token_managed − $/token_marginal) ``` - **Below V***: managed total cost is lower → use managed. - **Above V***: self-host total cost is lower → self-host *if* the two side-conditions hold. - The term people forget is **`ops_labor`** — a fraction-of-FTE DevOps cost (backups, monitoring, upgrades, on-call). Cost it explicitly; it is rarely zero. ### 2.4 The two side-conditions that invalidate "above V* → self-host" 1. **Utilization** — idle GPUs invert the math. A GPU billed 24/7 but used 20% of the time has 5× the effective $/token. Self-host only wins at *high, steady* utilization. 2. **Ops capacity** — the GPU math assumes someone can actually run it. No DevOps owner → outage + opportunity cost erases the savings (OP-3). ### 2.5 The throughput feedback loop (OP-4) Self-host volume isn't free of limits. A platform has a **per-replica ceiling** — Dify's is **~10 QPS/pod**, gated by per-node DB queries [dify §6.2]; vLLM's is gated by KV-cache occupancy + preemption [vllm OP-5]. To hit volume `V` you need `replicas = peak_QPS / ceiling`, and **that replica count feeds back into the GPU term of `V*`.** High volume can need so many replicas that the crossover moves against you. ### 2.6 Reversibility is a design choice (OP-7) The decision is **not permanent**. Keep a **thin OpenAI-compatible API surface** on both sides so flipping managed↔self-host is a config change, not a migration. Define the **flip trigger** up front (spend > V* for N months; new compliance rule; lost ops owner). Re-evaluate quarterly — both your volume and the price/quality frontier move (May 2026 stamp; re-measure). --- ## 3. SOP 工作流 (Standard Operating Procedure) ``` [Step 0] Confirm this is a "where to run" question ├─ "which engine?" → [[agentsop-llm-engine-selection]], stop ├─ "how to build Dify?" → [[agentsop-dify]], stop └─ "managed vs our own?"→ continue [Step 1] COMPLIANCE GATE first (OP-2) ── binary, overrides cost ├─ Regulated / residency / air-gap / contract boundary? │ ├─ YES, and no managed in-region/BAA/VPC tier → self-host MANDATORY → Step 3 │ └─ YES, but a compliant managed tier exists → managed re-opened → Step 2 └─ NO → Step 2 [Step 2] VOLUME SLIDER (OP-1) ── compute the crossover ├─ Estimate monthly volume (reqs × tokens) ├─ managed_cost = V × $/token ├─ selfhost_cost = (GPU + OPS_LABOR + infra) + V × marginal ← include ops! ├─ V* = fixed / (managed_$tok − marginal_$tok) ├─ V below V* → managed wins → Step 5 (plan fallback) └─ V above V* → self-host candidate → Step 3 [Step 3] HONEST OPS-CAPACITY CHECK (OP-3) ── the trap door ├─ DevOps / SRE / on-call owner exists? ─ no → managed wins anyway → Step 5 ├─ Can run external Postgres/Redis/VectorDB + K8s? ─ no → managed or hire first └─ yes → Step 4 [Step 4] THROUGHPUT-HEADROOM CHECK (OP-4) ── re-feed into cost ├─ replicas = peak_QPS / per-replica_ceiling (~10 QPS/pod Dify; KV-bound vLLM) ├─ Re-run V* with the TRUE replica count (GPU term grows) ├─ still favorable → self-host → Step 5 └─ flipped unfavorable → reconsider managed or HYBRID (Step 4b) [Step 4b] HYBRID option (OP-5) ── if volume is spiky or tiered └─ self-host the steady floor + burst/overflow to managed; or compliance/premium → self-host, bulk → managed [Step 5] PLAN THE FALLBACK (OP-6, OP-7) ├─ Lock-in / license audit (same-image? egress? contract minimums?) ├─ Thin OpenAI-compatible abstraction so the flip is config, not migration ├─ Define the bidirectional flip trigger └─ Schedule quarterly re-evaluation ``` > The ordering is load-bearing: **gate (Step 1) before slider (Step 2) before capacity (Step 3) before throughput (Step 4)**. A compliance NO short-circuits everything; a missing ops owner short-circuits a favorable GPU cost. --- ## 4. 操作模型 (Operation Model: Trigger / Action / Output / Evidence) ### OP-1: Volume-crossover calculation - **Trigger**: cost is a stated pressure; you can estimate monthly volume. - **Action**: estimate `V = reqs/mo × (in+out tokens)`; `managed = V × $/tok`; `selfhost = (GPU + ops_labor + infra) + V × marginal`; solve `V* = fixed / (managed_$tok − marginal_$tok)`. Place projected `V` relative to `V*`. **Include the ops-labor term** — do not assume it's zero. - **Output**: a crossover `V*` and which side projected volume sits on, with ops labor explicit. - **Evidence**: small team / small usage → Cloud $59–159/mo beats 养 DevOps; millions of calls → self-host [dify Case 3 决策矩阵; architjn.com/blog/dify-cloud-pricing-plans]. ### OP-2: Compliance / data-residency gate - **Trigger**: data is regulated, customer-confidential, or contractually region/boundary-bound. - **Action**: ask whether any law/contract/cert (HIPAA, finance, GDPR residency, air-gap, gov) **forbids data leaving your boundary**. YES → managed path **fails** regardless of volume → self-host (or a managed in-region/BAA/VPC tier, if one satisfies the boundary — check before defaulting to self-host). Evaluate **before** the volume slider. - **Output**: a pass/fail on the managed path from compliance alone, decided first. - **Evidence**: 合规/数据驻留 (金融、医疗) → self-host; air-gapped/内网 → self-host [dify Case 3]. Multi-tenant prefix privacy → cache salting [vllm OP-4 caveat]. ### OP-3: Honest ops-capacity check - **Trigger**: self-host looks attractive on cost; validate you can actually run it. - **Action**: self-host is **not** "docker compose up done" — it's reverse proxy + HTTPS + backups + monitoring + upgrade strategy + on-call. Score the team honestly: is there a DevOps/SRE owner? Production self-host means **external** PostgreSQL/Redis/Vector DB (not the bundled containers), K8s + Helm or equivalent, a tested upgrade path. No ops capacity → the "cheaper at scale" GPU math is a trap (outage + opportunity cost). - **Output**: a go/no-go on self-host based on real ops capacity, with the unbundling requirement flagged. - **Evidence**: 自部署 != docker compose up; 生产 K8s+Helm + 外部 PostgreSQL/Redis/Vector DB [dify Case 3 实操要点]. vLLM needs `2+N` CPU cores per N GPUs or the API server bottlenecks before the GPU [vllm Anti-pattern 6]. ### OP-4: Throughput-headroom reality check (self-host) - **Trigger**: self-host chosen on cost grounds; confirm the stack sustains the volume that justified it. - **Action**: find the per-replica ceiling (Dify **~10 QPS/pod**, node-level DB-query bottleneck; vLLM KV-cache occupancy + preemption). Compute `replicas = peak_QPS / ceiling`, then **re-feed that replica count into OP-1's GPU+ops term**. If the replica count is large, re-run the crossover; the "cheaper at scale" assumption may not survive. - **Output**: a replica count required for the volume, fed back into `V*`. - **Evidence**: ~10 QPS/pod hard limit, 每节点单独 DB query [dify §6.2]; triage `num_requests_waiting` / KV occupancy / preemption [vllm OP-5]. ### OP-5: Hybrid — self-host baseline + managed burst - **Trigger**: volume justifies self-host for the baseline but traffic is spiky, or a premium/compliance tier coexists with a bulk tier. - **Action**: provision self-host for the **steady-state floor** (high utilization keeps GPU $/token low); route **burst/overflow to a managed API** so you don't over-provision idle GPUs for peaks. Optionally split by tier: compliance/premium → self-host; bulk/non-sensitive → managed. Keep one gateway so the split is config, not a rewrite. - **Output**: a baseline-self-host + burst-managed routing plan that keeps utilization high and caps over-provisioning. - **Evidence**: two-replica tiering (FP8 premium / AWQ free) [vllm Dilemma 2]; add a replica / data-parallel pod when SLA violated under load [vllm Step 7]. ### OP-6: Same-image / lock-in audit - **Trigger**: evaluating a self-hostable platform's paid tiers, or worried about managed-vendor lock-in. - **Action**: for self-hostable platforms, check whether **Community / Premium / Enterprise are the same Docker image** gated by env vars / license keys (then the open tier is often technically sufficient — the paywall buys *support*, not *capability*). For managed APIs, audit lock-in surface: proprietary model, proprietary fine-tune format, egress cost, contract minimums. Prefer vendor-neutral self-host engines (vLLM, open weights) so the door stays open both ways. Confirm the license permits your commercial use. - **Output**: a lock-in / license assessment per candidate, plus whether paid self-host tiers differ in substance. - **Evidence**: Community/Premium/Enterprise 用同一 Docker image, 差异在环境变量 [dify Case 3]; license 是 Apache-like "not really" — 确认条款 [dify §6.4]; TensorRT-LLM beats vLLM 30–50% but **locks you in** [vllm §7]. ### OP-7: Fallback / exit plan - **Trigger**: a decision is made (either way); de-risk before scaling on it. - **Action**: if managed, define the **flip-to-self-host** trigger (spend > V* for N months; new compliance rule) and keep an open-weights equivalent identified. If self-host, define the **flip-to-managed** trigger (ops/outage cost exceeds savings; lost DevOps owner) and wire a managed fallback into the gateway. Keep the abstraction **thin (OpenAI-compatible)** so the flip is config, not migration. Re-evaluate quarterly. - **Output**: a documented bidirectional flip trigger + a thin abstraction so reversing is cheap. - **Evidence**: staged, reversible engine choices [vllm §7 common pattern]; bottleneck-driven re-evaluable decision [dify §1 判断公式]. --- ## 5. 困境决策案例 (Dilemma Cases) ### Dilemma 1: Low volume, but data-residency requires self-host **Situation**: A regional healthcare startup runs a doc-grounded copilot. Volume is ~50k requests/mo — well **below** the cost crossover `V*`, where a managed API ($/token) would clearly be cheaper than buying and running a GPU. But patient data is bound by data-residency + air-gap rules. **Tension**: - The **volume slider** (OP-1) says *managed* — at this volume the GPU + ops floor is far more expensive than per-token spend. - The **compliance gate** (OP-2) says *managed is forbidden* — data cannot leave the boundary. **Resolution heuristic**: - **The gate overrides the slider.** Evaluate compliance first; a hard residency/air-gap NO fails the managed path *regardless of how cheap it is* [dify Case 3: 合规/数据驻留, air-gapped → self-host]. - **But right-size it.** Compliance forcing self-host does **not** license self-host *scale*. Provision the **smallest adequate** stack — single replica, modest GPU — and accept the ops burden as the cost of compliance, not as an excuse to over-build (OP-1's `V*` still tells you how *small* you can go). - **Check the escape hatch first**: if the managed provider offers an **in-region / VPC / BAA** tier that satisfies the boundary, that re-opens the (cheaper, zero-ops) managed path. Don't default to self-host before checking [OP-2]. **Evidence**: [dify Case 3 决策矩阵 (合规/数据驻留, air-gapped → self-host)]; [vllm OP-4 caveat (multi-tenant privacy / cache salting)]. ### Dilemma 2: Dify self-host vs cloud — when is running your own worth the ops cost? (canonical) **Situation**: A team is choosing between **Dify Cloud Pro/Team ($59–159/mo)** and **self-deploying Dify on Docker**. The pitch for self-host: "we own the data, and at scale it's cheaper than a subscription." **Tension**: - **Cost slider**: team ≤ 3, modest usage → Cloud ($59–159/mo) is **cheaper than diverting a DevOps engineer** [dify Case 3]. Millions of API calls → subscription cost **overruns infra** → self-host wins on raw unit cost [dify Case 3]. - **Hidden ops cost**: self-host is **not "docker compose up"** — it's reverse proxy + HTTPS + backups + monitoring + upgrades, and *production* needs **external** PostgreSQL/Redis/Vector DB (not the bundled containers), plus K8s + Helm [dify Case 3 实操要点]. - **Throughput ceiling**: Dify caps at **~10 QPS/pod** (per-node DB-query bottleneck) [dify §6.2] — so the "high volume" that justified self-host needs **many replicas**, which feeds back into the GPU+ops term and can move `V*` against you (OP-4).