hermes-fly-ops

name	hermes-fly-ops
description	Operate and debug this Hermes Agent Fly.io deployment repo. Use when running or updating Makefile targets, validating .env and Fly config, syncing secrets, deploying Hermes or Open WebUI, smoke testing Cloudflare/OIDC/private networking/Modal, reading logs, or troubleshooting startup, auth, host-gate, model, or sandbox issues in this repository.

Hermes Fly Ops

Use this skill for repo-specific deployment and debugging work around the Hermes Agent private Fly app, Open WebUI public Fly app, Cloudflare Access/OIDC, and Modal sandbox execution.

Safety Rules

• Treat make sync-secrets, make deploy*, and make destroy as live operations. Run them only when the user asks for deployment, secret sync, or teardown.
• Do not print secret values from .env, Fly secrets, Modal tokens, OAuth client secrets, provider API keys, or shared bearer keys.
• Prefer targeted checks before live changes: make -n <target>, make validate, make check-oidc, sh -n <script>, and git diff --check.
• Preserve the production posture unless the user asks otherwise: Hermes private, Open WebUI public behind Cloudflare Access, Open WebUI OIDC enabled, local password login disabled, and Modal as the terminal backend.

Command Map

Use these Makefile targets instead of reconstructing long fly commands:

make help                         # list supported repo targets
make check-tools                  # verify fly, docker, and optional modal CLI
make validate                     # validate both fly.toml files
make check-oidc                   # verify Cloudflare OIDC discovery URL and JSON
make secrets-help                 # print required .env and Fly secret names
make domain-help                  # print Fly cert and Cloudflare domain steps
make create-apps                  # create Fly apps if missing
make create-volumes               # create persistent Fly volumes
make sync-secrets                 # import Hermes and Open WebUI secrets
make sync-hermes-secrets          # import Hermes-only secrets
make sync-openwebui-secrets       # import Open WebUI-only secrets
make deploy                       # deploy Hermes, then Open WebUI
make deploy-hermes                # deploy only Hermes
make deploy-openwebui             # deploy only Open WebUI with production auth defaults
make deploy-openwebui-oidc-bootstrap
make smoke                        # run all smoke checks
make smoke-hermes                 # private Hermes health check inside Fly
make smoke-modal                  # Modal auth check from deployed Hermes
make smoke-openwebui              # Open WebUI-to-Hermes model API check
make smoke-openwebui-hosts        # custom host reachable; raw fly.dev returns 404
make smoke-hermes-private-ips     # Hermes must not have public Fly IPs
make smoke-private-hermes         # public Hermes health endpoint must not succeed
make status                       # status for both Fly apps
make logs-hermes                  # stream Hermes logs
make logs-openwebui               # stream Open WebUI logs
make destroy DESTROY_CONFIRM=destroy-<HERMES_APP>-<OPENWEBUI_APP>

Normal Workflows

For local validation after edits:

sh -n hermes/scripts/fly-entrypoint.sh
sh -n openwebui/scripts/fly-entrypoint.sh
make validate
git diff --check

For first-time deployment:

cp .env.example .env
make check-tools
make create-apps
make create-volumes
make check-oidc
make sync-secrets
make deploy
make domain-help
make smoke

For a routine config or image update:

make validate
make deploy
make smoke

For Open WebUI OIDC migration from an existing password admin:

make deploy-openwebui-oidc-bootstrap
# User signs in once through OIDC as OPENWEBUI_ADMIN_EMAIL.
make deploy-openwebui
make smoke-openwebui-hosts

Do not leave production in bootstrap mode. The normal make deploy-openwebui target restores SSO-only defaults.

Debugging Tips

Start with make status, then choose the narrowest log or smoke target that matches the symptom.

OIDC discovery fails:

• Run make check-oidc.
• Confirm OPENID_PROVIDER_URL ends with /.well-known/openid-configuration.
• Do not use the Cloudflare team root, the self-hosted Access app URL, or /oauth/oidc/login as the provider URL.

Cloudflare reports Invalid redirect_uri:

• Confirm the Generic OIDC SaaS app allows exactly <OPENWEBUI_URL>/oauth/oidc/callback.
• Confirm OPENWEBUI_URL has no accidental path or trailing slash confusion; the Makefile derives OPENID_REDIRECT_URI from it.

OIDC returns to Open WebUI but login fails:

• Run make logs-openwebui.
• If logs mention OAuth state or mismatching_state, keep WEBUI_SESSION_COOKIE_SAME_SITE=lax; do not switch it to strict.
• If an existing local admin conflicts with the OIDC identity, use the bootstrap workflow once, then immediately redeploy production defaults.

Open WebUI is unreachable or not ready:

• Run make logs-openwebui and check the Caddy/Open WebUI startup path.
• Keep the Open WebUI FastAPI lifespan enabled; startup lifecycle sets readiness and initializes background services.
• If the custom host returns 404, check OPENWEBUI_URL and the deployed OPENWEBUI_PUBLIC_HOST.
• If the raw https://<OPENWEBUI_APP>.fly.dev/ host does not return 404, inspect openwebui/scripts/fly-entrypoint.sh and make smoke-openwebui-hosts.

Open WebUI cannot reach Hermes:

• Run make smoke-openwebui.
• Confirm HERMES_APP matches the deployed Hermes app name; Open WebUI calls http://<HERMES_APP>.internal:8642/v1.
• Confirm HERMES_API_SHARED_KEY was synced to Hermes as API_SERVER_KEY and to Open WebUI as OPENAI_API_KEY.

Hermes appears public:

• Run make smoke-private-hermes.
• Run make smoke-hermes-private-ips.
• Inspect hermes/fly.toml; it should not have http_service or public services.
• If make smoke-hermes-private-ips fails, remove the public Fly IP assignments before production.

Modal auth fails:

• Run make sync-hermes-secrets, then make smoke-modal.
• Confirm .env uses MODAL_TOKEN_ID and MODAL_TOKEN_SECRET; legacy aliases are only normalized at Makefile boundaries.
• Keep terminal.env_passthrough: [] unless a specific variable has been reviewed.

Model or provider calls fail:

• Confirm MODEL_DEFAULT, HERMES_MODEL_PROVIDER, and provider key variables in .env.
• For the default OpenRouter path, OPENROUTER_API_KEY must be synced to Hermes.
• Use Hermes logs for provider-side errors and Open WebUI logs for frontend/model-list errors.

Fly app or volume confusion:

• Remember fly.toml includes default app names, but Makefile deploy targets pass --app $(HERMES_APP) and --app $(OPENWEBUI_APP).
• Keep both apps single-instance unless Redis/shared DB and Hermes state handling are redesigned.
• Volumes are persistent; destructive rebuilds require the explicit make destroy DESTROY_CONFIRM=... flow.