메뉴
Sage Continuum & Waggle: edge computing platform — plugin dev, ECR submission, data APIs, job scheduling, node management, testing.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Use for any question about a codebase, its architecture, file relationships, or project content — especially when graphify-out/ exists, where the question should be treated as a graphify query first. Turns any input (code, docs, papers, images, videos) into a persistent knowledge graph with god nodes, community detection, and query/path/explain tools.
Validate and use packed sequences and long-context training in Megatron-Bridge, distinguishing offline packed SFT for LLMs from in-batch packing for VLMs, and applying the right CP constraints.
Run the full DEFT AOI improvement loop for NVIDIA TAO VisualChangeNet / ChangeNet PCB inspection models: baseline evaluate, RCA, Cosmos AnomalyGen / AMP synthetic defects, k-NN mining, retraining, and deployment gating until FAR / recall KPI targets are met. Use for prompts like "run the DEFT loop", "fine-tune until FAR below 0.1% at recall=100%", or "improve my AOI ChangeNet model with RCA and synthetic defects"; do not use for standalone TAO training, one-off inference, generic anomaly generation, or RCA-only analysis.
Use for VSS alert workflows — real-time monitoring, Alert-Bridge subscriptions, Slack notifications, incident queries, camera onboarding. Not for non-alert analytics.
Build and publish a Gradio demo on Hugging Face Spaces for a user-provided LoRA. Use when someone asks to create, generate, ship, or publish a Space, demo, Gradio app, or playground for a LoRA — including LoRAs for Qwen-Image, Qwen-Image-Edit, LTX-Video, Wan, FLUX, SDXL, or other diffusion base models. Also triggers when someone describes a LoRA they trained or hosts on the Hub and wants to share it. Covers picking the right base pipeline and `diffusers` inference recipe, designing a UI tailored to the LoRA's task and inputs (Union/multi-task control, edit, video, image, etc.), respecting model-card recommendations (trigger words, steps, guidance, LoRA scale, example inputs), and shipping to ZeroGPU hardware as a private Space by default.
| name | sage-waggle |
| description | Sage Continuum & Waggle: edge computing platform — plugin dev, ECR submission, data APIs, job scheduling, node management, testing. |
| tags | ["edge-computing","iot","sage","waggle","scientific-compute"] |
| triggers | ["User mentions Sage, Waggle, sagecontinuum, waggle-sensor, Beehive, Beekeeper, WES","Tasks involving edge computing plugins that publish sensor data","Querying environmental/scientific sensor data from distributed nodes","Scheduling jobs on edge nodes (sesctl, pluginctl)","Working with sage-data-client or pywaggle","Developing Docker-based edge plugins","Using the Sage MCP server"] |
| globs | ["*sage*","*waggle*","*plugin*sage*","*beehive*"] |
references/data-query-first-report-and-history.md, references/data-viz-and-honesty.md). Key refs: references/stack-architecture-map.md, references/pywaggle2-producer-consumer-architecture.md, references/frame-anchored-batch-consumers-and-watchers.md, references/frame-anchored-vs-window-pollers.md, references/node-ssh-access-and-gpsd-probe.md, references/node-identity-and-upload-contract.md, references/pywaggle2-nodeinfo-gps-design.md, references/wes-pod-config-and-manifest-exposure.md, references/pluginctl-sideload-and-node-build.md, references/node-sideload-test-and-restore.md, references/audio-plugin-multinode-testing.md, references/storage-upload-health-verification.md, references/ci-handoff-doc-discipline.md, references/thor-arm64-deploy-pipeline.md, references/scheduling-continuous-vs-oneshot-and-gpu-contention.md, references/producer-consumer-gpu-and-shared-mounts.md.| API | Endpoint | Auth | Notes |
|---|---|---|---|
| Data query (timeseries) | POST https://data.sagecontinuum.org/api/v1/query | None (public) | Beehive timeseries from plugin.publish(). NDJSON. Filters: plugin, name, vsn, sensor. See references/timeseries-data-query-api.md |
| Manifests (rich) | GET https://auth.sagecontinuum.org/manifests/ · .../manifests/<vsn> | None | Full node hardware + sensor URIs. Collection needs trailing /; single VSN slash optional. Prefer per-VSN (full list ≈2MB+). ?project= |
| Nodes (beta) | GET https://auth.sagecontinuum.org/api/v-beta/nodes/ · .../nodes/<vsn> | None | Flatter node card (type, site, partner, focus, modem). Filters: ?phase=, ?project__name= (comma=OR) |
| Edge Scheduler | https://es.sagecontinuum.org | Bearer token | Job submission/management |
| MCP Server (Sage) | https://mcp.sagecontinuum.org/mcp | None (read-only); Bearer for jobs | 29 Sage tools — references/mcp-tools.md |
| MCP Server (GitHub) | https://api.githubcopilot.com/mcp/ | Bearer GitHub PAT | Repos/issues/PRs/Actions — registry · references/github-mcp-server.md |
| MCP Server (Hugging Face) | https://huggingface.co/mcp | Bearer HF token | Models/datasets/Spaces/papers/docs/Jobs — docs · references/huggingface-mcp-server.md |
| MCP Server (Milvus SDK helper) | https://sdk.milvus.io/mcp/ | None (Accept: text/event-stream) | Prefer Milvus Lite + MilvusClient — MCP docs · Lite · references/milvus-sdk-helper-mcp.md |
| Portal | https://portal.sagecontinuum.org | Browser login | Node management, token generation |
| ECR (Edge Code Repo) | Portal: https://portal.sagecontinuum.org/apps · API: GET https://ecr.sagecontinuum.org/api/apps?public=true | List public: none | Public plugins/apps available to schedule. Per-app: /api/apps/<ns>/<name> · /api/apps/<ns>/<name>/<ver>. See references/ecr-public-apps-api.md |
Auth / manifest details + related routes (/computes/, /sensors/, …): references/auth-api-manifests-and-nodes.md. App source: waggle-auth-app.
Auth tokens: get from portal.sagecontinuum.org/account/access. Format: Authorization: Bearer {token}.
Camp default (Thor): prefer
sudo pluginctl build .→sudo pluginctl runfor on-node development. Start withreferences/pluginctl-camp-guide.md. Use rawpodman buildonly for ECR-bypass side-load workflows (seereferences/pluginctl-sideload-and-node-build.md).
Official docs (prefer these URLs when citing Sage):
references/sage-docs-index.md — pick a URL, then fetch the live page for full contentwaggle-sensor org): references/waggle-sensor-repos-index.md — summary + URL per public repo (private skipped)sagecontinuum org): references/sagecontinuum-repos-index.md — summary + URL per public repo (private skipped)references/nvidia-jetson-thor-docs-index.md — product + JetPack + Jetson Linux r39.2 Developer Guide (summary + URL; prefer Thor pages)references/nvidia-skills-index.md — NVIDIA/skills (jetson-*, DeepStream, TAO, …)references/duckdb-docs-index.md — duckdb.org/docs summary + URL (Python/CLI, SQL, CSV/Parquet, guides)Plugins are Docker containers using pywaggle — the official Python SDK for Waggle plugins (waggle.plugin, cameras, microphones, uploads). Prefer that GitHub repo as the source of truth for API, docs, and source layout (src/waggle/…).
Install:
pip install -U 'pywaggle[all]' # core + audio + vision
# or subsets: pywaggle | pywaggle[audio] | pywaggle[vision]
Docs / source:
Minimal pattern:
from waggle.plugin import Plugin
import time
with Plugin() as plugin:
while True:
value = read_sensor()
plugin.publish("env.measurement", value, meta={"units": "C"})
time.sleep(30)
app.py — main applicationDockerfile — based on waggle/plugin-base:1.1.1-base (ARM64) or -ml variant for GPU.dockerignore — excludes tests/, ecr-meta/, jobs/, pycache/, *.pyc, *.md, .git/ from Docker build contextsage.yaml — metadata (name, description, version, authors, resources)ecr-meta/ — ECR submission metadataoverview.md — pedantic/instructive documentation (mini tutorial style)jobs/ — per-plugin job YAMLs for edge scheduler deployment (self-contained)tests/ — self-contained test suite (local test with real GPU inference, test images, harness copy, run script). See references/testing-patterns.mdfrom waggle.data.vision import Camera
cam = Camera("bottom_camera") # Named (resolved via node data config — only on real Sage nodes)
cam = Camera("rtsp://admin:pass@192.168.1.100:554/h264Preview_01_main") # RTSP IP camera
cam = Camera("/path/image.jpg") # Static image — bare path, works on host only (NOT in Docker — see pitfall below)
cam = Camera("file:///path/image.jpg") # Static image in Docker (bare paths trigger WES config lookup → FileNotFoundError)
sample = cam.snapshot() # ImageSample: .data (numpy HWC RGB), .timestamp (ns), .save()
from waggle.data.audio import Microphone
mic = Microphone("my_microphone")
sample = mic.record(duration=5) # AudioSample: .data (numpy), .timestamp, .save()
For audio-based plugins (bird classification, sound event detection), the pattern differs from vision plugins:
Audio input modes:
Microphone("mic_name").record(duration=3) for named node microphonessounddevice or pyaudio for direct recording on host--audio-dir flag for batch testing against WAV/MP3 files--camera URL — extract audio from IP camera via ffmpeg. The --camera CLI flag accepts any ffmpeg-compatible audio source URL. The plugin auto-detects the stream type and adds appropriate ffmpeg flags. See references/camera-audio-capabilities.md for the full camera comparison. Supported patterns:
# Mobotix MxPEG (auto-detected by "faststream" + "MxPEG" in URL, adds -f mxg)
--camera "http://user:pass@IP/control/faststream.jpg?stream=MxPEG&needlength"
# RTSP (auto-detected by rtsp:// scheme, adds -rtsp_transport tcp)
--camera "rtsp://user:pass@IP/profile1/media.smp"
# Any HTTP audio stream
--camera "http://IP/audio.cgi"
Microphone class does NOT support network cameras — the plugin uses subprocess ffmpeg for camera audio.Key differences from vision plugins:
BirdNET V2.4 integration pattern (pip install birdnet):
import birdnet
# Load models (auto-download ~77MB acoustic + ~46MB geo on first use)
model = birdnet.load("acoustic", "2.4", "tf")
geo = birdnet.load("geo", "2.4", "tf")
# Species filtering by location + week
species = geo.predict(lat, lon, week=22, min_confidence=0.03)
species_set = species.to_set() # NOT to_list()
# File-based prediction
predictions = model.predict(
"recording.wav",
top_k=5,
default_confidence_threshold=0.25, # NOT min_confidence
sigmoid_sensitivity=1.0,
custom_species_list=species_set,
overlap_duration_s=0.0,
bandpass_fmin=0, # low-freq cutoff Hz (e.g. 150 to cut wind)
bandpass_fmax=15000, # high-freq cutoff Hz (e.g. 4000 for 8kHz camera mic)
batch_size=1, # increase for parallel processing of long recordings
)
df = predictions.to_dataframe()
# Columns: input, start_time, end_time, species_name, confidence
# species_name format: "Scientific name_Common name"
API pitfalls discovered:
model.predict() uses default_confidence_threshold, NOT min_confidencegeo.predict() returns GeoPredictionResult — use .to_set() (not .to_list())~/.local/share/birdnet/__version__ attribute on the birdnet modulebirdnet.load("acoustic", "2.4", "tf") — the "tf" backend uses TFLite on ARM64 (CPU only, no GPU)python:3.12-slim (not NVIDIA base) since BirdNET is CPU-only on ARM64Audio plugin app.py pattern (proven in birdnet repo):
BirdNETClassifier)--input <file> — read from a local audio file (testing)--camera <url> — capture from network camera via ffmpeg subprocessMicrophone--camera flag accepts any ffmpeg-compatible URL; auto-detects Mobotix MxPEG
(adds -f mxg) and RTSP (adds -rtsp_transport tcp). Credentials in URL are
masked in log output (splits on @, shows only the host portion).record_from_camera() function: spawns ffmpeg, converts to 48 KHz mono WAV in tmpdir,
raises RuntimeError on ffmpeg failure or empty output. Cleanup via shutil.rmtree in finally block.--dry-run for testing without pywaggle (import Plugin only when needed)--interval for gap between cycles + --num-recordings for bounded runs--num-recordings N (default 1) = run exactly N cycles then exit; --num-recordings 0 = loop forever (requires --interval > 0)--num_rec + --silence_int semantics:
old: analyze.py --num_rec 6 --sound_int 5 --silence_int 1
new: app.py --num-recordings 6 --duration 5 --interval 1env.detection.audio.<scientific_name> (lowercase, underscored)--lat/--lon default to -1 which triggers auto-detection from the node manifest (/etc/waggle/node-manifest-v2.json). --week defaults to auto (calculate BirdNET week from current date: (month-1)*4 + min(4, ceil(day/7.5)), range 1–48). Resolve both at startup before constructing the classifier, so each scheduler invocation gets the correct location and week. Use explicit values to override for testing (--lat 41.72 --lon -87.98 --week 25). Use --week -1 for year-round (no seasonal filter). The --week argparse type should be str (not int) to accept both auto and numeric values.argparse.ArgumentDefaultsHelpFormatter + grouped args (audio, model, location, runtime)finally blocks (record to tmpdir, shutil.rmtree after)run_cycle(plugin=None) + run_loop(plugin=None) —
eliminates the old duplicate run_once/run_with_plugin code pathsAudio plugin Dockerfile differs from vision plugins:
python:3.12-slim (not NVIDIA — BirdNET is CPU-only on ARM64)ffmpeg libsndfile1 libasound2-devRUN python3 -c "import birdnet; birdnet.load(...)"See references/audio-classification-models.md for the full model survey.
Camera device resolution (resolve_device() chain):
://) → WES data config lookup (/run/waggle/data-config.json, Sage nodes only)rtsp://, http://) → passed directly to cv2.VideoCapture()file://path → local fileNamed cameras (bottom_camera, top_camera) are aliases defined in the node's /run/waggle/data-config.json — a JSON array where each entry has match.id (the name) and handler.args.url (the actual RTSP/HTTP URL). This config is managed by WES and only exists on real Sage nodes.
For IP cameras not registered in WES (e.g. a Reolink on the same network), pass the RTSP URL directly to --stream. See references/camera-rtsp-patterns.md for vendor-specific URL formats, data-config.json format, Docker-on-Thor QA testing workflow, and troubleshooting.
Plugins support four input modes via argparse:
--stream <camera|rtsp|image> — live camera, RTSP URL, or single test image (default: bottom_camera)--image-dir <path> — batch-process all images in a directory (overrides --stream)--snapshot-url <http-url> — fetch a JPEG snapshot from an HTTP URL each cycle (overrides --stream). Works with Reolink CGI API, generic IP camera snapshot endpoints, or any URL returning a JPEG. Credentials go in the query string. See references/reolink-http-snapshot.md.--continuous Y|N — loop (camera/snapshot-url) or single-shot. WARNING: the main loop's if continuous != "Y": break must be guarded with and not using_image_dir — otherwise --image-dir mode only processes the first image. See Pitfalls.Note: Not all plugins implement --image-dir. YOLO and BioCLIP have it; vLLM only has --stream (accepts a camera name, RTSP URL, or single image path). For batch testing a plugin without --image-dir, loop in shell: for img in tests/test-images/*.jpg; do python3 app.py --stream "$img" --continuous N; done — but note each invocation may restart expensive resources (e.g. vLLM server subprocess).
When a plugin wraps a third-party ML library (Ultralytics YOLO, vLLM, etc.), expose the library's key tuning parameters as CLI flags rather than hardcoding them. Pattern:
model(frame, imgsz=args.imgsz, half=args.half, ...))Flag | Type | Default | Description columnshttps://docs.ultralytics.com/modes/predict/#inference-arguments)inputs: sectionExample YOLO flags exposed from Ultralytics: --imgsz (input resolution), --half (FP16), --max-det (max detections), --augment (TTA), --agnostic-nms (class-agnostic NMS). Each help string includes a link to Ultralytics docs so students can read the full parameter reference.
Use iter_image_dir() helper to yield (path, frame, timestamp) tuples from a directory, matching the Camera snapshot interface. Always filter by IMAGE_EXTENSIONS set. See templates/ml-plugin-app.py for the full pattern.
with Plugin() as plugin:
plugin.publish("env.temperature", 23.5)
plugin.publish("env.count.car", 12, timestamp=ts, meta={"camera": "bottom", "model": "yolov7"})
plugin.upload_file("annotated.jpg") # Upload large files (images, video)
Meta value rule: all values in meta={} must be str. pywaggle's valid_meta() raises ValueError on int/float/bool. Always str() wrap numeric meta.
When publishing aggregate records (e.g. env.count.total), include the full class breakdown in meta so each record is self-describing without cross-referencing per-class records:
classes_summary = ",".join(f"{c}:{n}" for c, n in sorted(counts.items()))
plugin.publish("env.count.total", total, timestamp=ts,
meta={"camera": source_name, "model": args.model,
"classes": classes_summary if classes_summary else "none",
"num_classes": str(len(counts))})
Result: {"name":"env.count.total","value":3,"meta":{"classes":"bottle:1,person:2","num_classes":"2",...}}
export PYWAGGLE_LOG_DIR=./test-run
python3 app.py --stream test-image.jpg --continuous N
cat test-run/data.ndjson # Published measurements captured here (NDJSON: one JSON object per line)
ls test-run/uploads/ # Uploaded files captured here as {timestamp}-{original_name}
PYWAGGLE_LOG_DIR is a built-in pywaggle feature (not custom code). When set, pywaggle redirects all plugin.publish() to data.ndjson and all plugin.upload_file() to uploads/ inside that directory instead of sending to Beehive. Without it, pywaggle tries to write to /run/waggle/uploads/ which only exists on real nodes — on dev machines you get PermissionError: [Errno 13] Permission denied: '/run/waggle'. The local test runners set it automatically; when running app.py directly, always export PYWAGGLE_LOG_DIR=./output/<name> first.
Important: all meta dict values MUST be strings. plugin.publish("topic", 42, meta={"count": str(n)}) — not {"count": n}.
For testing ML plugins, see references/testing-patterns.md. All tests require GPU and run real model inference — no mocked unit tests. Each plugin has a self-contained tests/ directory with its own test file, test images, harness copy, and run-tests.sh. A top-level tests/run-all-tests.sh auto-discovers and runs all plugin tests (GPU required). Test images live in plugins/<name>/tests/test-images/ (flat directory, committed to git). All plugins share the same set of real test images — no synthetic generation step needed.
docker run with -e PYWAGGLE_LOG_DIR=/output: data stays local. plugin.publish() writes to data.ndjson, plugin.upload_file() writes to uploads/ inside the mounted volume. Nothing reaches Beehive.pluginctl deploy (no PYWAGGLE_LOG_DIR): data goes to the real Sage pipeline. plugin.publish() sends measurements via RabbitMQ → Beehive → time-series DB (queryable at data.sagecontinuum.org). plugin.upload_file() sends files to the object store (Open Storage Network, S3-compatible). This is production data flow.Use docker run + PYWAGGLE_LOG_DIR for testing/debugging. Use pluginctl deploy for real deployments.
ssh waggle-dev-node-V032 # Dev nodes use V0xx format
sudo pluginctl build . # Build Docker image from current dir
sudo pluginctl deploy -n my-counter 10.31.81.1:5000/local/my-plugin # Deploy (use descriptive names — visible across Sage)
sudo pluginctl ps # List running plugins
sudo pluginctl logs <plugin-id> # View logs (sudo required on Thor)
sudo pluginctl rm <plugin-id> # Remove plugin
Note: ALL pluginctl commands require sudo on Thor (k3s kubeconfig is root-only).
| Image tag | Use case | Size (approx) | Arch |
|---|---|---|---|
1.1.1-base | Minimal Python, no ML | ~280MB | multi-arch |
1.1.1-ml | ML with CUDA | ~1.6GB arm64 / ~3.5GB amd64 | multi-arch |
1.1.1-ml-torch1.9.0 | PyTorch 1.9 | ~2.6GB arm64 / ~5.3GB amd64 | multi-arch |
1.1.1-ml-tensorflow2.3-arm64 | TensorFlow 2.3 | ~1.2GB | arm64 only |
1.1.1-ml-tensorflow2.3-amd64 | TensorFlow 2.3 | ~2.6GB | amd64 only |
1.1.1-ml-dev | Dev/debug ML | ~1.6GB | arm64 |
1.1.1-ros2-foxy | ROS2 robotics | varies | varies |
For modern ML models (YOLO 8+, BioCLIP, vLLM, transformers), use NVIDIA PyTorch images instead of waggle/plugin-base:
| Image | PyTorch | CUDA | GPU Support | Notes |
|---|---|---|---|---|
nvcr.io/nvidia/pytorch:25.08-py3 | 2.8 | 13.0 | Blackwell native: sm_110 (Thor) + sm_120/sm_121 (DGX Spark) | Recommended — covers both Thor and DGX Spark |
nvcr.io/nvidia/pytorch:25.04-py3 | 2.7 | 12.9 | sm_120/sm_121 only — NO sm_110 | ⚠️ Fails on Thor ("sm_110 is not compatible") |
nvcr.io/nvidia/pytorch:25.03-py3 | 2.7 | 12.8.1 | Blackwell sm_120 only | No Thor support |
nvcr.io/nvidia/pytorch:25.01-py3 | 2.6 | 12.8 | Blackwell sm_120 (first) | Minor caveats (cuSPARSELt), no Thor |
nvcr.io/nvidia/pytorch:24.06-py3 | 2.4 | 12.4 | Hopper max (sm_90) | ⚠️ WRONG for Blackwell — silently falls back to CPU |
nvcr.io/nvidia/l4t-pytorch:* | varies | varies | Jetson-specific | ARM64 only |
CRITICAL: On Blackwell GPUs, using 24.06-py3 causes PyTorch to silently fall back to CPU — inference appears to hang (extremely slow). On Thor nodes specifically, 25.04-py3 also fails (no sm_110 cubins). Always use 25.08-py3 or newer for any deployment targeting both DGX Spark and Thor. The 25.08 image requires driver R575+ (DGX Spark: 580.159, Thor: 580.00 — both compatible).
All NGC PyTorch containers are multi-arch (AMD64 + ARM64 SBSA) via manifests — docker pull auto-selects the right architecture. The pytorch/pytorch:* Docker Hub images are AMD64-only (no ARM64).
Pre-download model weights at build time to explicit paths (edge nodes may lack internet). Use curl -L -o /app/models/<name>.pt <url> for YOLO, huggingface-cli download for HF models. Do NOT rely on ultralytics auto-download (caches to ~/.config/Ultralytics/ — path changes between versions). Set TRANSFORMERS_OFFLINE=1 and HF_DATASETS_OFFLINE=1 in Dockerfile. See references/ml-plugin-patterns.md for baking patterns and the yolov7-fire ECR reference.
Each plugin's overview.md should end with a "Further Reading" appendix that helps students go beyond the plugin's scope. Topics to cover:
--model custom.ptThis is especially important for student-facing docs. The plugin demonstrates one use case; the appendix shows the frontier.
Each plugin should be independently explorable by a student. A student can copy any single plugin directory out of the monorepo and have everything needed to understand, test, and deploy it: app code, Dockerfile, metadata, documentation, job specs, and tests. Shared infrastructure (venv, top-level test runner) is the exception, not the rule. When in doubt about where a file belongs, put it inside the plugin directory.
Each plugin is submitted to ECR as an independent entry. Verify before submission: (1) app.py imports only stdlib + pip packages, no sibling plugin imports, (2) Dockerfile COPYs only requirements.txt and app.py from its own directory, (3) no relative paths reaching into other plugin dirs, (4) test_harness.py is copied (not symlinked) into each plugin's tests/ dir. Run grep -rn 'other-plugin-name' plugins/this-plugin/ to verify.
Each standalone plugin repo should have a Makefile with at minimum:
IMAGE := plugin-name
VERSION := 0.1.0
TAG := $(IMAGE):$(VERSION)
build: docker build -t $(TAG) .
test: build test-docker # default target: build + validate
test-docker: bash tests/run-tests.sh --docker
test-native: bash tests/run-tests.sh
clean: docker rmi $(TAG) 2>/dev/null || true
make test is the single command for build + validate. make help with self-documenting ## comment targets. Exit code 0/1 for CI integration. NOTE: no audio: or download: targets — all test assets (images AND audio) must be committed to git. No runtime downloads.
Each plugin repo should match this layout:
sage-<plugin>/
├── app.py — main application
├── Dockerfile — NVIDIA base, pip constraints, model bake, patch scripts
├── DOCKER-BUILD.md — build, test, deploy, ECR submission guide
├── DEPLOY-AND-RUN.md — pluginctl one-shot test + sesctl scheduled deployment guide
├── THOR-TESTING.md — quick start for Thor node testing
├── requirements.txt — pip dependencies
├── sage.yaml — ECR metadata, inputs, version
├── overview.md — pedantic tutorial-style documentation
├── .gitignore — tests/output/, output/, *.pt, __pycache__, ._*, .DS_Store
├── .dockerignore — tests/, ecr-meta/, jobs/, *.md, .git/
├── patch_pybioclip.py — (BioCLIP only) pybioclip monkey-patch
├── ecr-meta/ — ECR submission materials (6 files)
├── jobs/ — per-plugin job YAMLs
└── tests/
├── run-tests.sh — standalone test runner (no monorepo deps)
├── test_<name>_local.py — real GPU inference test
├── test_harness.py — test utilities (copied, not shared)
├── test-images/ — committed test images
└── output/ — gitignored test output
run-tests.sh must work standalone — no references to monorepo venv paths or parent directories. The test script should use the system Python or a local venv, not assume ../../tests/.venv exists.
waggle-sensor/edge-app-templatewaggle-sensor/cookiecutter-sage-appSage plugins follow a one-shot execution model: the scheduler fires a container, it processes, publishes, and exits. Pods are ephemeral — no persistent filesystem between runs. One-shot cron is standard; continuous pods are the exception (see references/job-scheduling-and-liveness.md). Publish a heartbeat every cycle or quiet jobs look dead. arm64/Thor: portal build crashes (QEMU), push denied — build local+sideload, see references/ecr-arm64-thor-deployment.md. Three scheduling modes:
| Mode | YAML | Use case |
|---|---|---|
| Cronjob | schedule: "*/10 * * * *" | Most common — periodic sampling |
| Lambda | when: {name: ..., cond: ...} | Data-driven triggers |
| Always | schedule: "always" | Continuous (rare, discouraged for GPU) |
k3s + containerd cache Docker image layers locally after first pull. Two plugins sharing the same base image (e.g. both on nvcr.io/nvidia/pytorch:24.06-py3) only download unique layers for the second one. Always use explicit version tags, never :latest — with :latest, every cron tick triggers a registry check.
Dockerfile layer ordering matters: place rarely-changing layers (base, requirements, model weights) BEFORE frequently-changing layers (app.py). Changing a layer invalidates everything below it. Put COPY app.py /app/ as the LAST layer so code changes don't trigger model re-downloads.
See references/runtime-packaging-patterns.md for full details: pod lifecycle timeline, containerd caching mechanics, imagePullPolicy defaults, cold-start optimization checklist, and analysis of 4 production reference plugins.
Official docs:
references/sesctl-ecr-validation.mdFormat: action : condition
Actions:
schedule(image) — run a plugin containerpublish(topic, value) — publish a messageset(variable, value=X) — set a variableCondition functions:
v(measurement, sensor=, since="-1h") — get measurement valuetime(unit) — current time ("hour", "minute", etc.)cronjob(name, crontab) — cron schedule (name must be unique per job)after(name, since="-1d") — true after a named eventrate(measurement, since, window, unit) — rate of changeExample rules:
schedule(object-counter): cronjob('run-counter', '*/5 * * * *')
schedule(object-counter): v('env.temperature') > 30.0
export SES_HOST=https://es.sagecontinuum.org
export SES_USER_TOKEN=<token-from-portal>
sesctl create -f job.yaml # flag is -f/--file-path, NOT --from-file; returns numeric job ID
sesctl stat # list jobs; sesctl stat -j <id> for one
sesctl submit -j <job-id> # submit/activate by numeric ID, NOT by name
sesctl rm -j <job-id> # remove by ID
See references/job-scheduling-and-liveness.md for: ECR app metadata vs Docker image (both must exist for SES — two distinct failure modes), one-shot cron vs continuous pods, pod namespace meaning (default=pluginctl, ses=scheduler), heartbeat/observability pattern, sesctl flag corrections, sage.yaml float type reality, avian-diversity-monitoring baseline schedules, and BioCLIP cold-start considerations.
create returns a numeric ID; capture it. Always run sesctl <subcmd> --help — this CLI's surface drifts from the published web docs.
Deployment model, namespace diagnostics, ECR gate, reading job schedules: see
references/deployment-and-diagnostics.md. Quick diagnostics: pods in thesesnamespace = scheduler-launched (official);defaultnamespace = hand-deployed via pluginctl (adefaultpod with multi-day uptime is a continuous test pod, NOT a scheduled job).sesctl submitrequires the app to be registered in the ECR catalog (portal build) even thoughpluginctl/docker pull succeed without it — hence400 ... does not exist in ECRon submit. Most Sage jobs are cron one-shot, not long-running; weigh cold-start vs warm-pod before choosing--continuous Y. Reolink audio auth (BirdNET etc.): FLV/BCS needs creds as query params cron-job liveness checks — seereferences/job-scheduling-and-liveness.md.**
Scalar / event data from Sage plugins lands in Beehive and is queried at:
POST https://data.sagecontinuum.org/api/v1/query (public, no auth) → NDJSON.
Full patterns: references/timeseries-data-query-api.md. Tutorial: Access and use data.
plugin-iiocurl https://data.sagecontinuum.org/api/v1/query \
-d '{"start":"-30m","filter":{"plugin":".*plugin-iio.*"}}'
import sage_data_client
df = sage_data_client.query(
start="-30m",
filter={
"plugin": ".*plugin-iio.*",
},
)
import sage_data_client
df = sage_data_client.query(
start="-1h",
filter={"name": "env.temperature", "vsn": "W030"}
)
# Returns pandas DataFrame with: timestamp, name, value, meta (sensor, vsn, node, plugin)
curl -s -X POST https://data.sagecontinuum.org/api/v1/query -d '
{
"start": "-1h",
"filter": {
"vsn": "H00F",
"name": "env.count.*"
}
}'
Filters: name (measurement), sensor (hardware), vsn (node ID), plugin (source plugin). Supports wildcards / regex (e.g. ".*plugin-iio.*").
Large files (images, audio): stored on Open Storage Network (S3-compatible object store), not in the timeseries DB.
references/cloud-trigger-notifications.md for the full pattern, Slack webhook + image upload examples, secret management, and reference implementations (hummingbird-watcher, wildfire-trigger, severe-weather-trigger).references/sage-data-web-viz.md · 3D globe: references/sage-data-3d-globe-viz.md · template: templates/sage-cors-proxy-server.pyreferences/sharing-a-sage-knowledge-bundle.mdsagecontinuum — public repo catalog: references/sagecontinuum-repos-index.md (sage-data-client, sage-gui, sage-storage-*, sage-object-store, …; private skipped)waggle-sensor — public repo catalog: references/waggle-sensor-repos-index.md; pywaggle SDK: https://github.com/waggle-sensor/pywaggle (also waggle-edge-stack, edge-scheduler / pluginctl+sesctl, plugin-base, virtual-waggle, sage-mcp)Wire up the Sage MCP server as a native Hermes tool so all 29 tools are callable directly (no curl/JSON-RPC):
# Non-interactive (no auth, enable all tools):
printf 'n\nY\n' | hermes mcp add sage --url "https://mcp.sagecontinuum.org/mcp"
# Verify:
hermes mcp list
After adding, start a new session. Tools appear as mcp_sage_* (e.g. mcp_sage_list_available_nodes, mcp_sage_find_plugins_for_task). No auth needed for read-only operations (data queries, node listing, plugin search, docs). Job submission tools (submit_sage_job, submit_plugin_job) require a Bearer token configured via portal.
Official server: MCP Registry — GitHub · remote endpoint https://api.githubcopilot.com/mcp/.
Shipped in profile mcp.json as github with enabled: false until you add a PAT. Setup: references/github-mcp-server.md.
hermes mcp add github --url "https://api.githubcopilot.com/mcp/"
# When prompted, use Authorization Bearer <GITHUB_PAT>
hermes mcp list
Use for live GitHub repos/issues/PRs (e.g. waggle-sensor/pywaggle). Prefer /mcp/readonly or a read-only PAT when you only need browse access.
Official server: Hugging Face MCP docs · remote endpoint https://huggingface.co/mcp.
Shipped in profile mcp.json as huggingface with enabled: false until you add an HF token. Toggle tools/Spaces at settings/mcp. Setup: references/huggingface-mcp-server.md.
hermes mcp add huggingface --url "https://huggingface.co/mcp"
# When prompted, use Authorization Bearer <HF_TOKEN>
hermes mcp list
Use for Hub models/datasets/Spaces/papers, HF documentation search, and Hub Jobs. Prefer Sage MCP for nodes/data; GitHub MCP for repos/PRs.
Official helper: milvus.io docs · endpoint https://sdk.milvus.io/mcp/ (header Accept: text/event-stream). Shipped enabled in mcp.json as sdk-code-helper. Setup notes: references/milvus-sdk-helper-mcp.md.
hermes mcp add sdk-code-helper --url "https://sdk.milvus.io/mcp/"
hermes mcp list
Use when generating vector-search code: prefer Milvus Lite (pip install -U "pymilvus[milvus-lite]", MilvusClient("./demo.db")) and current MilvusClient APIs — not full Milvus Standalone/Docker unless the user asks.
~/AI-projects/Sage-agents/sage-agents.md (15K+ bytes of detailed research)authors fields and ecr-meta credits.virtual-waggle (simulated node environment)hermes mcp add sage --url <url> prompts for auth token and tool filtering. Pipe printf 'n\nY\n' for no-auth, enable-all-tools. Must start new session after adding.0.1.0 → 0.2.0) across sage.yaml, job YAMLs, Dockerfiles, and docs, skip generic tutorial/example files that use the old version as a hypothetical placeholder (e.g. docs/sage-runtime-packaging-tutorial.md using my-plugin:0.1.0 as a generic example). Use replace_all=True per-file rather than a blind global sed to avoid corrupting unrelated examples./manifests/); single node is /manifests/<vsn> (slash optional). Prefer per-VSN; also use /api/v-beta/nodes/<vsn> for flatter site/type metadata — see references/auth-api-manifests-and-nodes.md-L (follow redirects): curl -L -u <username>:<portal-access-token> -o output.jpg <url> — token from portal account page. The Sage storage API at storage.sagecontinuum.org returns a 302 redirect to the actual backend (nrdstor.nationalresearchplatform.org) with a signed JWT in the query string. Without -L, curl gets an empty 302 response and writes a 0-byte file. Always use -L when downloading from Sage storage. In Python, use subprocess.run(["curl", "-s", "-f", "-L", "-u", ...]) rather than urllib — the NRP backend may do a double redirect that urllib can't follow.portal.sagecontinuum.org/account/access. Format: curl -u <portal-username>:<access-token>.plugin.publish() calls valid_meta() which enforces isinstance(v, str) for every meta dict value. Passing int/float meta values (e.g. {"confidence": 0.95}) silently raises ValueError. Always wrap: meta={"confidence": str(score), "count": str(n)}[a-z0-9_] joined by dots: YOLO COCO classes include multi-word names with spaces (dining table, traffic light, potted plant, hot dog, fire hydrant, stop sign, tennis racket, cell phone, teddy bear, etc.). Using these directly in plugin.publish(f"env.count.{cls_name}", ...) raises ValueError: publish name invalid. Sanitize before publishing: safe_name = cls_name.replace(" ", "_").replace("-", "_"). This bug is insidious — the per-class publish calls before the offending class succeed, but env.count.total and upload_file() never run for that image, making it appear to have zero detections.test_harness.py is a library, not a test: When writing test discovery scripts (like run-all-tests.sh), exclude test_harness.py from the glob test_*.py. It's a shared utility module copied into each plugin's tests/ directory — running it directly fails or produces misleading results. Pattern: find . -name "test_*_unit.py" (not test_*.py).tests/test_harness.py. pytest's default import mode (prepend) treats them as the same module — the second import silently gets the first plugin's harness, causing mysterious failures. Fix: add pyproject.toml at repo root with [tool.pytest.ini_options] import_mode = "importlib". Do NOT try adding __init__.py to test dirs — it turns them into packages and breaks relative imports (ModuleNotFoundError on test_harness).torch via sys.modules poisons numpy (reimport fails with "cannot load module more than once"). Instead, test publish logic directly by calling pywaggle APIs with canned detection results. See references/testing-patterns.mdTreeOfLifeClassifier.predict() always returns ranked predictions with scores, even for empty scenes. An IR nighttime image of a hummingbird feeder (no bird present) returns "Archilochus colubris" at ~19% — it learned the feeder shape association. Use --min-confidence 0.5 (or higher) to gate publishing so it only reports when genuinely confident. Without this, the plugin publishes species predictions every cycle regardless of content. For detection-triggered species ID, consider using YOLO as the trigger (detects "bird") and BioCLIP as enrichment (identifies species) — correlate by timestamp in a cloud-side watcher.cv2.putText with orange BGR (0, 165, 255) on a black background rectangle. Above threshold: show top-5 predictions in top-left corner. Below threshold: show "No confident species prediction (best: X%)" at bottom. Scale font to image size: scale = max(0.5, min(w, h) / 1000.0).pybioclip>=2.1.5 provides TreeOfLifeClassifier that handles model loading, taxonomy, and text embeddings automatically. Returns list-of-dicts with rank keys (kingdom...species) + score. Much cleaner than manual open_clip + pre-computed embeddings. The model_str parameter accepts any HuggingFace model string (hf-hub:imageomics/bioclip-2, hf-hub:imageomics/bioclip-2.5-vith14, etc.) — but BioCLIP 2.5 requires a monkey-patch until pybioclip adds native support (see pitfall below). Text embeddings for 2.5 exist at https://huggingface.co/datasets/imageomics/TreeOfLife-200M/tree/main/embeddings (3 GB npy + 80 MB json). See references/ml-plugin-patterns.md.--min-confidence threshold in production mode (camera/snapshot-url). In test mode (--image-dir), upload ALL images with annotations so every result can be reviewed. This matches YOLO's behavior: YOLO only saves images when it detects objects; BioCLIP should only save when it has a confident species prediction. The guard: if using_image_dir: around the below-threshold upload block.--model default in app.py, (2) update the TreeOfLifeClassifier(model_str=...) line in Dockerfile's model-bake step, (3) update patch_pybioclip.py if the new model isn't in pybioclip's TOL_MODELS, (4) bump sage.yaml version. Key differences: BioCLIP 2 uses ViT-L/14 (~430M params, ~2.5GB VRAM, memory=8Gi/16Gi), BioCLIP 2.5 uses ViT-H/14 (~1B+ params, ~5-7GB VRAM, memory=16Gi/32Gi, +5.7% accuracy). Both use 224x224 input. The model string format is hf-hub:imageomics/<model-name>. Always tag the repo before upgrading (git tag v<old>) so you can revert. Tested: BioCLIP 2.5 detects hummingbirds at 99.95-100% confidence vs 97% for BioCLIP 2. Confidence higher on 32/34 test images.
CRITICAL: pybioclip TOL_MODELS whitelist blocks new models. pybioclip 2.1.5 hardcodes TOL_MODELS in bioclip/_constants.py (NOT predict.py). Format: TOL_MODELS = {BIOCLIP_V1_MODEL_STR: TOL10M_HF_DATAFILE_REPO, BIOCLIP_V2_MODEL_STR: TOL200M_HF_DATAFILE_REPO}. TreeOfLifeClassifier raises ValueError: TreeOfLife predictions are only supported for... for any model not in the dict. Additionally, get_txt_emb() and get_txt_names() in predict.py hardcode embeddings/txt_emb_species.{npy,json} filenames, but newer models use model-specific filenames (e.g. txt_emb_bioclip-2.5-vith14.{npy,json} in the imageomics/TreeOfLife-200M HuggingFace dataset repo under embeddings/). The embeddings ARE available — check https://huggingface.co/api/datasets/imageomics/TreeOfLife-200M/tree/main/embeddings for the full list. predict.py imports TOL_MODELS via from ._constants import ( on one line, then HF_DATAFILE_REPO_TYPE, BIOCLIP_MODEL_STR, TOL_MODELS, on the next (indented continuation). Fix: use a separate Python patch script (not inline RUN — multi-line Python in Dockerfile breaks the parser with unknown instruction errors). The script patches two files: (1) bioclip/_constants.py — add the new model to TOL_MODELS dict and add a TOL_EMB_FILES dict mapping model strings to (npy, json) filename tuples; (2) bioclip/predict.py — add TOL_EMB_FILES to the import line and override the hardcoded txt_emb_species.{npy,json} in get_txt_emb() / get_txt_names() with lookups from TOL_EMB_FILES. Dockerfile pattern: COPY patch_pybioclip.py /tmp/patch_pybioclip.py then RUN python3 /tmp/patch_pybioclip.py && rm /tmp/patch_pybioclip.py. String matching pitfalls (each caused a build failure): (1) TOL_MODELS is in _constants.py (NOT predict.py) and uses variable references (BIOCLIP_V2_MODEL_STR), not string literals — matching 'hf-hub:imageomics/bioclip-2' fails; (2) the import in predict.py uses relative form from ._constants import ( — match the exact indented continuation line HF_DATAFILE_REPO_TYPE, BIOCLIP_MODEL_STR, TOL_MODELS,, not the from line itself; (3) always use docker run --entrypoint python3 <image> -c "import bioclip._constants; ..." to inspect the actual installed source before writing the patch script. See the sage-bioclip repo's patch_pybioclip.py for the complete working patch. This is fragile — when pybioclip upstream adds support, remove the patch.--trust-remote-code for Qwen models. Use non-default port (e.g. 8199) to avoid conflicts--disable-log-requests was removed; use --no-enable-log-requests. The old flag causes the vLLM subprocess to exit silently (unrecognized argument), leaving a <defunct> zombie process. Symptom: server PID shows <defunct> in ps, port never starts listening, no error in parent process. Always verify vLLM server is actually running after Popen launch (check ps for <defunct>, check port with ss -tlnp). When upgrading vLLM, run --help to verify flag names haven't changed.subprocess.PIPE. PIPE can cause deadlocks on large output and makes debugging harder. Log file lets you inspect server startup errors post-mortem. Both stdout and stderr should go to DEVNULL — routing only one to PIPE while the other goes to DEVNULL is still a deadlock risk if the PIPE buffer fills.--continuous N breaks --image-dir batch mode: The common main-loop pattern if args.continuous != "Y": break fires after the FIRST image even in --image-dir mode, so only one file gets processed. The fix: if args.continuous != "Y" and not using_image_dir: break — let the directory iterator's StopIteration handle the exit. This bug is silent and hard to spot because the plugin exits cleanly with valid output — it just only processed one image instead of all of them. Always verify the NDJSON record count matches the image count after batch runs.--image-dir and --stream modes separately: Plugin app.py files typically have multiple input modes (single image, directory glob, camera stream). Code paths for --image-dir (using os.listdir, os.path.join) may reference os without importing it if the initial development only tested --stream mode. Always run all CLI modes during local testing to catch missing imports.&width=640&height=360 to the snapshot URL to get sub-stream resolution (~12KB). That's a 38x bandwidth reduction — critical for LTE-connected cameras. YOLO resizes to 640px anyway so full-res snapshots waste bandwidth with zero accuracy benefit.--gpu-memory-utilization 0.58 for 32B models. Both 0.80 and 0.85 OOM.--enforce-eager required for large models on unified memory: CUDA graph capture consumes ~5GB extra memory. For 32B+ models on 128GB unified memory, this causes OOM even at conservative GPU fractions. Add --enforce-eager to skip graph capture (~10-15% throughput cost, prevents OOM)..pt / .safetensors model weights to git: Model weight files (e.g. yolo11x.pt, 110MB) get auto-downloaded by libraries like ultralytics into the working directory. Add *.pt *.pth *.bin *.safetensors to .gitignore. Models are baked into Docker images at build time via Dockerfile RUN curl — they don't belong in the source repo.pathlib.Path(module.__file__) to find the actual installed location (e.g. /usr/local/lib/python3.12/dist-packages/bioclip/predict.py). Always read the installed source to get exact string matches — don't assume formatting from GitHub. Common traps: (1) dict definitions may use variable references not string literals; (2) imports may use relative form (from ._constants import) with specific indentation on continuation lines; (3) the code may be spread across multiple files (_constants.py vs predict.py). Use assert old in src before each replacement to fail fast on mismatches.RUN python3 -c "import foo\\nbar()" with actual newlines causes unknown instruction: IMPORT errors — Docker interprets each line as a Dockerfile instruction. Always use a separate .py script file: COPY script.py /tmp/ then RUN python3 /tmp/script.py && rm /tmp/script.py. Single-line Python with \ continuations works only for trivial 1-2 line scripts. This is the pattern used for patch_pybioclip.py.FROM are single-stage builds, even if they have multiple RUN steps (download model, warmup, install deps). "Multi-stage" specifically means multiple FROM statements. Overview.md docs have been corrected but watch for this when writing new plugin documentation. Multiple RUN layers ≠ multi-stage build.README.md (GitHub landing page), (2) ecr-meta/README.md (ECR portal usage), (3) ecr-meta/ecr-science-description.md (ECR portal science tab), (4) sage.yaml inputs section. Also update jobs/*.yaml if args changed. NEVER commit code changes without proactively checking all four doc files — Pete considers this a serious oversight and will call it out. After every code change, check all four files plus jobs/*.yaml without being asked. The rule: if you touch app.py's argparse or behavior, the same commit (or at minimum the same PR) must touch all doc surfaces. Verify with git diff --stat before committing — if only .py files changed but args were added/modified, the commit is incomplete. This applies to ALL plugin repos (sage-yolo, sage-bioclip, birdnet, etc.) — it's a universal standard, not a per-project preference.testing.command pointing to current test files. Use delegate_task to parallelize auditing all plugins' overview.md + sage.yaml files against code simultaneously.Flag | Type | Default | Description. Test runner CLI options also get their own table. When adding argparse flags from a third-party library, the help text should include a direct URL to the upstream docs. sage.yaml inputs: must list every argparse flag. Job YAML examples should show all commonly-used flags.opencv-python-headless in requirements.txt for edge plugins (no GUI on nodes). If overview.md warns about switching to headless, that warning is probably backwards — check requirements.txt first.._ resource fork files break cv2.imread: When test images are copied via macOS Finder or Samba, ._* resource fork files appear alongside each image. These have valid image extensions (e.g. ._test-image001.jpg) but are not real images — cv2.imread() returns None, causing assertion failures in tests. Always filter with not name.startswith(".") when globbing test image directories — in EVERY place that iterates images: app.py:iter_image_dir(), test_harness.py:get_test_images(), local test runner count_images(), and integration test image lists. Add ._* to .gitignore to prevent committing them. The files keep coming back via Samba — add veto files = /._*/.DS_Store/ and delete veto files = yes to smb.conf [global] to block creation server-side.bind interfaces only = yes with interfaces = tailscale0 does NOT work — Tailscale's point-to-point interface lacks broadcast capability and Samba silently falls back to loopback only. Fix: remove bind interfaces only, use hosts allow = 100.64.0.0/10 127.0.0.1 + hosts deny = 0.0.0.0/0 under [global] instead. This restricts access to Tailscale CGNAT range without needing interface binding. To prevent macOS from creating ._* resource forks and .DS_Store files on the server, add veto files = /._*/.DS_Store/ and delete veto files = yes under [global]. See references/direct-node-testing.md.Camera("/images/test.jpg") inside a Docker container triggers pywaggle's named-stream lookup (no :// scheme) → FileNotFoundError: /run/waggle/data-config.json. Fix: use file:// prefix: Camera("file:///images/test.jpg"). On the host (not in Docker), bare paths work because resolve_device_from_data_config falls through to resolve_device_from_path. This only bites you in Docker where /run/waggle/data-config.json doesn't exist.himalaya flag add -a sage <ID> seen (positional arg, NOT --flag seen).~/AI-projects/tmux-logs/ directory (NOT the bare ~/AI-projects/ root, NOT inside a project repo). Filename convention: hermes-yolo-session-YYYY-MM-DD-partN-ansi.txt — hermes-yolo-session- prefix, ISO date, then partN (start a fresh part1 for each new date; bump the part number for additional captures the same day as the tmux history buffer rolls over). Capture full scrollback WITH ANSI color codes: cd ~/AI-projects/tmux-logs && tmux capture-pane -t <session>:<win>.<pane> -e -p -S - > hermes-yolo-session-YYYY-MM-DD-partN-ansi.txt. The -e flag preserves ANSI SGR codes (verify with grep -c $'\033' <file>); -S - grabs the entire history buffer (find the pane via tmux list-panes -a -F '#{session_name}:#{window_index}.#{pane_index} hist=#{history_size}'). Caveat: tmux only retains what's within its history-limit; anything scrolled off before capture is unrecoverable — that's why old sessions were split into part1–7.wes-upload-agent pod scans /media/plugin-data/uploads/<job>/<plugin>/<version>/ on the host, rsyncs to beehive-uploads.sagecontinuum.org, and deletes immediately. There is no practical window to intercept files on-node before they're cleaned up. Don't try to race it; use the Sage data API to find upload URLs instead.curl -L -u user:token <url> returns 404 ("Unable to open ... no such file or directory") from the NRP backend. Old files may still work while new ones 404. Debug: (1) check wes-upload-agent logs on node — if it shows "uploaded all files found" with no errors, the node side is fine; (2) try downloading from a different node's recent uploads — if those also 404, it's a global NRP issue; (3) dig nrdstor.nationalresearchplatform.org to see which replica you're hitting. This is a Sage infrastructure issue — escalate to the cyberinfrastructure team.plugin.upload_file(tmp_path, ...), the file at tmp_path may no longer exist — pywaggle moves it to the upload directory. Guard any cleanup: if os.path.exists(tmp_path): os.unlink(tmp_path).sudo kubectl logs wes-upload-agent-<id> --tail=30 — if it shows "uploading/cleaning/done" cycles with no errors, the node side is fine and the problem is downstream. The upload agent uses rsync to beehive-uploads.sagecontinuum.org. Upload files live briefly at hostPath: /media/plugin-data/uploads/<job>/<plugin>/<version>/ before being rsynced and deleted — the window is too short to intercept. Alternative for image delivery when NRP is down: SSH to the node and grab a fresh camera snapshot directly (curl the Reolink HTTP snapshot URL from the host).portal.sagecontinuum.org/account/access. Format: curl -u <portal-username>:<access-token>.clean.sh for pre-transfer cleanup: The repo includes a clean.sh script that removes test outputs, downloaded model weights (.pt/.pth/.bin/.safetensors), __pycache__, macOS junk (._*, .DS_Store), and .pytest_cache. Run ./clean.sh for a dry run, ./clean.sh --force to delete. Always run before git add, rsync to another machine, or archiving.slack_sdk (pip install slack_sdk) and files_upload_v2() — this is the current API (old files.upload retired March 2025). Requires creating a Slack app with files:write + chat:write scopes, installing it, and inviting the bot to the channel. Store the bot token and channel ID in secrets/bot-secrets (shell export format, gitignored, chmod 600). See references/cloud-trigger-notifications.md for the full setup and working code.curl https://hooks.slack.com/ returning 302. The recommended pattern is still a cloud-side watcher, not a host-side process. See references/cloud-trigger-notifications.md.output/: When running docker run with -e PYWAGGLE_LOG_DIR=/output -v $(pwd)/output/yolo-docker-test:/output, output goes to <repo>/output/ not <repo>/tests/output/. Make sure .gitignore includes top-level output/ in addition to tests/output/ and plugins/*/tests/output/.torch.cuda.is_available() returns False even though torch.version.cuda reports 13.0, _is_compiled() is True, and device_count() returns 1. Root cause: /dev/nvmap is cr--r----- root:video — the Tegra memory manager requires video group. torch.cuda.init() → NvRmMemInitNvmap Permission denied → RuntimeError: No CUDA GPUs are available. Note: /dev/nvidia* are rw-rw-rw- — the blocker is specifically /dev/nvmap. Debug: (1) ls -la /dev/nvmap, (2) groups, (3) getent group video. The NvRmMemInit stderr messages look like noise but ARE the failure signal. Sage docs warn: "Do not run any app or install packages directly on the node." Use sudo pluginctl build/deploy which gives containers proper GPU access. Direct python3 app.py requires video group (sudo usermod -aG video <user> + re-login)./etc/rancher/k3s/k3s.yaml is root-only. ALL pluginctl commands require sudo on Thor: sudo pluginctl build ., sudo pluginctl deploy, sudo pluginctl ps, sudo pluginctl logs, sudo pluginctl rm. Without sudo, commands fail with permission errors (e.g. pods "name" is forbidden). Docker on Thor also requires sudo.-n <name> already exists, pluginctl deploy tries to patch the running pod in place. Kubernetes only permits changing a handful of fields on a running pod (spec.containers[*].image, initContainers[*].image, activeDeadlineSeconds, tolerations, terminationGracePeriodSeconds). Any other change (volumes, the kube-api-access-* mount, args, env) is rejected with Error: Pod "<name>" is invalid: spec: Forbidden: pod updates may not change fields other than ... plus a diff showing the removed/changed fields. Fix: delete the existing pod first, then deploy fresh: sudo pluginctl rm <name> → verify gone (sudo pluginctl ps or sudo kubectl get pods | grep <name>) → redeploy. If the pod is stuck Terminating, force-delete: sudo kubectl delete pod <name> --grace-period=0 --force. This bites whenever you redeploy with changed args/camera URL — the args change is exactly the kind of non-image field k8s won't patch.--resource 'memory=8Gi,limit.memory=16Gi', the pod gets OOMKilled (exit code 137). The resource flag syntax uses bare k8s names (memory=8Gi), NOT resource.memory=8Gi (that's for selectors). Check OOMKilled with: sudo kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].state}'. Model-specific resource requirements: YOLO11x needs memory=8Gi,limit.memory=16Gi. BioCLIP 2 (ViT-L/14) needs memory=8Gi,limit.memory=16Gi. BioCLIP 2.5 Huge (ViT-H/14) needs memory=16Gi,limit.memory=32Gi — OOMs at 8Gi/16Gi during text embeddings loading (~3 GB .npy file + ~4-5 GB model weights). Always check sudo kubectl get pods | grep <name> for OOMKilled status after first deploy of a new model version.ControlPersist 10m on the jump host (sage-vpn) and a passphrase-protected IdentityFile, connections through ProxyJump expire after 10 minutes of idle. The next SSH command needs the agent to provide the key again, but the agent may have been restarted or the SSH_AUTH_SOCK changed. Symptoms: "Permission denied" errors on SSH commands that worked minutes ago, especially during long background operations (Docker builds, k3s imports). Root cause: the *.sage host block uses ProxyJump sage-vpn but has no IdentityFile of its own — it relies entirely on the agent. When the control master expires and must reconnect, both the jump and the target need the key. Fix: add IdentityFile ~/.ssh/sage_key and IdentitiesOnly yes to the *.sage block so SSH can find the key without the agent. Keep User root as default (override with beckman@ on command line as needed). Save backup: cp ~/.ssh/config ~/.ssh/config.old. For long operations (docker build, k3s import), always use nohup on Thor so the work survives SSH disconnections:
ssh beckman@node-H00F.sage "cd /tmp/sage-bioclip && nohup sudo docker build -t bioclip:0.2.0 . > /tmp/build.log 2>&1 &"
# Check progress:
ssh beckman@node-H00F.sage "tail -5 /tmp/build.log"
nohup bash -c 'sudo docker save IMAGE | sudo k3s ctr images import - > /tmp/import.log 2>&1 && echo DONE >> /tmp/import.log' &linux/amd64 and linux/arm64 from sage.yaml's source.architectures. The arm64 build uses QEMU emulation on an amd64 host, which crashes (qemu: uncaught target signal 6 (Aborted) - core dumped) when the NVIDIA PyTorch base image tries to import torch during any RUN step. This affects ALL plugins using nvcr.io/nvidia/pytorch:* as base. The amd64 build succeeds fine. Confirmed: sage-bioclip v0.3.0 failed on ECR with this exact error at the pip constraints RUN step (first step that imports torch). Fix options: (1) Remove linux/arm64 from sage.yaml architectures — ECR builds amd64 only, build arm64 locally on Thor/DGX Spark; (2) Ask ECR team for native arm64 builder (no QEMU); (3) Use a lighter base image for arm64 that doesn't require GPU at build time. Until fixed upstream, arm64 images must be built locally and transferred via docker save | k3s ctr images import.docker login, docker tag, or docker push against registry.sagecontinuum.org. The workflow is: push code to GitHub → register on portal.sagecontinuum.org → ECR builds → get registry tag from "Tags" tab. For pre-ECR testing on Thor nodes, use docker save | gzip + scp + sudo k3s ctr images import. sesctl submit validates the image against the ECR app catalog, but pluginctl deploy does NOT — a pluginctl-runnable image can still fail sesctl submit with 400 ... does not exist in ECR if it was never built in the portal pipeline. See references/docker-build-deploy.md and references/sesctl-ecr-validation.md.docker build with pip install fails with DNS resolution errors. Build Docker images on a machine WITH internet (DGX Spark, CI server), then transfer or publish to ECR. Exception: some Thor nodes DO have Docker installed and outbound internet access. On those nodes, clone the repo and build directly (git clone + sudo docker build) for faster iteration — skip the save/scp/load dance. Check with curl -s https://pypi.org > /dev/null and sudo docker --version. All Docker commands on Thor require sudo. See references/docker-build-deploy.md for both workflows.nvidia-container-toolkit package may be installed but Docker doesn't know about the nvidia runtime until configured. docker run --runtime=nvidia fails with unknown or invalid runtime name: nvidia. Fix (one-time per machine): sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker. Verify: docker info | grep -i runtime should show nvidia. Thor nodes have this pre-configured by Sage. See references/docker-build-deploy.md.--runtime=nvidia not --gpus all for portable Docker commands: Thor nodes use an older NVIDIA Container Runtime Hook that does NOT support --gpus all — it errors with invoking the NVIDIA Container Runtime Hook directly is not supported. Please use the NVIDIA Container Runtime (e.g. specify the --runtime=nvidia flag) instead. DGX Spark supports both flags. Always use --runtime=nvidia in documentation and scripts so commands work on both DGX Spark and Thor without modification. --gpus all is a newer nvidia-container-toolkit feature; --runtime=nvidia is the older/universal path that both setups support.ImportError: numpy.core.multiarray failed to import. Fix in Dockerfile (works for both numpy 1.x and 2.x bases): pip uninstall -y opencv-python opencv-python-headless 2>/dev/null; rm -rf /usr/local/lib/python3.*/dist-packages/cv2* && pip install --no-cache-dir opencv-python-headless>=4.8.0. Three things that DON'T work: (1) --force-reinstall alone — leaves stale .so files. (2) pip uninstall alone — also leaves stale files. (3) Just adding opencv-python-headless to requirements.txt — the base image's GUI opencv takes priority. The rm -rf cv2* with a python3.* glob (not hardcoded version) is essential. Use python3 not python in ENTRYPOINT for Python 3.12+ base images.requirements.txt lists torch>=2.0.0, numpy>=1.24.0, or ultralytics (which depends on all of them), pip replaces the base image's NVIDIA-compiled packages with generic PyPI versions. Three things MUST be frozen via pip constraints file: (1) torch — generic PyPI torch lacks Blackwell GPU kernels. Symptom: RuntimeError: GET was unable to find an engine to execute this computation. (2) torchvision — must match the NVIDIA torch build. (3) numpy — the base image ships numpy 1.26.4 and torch is compiled against numpy 1.x C ABI. If pip upgrades to numpy 2.x, torch.from_numpy() crashes with RuntimeError: Numpy is not available. Fix: remove torch and numpy from requirements.txt, then freeze all three in Dockerfile: TORCH_VER=$(...) && TV_VER=$(...) && NP_VER=$(python3 -c "import numpy; print(numpy.__version__)") && printf "torch==${TORCH_VER}\ntorchvision==${TV_VER}\nnumpy==${NP_VER}\n" > /tmp/constraints.txt && pip install -c /tmp/constraints.txt -r requirements.txt. Use the constraints file for ALL pip install commands in the Dockerfile. Note: the 25.08-py3 base image does NOT include torchaudio — don't try to freeze it. See references/docker-build-deploy.md for the complete Dockerfile pattern.sm_80 sm_86 sm_90 sm_100 sm_120 — sm_110 is MISSING. On Thor: "NVIDIA Thor with CUDA capability sm_110 is not compatible" warning + CPU fallback. The 25.08-py3 image (CUDA 13.0) adds sm_110 support and covers BOTH machines. Use 25.08-py3 or newer for any deployment targeting Thor. Using 24.06-py3 on either machine silently falls back to CPU (sm_90 max). The symptom is deceptive — model loads fine, inference appears to hang (extremely slow, not a crash).python3 app.py directly on a node (not in the Docker container), Ultralytics will download model weights (e.g. yolo11x.pt, ~130MB) to the current directory on first use. This is expected behavior for local testing — the Dockerfile bakes the model in so production containers never download. The download is a one-time cost; subsequent runs use the cached .pt file. clean.sh --force removes downloaded weights./flv?port=1935&app=bcs&stream=...) does NOT accept HTTP basic auth in the URL (http://user:pass@ip/...). That form makes ffmpeg fail instantly with Error opening input: End of file / exit 187. Credentials MUST be passed as query parameters: ...&user=USER&password=PASS. Confirmed working on a Reolink RLC-811A hummingcam (example host CAMERA_IP:PORT, user CAMERA_USER): http://CAMERA_IP:PORT/flv?port=1935&app=bcs&stream=channel0_sub.bcs&user=CAMERA_USER&password=CAMERA_PASSWORD (get the actual node camera IP/user/password from your instructor or the node owner — never hard-code a real credential in a skill or repo). Shell escaping: always wrap the whole --camera URL in SINGLE quotes when running pluginctl deploy by hand — a password containing ! triggers bash history expansion under double quotes (and &/? need protection too). Note this is Reolink-specific: the Mobotix M16 MxPEG stream DOES use basic auth (http://user:pass@ip/control/faststream.jpg?stream=MxPEG&needlength) and that form is confirmed working. See references/reolink-audio-capture.md.