| name | sage-waggle |
| description | Sage Continuum & Waggle: edge computing platform — plugin dev, ECR submission, data APIs, job scheduling, node management, testing. |
| tags | ["edge-computing","iot","sage","waggle","scientific-compute"] |
| triggers | ["User mentions Sage, Waggle, sagecontinuum, waggle-sensor, Beehive, Beekeeper, WES","Tasks involving edge computing plugins that publish sensor data","Querying environmental/scientific sensor data from distributed nodes","Scheduling jobs on edge nodes (sesctl, pluginctl)","Working with sage-data-client or pywaggle","Developing Docker-based edge plugins","Using the Sage MCP server"] |
| globs | ["*sage*","*waggle*","*plugin*sage*","*beehive*"] |
Sage Continuum & Waggle Platform
Architecture Overview
- Edge Nodes — ARM64/x86; WES/k3s. Prefer honesty: real VSNs/names only (
references/data-query-first-report-and-history.md, references/data-viz-and-honesty.md). Key refs: references/stack-architecture-map.md, references/pywaggle2-producer-consumer-architecture.md, references/frame-anchored-batch-consumers-and-watchers.md, references/frame-anchored-vs-window-pollers.md, references/node-ssh-access-and-gpsd-probe.md, references/node-identity-and-upload-contract.md, references/pywaggle2-nodeinfo-gps-design.md, references/wes-pod-config-and-manifest-exposure.md, references/pluginctl-sideload-and-node-build.md, references/node-sideload-test-and-restore.md, references/audio-plugin-multinode-testing.md, references/storage-upload-health-verification.md, references/ci-handoff-doc-discipline.md, references/thor-arm64-deploy-pipeline.md, references/scheduling-continuous-vs-oneshot-and-gpu-contention.md, references/producer-consumer-gpu-and-shared-mounts.md.
- Beehive (Cloud) — Receives data uploads, stores in time-series DB + object store. Runs RabbitMQ message bus.
- Beekeeper — Node identity, registration, provisioning, reverse SSH tunnels for management.
Key APIs
| API | Endpoint | Auth | Notes |
|---|
| Data query (timeseries) | POST https://data.sagecontinuum.org/api/v1/query | None (public) | Beehive timeseries from plugin.publish(). NDJSON. Filters: plugin, name, vsn, sensor. See references/timeseries-data-query-api.md |
| Manifests (rich) | GET https://auth.sagecontinuum.org/manifests/ · .../manifests/<vsn> | None | Full node hardware + sensor URIs. Collection needs trailing /; single VSN slash optional. Prefer per-VSN (full list ≈2MB+). ?project= |
| Nodes (beta) | GET https://auth.sagecontinuum.org/api/v-beta/nodes/ · .../nodes/<vsn> | None | Flatter node card (type, site, partner, focus, modem). Filters: ?phase=, ?project__name= (comma=OR) |
| Edge Scheduler | https://es.sagecontinuum.org | Bearer token | Job submission/management |
| MCP Server (Sage) | https://mcp.sagecontinuum.org/mcp | None (read-only); Bearer for jobs | 29 Sage tools — references/mcp-tools.md |
| MCP Server (GitHub) | https://api.githubcopilot.com/mcp/ | Bearer GitHub PAT | Repos/issues/PRs/Actions — registry · references/github-mcp-server.md |
| MCP Server (Hugging Face) | https://huggingface.co/mcp | Bearer HF token | Models/datasets/Spaces/papers/docs/Jobs — docs · references/huggingface-mcp-server.md |
| MCP Server (Milvus SDK helper) | https://sdk.milvus.io/mcp/ | None (Accept: text/event-stream) | Prefer Milvus Lite + MilvusClient — MCP docs · Lite · references/milvus-sdk-helper-mcp.md |
| Portal | https://portal.sagecontinuum.org | Browser login | Node management, token generation |
| ECR (Edge Code Repo) | Portal: https://portal.sagecontinuum.org/apps · API: GET https://ecr.sagecontinuum.org/api/apps?public=true | List public: none | Public plugins/apps available to schedule. Per-app: /api/apps/<ns>/<name> · /api/apps/<ns>/<name>/<ver>. See references/ecr-public-apps-api.md |
Auth / manifest details + related routes (/computes/, /sensors/, …): references/auth-api-manifests-and-nodes.md. App source: waggle-auth-app.
Auth tokens: get from portal.sagecontinuum.org/account/access. Format: Authorization: Bearer {token}.
Plugin Development
Camp default (Thor): prefer sudo pluginctl build . → sudo pluginctl run for on-node development. Start with references/pluginctl-camp-guide.md. Use raw podman build only for ECR-bypass side-load workflows (see references/pluginctl-sideload-and-node-build.md).
Official docs (prefer these URLs when citing Sage):
Plugins are Docker containers using pywaggle — the official Python SDK for Waggle plugins (waggle.plugin, cameras, microphones, uploads). Prefer that GitHub repo as the source of truth for API, docs, and source layout (src/waggle/…).
Install:
pip install -U 'pywaggle[all]'
Docs / source:
Minimal pattern:
from waggle.plugin import Plugin
import time
with Plugin() as plugin:
while True:
value = read_sensor()
plugin.publish("env.measurement", value, meta={"units": "C"})
time.sleep(30)
Plugin file structure
app.py — main applicationDockerfile — based on waggle/plugin-base:1.1.1-base (ARM64) or -ml variant for GPU.dockerignore — excludes tests/, ecr-meta/, jobs/, pycache/, *.pyc, *.md, .git/ from Docker build contextsage.yaml — metadata (name, description, version, authors, resources)ecr-meta/ — ECR submission metadataoverview.md — pedantic/instructive documentation (mini tutorial style)jobs/ — per-plugin job YAMLs for edge scheduler deployment (self-contained)tests/ — self-contained test suite (local test with real GPU inference, test images, harness copy, run script). See references/testing-patterns.md
Camera & Microphone (pywaggle abstractions)
from waggle.data.vision import Camera
cam = Camera("bottom_camera")
cam = Camera("rtsp://admin:pass@192.168.1.100:554/h264Preview_01_main")
cam = Camera("/path/image.jpg")
cam = Camera("file:///path/image.jpg")
sample = cam.snapshot()
from waggle.data.audio import Microphone
mic = Microphone("my_microphone")
sample = mic.record(duration=5)
Audio Plugin Development
For audio-based plugins (bird classification, sound event detection),
the pattern differs from vision plugins:
Audio input modes:
- pywaggle Microphone —
Microphone("mic_name").record(duration=3) for named node microphones
- Direct ALSA/PulseAudio —
sounddevice or pyaudio for direct recording on host
- File-based —
--audio-dir flag for batch testing against WAV/MP3 files
- Network camera audio via
--camera URL — extract audio from IP camera via ffmpeg. The --camera CLI flag accepts any ffmpeg-compatible audio source URL. The plugin auto-detects the stream type and adds appropriate ffmpeg flags. See references/camera-audio-capabilities.md for the full camera comparison. Supported patterns:
--camera "http://user:pass@IP/control/faststream.jpg?stream=MxPEG&needlength"
--camera "rtsp://user:pass@IP/profile1/media.smp"
--camera "http://IP/audio.cgi"
IMPORTANT: Mobotix M16 cameras may refuse RTSP (port 554 closed) but serve audio fine via MxPEG HTTP. Always try the MxPEG URL first. Tested: M16 delivers pcm_alaw at 8 KHz (4 KHz Nyquist) — marginal for BirdNET, which needs frequencies up to 8-15 KHz. A USB mic at 48 KHz is significantly better. pywaggle's Microphone class does NOT support network cameras — the plugin uses subprocess ffmpeg for camera audio.
Key differences from vision plugins:
- Audio chunks are typically 3 seconds (vs single-frame snapshots for vision)
- Sample rate matters: BirdNET requires 48 kHz, resampling if needed
- Model formats: TFLite (CPU, ARM64) vs PyTorch (GPU); GPU inference on ARM64
is not always available (see BirdNET ARM64 limitation)
- Audio files can be large — consider recording to tmpfs and cleaning up
- Noise rejection: most audio classifiers have "non-event" classes for
filtering environmental noise, human speech, etc.
BirdNET V2.4 integration pattern (pip install birdnet):
import birdnet
model = birdnet.load("acoustic", "2.4", "tf")
geo = birdnet.load("geo", "2.4", "tf")
species = geo.predict(lat, lon, week=22, min_confidence=0.03)
species_set = species.to_set()
predictions = model.predict(
"recording.wav",
top_k=5,
default_confidence_threshold=0.25,
sigmoid_sensitivity=1.0,
custom_species_list=species_set,
overlap_duration_s=0.0,
bandpass_fmin=0,
bandpass_fmax=15000,
batch_size=1,
)
df = predictions.to_dataframe()
API pitfalls discovered:
model.predict() uses default_confidence_threshold, NOT min_confidencegeo.predict() returns GeoPredictionResult — use .to_set() (not .to_list())- Model auto-downloads on first use to
~/.local/share/birdnet/
- No
__version__ attribute on the birdnet module
birdnet.load("acoustic", "2.4", "tf") — the "tf" backend uses TFLite on ARM64 (CPU only, no GPU)- Audio plugin Dockerfile should use
python:3.12-slim (not NVIDIA base) since BirdNET is CPU-only on ARM64
Audio plugin app.py pattern (proven in birdnet repo):
- Wrap the model in a classifier class (e.g.
BirdNETClassifier)
- Three audio source modes via CLI flags (in priority order):
--input <file> — read from a local audio file (testing)--camera <url> — capture from network camera via ffmpeg subprocess- (default) — record from USB microphone via pywaggle
Microphone
- The
--camera flag accepts any ffmpeg-compatible URL; auto-detects Mobotix MxPEG
(adds -f mxg) and RTSP (adds -rtsp_transport tcp). Credentials in URL are
masked in log output (splits on @, shows only the host portion).
record_from_camera() function: spawns ffmpeg, converts to 48 KHz mono WAV in tmpdir,
raises RuntimeError on ffmpeg failure or empty output. Cleanup via shutil.rmtree in finally block.- Support
--dry-run for testing without pywaggle (import Plugin only when needed)
- Support
--interval for gap between cycles + --num-recordings for bounded runs
--num-recordings N (default 1) = run exactly N cycles then exit; --num-recordings 0 = loop forever (requires --interval > 0)- This matches the old plugin's
--num_rec + --silence_int semantics:
old: analyze.py --num_rec 6 --sound_int 5 --silence_int 1
new: app.py --num-recordings 6 --duration 5 --interval 1
- Publish per-species detections + JSON summary per cycle
- Topic format:
env.detection.audio.<scientific_name> (lowercase, underscored)
- eBird geo-filtering is fully automatic for live deployments:
--lat/--lon default to -1 which triggers auto-detection from the node manifest (/etc/waggle/node-manifest-v2.json). --week defaults to auto (calculate BirdNET week from current date: (month-1)*4 + min(4, ceil(day/7.5)), range 1–48). Resolve both at startup before constructing the classifier, so each scheduler invocation gets the correct location and week. Use explicit values to override for testing (--lat 41.72 --lon -87.98 --week 25). Use --week -1 for year-round (no seasonal filter). The --week argparse type should be str (not int) to accept both auto and numeric values.
- Use
argparse.ArgumentDefaultsHelpFormatter + grouped args (audio, model, location, runtime)
- Clean up temp recording files in
finally blocks (record to tmpdir, shutil.rmtree after)
- Main loop refactored into
run_cycle(plugin=None) + run_loop(plugin=None) —
eliminates the old duplicate run_once/run_with_plugin code paths
Audio plugin Dockerfile differs from vision plugins:
- Base image:
python:3.12-slim (not NVIDIA — BirdNET is CPU-only on ARM64)
- System deps:
ffmpeg libsndfile1 libasound2-dev
- Pre-download models at build time:
RUN python3 -c "import birdnet; birdnet.load(...)"
- Image size: ~2.9 GB (TensorFlow) vs ~10+ GB for vision plugins on NVIDIA base
- No GPU, no CUDA, no pip constraints file needed
See references/audio-classification-models.md for the full model survey.
Camera device resolution (resolve_device() chain):
- Named stream (no
://) → WES data config lookup (/run/waggle/data-config.json, Sage nodes only)
- URL with scheme (
rtsp://, http://) → passed directly to cv2.VideoCapture()
file://path → local file- Path object → local file
Named cameras (bottom_camera, top_camera) are aliases defined in the node's /run/waggle/data-config.json — a JSON array where each entry has match.id (the name) and handler.args.url (the actual RTSP/HTTP URL). This config is managed by WES and only exists on real Sage nodes.
For IP cameras not registered in WES (e.g. a Reolink on the same network), pass the RTSP URL directly to --stream. See references/camera-rtsp-patterns.md for vendor-specific URL formats, data-config.json format, Docker-on-Thor QA testing workflow, and troubleshooting.
Plugin CLI input modes
Plugins support four input modes via argparse:
--stream <camera|rtsp|image> — live camera, RTSP URL, or single test image (default: bottom_camera)--image-dir <path> — batch-process all images in a directory (overrides --stream)--snapshot-url <http-url> — fetch a JPEG snapshot from an HTTP URL each cycle (overrides --stream). Works with Reolink CGI API, generic IP camera snapshot endpoints, or any URL returning a JPEG. Credentials go in the query string. See references/reolink-http-snapshot.md.--continuous Y|N — loop (camera/snapshot-url) or single-shot. WARNING: the main loop's if continuous != "Y": break must be guarded with and not using_image_dir — otherwise --image-dir mode only processes the first image. See Pitfalls.
Note: Not all plugins implement --image-dir. YOLO and BioCLIP have it; vLLM only has --stream (accepts a camera name, RTSP URL, or single image path). For batch testing a plugin without --image-dir, loop in shell: for img in tests/test-images/*.jpg; do python3 app.py --stream "$img" --continuous N; done — but note each invocation may restart expensive resources (e.g. vLLM server subprocess).
Exposing third-party library parameters
When a plugin wraps a third-party ML library (Ultralytics YOLO, vLLM, etc.), expose the library's key tuning parameters as CLI flags rather than hardcoding them. Pattern:
- Add argparse flags with descriptive help text including the default value and a URL to the upstream docs
- Pass the flag values through to the library's API call (e.g.
model(frame, imgsz=args.imgsz, half=args.half, ...))
- Document all flags in overview.md's Configuration Reference table with consistent
Flag | Type | Default | Description columns
- Add a callout linking to the upstream docs page (e.g.
https://docs.ultralytics.com/modes/predict/#inference-arguments)
- Add corresponding entries to sage.yaml's
inputs: section
Example YOLO flags exposed from Ultralytics: --imgsz (input resolution), --half (FP16), --max-det (max detections), --augment (TTA), --agnostic-nms (class-agnostic NMS). Each help string includes a link to Ultralytics docs so students can read the full parameter reference.
Use iter_image_dir() helper to yield (path, frame, timestamp) tuples from a directory, matching the Camera snapshot interface. Always filter by IMAGE_EXTENSIONS set. See templates/ml-plugin-app.py for the full pattern.
Plugin publish patterns
with Plugin() as plugin:
plugin.publish("env.temperature", 23.5)
plugin.publish("env.count.car", 12, timestamp=ts, meta={"camera": "bottom", "model": "yolov7"})
plugin.upload_file("annotated.jpg")
Meta value rule: all values in meta={} must be str. pywaggle's valid_meta() raises ValueError on int/float/bool. Always str() wrap numeric meta.
Self-describing publish records
When publishing aggregate records (e.g. env.count.total), include the full class breakdown in meta so each record is self-describing without cross-referencing per-class records:
classes_summary = ",".join(f"{c}:{n}" for c, n in sorted(counts.items()))
plugin.publish("env.count.total", total, timestamp=ts,
meta={"camera": source_name, "model": args.model,
"classes": classes_summary if classes_summary else "none",
"num_classes": str(len(counts))})
Result: {"name":"env.count.total","value":3,"meta":{"classes":"bottle:1,person:2","num_classes":"2",...}}
Local testing (no node required)
export PYWAGGLE_LOG_DIR=./test-run
python3 app.py --stream test-image.jpg --continuous N
cat test-run/data.ndjson
ls test-run/uploads/
PYWAGGLE_LOG_DIR is a built-in pywaggle feature (not custom code). When set, pywaggle redirects all plugin.publish() to data.ndjson and all plugin.upload_file() to uploads/ inside that directory instead of sending to Beehive. Without it, pywaggle tries to write to /run/waggle/uploads/ which only exists on real nodes — on dev machines you get PermissionError: [Errno 13] Permission denied: '/run/waggle'. The local test runners set it automatically; when running app.py directly, always export PYWAGGLE_LOG_DIR=./output/<name> first.
Important: all meta dict values MUST be strings. plugin.publish("topic", 42, meta={"count": str(n)}) — not {"count": n}.
For testing ML plugins, see references/testing-patterns.md. All tests require GPU and run real model inference — no mocked unit tests. Each plugin has a self-contained tests/ directory with its own test file, test images, harness copy, and run-tests.sh. A top-level tests/run-all-tests.sh auto-discovers and runs all plugin tests (GPU required). Test images live in plugins/<name>/tests/test-images/ (flat directory, committed to git). All plugins share the same set of real test images — no synthetic generation step needed.
pluginctl deploy vs docker run: where data goes
docker run with -e PYWAGGLE_LOG_DIR=/output: data stays local. plugin.publish() writes to data.ndjson, plugin.upload_file() writes to uploads/ inside the mounted volume. Nothing reaches Beehive.pluginctl deploy (no PYWAGGLE_LOG_DIR): data goes to the real Sage pipeline. plugin.publish() sends measurements via RabbitMQ → Beehive → time-series DB (queryable at data.sagecontinuum.org). plugin.upload_file() sends files to the object store (Open Storage Network, S3-compatible). This is production data flow.
Use docker run + PYWAGGLE_LOG_DIR for testing/debugging. Use pluginctl deploy for real deployments.
Plugin CLI tools (on-node via SSH)
ssh waggle-dev-node-V032
sudo pluginctl build .
sudo pluginctl deploy -n my-counter 10.31.81.1:5000/local/my-plugin
sudo pluginctl ps
sudo pluginctl logs <plugin-id>
sudo pluginctl rm <plugin-id>
Note: ALL pluginctl commands require sudo on Thor (k3s kubeconfig is root-only).
Docker base images (waggle/plugin-base on Docker Hub)
| Image tag | Use case | Size (approx) | Arch |
|---|
1.1.1-base | Minimal Python, no ML | ~280MB | multi-arch |
1.1.1-ml | ML with CUDA | ~1.6GB arm64 / ~3.5GB amd64 | multi-arch |
1.1.1-ml-torch1.9.0 | PyTorch 1.9 | ~2.6GB arm64 / ~5.3GB amd64 | multi-arch |
1.1.1-ml-tensorflow2.3-arm64 | TensorFlow 2.3 | ~1.2GB | arm64 only |
1.1.1-ml-tensorflow2.3-amd64 | TensorFlow 2.3 | ~2.6GB | amd64 only |
1.1.1-ml-dev | Dev/debug ML | ~1.6GB | arm64 |
1.1.1-ros2-foxy | ROS2 robotics | varies | varies |
NVIDIA base images (GPU ML plugins)
For modern ML models (YOLO 8+, BioCLIP, vLLM, transformers), use NVIDIA PyTorch images instead of waggle/plugin-base:
| Image | PyTorch | CUDA | GPU Support | Notes |
|---|
nvcr.io/nvidia/pytorch:25.08-py3 | 2.8 | 13.0 | Blackwell native: sm_110 (Thor) + sm_120/sm_121 (DGX Spark) | Recommended — covers both Thor and DGX Spark |
nvcr.io/nvidia/pytorch:25.04-py3 | 2.7 | 12.9 | sm_120/sm_121 only — NO sm_110 | ⚠️ Fails on Thor ("sm_110 is not compatible") |
nvcr.io/nvidia/pytorch:25.03-py3 | 2.7 | 12.8.1 | Blackwell sm_120 only | No Thor support |
nvcr.io/nvidia/pytorch:25.01-py3 | 2.6 | 12.8 | Blackwell sm_120 (first) | Minor caveats (cuSPARSELt), no Thor |
nvcr.io/nvidia/pytorch:24.06-py3 | 2.4 | 12.4 | Hopper max (sm_90) | ⚠️ WRONG for Blackwell — silently falls back to CPU |
nvcr.io/nvidia/l4t-pytorch:* | varies | varies | Jetson-specific | ARM64 only |
CRITICAL: On Blackwell GPUs, using 24.06-py3 causes PyTorch to silently fall back to CPU — inference appears to hang (extremely slow). On Thor nodes specifically, 25.04-py3 also fails (no sm_110 cubins). Always use 25.08-py3 or newer for any deployment targeting both DGX Spark and Thor. The 25.08 image requires driver R575+ (DGX Spark: 580.159, Thor: 580.00 — both compatible).
All NGC PyTorch containers are multi-arch (AMD64 + ARM64 SBSA) via manifests — docker pull auto-selects the right architecture. The pytorch/pytorch:* Docker Hub images are AMD64-only (no ARM64).
Pre-download model weights at build time to explicit paths (edge nodes may lack internet). Use curl -L -o /app/models/<name>.pt <url> for YOLO, huggingface-cli download for HF models. Do NOT rely on ultralytics auto-download (caches to ~/.config/Ultralytics/ — path changes between versions). Set TRANSFORMERS_OFFLINE=1 and HF_DATASETS_OFFLINE=1 in Dockerfile. See references/ml-plugin-patterns.md for baking patterns and the yolov7-fire ECR reference.
"Further Reading" appendix pattern
Each plugin's overview.md should end with a "Further Reading" appendix that helps students go beyond the plugin's scope. Topics to cover:
- Custom-trained models — how to fine-tune on domain-specific data and deploy with
--model custom.pt
- Temporal analysis — the plugin's limitations (single-frame, no tracking) and how the library supports tracking/temporal features
- Other tasks — table of the library's task modes (detect, segment, pose, classify) with use cases, even though the plugin only uses one
- Relevant blog posts / papers — links to upstream vendor articles showing real-world applications in the plugin's domain
This is especially important for student-facing docs. The plugin demonstrates one use case; the appendix shows the frontier.
Design principle: self-contained teaching units
Each plugin should be independently explorable by a student. A student can copy any single plugin directory out of the monorepo and have everything needed to understand, test, and deploy it: app code, Dockerfile, metadata, documentation, job specs, and tests. Shared infrastructure (venv, top-level test runner) is the exception, not the rule. When in doubt about where a file belongs, put it inside the plugin directory.
Design principle: no cross-plugin dependencies for ECR
Each plugin is submitted to ECR as an independent entry. Verify before submission: (1) app.py imports only stdlib + pip packages, no sibling plugin imports, (2) Dockerfile COPYs only requirements.txt and app.py from its own directory, (3) no relative paths reaching into other plugin dirs, (4) test_harness.py is copied (not symlinked) into each plugin's tests/ dir. Run grep -rn 'other-plugin-name' plugins/this-plugin/ to verify.
Build pipeline (Makefile)
Each standalone plugin repo should have a Makefile with at minimum:
IMAGE := plugin-name
VERSION := 0.1.0
TAG := $(IMAGE):$(VERSION)
build: docker build -t $(TAG) .
test: build test-docker # default target: build + validate
test-docker: bash tests/run-tests.sh --docker
test-native: bash tests/run-tests.sh
clean: docker rmi $(TAG) 2>/dev/null || true
make test is the single command for build + validate. make help with self-documenting ## comment targets. Exit code 0/1 for CI integration. NOTE: no audio: or download: targets — all test assets (images AND audio) must be committed to git. No runtime downloads.
Standalone repo structure (proven pattern for ECR)
Each plugin repo should match this layout:
sage-<plugin>/
├── app.py — main application
├── Dockerfile — NVIDIA base, pip constraints, model bake, patch scripts
├── DOCKER-BUILD.md — build, test, deploy, ECR submission guide
├── DEPLOY-AND-RUN.md — pluginctl one-shot test + sesctl scheduled deployment guide
├── THOR-TESTING.md — quick start for Thor node testing
├── requirements.txt — pip dependencies
├── sage.yaml — ECR metadata, inputs, version
├── overview.md — pedantic tutorial-style documentation
├── .gitignore — tests/output/, output/, *.pt, __pycache__, ._*, .DS_Store
├── .dockerignore — tests/, ecr-meta/, jobs/, *.md, .git/
├── patch_pybioclip.py — (BioCLIP only) pybioclip monkey-patch
├── ecr-meta/ — ECR submission materials (6 files)
├── jobs/ — per-plugin job YAMLs
└── tests/
├── run-tests.sh — standalone test runner (no monorepo deps)
├── test_<name>_local.py — real GPU inference test
├── test_harness.py — test utilities (copied, not shared)
├── test-images/ — committed test images
└── output/ — gitignored test output
run-tests.sh must work standalone — no references to monorepo venv paths or parent directories. The test script should use the system Python or a local venv, not assume ../../tests/.venv exists.
Template repos
waggle-sensor/edge-app-templatewaggle-sensor/cookiecutter-sage-app
Container Runtime & Scheduling Model
Sage plugins follow a one-shot execution model: the scheduler fires a container, it processes, publishes, and exits. Pods are ephemeral — no persistent filesystem between runs. One-shot cron is standard; continuous pods are the exception (see references/job-scheduling-and-liveness.md). Publish a heartbeat every cycle or quiet jobs look dead. arm64/Thor: portal build crashes (QEMU), push denied — build local+sideload, see references/ecr-arm64-thor-deployment.md. Three scheduling modes:
| Mode | YAML | Use case |
|---|
| Cronjob | schedule: "*/10 * * * *" | Most common — periodic sampling |
| Lambda | when: {name: ..., cond: ...} | Data-driven triggers |
| Always | schedule: "always" | Continuous (rare, discouraged for GPU) |
k3s + containerd cache Docker image layers locally after first pull. Two plugins sharing the same base image (e.g. both on nvcr.io/nvidia/pytorch:24.06-py3) only download unique layers for the second one. Always use explicit version tags, never :latest — with :latest, every cron tick triggers a registry check.
Dockerfile layer ordering matters: place rarely-changing layers (base, requirements, model weights) BEFORE frequently-changing layers (app.py). Changing a layer invalidates everything below it. Put COPY app.py /app/ as the LAST layer so code changes don't trigger model re-downloads.
See references/runtime-packaging-patterns.md for full details: pod lifecycle timeline, containerd caching mechanics, imagePullPolicy defaults, cold-start optimization checklist, and analysis of 4 production reference plugins.
Job Scheduling (sesctl)
Official docs:
Science rule syntax
Format: action : condition
Actions:
schedule(image) — run a plugin containerpublish(topic, value) — publish a messageset(variable, value=X) — set a variable
Condition functions:
v(measurement, sensor=, since="-1h") — get measurement valuetime(unit) — current time ("hour", "minute", etc.)cronjob(name, crontab) — cron schedule (name must be unique per job)after(name, since="-1d") — true after a named eventrate(measurement, since, window, unit) — rate of change
Example rules:
schedule(object-counter): cronjob('run-counter', '*/5 * * * *')
schedule(object-counter): v('env.temperature') > 30.0
Job YAML (see templates/job.yaml for full example)
sesctl CLI
export SES_HOST=https://es.sagecontinuum.org
export SES_USER_TOKEN=<token-from-portal>
sesctl create -f job.yaml
sesctl stat
sesctl submit -j <job-id>
sesctl rm -j <job-id>
See references/job-scheduling-and-liveness.md for: ECR app metadata vs Docker image (both must exist for SES — two distinct failure modes), one-shot cron vs continuous pods, pod namespace meaning (default=pluginctl, ses=scheduler), heartbeat/observability pattern, sesctl flag corrections, sage.yaml float type reality, avian-diversity-monitoring baseline schedules, and BioCLIP cold-start considerations.
create returns a numeric ID; capture it. Always run sesctl <subcmd> --help — this CLI's surface drifts from the published web docs.
Deployment model, namespace diagnostics, ECR gate, reading job schedules:
see references/deployment-and-diagnostics.md. Quick diagnostics:
pods in the ses namespace = scheduler-launched (official); default
namespace = hand-deployed via pluginctl (a default pod with multi-day uptime
is a continuous test pod, NOT a scheduled job). sesctl submit requires the
app to be registered in the ECR catalog (portal build) even though
pluginctl/docker pull succeed without it — hence 400 ... does not exist in ECR on submit. Most Sage jobs are cron one-shot, not long-running; weigh
cold-start vs warm-pod before choosing --continuous Y.
Reolink audio auth (BirdNET etc.): FLV/BCS needs creds as query params
cron-job liveness checks — see references/job-scheduling-and-liveness.md.**
Data Access (timeseries)
Scalar / event data from Sage plugins lands in Beehive and is queried at:
POST https://data.sagecontinuum.org/api/v1/query (public, no auth) → NDJSON.
Full patterns: references/timeseries-data-query-api.md. Tutorial: Access and use data.
Example — recent samples from plugin-iio
curl https://data.sagecontinuum.org/api/v1/query \
-d '{"start":"-30m","filter":{"plugin":".*plugin-iio.*"}}'
import sage_data_client
df = sage_data_client.query(
start="-30m",
filter={
"plugin": ".*plugin-iio.*",
},
)
By node / measurement
import sage_data_client
df = sage_data_client.query(
start="-1h",
filter={"name": "env.temperature", "vsn": "W030"}
)
curl -s -X POST https://data.sagecontinuum.org/api/v1/query -d '
{
"start": "-1h",
"filter": {
"vsn": "H00F",
"name": "env.count.*"
}
}'
Filters: name (measurement), sensor (hardware), vsn (node ID), plugin (source plugin). Supports wildcards / regex (e.g. ".*plugin-iio.*").
Large files (images, audio): stored on Open Storage Network (S3-compatible object store), not in the timeseries DB.
Triggers
- Cloud-to-edge: data arrival in Beehive triggers edge job (Lambda Triggers)
- Edge-to-cloud: edge data triggers HPC/cloud compute via sage-data-client polling
- External notifications (Slack, email, etc.): run a watcher script externally that polls the data API and reacts. Containers on Sage nodes are network-restricted and cannot reach external services. Host processes on some nodes (e.g. Thor via SSH) CAN reach external URLs — but the recommended pattern is a cloud-side watcher, not a host-side process. See
references/cloud-trigger-notifications.md for the full pattern, Slack webhook + image upload examples, secret management, and reference implementations (hummingbird-watcher, wildfire-trigger, severe-weather-trigger).
- Shareable web viz of fleet data (public API + CORS proxy, themes, WebGL fallback):
references/sage-data-web-viz.md · 3D globe: references/sage-data-3d-globe-viz.md · template: templates/sage-cors-proxy-server.py
- Share a Sage knowledge bundle (skills tap, secret scrubbing, starter README):
references/sharing-a-sage-knowledge-bundle.md
GitHub Organizations
sagecontinuum — public repo catalog: references/sagecontinuum-repos-index.md (sage-data-client, sage-gui, sage-storage-*, sage-object-store, …; private skipped)waggle-sensor — public repo catalog: references/waggle-sensor-repos-index.md; pywaggle SDK: https://github.com/waggle-sensor/pywaggle (also waggle-edge-stack, edge-scheduler / pluginctl+sesctl, plugin-base, virtual-waggle, sage-mcp)
Hermes Native MCP Integration
Sage MCP (pre-wired)
Wire up the Sage MCP server as a native Hermes tool so all 29 tools are callable directly (no curl/JSON-RPC):
printf 'n\nY\n' | hermes mcp add sage --url "https://mcp.sagecontinuum.org/mcp"
hermes mcp list
After adding, start a new session. Tools appear as mcp_sage_* (e.g. mcp_sage_list_available_nodes, mcp_sage_find_plugins_for_task). No auth needed for read-only operations (data queries, node listing, plugin search, docs). Job submission tools (submit_sage_job, submit_plugin_job) require a Bearer token configured via portal.
GitHub MCP (optional — enable when needed)
Official server: MCP Registry — GitHub · remote endpoint https://api.githubcopilot.com/mcp/.
Shipped in profile mcp.json as github with enabled: false until you add a PAT. Setup: references/github-mcp-server.md.
hermes mcp add github --url "https://api.githubcopilot.com/mcp/"
hermes mcp list
Use for live GitHub repos/issues/PRs (e.g. waggle-sensor/pywaggle). Prefer /mcp/readonly or a read-only PAT when you only need browse access.
Hugging Face MCP (optional — enable when needed)
Official server: Hugging Face MCP docs · remote endpoint https://huggingface.co/mcp.
Shipped in profile mcp.json as huggingface with enabled: false until you add an HF token. Toggle tools/Spaces at settings/mcp. Setup: references/huggingface-mcp-server.md.
hermes mcp add huggingface --url "https://huggingface.co/mcp"
hermes mcp list
Use for Hub models/datasets/Spaces/papers, HF documentation search, and Hub Jobs. Prefer Sage MCP for nodes/data; GitHub MCP for repos/PRs.
Milvus SDK Code Helper (pre-wired)
Official helper: milvus.io docs · endpoint https://sdk.milvus.io/mcp/ (header Accept: text/event-stream). Shipped enabled in mcp.json as sdk-code-helper. Setup notes: references/milvus-sdk-helper-mcp.md.
hermes mcp add sdk-code-helper --url "https://sdk.milvus.io/mcp/"
hermes mcp list
Use when generating vector-search code: prefer Milvus Lite (pip install -U "pymilvus[milvus-lite]", MilvusClient("./demo.db")) and current MilvusClient APIs — not full Milvus Standalone/Docker unless the user asks.
Working with This Project
- Project notes live at
~/AI-projects/Sage-agents/sage-agents.md (15K+ bytes of detailed research)
- Pete Beckman leads the Sage project (Northwestern University, pete.beckman@northwestern.edu — no longer at ANL). He has deep domain expertise and works on ML plugins for Thor nodes (128GB unified memory, aarch64, GB10 Blackwell). Use his Northwestern email in all sage.yaml
authors fields and ecr-meta credits.
- DGX Spark and Thor nodes share the same 128GB unified memory architecture — model sizing for one applies to the other
- When developing plugins, test with
virtual-waggle (simulated node environment)
- Data API is public and unauthenticated — good for quick verification
- Node IDs look like W030, W09E, W0A0 (hex-style short codes called VSN)
Pitfalls
