keymaster

name

keymaster

description

ALWAYS ACTIVE - Intelligent API key rotation and fallback system for NVIDIA LLM requests. Automatically handles rate limiting, timeouts, and context compaction when switching between multiple API keys. Integrates with OpenClaw's agent-orchestrator for seamless long-running multi-agent tasks. MANDATORY TRIGGERS: rate limit, timeout, key rotation, fallback, nvidia api, multi-agent, context compaction

Keymaster

Intelligent API key fallback system that rotates through NVIDIA keys with context preservation and automatic compaction.

Keymaster integrates with your existing OpenClaw configuration - it reads keys from ~/.openclaw/openclaw.json and ~/.openclaw/agents/main/agent/auth-profiles.json.

---

Overview

Keymaster wraps all LLM requests and manages a pool of NVIDIA API keys. When one key hits a rate limit or timeout, it automatically falls back to the next key while preserving conversation context.

---

NEW: OpenClaw Proxy (v2.0)

Keymaster now includes a transparent proxy that sits between OpenClaw and NVIDIA API:

OpenClaw → Local Proxy (Port 8787) → Rotates Keys → NVIDIA API

This solves the core problem: OpenClaw itself can now run 60+ minute tasks by automatically rotating through all your keys.

How It Works

1. Start the proxy (keymaster start)
2. OpenClaw connects to proxy (configured automatically)
3. Use OpenClaw normally - keys rotate automatically behind the scenes

Proxy Features

• Transparent - OpenClaw doesn't know it's using a proxy
• Streaming support - Full SSE passthrough
• Random key selection - Better distribution than round-robin
• Cyclical rotation - When key 5 rate-limited, cycles back to key 1
• Header-based cooldown - Uses rate-limit headers when available
• Retry logic - 5 retries with exponential backoff
• In-memory state - No file I/O bottlenecks

Quick Proxy Setup

# 1. Configure with your keys
# Edit ~/.openclaw/openclaw.json and ~/.openclaw/agents/main/agent/auth-profiles.json

# 2. Start the service
keymaster start

# 3. Check status
keymaster status

# 4. Use OpenClaw normally
openclaw "create me a complex dashboard"
# Works for 60+ minutes, rotating through all keys!

See Setup section below for details.

---

Configuration for Long-Running Tasks (24h Support)

Keymaster includes optimized settings for tasks that run 24+ hours. Here's the complete configuration structure:

Complete openclaw.json Configuration

{
  "auth": {
    "profiles": {
      "nvidia:primary": { "provider": "nvidia", "mode": "api_key" },
      "nvidia:secondary": { "provider": "nvidia", "mode": "api_key" },
      "nvidia:tertiary": { "provider": "nvidia", "mode": "api_key" },
      "nvidia:quaternary": { "provider": "nvidia", "mode": "api_key" },
      "nvidia:quinary": { "provider": "nvidia", "mode": "api_key" }
    },
    "order": {
      "nvidia": [
        "nvidia:primary",
        "nvidia:secondary",
        "nvidia:tertiary",
        "nvidia:quaternary",
        "nvidia:quinary"
      ]
    }
  },
  "models": {
    "mode": "merge",
    "providers": {
      "nvidia-key-1": {
        "baseUrl": "http://127.0.0.1:8787",
        "apiKey": "YOUR_KEY_1",
        "api": "openai-completions",
        "models": [{
          "id": "nvidia/nemotron-3-super-120b-a12b",
          "name": "Nemotron 3 Super",
          "reasoning": false,
          "input": ["text"],
          "cost": { "input": 0.000002, "output": 0.000008, "cacheRead": 0, "cacheWrite": 0 },
          "contextWindow": 256000,
          "maxTokens": 16384
        }]
      },
      "nvidia-key-2": {
        "baseUrl": "http://127.0.0.1:8787",
        "apiKey": "YOUR_KEY_2",
        "api": "openai-completions",
        "models": [{
          "id": "nvidia/nemotron-3-super-120b-a12b",
          "name": "Nemotron 3 Super",
          "reasoning": false,
          "input": ["text"],
          "cost": { "input": 0.000002, "output": 0.000008, "cacheRead": 0, "cacheWrite": 0 },
          "contextWindow": 256000,
          "maxTokens": 16384
        }]
      },
      "nvidia-key-3": {
        "baseUrl": "http://127.0.0.1:8787",
        "apiKey": "YOUR_KEY_3",
        "api": "openai-completions",
        "models": [{
          "id": "nvidia/nemotron-3-super-120b-a12b",
          "name": "Nemotron 3 Super",
          "reasoning": false,
          "input": ["text"],
          "cost": { "input": 0.000002, "output": 0.000008, "cacheRead": 0, "cacheWrite": 0 },
          "contextWindow": 256000,
          "maxTokens": 16384
        }]
      },
      "nvidia-key-4": {
        "baseUrl": "http://127.0.0.1:8787",
        "apiKey": "YOUR_KEY_4",
        "api": "openai-completions",
        "models": [{
          "id": "nvidia/nemotron-3-super-120b-a12b",
          "name": "Nemotron 3 Super",
          "reasoning": false,
          "input": ["text"],
          "cost": { "input": 0.000002, "output": 0.000008, "cacheRead": 0, "cacheWrite": 0 },
          "contextWindow": 256000,
          "maxTokens": 16384
        }]
      },
      "nvidia-key-5": {
        "baseUrl": "http://127.0.0.1:8787",
        "apiKey": "YOUR_KEY_5",
        "api": "openai-completions",
        "models": [{
          "id": "nvidia/nemotron-3-super-120b-a12b",
          "name": "Nemotron 3 Super",
          "reasoning": false,
          "input": ["text"],
          "cost": { "input": 0.000002, "output": 0.000008, "cacheRead": 0, "cacheWrite": 0 },
          "contextWindow": 256000,
          "maxTokens": 16384
        }]
      }
    }
  },
  "agents": {
    "defaults": {
      "model": {
        "primary": "nvidia-key-1/nvidia/nemotron-3-super-120b-a12b",
        "fallbacks": [
          "nvidia-key-2/nvidia/nemotron-3-super-120b-a12b",
          "nvidia-key-3/nvidia/nemotron-3-super-120b-a12b",
          "nvidia-key-4/nvidia/nemotron-3-super-120b-a12b",
          "nvidia-key-5/nvidia/nemotron-3-super-120b-a12b"
        ]
      },
      "models": {
        "nvidia-key-1/nvidia/nemotron-3-super-120b-a12b": {
          "alias": "Nemotron 3 Super (Key 1)"
        },
        "nvidia-key-2/nvidia/nemotron-3-super-120b-a12b": {
          "alias": "Nemotron 3 Super (Key 2)"
        },
        "nvidia-key-3/nvidia/nemotron-3-super-120b-a12b": {
          "alias": "Nemotron 3 Super (Key 3)"
        },
        "nvidia-key-4/nvidia/nemotron-3-super-120b-a12b": {
          "alias": "Nemotron 3 Super (Key 4)"
        },
        "nvidia-key-5/nvidia/nemotron-3-super-120b-a12b": {
          "alias": "Nemotron 3 Super (Key 5)"
        }
      },
      "bootstrapMaxChars": 20000,
      "bootstrapTotalMaxChars": 150000,
      "compaction": {
        "mode": "safeguard",
        "reserveTokensFloor": 20000,
        "memoryFlush": {
          "enabled": true,
          "softThresholdTokens": 4000,
          "systemPrompt": "Session nearing compaction. Store durable memories now.",
          "prompt": "Write any lasting notes to memory/YYYY-MM-DD.md; reply with NO_REPLY if nothing to store."
        }
      },
      "timeoutSeconds": 86400,
      "maxConcurrent": 3,
      "subagents": {
        "maxConcurrent": 2
      }
    }
  },
  "tools": {
    "profile": "full",
    "web": {
      "search": {
        "enabled": true
      },
      "fetch": {
        "enabled": true
      }
    },
    "loopDetection": {
      "enabled": true,
      "warningThreshold": 10,
      "criticalThreshold": 20,
      "globalCircuitBreakerThreshold": 30,
      "historySize": 30,
      "detectors": {
        "genericRepeat": true,
        "knownPollNoProgress": true,
        "pingPong": true
      }
    },
    "exec": {
      "backgroundMs": 10000,
      "timeoutSec": 86400,
      "cleanupMs": 1800000,
      "notifyOnExit": true,
      "notifyOnExitEmptySuccess": false,
      "applyPatch": {
        "enabled": true,
        "allowModels": [
          "nvidia/nemotron-3-super-120b-a12b:free"
        ]
      }
    }
  }
}

Complete auth-profiles.json

{
  "version": 1,
  "profiles": {
    "nvidia:primary": {
      "type": "api_key",
      "provider": "nvidia-key-1",
      "key": "YOUR_KEY_1",
      "priority": 1,
      "coordinator_priority": 1,
      "is_primary_coordinator": true,
      "can_act_as_coordinator": true,
      "role": "coordinator",
      "agent_mapping": "main",
      "model": "nvidia/nemotron-3-super-120b-a12b",
      "fallback_to": "nvidia:secondary",
      "fallback_chain": ["nvidia:secondary", "nvidia:tertiary", "nvidia:quaternary", "nvidia:quinary"]
    },
    "nvidia:secondary": {
      "type": "api_key",
      "provider": "nvidia-key-2",
      "key": "YOUR_KEY_2",
      "priority": 2,
      "coordinator_priority": 2,
      "can_act_as_coordinator": true,
      "role": "strategist",
      "agent_mapping": "charlie",
      "model": "nvidia/nemotron-3-super-120b-a12b",
      "fallback_to": "nvidia:tertiary",
      "fallback_chain": ["nvidia:tertiary", "nvidia:quaternary", "nvidia:quinary", "nvidia:primary"]
    },
    "nvidia:tertiary": {
      "type": "api_key",
      "provider": "nvidia-key-3",
      "key": "YOUR_KEY_3",
      "priority": 3,
      "coordinator_priority": 3,
      "can_act_as_coordinator": false,
      "role": "heavy_lifter",
      "agent_mapping": "echo",
      "model": "nvidia/nemotron-3-super-120b-a12b",
      "fallback_to": "nvidia:quaternary",
      "fallback_chain": ["nvidia:quaternary", "nvidia:quinary", "nvidia:primary", "nvidia:secondary"]
    },
    "nvidia:quaternary": {
      "type": "api_key",
      "provider": "nvidia-key-4",
      "key": "YOUR_KEY_4",
      "priority": 4,
      "coordinator_priority": 4,
      "can_act_as_coordinator": false,
      "role": "worker",
      "agent_mapping": "alpha",
      "model": "nvidia/nemotron-3-super-120b-a12b",
      "fallback_to": "nvidia:quinary",
      "fallback_chain": ["nvidia:quinary", "nvidia:primary", "nvidia:secondary", "nvidia:tertiary"]
    },
    "nvidia:quinary": {
      "type": "api_key",
      "provider": "nvidia-key-5",
      "key": "YOUR_KEY_5",
      "priority": 5,
      "coordinator_priority": 5,
      "can_act_as_coordinator": false,
      "role": "fixer",
      "agent_mapping": "delta",
      "model": "nvidia/nemotron-3-super-120b-a12b",
      "fallback_to": "nvidia:primary",
      "fallback_chain": ["nvidia:primary", "nvidia:secondary", "nvidia:tertiary", "nvidia:quaternary"]
    }
  },
  "lastGood": { "nvidia": "nvidia:primary" },
  "usageStats": {},
  "keymaster": {
    "enabled": true,
    "auto_rotation": true,
    "context_compaction": true,
    "compaction_threshold": 0.8,
    "cooldown_seconds": 60,
    "max_retries_per_key": 3,
    "state_persistence": true,
    "parallel_tool_calls": false,
    "timeout_seconds": 86400,
    "max_concurrent": 3
  }
}

Key Configuration Values

Setting	Value	Purpose
bootstrapMaxChars	20000	Maximum characters for bootstrap
bootstrapTotalMaxChars	150000	Total maximum bootstrap characters
timeoutSeconds	86400 (24h)	Maximum time for a single agent run
maxConcurrent	3	Prevents resource exhaustion
subagents.maxConcurrent	2	Max concurrent subagents
compaction.mode	"safeguard"	Auto-compact context before overflow
compaction.reserveTokensFloor	20000	Minimum tokens to reserve
tools.exec.timeoutSec	86400 (24h)	Tool execution timeout
tools.loopDetection.enabled	true	Detects infinite loops

---

Quick Setup (For LLM Assistants)

INSTRUCTION TO LLM: If a user says "set up keymaster with my keys" or provides API keys, follow this exact process:

Step 1: Collect User's API Keys

Ask the user for:

• How many NVIDIA API keys they have (supports 1-N keys)
• Each API key string (format: nvapi-...)
• Preferred model (default: nvidia/nemotron-3-super-120b-a12b)

Step 2: Create Config Files

CRITICAL: When updating existing openclaw.json:

• ONLY modify auth, models.providers, and agents.defaults sections
• NEVER touch gateway, channels, plugins, skills, hooks, or other sections
• PRESERVE all existing Telegram, Discord, Brave, and other integrations

Create:

1. ~/.openclaw/openclaw.json - Use the complete configuration above
2. ~/.openclaw/agents/main/agent/auth-profiles.json - Use the auth-profiles template above

Step 3: Install the Skill

# Clone the repo if not already present
git clone https://github.com/dommurphy155/Keymaster.git ~/.openclaw/skills/keymaster

# Run the installer
cd ~/.openclaw/skills/keymaster
bash install.sh

Step 4: Start the Service

# Start the service
keymaster start

# Check status
keymaster status

# Verify health
keymaster health

How It Works Now

Before (fails at 10 mins):
  You → OpenClaw → NVIDIA API (Key 1) → Rate Limit → ❌ STOPS

After (works 60+ mins):
  You → OpenClaw → Proxy (localhost:8787)
                     ↓
              Key 1 works 10 mins → 429
                     ↓
              Switch to Key 2 → works 10 mins → 429
                     ↓
              Switch to Key 3 → works 10 mins → 429
                     ↓
              ...cycles through all 5 keys...
                     ↓
              If all cooling, waits and cycles back to Key 1
                     ↓
              ✅ Task completes!

Proxy Commands:

# Check status
keymaster status

# View logs
tail -f ~/.openclaw/keymaster.log

# Stop service
keymaster stop

# Restart service
keymaster restart

---

Integration with OpenClaw

Keymaster is designed to work seamlessly with your existing OpenClaw setup:

~/.openclaw/openclaw.json           # Contains API keys and model providers
~/.openclaw/agents/main/agent/auth-profiles.json  # Contains key metadata and fallback chains
~/.openclaw/skills/keymaster/       # This skill
  ├── scripts/
  │   ├── key_pool_manager.py      # Uses your OpenClaw configs
  │   ├── request_wrapper.py        # Wraps requests with fallback
  │   ├── context_compactor.py     # Compacts context on rotation
  │   ├── state_manager.py         # Persists conversation state
  │   ├── orchestrator_bridge.py   # Integrates with agent-orchestrator
  │   └── __init__.py              # Main exports

Key Pool (5 Keys)

Key	Provider	Role	Coordinator
nvidia:primary	nvidia-key-1	coordinator	Yes
nvidia:secondary	nvidia-key-2	strategist	Yes
nvidia:tertiary	nvidia-key-3	heavy_lifter	No
nvidia:quaternary	nvidia-key-4	worker	No
nvidia:quinary	nvidia-key-5	fixer	No

Agent-Orchestrator Integration

Keymaster works seamlessly with the agent-orchestrator skill:

from keymaster.scripts.orchestrator_bridge import OrchestratorKeymaster

bridge = OrchestratorKeymaster(
    agent_path="/path/to/agent/workspace",
    agent_name="data-collector"
)

# Make requests with automatic key rotation
response = bridge.request(
    messages=[{"role": "user", "content": "Analyze this data..."}],
    task_id="task-123",
    model="nvidia/nemotron-3-super-120b-a12b"
)

The bridge automatically:

• Creates checkpoints before requests
• Writes status to outbox/keymaster_status.json
• Handles key rotation on failures
• Resumes from checkpoints after rotation

Multi-Agent Key Coordination

from keymaster.scripts.orchestrator_bridge import MultiAgentCoordinator

coordinator = MultiAgentCoordinator()

# Assign keys based on agent roles
coordinator.assign_key_to_agent("data-collector", "worker")
coordinator.assign_key_to_agent("analyst", "strategist")
coordinator.assign_key_to_agent("coordinator", "coordinator")

How It Works

1. Request Interception

Every LLM request goes through Keymaster wrapper. Uses your configured keys from OpenClaw.

2. Error Detection

Catch these errors for rotation:

• HTTP 429 (Rate Limited)
• HTTP 408/504 (Timeout)
• Connection errors
• Token limit exceeded
• NVIDIA NVCF errors

3. Automatic Rotation

On error:

1. Mark current key as cooling down (60s)
2. Save conversation state to disk
3. Compact context if >80% of 256k limit
4. Switch to next key in fallback_chain from auth-profiles.json
5. Retry request with preserved context

4. Context Compaction

When switching keys, if context >204800 tokens (80% of 256k):

• Summarize older conversation turns
• Preserve last 10 messages fully
• Keep system prompts intact
• Store compaction summary

5. State Persistence

Maintains ~/.openclaw/keymaster_state.json:

{
  "current_key": "nvidia:primary",
  "key_status": {
    "nvidia:primary": {"available": true, "cooldown_until": null, "role": "coordinator"},
    "nvidia:secondary": {"available": false, "cooldown_until": 1234567890, "role": "strategist"}
  },
  "conversation_state": {...},
  "compaction_history": [...]
}

Usage Patterns

Pattern 1: Direct Import (Recommended)

from keymaster.scripts import keymaster_request

response = keymaster_request(
    messages=[{"role": "user", "content": "Hello"}],
    model="nvidia/nemotron-3-super-120b-a12b",
    temperature=0.7,
    max_tokens=4096
)
print(response['content'])

Pattern 2: With Conversation Recovery

from keymaster.scripts import keymaster_request_with_recovery

response = keymaster_request_with_recovery(
    messages=messages,
    conversation_id="my-task-123",  # Enables checkpointing
    model="nvidia/nemotron-3-super-120b-a12b"
)

Pattern 3: Agent-Orchestrator Sub-Agent

from keymaster.scripts.orchestrator_bridge import OrchestratorKeymaster

bridge = OrchestratorKeymaster(
    agent_path="/home/user/.openclaw/workspace/agents/my-agent",
    agent_name="my-agent"
)

response = bridge.request(
    messages=messages,
    task_id="analyze-data",
    temperature=0.7
)

# Status automatically written to outbox/keymaster_status.json

Pattern 4: Role-Based Key Assignment

from keymaster.scripts.orchestrator_bridge import get_key_for_agent_role

# Get the best key for an agent's role
key_config = get_key_for_agent_role("heavy_lifter")
print(key_config['key_name'])  # nvidia:tertiary
print(key_config['provider'])   # nvidia-key-3

Scripts

key_pool_manager.py

Manages key rotation using your OpenClaw configs:

python3 ~/.openclaw/skills/keymaster/scripts/key_pool_manager.py stats
python3 ~/.openclaw/skills/keymaster/scripts/key_pool_manager.py rotate
python3 ~/.openclaw/skills/keymaster/scripts/key_pool_manager.py role nvidia:primary

request_wrapper.py

Wraps LLM API calls with fallback logic:

python3 ~/.openclaw/skills/keymaster/scripts/request_wrapper.py --test --prompt "Hello"

orchestrator_bridge.py

Integration with agent-orchestrator:

python3 ~/.openclaw/skills/keymaster/scripts/orchestrator_bridge.py --test-assign --role worker

Configuration

Keymaster reads from your existing OpenClaw configs:

• ~/.openclaw/openclaw.json - API keys and provider settings
• ~/.openclaw/agents/main/agent/auth-profiles.json - Key metadata, roles, fallback chains

No additional configuration needed - uses your existing setup!

Fallback Chain

Each key has a fallback chain defined in auth-profiles.json:

• primary → secondary → tertiary → quaternary → quinary
• secondary → tertiary → quaternary → quinary → primary
• tertiary → quaternary → quinary → primary → secondary
• quaternary → quinary → primary → secondary → tertiary
• quinary → primary → secondary → tertiary → quaternary

This creates a distributed rotation pattern where keys are cycled through evenly.

Error Handling

Recoverable Errors (trigger rotation)

• 429 Rate Limit
• 408/504 Timeout
• Connection reset
• Token limit exceeded
• NVCF rate limit errors

Non-recoverable Errors (fail immediately)

• 401 Unauthorized (bad key)
• 400 Bad Request
• 404 Model not found
• NVCF auth errors

Best Practices

1. Always use keymaster service - Start the service, let it manage rotation
2. Use conversation_id - For long tasks, pass conversation_id for checkpointing
3. Assign roles - Use role-based key assignment for multi-agent tasks
4. Monitor status - Check keymaster status for key status
5. Handle AllKeysExhaustedError - Catch this to implement your own retry logic

CLI Usage

# Check status
keymaster status

# Manual rotation
keymaster rotate

# Test request
keymaster health

# View logs
keymaster logs
keymaster logcat

# List keys
keymaster keys

# Show cooldowns
keymaster cooldowns

# Reset cooldowns
keymaster reset

# Enable auto-start
keymaster enable

# Disable auto-start
keymaster disable

Integration Check

Verify Keymaster is working with your OpenClaw setup:

from keymaster.scripts import is_keymaster_healthy, get_keymaster_stats

# Check health
health = is_keymaster_healthy()
print(health)
# {'healthy': True, 'available_keys': 5, 'current_key': 'nvidia:primary', ...}

# Get stats
stats = get_keymaster_stats()
print(stats)

Reference Files

• references/error_patterns.md - Error detection patterns
• references/compaction_strategies.md - Context summarization methods
• references/state_format.md - State file schema
• LLM_INSTRUCTIONS.md - Complete LLM setup guide
• SETUP.md - User-friendly setup guide
• QUICKSTART.md - Quick reference
• QUICKSTART-SERVICE.md - Service quickstart