# From AI Copilot to Commander
**AI Agents and RAG for Security Automation and Investigation โ a two-day hands-on
training, BSides Augusta 2026.**
Building AI systems that reliably run security investigations.
๐
**October 21โ22, 2026 ยท 09:00โ17:00 ET both days** (one-hour lunch each day)
๐ Bit room, first floor, Hull-McKnight building, Georgia Cyber Center. Check in at the
BSides Augusta desk on Wednesday morning.
Instructors: **Leo Meyerovich** and **Sindre Breda** (Graphistry / Louie.ai), with Louie.ai
staff on the floor. Sindre runs the room both days. Leo teaches Day 1 until the afternoon
and the first half of Day 2.
This is the full two-day course โ the BSides Canberra and BSides Las Vegas 2026 workshops
were one-day distillations of it.
๐ฆ **Repo:** <https://github.com/graphistry/bsides-augusta-2026>
๐ฌ **Chat:** the class Discord channel (invite sent by email). Links, questions and lab
help during and after the class.
๐ **Sign up first (free):** a [**Graphistry Hub**](https://hub.graphistry.com) account โ
you'll use it for the Louie paths and to render graphs.
---
## The idea
Modern security work is moving from *using* an AI assistant to *commanding* a fleet of
them. But agents are unreliable in a way that compounds: if each step of an
investigation is **70% reliable, a 10-step run lands at ~3%.** And you cannot simply carry
over AI coding practices โ assumptions like unit tests no longer apply. This course is
about what actually moves that number, and about being able to **prove** it moved.
Everything is built around one spine:
> **plan โ run โ eval**
Day 1 is the **copilot** half: run models on your own laptop, including a small decision
model, reach a real Splunk cyber range through an agent harness, and investigate a
multi-domain incident. Day 2 is the **commander** half: make the agent reliable with
skills and planning, build an eval loop that measures whether your changes helped (and
whether the score is even real), turn a cheap model into a calibrated triage router,
secure the whole thing, and ship a full investigation deliverable.
You leave with a working investigation agent, a reusable timelining skill, an eval loop,
a measured triage router, and a defensible, evidence-based opinion about what AI can and
cannot do in your SOC.
## Who it's for
Beginner to advanced. No prior AI, ML, or agent experience needed, no GPU required.
- **Analysts** (SOC, IR, DFIR, hunt, threat intel, vuln mgmt) โ go beyond copilots.
- **Engineers and platform teams** โ build agents, run models on-prem, connect your own tools.
- **Leaders** โ understand ROI, technology trajectories, and how to responsibly target outcomes.
Every lab has a **prompt-first (no-code)** path and a **code-first (Python)** path, and
three levels: **base**, **stretch**, and **hard mode** (e.g. an uncontaminated CTF, your own
MCP, subagents). If the base level feels easy, keep going.
## The labs
One incident โ **BOTSv3 Incident B2, an AWS key compromise** spanning email, CloudTrail,
IAM, and EC2 โ runs across the whole course, so each lab builds on the last.
| Lab | Day | Time | What you do |
|-----|-----|------|-------------|
| **[Lab 0 โ Setup](labs/lab0/lab0_guide.md)** | 1 | 35m | Reach Splunk three ways (MCP harness, `louie-py`, Louie desktop) and run your first query + graph. |
| **[Lab 1 โ DIY LLM + decision API](labs/lab1/lab1_guide.md)** | 1 | 60m | Run an OSS LLM locally and watch it hallucinate on SOC questions. Then run [Laya](https://github.com/NandhaKishorM/laya), an open-source Jev-compatible decision model, zero-shot on B2 CloudTrail events: it starts *worse than random* โ see what context fixes. |
| **[Lab 2 โ AI CTF](labs/lab2/lab2_guide.md)** | 1 | 60m | Investigate B2 โ **BYO agent** (Claude Code / OpenCode over MCP) or **running with Louie** โ across the six forensic questions, with vibes evals. |
| **[Lab 3 โ Skills, timelining & evals](labs/lab3/lab3_guide.md)** | 2 | 60m | Build a BOTS Q skill and a timelining skill, run it on the attack chain, score it against a reference, error-analyze and fix it. Code track: automate via the `louie-py` API. |
| **[Lab 4 โ Triage router & contamination](labs/lab4/lab4_guide.md)** | 2 | 60m | Turn Laya into a router that suppresses routine events before they reach the expensive agent: test / validation / holdout splits, temperature calibration, a recall-vs-suppression threshold. Then check whether your agent's CTF score is contamination. |
| **[Capstone โ IR report + graphs](labs/capstone/capstone_guide.md)** | 2 | 45m | In-room, instructor-reviewed: an IR report for B2 plus a timeline-entity graph and an infrastructure graph on Graphistry Hub. Alternative: craft malicious events that get past your own classifier. |
Full lab breakdown, tracks, and the mapping to the one-day BSides editions:
**[`labs/README.md`](labs/README.md)**.
## Schedule
Two days, 09:00โ17:00 ET, one-hour lunch each day. Room logistics and any day-of changes
go out on Discord.
### Day 1 โ Copilot: models, harnesses, investigation
| Time | Length | Block |
|------|--------|-------|
| 09:00โ09:40 | 40m | Intro, group intros & use cases |
| 09:40โ10:15 | 35m | **Lab 0** โ Setup: reach Splunk three ways |
| 10:15โ10:35 | 20m | Talk โ AI fundamentals: vectors, generation, decision APIs (Jev) |
| 10:35โ11:00 | 25m | Talk โ Self-hosting: MoE, quantization, open-weight models (incl. 10m break) |
| 11:00โ12:00 | 60m | **Lab 1** โ DIY LLM + decision API (local LLM, Laya zero-shot) |
| 12:00โ13:00 | 60m | Lunch |
| 13:00โ13:45 | 45m | Talk โ Modern harnesses, thick MCPs & the semantic layer |
| 13:45โ14:00 | 15m | Break |
| 14:00โ15:00 | 60m | **Lab 2** โ AI CTF: BOTSv3 B2 + vibes evals |
| 15:00โ15:45 | 45m | Talk โ Agents, skills & planning: BOTS baselines, plan.md, planning tricks |
| 15:45โ16:00 | 15m | Break |
| 16:00โ16:30 | 30m | Talk โ Maturity & evolution: evals & AI investigation factories |
| 16:30โ17:00 | 30m | Group discussion |
### Day 2 โ Commander: evals, security, delivery
| Time | Length | Block |
|------|--------|-------|
| 09:00โ09:30 | 30m | Morning discussion & recap (Day 1 leftovers: thick vs thin MCPs, the semantic layer) |
| 09:30โ10:15 | 45m | Talk โ Timelining, context graphs & RAG: timelining-first methodology, RAG fundamentals & memory, advanced retrieval & knowledge graphs |
| 10:15โ10:30 | 15m | Break |
| 10:30โ11:00 | 30m | Talk โ Evals: core (data splits, Cohen's kappa, eval-driven skill loops) & adversarial (contamination, eval integrity) |
| 11:00โ12:00 | 60m | **Lab 3** โ Skills, timelining & evals |
| 12:00โ13:00 | 60m | Lunch |
| 13:00โ13:45 | 45m | Talk โ Securing agents (threat models, the lethal trifecta, MCP auth, safeguards) & AI compliance |
| 13:45โ14:00 | 15m | Break |
| 14:00โ14:30 | 30m | Talk โ Planning II: investigation-informed planning, multi-agents, pipelines ([`resources/`](resources/README.md)) |
| 14:30โ15:30 | 60m | **Lab 4** โ Triage router & contamination (finish Lab 3 if behind) |
| 15:30โ16:15 | 45m | **Capstone** โ IR report + graphs (or: get past the classifier) |
| 16:15โ17:00 | 45m | Wrap & guided discussion: key takeaways, round-robin "one thing you'll try next week", take-home challenge, exit survey |
Excluding lunch, that is 14 hours: about 5h20m of labs, 7h30m of talks and discussion,
and 1h10m of breaks. The lab guides budget a little more time than the slots; use the lab
time for the base level and come back to stretch and hard mode after class.
## Three ways to reach Splunk
The course meets you at your comfort level โ pick a lane, or try them all:
- **BYO agent** โ your own coding agent (Claude Code, OpenCode, Codex) over the Splunk
**MCP** server. The raw harness experience, and the baseline everything else improves on.
- **`louie-py`** โ the code-first client: a few lines of Python that query Splunk and
return dataframes *and* Graphistry graphs.
- **Louie desktop** โ the same copilot in a no-code GUI ([download.louie.ai](https://download.louie.ai)).
## Getting started
```bash
git clone https://github.com/graphistry/bsides-augusta-2026.git
cd bsides-augusta-2026
```
Then do **[Lab 0](labs/lab0/lab0_guide.md)** before Day 1 if you can โ it installs
everything and catches account, connector, and credential problems early.
Bring a laptop (8 GB+ RAM) with a modern browser, Wi-Fi, and its power adapter. You'll need:
- **Splunk access** with BOTSv3 loaded (cyber-range credentials provided in-session)
- **Python 3.12+** โ required by the Splunk **MCP** server (`splunk-mcp-server2`), the
`louie-py` code path, and the Laya labs. **Not needed if you only use Louie desktop** (no code).
- **Claude Code CLI** or **OpenCode** (for the BYO-agent path; optional otherwise)
- A free **[Graphistry Hub](https://hub.graphistry.com)** account (for the Louie paths)
**Download these ahead of time** โ conference Wi-Fi might be slow/unavailable:
```bash
# Laya decision model (Labs 1 and 4) โ ~1 GB of weights, no GPU needed
pip install "louieai>=0.7" graphistry laya==0.4.8 pandas
python -c "import laya; laya.load()"
# Small local LLM (Lab 1) โ ~2 GB. Or use the Colab notebook, which installs Ollama for you.
ollama pull phi3:mini
```
Setup commands and credential handling: see **[`labs/README.md#setup`](labs/README.md#setup)**.
## Working through the labs with an AI assistant
These are teaching exercises. If you pair with an LLM, the intent is that it helps you
*write and debug code* โ not that it solves the CTF or opens the answer key for you. The
lab notebooks say so explicitly; the evals labs (Labs 3โ4) relax this since the result is
already known.
## Repository layout
```
labs/ the course โ one folder per lab, plus labs/README.md
resources/ reference material from the talks: PLAN.md task plan template,
skills/investigation-loop (Planning II)
lab0/ ... setup: reach Splunk three ways (+ mcp_config.json)
lab1/ ... DIY LLM + decision API: local OSS model, Laya zero-shot
lab2/ ... AI CTF: BOTSv3 B2 investigation
lab3/ ... skills, timelining, evals, API automation
lab4/ ... Laya triage router, contamination check (+ offline fallback data)
capstone/ .. in-room capstone: IR report + graphs
```
> **Credentials never get committed** โ `.env` / `env` and the instructor answer key
> are gitignored. Never paste live secrets into a notebook or commit.