| name | validate-ql-mcp-server-tools-queries |
| description | Validate that the CodeQL Development MCP Server reliably returns non-empty AST, CFG, and CallGraph data from the bundled tools queries. Use this skill to ensure the `codeql_query_run` tool properly executes PrintAST, PrintCFG, CallGraphFrom, and CallGraphTo queries for any supported language. |
Validate QL MCP Server Tools Queries
This skill provides a systematic approach to validating that the CodeQL Development MCP Server's tools queries (PrintAST, PrintCFG, CallGraphFrom, CallGraphTo) return meaningful, non-empty output for test code across all supported languages.
Why This Skill Exists
⚠️ CRITICAL: The most common failure mode in MCP server tool validation is tests that "go through the motions" without verifying that query output is substantive.
A test that:
- ✅ Invokes
codeql_query_run with PrintAST.ql
- ✅ Receives a successful HTTP response
- ❌ Does NOT verify the output contains actual AST nodes
...is a false positive that masks real bugs.
Supported Languages
The tools queries are available for all CodeQL-supported languages:
| Language | Tools Pack Location | Test Code Extension |
|---|
| actions | server/ql/actions/tools/ | .yml |
| cpp | server/ql/cpp/tools/ | .cpp |
| csharp | server/ql/csharp/tools/ | .cs |
| go | server/ql/go/tools/ | .go |
| java | server/ql/java/tools/ | .java |
| javascript | server/ql/javascript/tools/ | .js |
| python | server/ql/python/tools/ | .py |
| ruby | server/ql/ruby/tools/ | .rb |
| rust | server/ql/rust/tools/ | .rs |
| swift | server/ql/swift/tools/ | .swift |
Tools Queries Overview
Each language pack includes four standard tools queries:
PrintAST
Purpose: Outputs a hierarchical representation of the Abstract Syntax Tree.
Expected Output: A graph with labeled nodes representing code elements (functions, classes, statements, expressions).
Validation Check: Output must contain multiple semmle.label properties with actual node names.
PrintCFG
Purpose: Outputs the Control Flow Graph showing execution paths.
Expected Output: A graph with nodes (CFG nodes) and edges (control flow transitions).
Validation Check: Output must contain both nodes and edges predicates with non-zero results.
CallGraphFrom
Purpose: Shows outbound function calls from a specified function.
Expected Output: Problem-style results showing call sites and their targets.
Validation Check: Output must contain call relationships when test code has function calls.
CallGraphTo
Purpose: Shows inbound function calls to a specified function.
Expected Output: Problem-style results showing callers of the target function.
Validation Check: Output must contain caller relationships when test code has function definitions that are called.
Validation Workflow
Step 1: Select Language and Test Code
Choose a language and ensure valid test code exists:
server/ql/<language>/tools/test/PrintAST/
├── Example1.<ext> # Test source code
├── PrintAST.expected # Expected results
└── PrintAST.qlref # Reference to query
Step 2: Extract Test Database
Use codeql_test_extract to create the test database:
{
"testPath": "server/ql/<language>/tools/test/PrintAST",
"searchPath": ["server/ql/<language>/tools/src", "server/ql/<language>/tools/test"]
}
Step 3: Run PrintAST Query
Use codeql_query_run:
{
"queryPath": "server/ql/<language>/tools/src/PrintAST/PrintAST.ql",
"database": "server/ql/<language>/tools/test/PrintAST/PrintAST.testproj",
"outputFormat": "bqrs"
}
Step 4: Interpret Results
Use codeql_bqrs_interpret with graphtext format:
{
"file": "<path-to-results.bqrs>",
"format": "graphtext",
"output": "<path-to-output.txt>"
}
Step 5: Validate Non-Empty Output
CRITICAL: Parse the output and verify it contains substantive data:
- For
PrintAST: Count nodes with semmle.label > 0
- For
PrintCFG: Verify both nodes and edges sections exist
- For
CallGraphFrom/To: Count result rows > 0 (when calls exist)
Language-Specific Examples
JavaScript Example
Test Code (server/ql/javascript/tools/test/PrintAST/Example1.js):
class Example1 {
constructor(name = 'World') {
this.name = name;
}
static main(args = []) {
const example = new Example1();
example.demo();
}
demo() {
for (let i = 0; i < 3; i++) {
console.log(i);
}
}
}
Expected AST Output (non-empty validation):
- Must contain
ClassDefinition node for Example1
- Must contain
MethodDefinition nodes for constructor, main, demo
- Must contain
ForStatement node in demo method
Expected CFG Output:
- Must contain nodes for function entry/exit
- Must contain edges for loop control flow
Expected CallGraph Output:
CallGraphFrom(main) → should find call to Example1() and demo()CallGraphTo(demo) → should find caller main
C++ Example
Test Code (server/ql/cpp/tools/test/PrintAST/Example1.cpp):
class someClass {
public:
void f(void);
int g(int i, int j);
};
void fun3(someClass *sc) {
int i;
sc->f();
i = sc->g(1, 2);
}
Expected AST Output:
- Must contain
Class node for someClass
- Must contain
MemberFunction nodes for f and g
- Must contain
Function node for fun3
Expected CallGraph Output:
CallGraphFrom(fun3) → should find calls to f() and g()CallGraphTo(f) → should find caller fun3
Python Example
Test Code (server/ql/python/tools/test/PrintAST/Example1.py):
class Example1:
def __init__(self, name="World"):
self.name = name
def demo(self):
for i in range(3):
print(i)
def main():
example = Example1()
example.demo()
Expected AST Output:
- Must contain
ClassDef node for Example1
- Must contain
FunctionDef nodes for __init__, demo, main
- Must contain
For node in demo
Java Example
Test Code (server/ql/java/tools/test/PrintAST/Example1.java):
public class Example1 {
private String name;
public Example1(String name) {
this.name = name;
}
public void demo() {
for (int i = 0; i < 3; i++) {
System.out.println(i);
}
}
public static void main(String[] args) {
Example1 example = new Example1("World");
example.demo();
}
}
Expected AST Output:
- Must contain
ClassOrInterface node for Example1
- Must contain
Method nodes for constructor, demo, main
- Must contain
ForStmt node in demo
Validation Failure Indicators
| Symptom | Likely Cause | Resolution |
|---|
| Empty AST output | Database not extracted properly | Re-run codeql_test_extract |
| Only headers in output | Query ran but no results matched | Check query's source file selection logic |
| Empty CFG nodes/edges | Test code has no control flow | Add functions with loops/conditionals |
| Empty CallGraph | No function calls in test code | Add function calls or verify targetFunction param |
| BQRS decode error | Incompatible CodeQL version | Update CodeQL CLI and re-run |
| Query compilation error | Missing pack dependencies | Run codeql pack install in tools pack |
| "Nothing to extract" | Missing test source files | Run workshop's initialize-qltests.sh script |
| Pack not found | Older pack version not cached | Run codeql pack install in solutions directory |
Handling External Workshops
When validating against external workshops (outside the current workspace), follow these guidelines:
Pre-Validation Setup
-
Install pack dependencies: External workshops often use older CodeQL pack versions
cd /path/to/external/workshop
codeql pack install solutions
codeql pack install solutions-tests
-
Run initialization scripts: Many workshops have scripts that copy test files from shared directories
./initialize-qltests.sh
-
Verify test files exist: Check that test source files are present before running tests
ls solutions-tests/Exercise*/test.*
File Access Patterns
When the external workshop is outside your workspace:
- Use terminal commands (
cat, ls, head) to read files
- Use MCP tools (
codeql_query_run, codeql_test_run) with absolute paths
- Pre-built databases in the workshop root can be used for tools queries
Integration with Prompts and Agents
This skill is designed to be used with:
MCP Tools Used
This skill exercises the following MCP server tools:
| Tool | Purpose |
|---|
codeql_test_extract | Create test database from test code |
codeql_query_run | Execute tools queries |
codeql_bqrs_decode | Decode raw BQRS results |
codeql_bqrs_interpret | Convert results to readable formats |
codeql_pack_install | Install pack dependencies |
codeql_pack_ls | List available packs |
Important Implementation Notes
Server Logger Output Goes to stderr
All logger.info/warn/error/debug methods write to stderr (console.error), not stdout. This is required because in stdio transport mode, stdout is reserved exclusively for the MCP JSON-RPC protocol wire format. When validating server startup logs (e.g., confirming CODEQL_PATH resolution), always check stderr.
CODEQL_PATH Environment Variable
The MCP server resolves the CodeQL CLI binary at startup via resolveCodeQLBinary() in server/src/lib/cli-executor.ts. When CODEQL_PATH is set to an absolute path pointing to a valid codeql binary, the server uses that binary for all CodeQL CLI operations instead of searching PATH. This is validated per-OS in .github/workflows/build-and-test-client.yml (codeql-path-tests job).
STDIO Transport and stdin EOF
When the STDIO transport receives an immediate EOF on stdin (e.g., via </dev/null), the server exits cleanly with code 0. To keep the server alive for testing, use a named pipe (mkfifo) with a persistent background writer (sleep 300 > fifo &).
npm Package Includes Tool Query Source Packs
The published npm package (codeql-development-mcp-server) bundles all tool query source packs under ql/*/tools/src/. These are the same .ql, .qll, .md, codeql-pack.yml, and codeql-pack.lock.yml files — but never compiled .qlx bytecode (excluded by server/.npmignore).
Success Criteria
Validation passes when ALL of the following are true:
Related Resources
