validate-ql-mcp-server-tools-queries

name	validate-ql-mcp-server-tools-queries
description	Validate that the CodeQL Development MCP Server reliably returns non-empty AST, CFG, and CallGraph data from the bundled tools queries. Use this skill to ensure the `codeql_query_run` tool properly executes PrintAST, PrintCFG, CallGraphFrom, and CallGraphTo queries for any supported language.

Validate QL MCP Server Tools Queries

This skill provides a systematic approach to validating that the CodeQL Development MCP Server's tools queries (PrintAST, PrintCFG, CallGraphFrom, CallGraphTo) return meaningful, non-empty output for test code across all supported languages.

Why This Skill Exists

⚠️ CRITICAL: The most common failure mode in MCP server tool validation is tests that "go through the motions" without verifying that query output is substantive.

A test that:

• ✅ Invokes codeql_query_run with PrintAST.ql
• ✅ Receives a successful HTTP response
• ❌ Does NOT verify the output contains actual AST nodes

...is a false positive that masks real bugs.

Supported Languages

The tools queries are available for all CodeQL-supported languages:

Language	Tools Pack Location	Test Code Extension
actions	server/ql/actions/tools/	.yml
cpp	server/ql/cpp/tools/	.cpp
csharp	server/ql/csharp/tools/	.cs
go	server/ql/go/tools/	.go
java	server/ql/java/tools/	.java
javascript	server/ql/javascript/tools/	.js
python	server/ql/python/tools/	.py
ruby	server/ql/ruby/tools/	.rb
rust	server/ql/rust/tools/	.rs
swift	server/ql/swift/tools/	.swift

Tools Queries Overview

Each language pack includes four standard tools queries:

PrintAST

Purpose: Outputs a hierarchical representation of the Abstract Syntax Tree.

Expected Output: A graph with labeled nodes representing code elements (functions, classes, statements, expressions).

Validation Check: Output must contain multiple semmle.label properties with actual node names.

PrintCFG

Purpose: Outputs the Control Flow Graph showing execution paths.

Expected Output: A graph with nodes (CFG nodes) and edges (control flow transitions).

Validation Check: Output must contain both nodes and edges predicates with non-zero results.

CallGraphFrom

Purpose: Shows outbound function calls from a specified function.

Expected Output: Problem-style results showing call sites and their targets.

Validation Check: Output must contain call relationships when test code has function calls.

CallGraphTo

Purpose: Shows inbound function calls to a specified function.

Expected Output: Problem-style results showing callers of the target function.

Validation Check: Output must contain caller relationships when test code has function definitions that are called.

Validation Workflow

Step 1: Select Language and Test Code

Choose a language and ensure valid test code exists:

server/ql/<language>/tools/test/PrintAST/
├── Example1.<ext>       # Test source code
├── PrintAST.expected    # Expected results
└── PrintAST.qlref       # Reference to query

Step 2: Extract Test Database

Use codeql_test_extract to create the test database:

{
  "testPath": "server/ql/<language>/tools/test/PrintAST",
  "searchPath": ["server/ql/<language>/tools/src", "server/ql/<language>/tools/test"]
}

Step 3: Run PrintAST Query

Use codeql_query_run:

{
  "queryPath": "server/ql/<language>/tools/src/PrintAST/PrintAST.ql",
  "database": "server/ql/<language>/tools/test/PrintAST/PrintAST.testproj",
  "outputFormat": "bqrs"
}

Step 4: Interpret Results

Use codeql_bqrs_interpret with graphtext format:

{
  "file": "<path-to-results.bqrs>",
  "format": "graphtext",
  "output": "<path-to-output.txt>"
}

Step 5: Validate Non-Empty Output

CRITICAL: Parse the output and verify it contains substantive data:

• For PrintAST: Count nodes with semmle.label > 0
• For PrintCFG: Verify both nodes and edges sections exist
• For CallGraphFrom/To: Count result rows > 0 (when calls exist)

Language-Specific Examples

JavaScript Example

Test Code (server/ql/javascript/tools/test/PrintAST/Example1.js):

class Example1 {
  constructor(name = 'World') {
    this.name = name;
  }

  static main(args = []) {
    const example = new Example1();
    example.demo();
  }

  demo() {
    for (let i = 0; i < 3; i++) {
      console.log(i);
    }
  }
}

Expected AST Output (non-empty validation):

• Must contain ClassDefinition node for Example1
• Must contain MethodDefinition nodes for constructor, main, demo
• Must contain ForStatement node in demo method

Expected CFG Output:

• Must contain nodes for function entry/exit
• Must contain edges for loop control flow

Expected CallGraph Output:

• CallGraphFrom(main) → should find call to Example1() and demo()
• CallGraphTo(demo) → should find caller main

C++ Example

Test Code (server/ql/cpp/tools/test/PrintAST/Example1.cpp):

class someClass {
public:
    void f(void);
    int g(int i, int j);
};

void fun3(someClass *sc) {
    int i;
    sc->f();
    i = sc->g(1, 2);
}

Expected AST Output:

• Must contain Class node for someClass
• Must contain MemberFunction nodes for f and g
• Must contain Function node for fun3

Expected CallGraph Output:

• CallGraphFrom(fun3) → should find calls to f() and g()
• CallGraphTo(f) → should find caller fun3

Python Example

Test Code (server/ql/python/tools/test/PrintAST/Example1.py):

class Example1:
    def __init__(self, name="World"):
        self.name = name

    def demo(self):
        for i in range(3):
            print(i)

def main():
    example = Example1()
    example.demo()

Expected AST Output:

• Must contain ClassDef node for Example1
• Must contain FunctionDef nodes for __init__, demo, main
• Must contain For node in demo

Java Example

Test Code (server/ql/java/tools/test/PrintAST/Example1.java):

public class Example1 {
    private String name;

    public Example1(String name) {
        this.name = name;
    }

    public void demo() {
        for (int i = 0; i < 3; i++) {
            System.out.println(i);
        }
    }

    public static void main(String[] args) {
        Example1 example = new Example1("World");
        example.demo();
    }
}

Expected AST Output:

• Must contain ClassOrInterface node for Example1
• Must contain Method nodes for constructor, demo, main
• Must contain ForStmt node in demo

Validation Failure Indicators

Symptom	Likely Cause	Resolution
Empty AST output	Database not extracted properly	Re-run codeql_test_extract
Only headers in output	Query ran but no results matched	Check query's source file selection logic
Empty CFG nodes/edges	Test code has no control flow	Add functions with loops/conditionals
Empty CallGraph	No function calls in test code	Add function calls or verify targetFunction param
BQRS decode error	Incompatible CodeQL version	Update CodeQL CLI and re-run
Query compilation error	Missing pack dependencies	Run codeql pack install in tools pack
"Nothing to extract"	Missing test source files	Run workshop's initialize-qltests.sh script
Pack not found	Older pack version not cached	Run codeql pack install in solutions directory

Handling External Workshops

When validating against external workshops (outside the current workspace), follow these guidelines:

Pre-Validation Setup

1. Install pack dependencies: External workshops often use older CodeQL pack versions

   cd /path/to/external/workshop
   codeql pack install solutions
   codeql pack install solutions-tests
   

2. Run initialization scripts: Many workshops have scripts that copy test files from shared directories

   ./initialize-qltests.sh  # Copies from tests-common/ to test directories
   

3. Verify test files exist: Check that test source files are present before running tests

   ls solutions-tests/Exercise*/test.*
   

File Access Patterns

When the external workshop is outside your workspace:

• Use terminal commands (cat, ls, head) to read files
• Use MCP tools (codeql_query_run, codeql_test_run) with absolute paths
• Pre-built databases in the workshop root can be used for tools queries

Integration with Prompts and Agents

This skill is designed to be used with:

• Prompt: validate-ql-mcp-server-tools-via-workshop — uses this skill as part of workshop creation validation.
• Agent: mcp-enabled-ql-workshop-developer — orchestrates validation during workshop creation.
• Agent: ql-mcp-tool-tester — tests MCP tools including tools query execution.

MCP Tools Used

This skill exercises the following MCP server tools:

Tool	Purpose
codeql_test_extract	Create test database from test code
codeql_query_run	Execute tools queries
codeql_bqrs_decode	Decode raw BQRS results
codeql_bqrs_interpret	Convert results to readable formats
codeql_pack_install	Install pack dependencies
codeql_pack_ls	List available packs

Important Implementation Notes

Server Logger Output Goes to stderr

All logger.info/warn/error/debug methods write to stderr (console.error), not stdout. This is required because in stdio transport mode, stdout is reserved exclusively for the MCP JSON-RPC protocol wire format. When validating server startup logs (e.g., confirming CODEQL_PATH resolution), always check stderr.

CODEQL_PATH Environment Variable

The MCP server resolves the CodeQL CLI binary at startup via resolveCodeQLBinary() in server/src/lib/cli-executor.ts. When CODEQL_PATH is set to an absolute path pointing to a valid codeql binary, the server uses that binary for all CodeQL CLI operations instead of searching PATH. This is validated per-OS in .github/workflows/build-and-test-client.yml (codeql-path-tests job).

STDIO Transport and stdin EOF

When the STDIO transport receives an immediate EOF on stdin (e.g., via </dev/null), the server exits cleanly with code 0. To keep the server alive for testing, use a named pipe (mkfifo) with a persistent background writer (sleep 300 > fifo &).

npm Package Includes Tool Query Source Packs

The published npm package (codeql-development-mcp-server) bundles all tool query source packs under ql/*/tools/src/. These are the same .ql, .qll, .md, codeql-pack.yml, and codeql-pack.lock.yml files — but never compiled .qlx bytecode (excluded by server/.npmignore).

Success Criteria

Validation passes when ALL of the following are true:

• PrintAST returns ≥10 labeled AST nodes for test code
• PrintCFG returns both nodes and edges with ≥5 entries each
• CallGraphFrom returns ≥1 result when test code contains function calls
• CallGraphTo returns ≥1 result when test code contains called functions
• No MCP tool errors or timeouts occurred
• Results are consistent across multiple runs

Related Resources

• create-codeql-query-development-workshop — workshop creation that depends on tools queries.