Validate Ql MCP Server Tools Queries

Validate QL MCP Server Tools Queries

This skill provides a systematic approach to validating that the CodeQL Development MCP Server's tools queries (PrintAST, PrintCFG, CallGraphFrom, CallGraphTo) return meaningful, non-empty output for test code across all supported languages.

Why This Skill Exists

⚠️ CRITICAL: The most common failure mode in MCP server tool validation is tests that "go through the motions" without verifying that query output is substantive.

A test that:

• ✅ Invokes codeql_query_run with PrintAST.ql
• ✅ Receives a successful HTTP response
• ❌ Does NOT verify the output contains actual AST nodes

...is a false positive that masks real bugs.

Supported Languages

The tools queries are available for all CodeQL-supported languages:

Tools Queries Overview

Each language pack includes four standard tools queries:

PrintAST

Purpose: Outputs a hierarchical representation of the Abstract Syntax Tree.

Expected Output: A graph with labeled nodes representing code elements (functions, classes, statements, expressions).

Validation Check: Output must contain multiple semmle.label properties with actual node names.

PrintCFG

Purpose: Outputs the Control Flow Graph showing execution paths.

Expected Output: A graph with nodes (CFG nodes) and edges (control flow transitions).

Validation Check: Output must contain both nodes and edges predicates with non-zero results.

CallGraphFrom

Purpose: Shows outbound function calls from a specified function.

Expected Output: Problem-style results showing call sites and their targets.

Validation Check: Output must contain call relationships when test code has function calls.

CallGraphTo

Purpose: Shows inbound function calls to a specified function.

Expected Output: Problem-style results showing callers of the target function.

Validation Check: Output must contain caller relationships when test code has function definitions that are called.

Validation Workflow

Step 1: Select Language and Test Code

Choose a language and ensure valid test code exists:

server/ql/<language>/tools/test/PrintAST/
├── Example1.<ext>       # Test source code
├── PrintAST.expected    # Expected results
└── PrintAST.qlref       # Reference to query

Step 2: Extract Test Database

Use codeql_test_extract to create the test database:

{
  "testPath": "server/ql/<language>/tools/test/PrintAST",
  "searchPath": ["server/ql/<language>/tools/src", "server/ql/<language>/tools/test"]
}

Step 3: Run PrintAST Query

Use codeql_query_run:

{
  "queryPath": "server/ql/<language>/tools/src/PrintAST/PrintAST.ql",
  "database": "server/ql/<language>/tools/test/PrintAST/PrintAST.testproj",
  "outputFormat": "bqrs"
}

Step 4: Interpret Results

Use codeql_bqrs_interpret with graphtext format:

{
  "file": "<path-to-results.bqrs>",
  "format": "graphtext",
  "output": "<path-to-output.txt>"
}

Step 5: Validate Non-Empty Output

CRITICAL: Parse the output and verify it contains substantive data:

• For PrintAST: Count nodes with semmle.label > 0
• For PrintCFG: Verify both nodes and edges sections exist
• For CallGraphFrom/To: Count result rows > 0 (when calls exist)

Language-Specific Examples

JavaScript Example

Test Code (server/ql/javascript/tools/test/PrintAST/Example1.js):

class Example1 {
  constructor(name = 'World') {
    this.name = name;
  }

  static main(args = []) {
    const example = new Example1();
    example.demo();
  }

  demo() {
    for (let i = 0; i < 3; i++) {
      console.log(i);
    }
  }
}

Expected AST Output (non-empty validation):

• Must contain ClassDefinition node for Example1
• Must contain MethodDefinition nodes for constructor, main, demo
• Must contain ForStatement node in demo method

Expected CFG Output:

• Must contain nodes for function entry/exit
• Must contain edges for loop control flow

Expected CallGraph Output:

• CallGraphFrom(main) → should find call to Example1() and demo()
• CallGraphTo(demo) → should find caller main

C++ Example

Test Code (server/ql/cpp/tools/test/PrintAST/Example1.cpp):

class someClass {
public:
    void f(void);
    int g(int i, int j);
};

void fun3(someClass *sc) {
    int i;
    sc->f();
    i = sc->g(1, 2);
}

Expected AST Output:

• Must contain Class node for someClass
• Must contain MemberFunction nodes for f and g
• Must contain Function node for fun3

Expected CallGraph Output:

• CallGraphFrom(fun3) → should find calls to f() and g()
• CallGraphTo(f) → should find caller fun3

Python Example

Test Code (server/ql/python/tools/test/PrintAST/Example1.py):

class Example1:
    def __init__(self, name="World"):
        self.name = name

    def demo(self):
        for i in range(3):
            print(i)

def main():
    example = Example1()
    example.demo()

Expected AST Output:

• Must contain ClassDef node for Example1
• Must contain FunctionDef nodes for __init__, demo, main
• Must contain For node in demo

Java Example

Test Code (server/ql/java/tools/test/PrintAST/Example1.java):

public class Example1 {
    private String name;

    public Example1(String name) {
        this.name = name;
    }

    public void demo() {
        for (int i = 0; i < 3; i++) {
            System.out.println(i);
        }
    }

    public static void main(String[] args) {
        Example1 example = new Example1("World");
        example.demo();
    }
}

Expected AST Output:

• Must contain ClassOrInterface node for Example1
• Must contain Method nodes for constructor, demo, main
• Must contain ForStmt node in demo

Validation Failure Indicators

Handling External Workshops

When validating against external workshops (outside the current workspace), follow these guidelines:

Pre-Validation Setup

1. Install pack dependencies: External workshops often use older CodeQL pack versions

   cd /path/to/external/workshop
   codeql pack install solutions
   codeql pack install solutions-tests
   

2. Run initialization scripts: Many workshops have scripts that copy test files from shared directories

   ./initialize-qltests.sh  # Copies from tests-common/ to test directories
   

3. Verify test files exist: Check that test source files are present before running tests

   ls solutions-tests/Exercise*/test.*
   

File Access Patterns

When the external workshop is outside your workspace:

• Use terminal commands (cat, ls, head) to read files
• Use MCP tools (codeql_query_run, codeql_test_run) with absolute paths
• Pre-built databases in the workshop root can be used for tools queries

Integration with Prompts and Agents

This skill is designed to be used with:

• Prompt: validate-ql-mcp-server-tools-via-workshop — uses this skill as part of workshop creation validation.
• Agent: mcp-enabled-ql-workshop-developer — orchestrates validation during workshop creation.
• Agent: ql-mcp-tool-tester — tests MCP tools including tools query execution.

MCP Tools Used

This skill exercises the following MCP server tools:

Important Implementation Notes

Server Logger Output Goes to stderr

All logger.info/warn/error/debug methods write to stderr (console.error), not stdout. This is required because in stdio transport mode, stdout is reserved exclusively for the MCP JSON-RPC protocol wire format. When validating server startup logs (e.g., confirming CODEQL_PATH resolution), always check stderr.

CODEQL_PATH Environment Variable

The MCP server resolves the CodeQL CLI binary at startup via resolveCodeQLBinary() in server/src/lib/cli-executor.ts. When CODEQL_PATH is set to an absolute path pointing to a valid codeql binary, the server uses that binary for all CodeQL CLI operations instead of searching PATH. This is validated per-OS in .github/workflows/build-and-test-client.yml (codeql-path-tests job).

STDIO Transport and stdin EOF

When the STDIO transport receives an immediate EOF on stdin (e.g., via </dev/null), the server exits cleanly with code 0. To keep the server alive for testing, use a named pipe (mkfifo) with a persistent background writer (sleep 300 > fifo &).

npm Package Includes Tool Query Source Packs

The published npm package (codeql-development-mcp-server) bundles all tool query source packs under ql/*/tools/src/. These are the same .ql, .qll, .md, codeql-pack.yml, and codeql-pack.lock.yml files — but never compiled .qlx bytecode (excluded by server/.npmignore).

Success Criteria

Validation passes when ALL of the following are true:

• PrintAST returns ≥10 labeled AST nodes for test code
• PrintCFG returns both nodes and edges with ≥5 entries each
• CallGraphFrom returns ≥1 result when test code contains function calls
• CallGraphTo returns ≥1 result when test code contains called functions
• No MCP tool errors or timeouts occurred
• Results are consistent across multiple runs

Related Resources

• create-codeql-query-development-workshop — workshop creation that depends on tools queries.