Create Data Extensions Workflow

Create Data Extensions Workflow

Generate data extension YAML files to improve CodeQL's data flow coverage for project-specific APIs. Runs after database build and before analysis.

Task System

Create these tasks on workflow start:

TaskCreate: "Check for existing data extensions" (Step 1)
TaskCreate: "Query known sources and sinks" (Step 2) - blockedBy: Step 1
TaskCreate: "Identify missing sources and sinks" (Step 3) - blockedBy: Step 2
TaskCreate: "Create data extension files" (Step 4) - blockedBy: Step 3
TaskCreate: "Validate with re-analysis" (Step 5) - blockedBy: Step 4

Early Exit Points

Steps

Step 1: Check for Existing Data Extensions

Search the project for existing data extensions and model packs.

1. In-repo model packs — qlpack.yml or codeql-pack.yml with dataExtensions:

fd '(qlpack|codeql-pack)\.yml$' . --exclude codeql_*.db | while read -r f; do
  if grep -q 'dataExtensions' "$f"; then
    echo "MODEL PACK: $(dirname "$f") - $(grep '^name:' "$f")"
  fi
done

2. Standalone data extension files — .yml files with extensions: key:

rg -l '^extensions:' --glob '*.yml' --glob '!codeql_*.db/**' | head -20

3. Installed model packs:

codeql resolve qlpacks 2>/dev/null | grep -iE 'model|extension'

If any found: Report to user what was found and finish. These will be picked up by the run-analysis workflow's model pack detection (Step 2b).

If none found: Proceed to Step 2.

Step 2: Query Known Sources and Sinks

Run custom QL queries against the database to enumerate all sources and sinks CodeQL currently recognizes. This gives a direct inventory of what is modeled vs. what is not.

2a: Select Database and Language

DB_NAME=$(ls -dt codeql_*.db 2>/dev/null | head -1)
LANG=$(codeql resolve database --format=json -- "$DB_NAME" | jq -r '.languages[0]')
echo "Database: $DB_NAME, Language: $LANG"

DIAG_DIR="${DB_NAME%.db}-diagnostics"
mkdir -p "$DIAG_DIR"

2b: Write Source Enumeration Query

Use the Write tool to create $DIAG_DIR/list-sources.ql using the source template from diagnostic-query-templates.md. Pick the correct import block for $LANG.

2c: Write Sink Enumeration Query

Use the Write tool to create $DIAG_DIR/list-sinks.ql using the language-specific sink template from diagnostic-query-templates.md. The Concepts API differs significantly across languages — use the exact template for the detected language.

For Java: Also create $DIAG_DIR/qlpack.yml with a codeql/java-all dependency and run codeql pack install before executing queries. See the Java section in the templates reference.

2d: Run Queries

# Run sources query
codeql query run \
  --database="$DB_NAME" \
  --output="$DIAG_DIR/sources.bqrs" \
  -- "$DIAG_DIR/list-sources.ql"

codeql bqrs decode \
  --format=csv \
  --output="$DIAG_DIR/sources.csv" \
  -- "$DIAG_DIR/sources.bqrs"

# Run sinks query
codeql query run \
  --database="$DB_NAME" \
  --output="$DIAG_DIR/sinks.bqrs" \
  -- "$DIAG_DIR/list-sinks.ql"

codeql bqrs decode \
  --format=csv \
  --output="$DIAG_DIR/sinks.csv" \
  -- "$DIAG_DIR/sinks.bqrs"

2e: Summarize Results

echo "=== Known Sources ==="
wc -l < "$DIAG_DIR/sources.csv"
# Show unique source types
cut -d',' -f2 "$DIAG_DIR/sources.csv" | sort -u

echo "=== Known Sinks ==="
wc -l < "$DIAG_DIR/sinks.csv"
# Show unique sink kinds
cut -d',' -f2 "$DIAG_DIR/sinks.csv" | sort -u

Read both CSV files and present a summary to the user:

## CodeQL Known Models

### Sources (<N> total):
- remote: <count> (HTTP handlers, request parsing)
- local: <count> (CLI args, file reads)
- ...

### Sinks (<N> total):
- sql-execution: <count>
- command-execution: <count>
- file-access: <count>
- ...

Step 3: Identify Missing Sources and Sinks

This is the core analysis step. Cross-reference the project's API surface against CodeQL's known models.

3a: Map the Project's API Surface

Read source code to identify security-relevant patterns. Look for:

Use Grep to search for these patterns in source code:

# Examples for Python - adapt patterns per language
rg -n 'def (get_param|get_header|get_body|parse_request)' --type py
rg -n '(execute|query|raw_sql|cursor\.)' --type py
rg -n '(subprocess|os\.system|popen|exec)' --type py
rg -n '(open|read_file|write_file|path\.join)' --type py
rg -n '(render|template|html)' --type py
rg -n '(requests\.|urlopen|fetch|http_client)' --type py

3b: Cross-Reference Against Known Sources and Sinks

For each API pattern found in 3a, check if it appears in the source/sink CSVs from Step 2:

# Check if a specific file/function appears in known sources
grep -i "<function_or_file>" "$DIAG_DIR/sources.csv"

# Check if a specific file/function appears in known sinks
grep -i "<function_or_file>" "$DIAG_DIR/sinks.csv"

An API is "missing" if:

• It handles user input but does not appear in sources.csv
• It performs a dangerous operation but does not appear in sinks.csv
• It wraps/transforms tainted data but CodeQL has no summary model for it (these won't appear in either CSV — identify by reading the code for wrapper patterns around known sources/sinks)

3c: Report Gaps

Present findings to user:

## Data Flow Coverage Gaps

### Missing Sources (user input not tracked):
- `myapp.http.Request.get_param()` — custom parameter access
- `myapp.auth.Token.decode()` — untrusted token data

### Missing Sinks (dangerous operations not checked):
- `myapp.db.Connection.raw_query()` — SQL execution wrapper
- `myapp.shell.Runner.execute()` — command execution

### Missing Summaries (taint lost through wrappers):
- `myapp.cache.Cache.get()` — taint not propagated through cache
- `myapp.utils.encode_json()` — taint lost in serialization

Proceed to create data extension files?

Use AskUserQuestion:

header: "Extensions"
question: "Create data extension files for the identified gaps?"
options:
  - label: "Create all (Recommended)"
    description: "Generate extensions for all identified gaps"
  - label: "Select individually"
    description: "Choose which gaps to model"
  - label: "Skip"
    description: "No extensions needed, proceed to analysis"

If "Skip": Finish workflow.

If "Select individually": Use AskUserQuestion with multiSelect: true listing each gap.

Step 4: Create Data Extension Files

Generate YAML data extension files for the gaps confirmed by the user.

File Structure

Create files in a codeql-extensions/ directory at project root:

codeql-extensions/
  sources.yml       # sourceModel entries
  sinks.yml         # sinkModel entries
  summaries.yml     # summaryModel and neutralModel entries

YAML Format

All extension files follow this structure:

extensions:
  - addsTo:
      pack: codeql/<language>-all  # Target library pack
      extensible: <model-type>      # sourceModel, sinkModel, summaryModel, neutralModel
    data:
      - [<columns>]

Source Models

Columns: [package, type, subtypes, name, signature, ext, output, kind, provenance]

Java-specific format differences:

• subtypes: Use True / False (capitalized, Python-style), not true / false
• output for parameters: Use Parameter[N] (not Argument[N]) to mark method parameters as sources
• signature: Required for disambiguation — use Java type syntax: "(String)", "(String,int)"
• Parameter ranges: Use Parameter[0..2] to mark multiple consecutive parameters

Example (Python):

# codeql-extensions/sources.yml
extensions:
  - addsTo:
      pack: codeql/python-all
      extensible: sourceModel
    data:
      - ["myapp.http", "Request", true, "get_param", "", "", "ReturnValue", "remote", "manual"]
      - ["myapp.http", "Request", true, "get_header", "", "", "ReturnValue", "remote", "manual"]

Example (Java — note True, Parameter[N], and signature):

# codeql-extensions/sources.yml
extensions:
  - addsTo:
      pack: codeql/java-all
      extensible: sourceModel
    data:
      - ["com.myapp.controller", "ApiController", True, "search", "(String)", "", "Parameter[0]", "remote", "manual"]
      - ["com.myapp.service", "FileService", True, "upload", "(String,String)", "", "Parameter[0..1]", "remote", "manual"]

Sink Models

Columns: [package, type, subtypes, name, signature, ext, input, kind, provenance]

Note: column 7 is input (which argument receives tainted data), not output.

Example (Python):

# codeql-extensions/sinks.yml
extensions:
  - addsTo:
      pack: codeql/python-all
      extensible: sinkModel
    data:
      - ["myapp.db", "Connection", true, "raw_query", "", "", "Argument[0]", "sql-injection", "manual"]
      - ["myapp.shell", "Runner", false, "execute", "", "", "Argument[0]", "command-injection", "manual"]

Example (Java — note True and Argument[N] for sink input):

extensions:
  - addsTo:
      pack: codeql/java-all
      extensible: sinkModel
    data:
      - ["com.myapp.db", "QueryRunner", True, "execute", "(String)", "", "Argument[0]", "sql-injection", "manual"]

Summary Models

Columns: [package, type, subtypes, name, signature, ext, input, output, kind, provenance]

Example:

# codeql-extensions/summaries.yml
extensions:
  # Pass-through: taint propagates
  - addsTo:
      pack: codeql/python-all
      extensible: summaryModel
    data:
      - ["myapp.cache", "Cache", true, "get", "", "", "Argument[0]", "ReturnValue", "taint", "manual"]
      - ["myapp.utils", "JSON", false, "parse", "", "", "Argument[0]", "ReturnValue", "taint", "manual"]

  # Sanitizer: taint blocked
  - addsTo:
      pack: codeql/python-all
      extensible: neutralModel
    data:
      - ["myapp.security", "Sanitizer", "escape_html", "", "summary", "manual"]

neutralModel vs no model: If a function has no model at all, CodeQL may still infer flow through it. Use neutralModel to explicitly block taint propagation through known-safe functions.

Language-Specific Notes

Python: Use dotted module paths for package (e.g., myapp.db).

JavaScript: package is often "" for project-local code. Use the import path for npm packages.

Go: Use full import paths (e.g., myapp/internal/db). type is often "" for package-level functions.

Java: Use fully qualified package names (e.g., com.myapp.db).

C/C++: Use "" for package, put the namespace in type.

Write the Files

Use the Write tool to create each file. Only create files that have entries — skip empty categories.

Deploy the Extensions

Known limitation: --additional-packs and --model-packs flags do not work with pre-compiled query packs (bundled CodeQL distributions that cache java-all inside .codeql/libraries/). Extensions placed in a standalone model pack directory will be resolved by codeql resolve qlpacks but silently ignored during codeql database analyze.

Workaround — copy extensions into the library pack's ext/ directory:

Warning: Files copied into the ext/ directory live inside CodeQL's managed pack cache. They will be lost when packs are updated via codeql pack download or version upgrades. After any pack update, re-run this deployment step to restore the extensions.

# Find the java-all ext directory used by the query pack
JAVA_ALL_EXT=$(find "$(codeql resolve qlpacks 2>/dev/null | grep 'java-queries' | awk '{print $NF}' | tr -d '()')" \
  -path '*/.codeql/libraries/codeql/java-all/*/ext' -type d 2>/dev/null | head -1)

if [ -n "$JAVA_ALL_EXT" ]; then
  PROJECT_NAME=$(basename "$(pwd)")
  cp codeql-extensions/sources.yml "$JAVA_ALL_EXT/${PROJECT_NAME}.sources.model.yml"
  [ -f codeql-extensions/sinks.yml ] && cp codeql-extensions/sinks.yml "$JAVA_ALL_EXT/${PROJECT_NAME}.sinks.model.yml"
  [ -f codeql-extensions/summaries.yml ] && cp codeql-extensions/summaries.yml "$JAVA_ALL_EXT/${PROJECT_NAME}.summaries.model.yml"

  # Verify deployment — confirm files landed correctly
  DEPLOYED=$(ls "$JAVA_ALL_EXT/${PROJECT_NAME}".*.model.yml 2>/dev/null | wc -l)
  if [ "$DEPLOYED" -gt 0 ]; then
    echo "Extensions deployed to $JAVA_ALL_EXT ($DEPLOYED files):"
    ls -la "$JAVA_ALL_EXT/${PROJECT_NAME}".*.model.yml
  else
    echo "ERROR: Files were copied but verification failed. Check path: $JAVA_ALL_EXT"
  fi
else
  echo "WARNING: Could not find java-all ext directory. Extensions may not load."
  echo "Attempted path lookup from: codeql resolve qlpacks | grep java-queries"
  echo "Run 'codeql resolve qlpacks' manually to debug."
fi

For Python/JS/Go: The same limitation may apply. Locate the <lang>-all pack's ext/ directory and copy extensions there.

Alternative (if query packs are NOT pre-compiled): Use --additional-packs=./codeql-extensions with a proper model pack qlpack.yml:

# codeql-extensions/qlpack.yml
name: custom/<project>-extensions
version: 0.0.1
library: true
extensionTargets:
  codeql/<lang>-all: "*"
dataExtensions:
  - sources.yml
  - sinks.yml
  - summaries.yml

Step 5: Validate with Re-Analysis

Run a full security analysis with and without extensions to measure the finding delta. This is more reliable than re-running source/sink enumeration queries, which may not reflect the sourceModel extensible being used by taint-tracking queries.

5a: Run Baseline Analysis (without extensions)

RESULTS_DIR="${DB_NAME%.db}-results"
mkdir -p "$RESULTS_DIR"

# Baseline run (or skip if already run in a previous step)
codeql database analyze "$DB_NAME" \
  --format=sarif-latest \
  --output="$RESULTS_DIR/baseline.sarif" \
  --threads=0 \
  -- codeql/<lang>-queries:codeql-suites/<lang>-security-extended.qls

5b: Run Analysis with Extensions

# Clean cache to force re-evaluation
codeql database cleanup "$DB_NAME"

codeql database analyze "$DB_NAME" \
  --format=sarif-latest \
  --output="$RESULTS_DIR/with-extensions.sarif" \
  --threads=0 \
  --rerun \
  -- codeql/<lang>-queries:codeql-suites/<lang>-security-extended.qls

Use -vvv flag to verify extensions are being loaded — look for Loading data extensions in ... <your-extension-file>.yml in stderr.

5c: Compare Findings

BASELINE=$(python3 -c "import json; print(sum(len(r.get('results',[])) for r in json.load(open('$RESULTS_DIR/baseline.sarif')).get('runs',[])))")
WITH_EXT=$(python3 -c "import json; print(sum(len(r.get('results',[])) for r in json.load(open('$RESULTS_DIR/with-extensions.sarif')).get('runs',[])))")
echo "Findings: $BASELINE → $WITH_EXT (+$((WITH_EXT - BASELINE)))"

If counts did not increase: The extension YAML may have syntax errors or column values that don't match the code. Check:

Fix the extension files, re-deploy to ext/, and re-run 5b until counts increase.

Final Output

## Data Extensions Created

**Database:** $DB_NAME
**Language:** <LANG>

### Files Created:
- codeql-extensions/sources.yml — <N> source models
- codeql-extensions/sinks.yml — <N> sink models
- codeql-extensions/summaries.yml — <N> summary/neutral models

### Model Coverage:
- Sources: <BEFORE> → <AFTER> (+<DELTA>)
- Sinks: <BEFORE> → <AFTER> (+<DELTA>)

### Usage:
Extensions deployed to `<lang>-all` ext/ directory (auto-loaded).
Source files in `codeql-extensions/` for version control.
Run the run-analysis workflow to use them.

References

• Threat models reference — control which source categories are active during analysis
• CodeQL data extensions
• Customizing library models