Parser And Nodes

Parser & Node-types — operational

Companion docs: knowledge/idioms/parser-pipeline.md, knowledge/idioms/node-types-and-lists.md.

0. The three-tree pipeline — AST → Query → Plan

Every SQL statement flows through three distinct tree shapes; nodes in each header live or die at a specific stage. Confuse the stages and your patch will silently fail in a downstream walker.

SQL text
  │
  │  gram.y / scan.l / kwlist.h
  ▼
1. Raw parsetree (AST)         — parsenodes.h utility-stmt nodes
   Examples: SelectStmt,         + A_Expr / A_Const / ColumnRef /
   InsertStmt, A_Expr,             FuncCall (raw expression shapes)
   ColumnRef, FuncCall          → walker: raw_expression_tree_walker
                                  No types resolved, no relations
                                  looked up, no operators resolved.
  │
  │  parser/analyze.c           → transformStmt → transform<Foo>Stmt
  │  parser/parse_*.c           → adds rangetable, resolves names,
  │                               picks operator + cast OIDs,
  │                               applies coercions
  ▼
2. Query tree                  — parsenodes.h Query + analyzed
   Examples: Query, RangeTblEntry, expression nodes (primnodes.h)
   Var, OpExpr, Const, FuncExpr,  Examples: Var, OpExpr, Const,
   TargetEntry, FromExpr, JoinExpr  FuncExpr, Aggref, WindowFunc
                                → walker: expression_tree_walker
                                  + query_tree_walker
                                  + QTW_* flags for rtable/CTE descent
  │
  │  rewrite/                  → rules, views (Query → Query)
  │  optimizer/planner.c        → planner: Query → Plan
  ▼
3. Plan tree                   — plannodes.h
   Examples: SeqScan,            Examples: SeqScan, IndexScan, HashJoin,
   IndexScan, HashJoin,            Sort, Agg, Limit, Gather, GatherMerge
   Sort, Agg, Limit             → walker: plan_tree_walker
                                + planstate_tree_walker (executor-time)
                                  Shipped to parallel workers via
                                  nodeToString / stringToNode.

Operational rules per shape:

The executor-and-planner skill covers the Plan-tree side (createplan, add_path costing, ExecInit/ProcNode dispatch). This skill covers raw parsetree and Query tree work, plus the node-machinery infrastructure (gen_node_support.pl, copy/equal/out/read funcs).

Tools for working with an existing tree

• copyObject(p) — deep copy (preserves argument type via typeof_unqual macro, nodes.h:228-233). Dispatcher copyObjectImpl in copyfuncs.c:177-212, with check_stack_depth() guard (:185). By-ref Datums go through _copyConst → datumCopy; T_List is deep-copied via list_copy_deep, T_IntList/T_OidList/T_XidList shallow via list_copy. Extensible nodes dispatch to a registered nodeCopy callback (_copyExtensibleNode). Allocated in CurrentMemoryContext — caller controls the lifetime.
• equal(a, b) — deep structural compare (equalfuncs.c).
• nodeToString(p) / stringToNode(s) — Lisp-ish text serialization; load-bearing for plan cache, rule storage (pg_rewrite.ev_action), parallel-worker plan shipping. The round-trip copyObject(stringToNode(nodeToString(p))) underpins the debug_write_read_parse_plan_trees GUC.
• expression_tree_walker / _mutator — polymorphic recursion over expression trees (post-analysis: Var, OpExpr, …). raw_expression_tree_walker for the pre-analysis shape (A_Expr, ColumnRef). query_tree_walker / _mutator wrap for Query, with QTW_* flags (nodeFuncs.h:22-34) to control rtable/CTE descent. planstate_tree_walker is the executor-time analogue. Mutator contract: return NULL to delete a subtree; walker contract: callback returns true to short-circuit, false to keep descending. The macro wrappers (nodeFuncs.h:155-183) let callbacks be typed without -Wincompatible-pointer-types warnings.

When you're adding a new SQL statement

Touch four layers, in order. Build between each so you fail fast.

1. Grammar — src/backend/parser/gram.y

   • Add the keyword(s) to unreserved_keyword / col_name_keyword / etc. AND to src/include/parser/kwlist.h (alphabetical, with category).
   • Add a top-level rule producing your new *Stmt node, wire it into stmt: (search for stmt: in gram.y). Use makeNode(FooStmt) in the action, set the location with @1, fill fields from $N.
   • Match an existing simple statement as template (e.g. DropStmt, VariableSetStmt). Do NOT invent fresh helpers in gram.y if parser/parse_utilcmd.c already has one.

2. Lexer — src/backend/parser/scan.l (only if you introduced a new token shape; new keywords alone do not need scan.l changes — they go through kwlist.h).

3. Parse node — src/include/nodes/parsenodes.h (utility stmts) or primnodes.h (expressions) or plannodes.h (plan nodes)

   • First field NodeTag type; (or another node struct, for "inheritance").
   • Last field for top-level statements is typically ParseLoc location;.
   • Annotate fields if needed: pg_node_attr(...) on the struct, per-field attrs like query_jumble_ignore, equal_ignore. See nodes.h:43-125 for the full list.

4. Analyze / execute

   • Optimizable statement → add a transformFooStmt(ParseState *, FooStmt *) in parser/analyze.c and wire it into the switch (nodeTag(parseTree)) inside transformStmt (analyze.c:334-451, switch at :368). Caution (analyze.c:363-367): any change to that switch must also be reflected in stmt_requires_parse_analysis() (:469-505) and analyze_requires_snapshot() (:513-529) — three sites, one logical change.
   • Utility statement → no transform needed; just dump into a Query with CMD_UTILITY (the default path). Execution lives in src/backend/commands/ via ProcessUtility / standard_ProcessUtility (src/backend/tcop/utility.c).

5. Run gen_node_support.pl (done automatically by the build) and verify the generated copyfuncs.funcs.c, equalfuncs.funcs.c, outfuncs.funcs.c, readfuncs.funcs.c chunks for your new struct look sane.

6. Add a regression test under src/test/regress/sql/ (and a matching expected file). See .claude/skills/testing/SKILL.md.

When you're adding a new Node type (no SQL change)

From src/backend/nodes/README "Steps to Add a Node":

1. Put the struct in the right header. The file must be in gen_node_support.pl's @all_input_files list (parsenodes.h, primnodes.h, plannodes.h, pathnodes.h, execnodes.h, value.h, etc. — see gen_node_support.pl:53-77). The T_Foo tag is generated automatically into nodes/nodetags.h (do NOT hand-edit it).
2. First field is NodeTag type; — unless you "inherit" by embedding another node struct as the first field (e.g. Plan plan; in a new plan node).
3. Build. Inspect the generated copyfuncs.funcs.c / equalfuncs.funcs.c / outfuncs.funcs.c / readfuncs.funcs.c / queryjumblefuncs.funcs.c entries for your node. Add pg_node_attr(...) and per-field attrs to fix anything wrong.
4. If a field needs special treatment, common annotations:
   • pg_node_attr(custom_copy_equal) — write your own bodies in copyfuncs.c / equalfuncs.c and the generator will skip yours.
   • pg_node_attr(no_copy_equal, no_read, no_query_jumble) — opt out entirely (executor state nodes typically use nodetag_only).
   • Per-field: array_size(otherfield), copy_as(VALUE), equal_ignore, read_write_ignore, query_jumble_ignore, query_jumble_location.
5. If your node is an expression (lives in primnodes.h or has child Expr fields), add cases to every giant switch in nodeFuncs.c:
   • exprType (:42+), exprTypmod (:304+), exprCollation / exprSetCollation / exprInputCollation (:826+, :1140+, :1092+), exprLocation (:1403+) — type/typmod/collation/location introspection.
   • expression_tree_walker_impl (:2111+), expression_tree_mutator_impl (:3018+) — recursion into children.
   • raw_expression_tree_walker_impl (:4115+) — only if the node can appear in raw parsetrees.
   • set_opfuncid family (:1890+) — only if your node holds an operator OID needing resolution. Grep T_<SiblingNode> (e.g. T_OpExpr) to find every site; five-to-eight maintenance points per node, miss one and walkers silently misbehave.
6. Adding a node type renumbers existing tags. Recompile the whole tree (--enable-depend helps). No initdb needed — node numbers never go to disk. BUT if the node can appear in a stored parse tree (rule actions, view definitions), bump CATALOG_VERSION_NO in src/include/catalog/catversion.h.
7. Test with debug_copy_parse_plan_trees=on, debug_write_read_parse_plan_trees=on, debug_raw_expression_coverage_test=on (set via PG_TEST_INITDB_EXTRA_OPTS).

When you're modifying an existing Node type

• Adding a field to a node that appears in stored catalog parse trees (e.g. Query, anything in views / rules) → bump catversion. Add a pg_node_attr if the new field needs special read/write handling (e.g. read_as(...) for backward-compat when reading old serialized trees from extensions — usually not needed because catversion changes invalidate stored trees).
• Removing or reordering fields → also catversion bump if stored.
• Changing only Plan/Path internal fields (never serialized to catalogs) → no catversion bump needed. But Plan nodes ARE serialized to parallel workers, so you still need correct out/read funcs.

Pre-submit smoke

cd dev/build-debug && ninja && \
  meson test --suite regress -q

Plus the debug-tree GUCs above on a separate initdb if you touched anything tree-shape related.

Node-type reference table

The major node families and where each lives. When in doubt about which header your new field belongs in, check this table.

nodes/nodetags.h is auto-generated by gen_node_support.pl; never hand-edit it. A new struct in any of the headers above (with the file listed in gen_node_support.pl @all_input_files) gets its T_Foo tag automatically.

Walker / mutator quick reference

Conventions to remember:

• Walker callback returns bool. Return true to short-circuit the whole walk; return false to continue descending.
• Mutator callback returns Node *. Return NULL to delete a subtree; return the input pointer to keep it unchanged; return a new node to replace.
• QTW_* flags (nodeFuncs.h:22-34) control rtable / CTE descent. The default skips RTE subqueries — opt in with QTW_EXAMINE_RTES_BEFORE / _AFTER when you actually want them.
• The typed macro wrappers (nodeFuncs.h:155-183) let your callback be typed (bool foo_walker(Node *n, MyCtx *ctx)) without -Wincompatible-pointer-types warnings.

Files-examined rows

Cross-references

• .claude/skills/executor-and-planner/SKILL.md — Plan-tree side of the three-tree pipeline; createplan.c, ExecInitNode/ExecProcNode for new plan nodes.
• .claude/skills/catalog-conventions/SKILL.md — CATALOG_VERSION_NO bump rules when serialized parse-tree fields change.
• .claude/skills/testing/SKILL.md — debug_copy_parse_plan_trees, debug_write_read_parse_plan_trees, debug_raw_expression_coverage_test GUCs for tree-shape regressions.
• .claude/skills/coding-style/SKILL.md — node-header style, pg_node_attr placement, T_Foo enum convention.
• .claude/skills/fmgr-and-spi/SKILL.md — function-call node (FuncExpr) interaction with fmgr; SPI's parse-tree consumption.
• knowledge/idioms/parser-pipeline.md — long-form pipeline narrative.
• knowledge/idioms/node-types-and-lists.md — List / foreach patterns + node-tag invariants.
• source/src/backend/nodes/README — canonical "Steps to Add a Node" reference.
• source/src/backend/parser/README — gram.y conventions, shift/reduce conflict resolution.