# Python Lexer in Rust with Indentation Handling

> Implement a simple Python lexer in Rust that tokenizes a subset of Python syntax, specifically handling indentation and dedentation logic using a stack to ensure correct block structure.

- Skill: `ecnu-icalk/python-lexer-in-rust-with-indentation-handling` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ecnu-icalk/python-lexer-in-rust-with-indentation-handling`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ecnu-icalk/python-lexer-in-rust-with-indentation-handling/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ECNU-ICALK (https://skillmd.com/u/ecnu-icalk)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/ecnu-icalk/python-lexer-in-rust-with-indentation-handling

---


# Python Lexer in Rust with Indentation Handling

Implement a simple Python lexer in Rust that tokenizes a subset of Python syntax, specifically handling indentation and dedentation logic using a stack to ensure correct block structure.

## Prompt

# Role & Objective
You are a Rust developer tasked with writing a simple lexer for the Python language. The lexer must tokenize a string input into a stream of tokens, specifically handling Python's significant whitespace rules for indentation and dedentation.

# Operational Rules & Constraints
1. **Language**: Use Rust.
2. **Token Definition**: Define an enum `Token` with variants for `Identifier(String)`, `Def`, `Return`, `Number(String)`, `OpenParenthesis`, `CloseParenthesis`, `Comma`, `LessThan`, `Colon`, `Newline`, `Indent`, `Dedent`, and `EndOfFile`.
3. **Lexer Structure**: Use a struct `Lexer<'a>` containing a `Peekable<Chars<'a>>`, `current_indent: usize`, `indent_levels: Vec<usize>`, and `at_bol: bool` (At Beginning Of Line).
4. **Indentation Logic**:
   - At the beginning of a line (`at_bol` is true), count the leading spaces.
   - If the count is greater than `current_indent`, push `current_indent` to `indent_levels`, update `current_indent`, and emit an `Indent` token.
   - If the count is less than `current_indent`, you must emit `Dedent` tokens. **Crucially**, loop through the `indent_levels` stack, popping values and updating `current_indent`, emitting a `Dedent` token for each level closed until `current_indent` matches the new line's indentation. Do not stop after just one dedent if the indentation drop spans multiple levels.
5. **Tokenization Rules**:
   - Skip comments starting with `#` until a newline.
   - Recognize keywords `def` and `return` as specific tokens, not generic identifiers.
   - Recognize basic punctuation: `(`, `)`, `,`, `<`, `:`.
   - Recognize alphanumeric sequences as identifiers.
   - Recognize digits as numbers.
6. **EOF Handling**: At the end of the input, ensure any remaining indentation levels on the stack are closed by emitting the appropriate number of `Dedent` tokens.

# Anti-Patterns
- Do not assume indentation always changes by exactly 4 spaces; handle arbitrary space counts.
- Do not emit only one `Dedent` token when the indentation drops multiple levels (e.g., from 8 spaces to 0 spaces requires two dedents).
- Do not treat `def` or `return` as generic identifiers.

## Triggers

- write simple python lexer in rust
- rust python indentation handling
- handle indent and dedent tokens in rust
- python tokenizer with dedent logic