lexical-analysis-tokenization
Summary
Tokenize domain-specific query strings (e.g., MassQL) into a stream of meaningful lexical units (keywords, operators, numeric literals, identifiers) to enable downstream parsing and AST construction. This is the first stage of a compiler/interpreter pipeline for mass-spectrometry-centric query languages.
When to use
You have a mass-spectrometry query string written in MassQL (or similar domain-specific SQL-inspired syntax) that must be converted into structured form for execution. The input is raw, unparsed text containing SQL keywords, MS-specific operators (e.g., MS1MZ, MS2PREC, TOLERANCEMZ, INTENSITYPERCENT), numeric m/z and intensity thresholds, and logical connectives (AND, OR). Tokenization is necessary before you can build an Abstract Syntax Tree (AST) or validate query semantics.
When NOT to use
- Input is already a pre-tokenized or parsed representation (e.g., JSON-serialized query object, AST). Skip directly to semantic validation or execution.
- You only need to execute a pre-compiled query; the query text has already been tokenized and validated by an earlier stage of the pipeline.
- Input is free-form natural language prose, not a formal query syntax. Domain-specific tokenization assumes formal grammar; unstructured text requires NLP preprocessing instead.
Inputs
- MassQL query string (raw text)
- MassQL grammar specification (keywords, operators, token patterns)
Outputs
- Token stream (sequence of (token_type, token_value, position) tuples)
- Lexical error report (if unrecognized input encountered)
How to apply
Implement a lexer that scans the input MassQL query string character-by-character, recognizing and classifying tokens according to the MassQL grammar. Group characters into meaningful units: (1) SQL and MS-specific keywords (QUERY, WHERE, MS1DATA, MS2DATA, scaninfo, PREC, MZ); (2) comparison and logical operators (=, AND, +, −); (3) numeric literals (m/z values, tolerance thresholds, intensity percentages); (4) special syntax (parentheses, colons for key-value pairs like TOLERANCEMZ=0.1). Use a whitespace-aware strategy to delimit tokens. Emit a stream of (token_type, token_value) pairs, preserving enough positional information to generate diagnostic error messages if parsing fails later. Validate that all tokens are recognized; if an unrecognized character sequence is encountered, halt with a clear error indicating the location and unexpected input.
Related tools
Examples
from massql import msql_engine; results_df = msql_engine.process_query('QUERY MS2DATA WHERE MS1MZ=572.828 TOLERANCEMZ=0.1 AND MS2PREC=572.828', 'sample.mzML')
Evaluation signals
- All valid MassQL tokens (SQL keywords like QUERY, WHERE; MS-specific operators like MS1MZ, MS2DATA, TOLERANCEMZ; numeric literals; parentheses) are correctly classified and emitted in order.
- Whitespace and comment regions are properly skipped; output token stream contains no spurious whitespace tokens.
- Compound tokens (e.g., 'MS1MZ=0.1' vs. separate 'MS1MZ' and '=' and '0.1' tokens) are split or grouped according to the grammar specification; test with multi-clause queries (AND chains with nested parentheses).
- Unrecognized input (typos, invalid operators, malformed numeric literals) triggers a lexical error with source location (line, column), enabling user diagnosis.
- Token stream is sufficient to reconstruct a recognizable approximation of the original query (round-trip test): serialize tokens back to text and compare against normalized input.
Limitations
- Lexical analysis does not validate semantic correctness (e.g., whether MS1MZ tolerance is physically reasonable or whether a referenced filter exists). Semantic checks are deferred to the parser and AST validator.
- Error recovery is minimal: the lexer typically stops at the first unrecognized token. Batch error collection (reporting all lexical issues in one pass) would require more sophisticated tokenization.
- The lexer assumes well-formed input encoding (UTF-8 or ASCII); binary or mixed-encoding query strings may cause silent misclassification.
- No support for comments or query metadata (e.g., /* comment */ syntax) unless explicitly defined in the grammar.
Evidence
- [other] Implement a lexer to tokenize the input MassQL query string, identifying SQL keywords, MS-specific operators, and numeric literals.: "Implement a lexer to tokenize the input MassQL query string, identifying SQL keywords, MS-specific operators, and numeric literals."
- [readme] MassQL is inspired by SQL, but it attempts to bake in assumptions of mass spectrometry to make querying much more natural for mass spectrometry users.: "MassQL is inspired by SQL, but it attempts to bake in assumptions of mass spectrometry to make querying much more natural for mass spectrometry users."
- [other] Define the MassQL grammar as an extended SQL syntax incorporating mass-spectrometry-specific constructs (e.g., spectrum filters, m/z ranges, intensity thresholds, fragmentation patterns): "Define the MassQL grammar as an extended SQL syntax incorporating mass-spectrometry-specific constructs (e.g., spectrum filters, m/z ranges, intensity thresholds, fragmentation patterns)"
- [other] Serialize the AST to JSON or canonical text format and report parsing success/failure with diagnostic error messages.: "Serialize the AST to JSON or canonical text format and report parsing success/failure with diagnostic error messages."
1---2name: lexical-analysis-tokenization3description: Use when you have a mass-spectrometry query string written in MassQL (or similar domain-specific SQL-inspired syntax) that must be converted into structured form for execution. The input is raw, unparsed text containing SQL keywords, MS-specific operators (e.4license: CC-BY-4.05---67# lexical-analysis-tokenization89## Summary1011Tokenize domain-specific query strings (e.g., MassQL) into a stream of meaningful lexical units (keywords, operators, numeric literals, identifiers) to enable downstream parsing and AST construction. This is the first stage of a compiler/interpreter pipeline for mass-spectrometry-centric query languages.1213## When to use1415You have a mass-spectrometry query string written in MassQL (or similar domain-specific SQL-inspired syntax) that must be converted into structured form for execution. The input is raw, unparsed text containing SQL keywords, MS-specific operators (e.g., MS1MZ, MS2PREC, TOLERANCEMZ, INTENSITYPERCENT), numeric m/z and intensity thresholds, and logical connectives (AND, OR). Tokenization is necessary before you can build an Abstract Syntax Tree (AST) or validate query semantics.1617## When NOT to use1819- Input is already a pre-tokenized or parsed representation (e.g., JSON-serialized query object, AST). Skip directly to semantic validation or execution.20- You only need to execute a pre-compiled query; the query text has already been tokenized and validated by an earlier stage of the pipeline.21- Input is free-form natural language prose, not a formal query syntax. Domain-specific tokenization assumes formal grammar; unstructured text requires NLP preprocessing instead.2223## Inputs2425- MassQL query string (raw text)26- MassQL grammar specification (keywords, operators, token patterns)2728## Outputs2930- Token stream (sequence of (token_type, token_value, position) tuples)31- Lexical error report (if unrecognized input encountered)3233## How to apply3435Implement a lexer that scans the input MassQL query string character-by-character, recognizing and classifying tokens according to the MassQL grammar. Group characters into meaningful units: (1) SQL and MS-specific keywords (QUERY, WHERE, MS1DATA, MS2DATA, scaninfo, PREC, MZ); (2) comparison and logical operators (=, AND, +, −); (3) numeric literals (m/z values, tolerance thresholds, intensity percentages); (4) special syntax (parentheses, colons for key-value pairs like TOLERANCEMZ=0.1). Use a whitespace-aware strategy to delimit tokens. Emit a stream of (token_type, token_value) pairs, preserving enough positional information to generate diagnostic error messages if parsing fails later. Validate that all tokens are recognized; if an unrecognized character sequence is encountered, halt with a clear error indicating the location and unexpected input.3637## Related tools3839- **MassQL** (Domain-specific query language whose syntax defines the lexical grammar (keywords, operators, literals) to be tokenized) — https://github.com/mwang87/MassQueryLanguage4041## Examples4243```44from massql import msql_engine; results_df = msql_engine.process_query('QUERY MS2DATA WHERE MS1MZ=572.828 TOLERANCEMZ=0.1 AND MS2PREC=572.828', 'sample.mzML')45```4647## Evaluation signals4849- All valid MassQL tokens (SQL keywords like QUERY, WHERE; MS-specific operators like MS1MZ, MS2DATA, TOLERANCEMZ; numeric literals; parentheses) are correctly classified and emitted in order.50- Whitespace and comment regions are properly skipped; output token stream contains no spurious whitespace tokens.51- Compound tokens (e.g., 'MS1MZ=0.1' vs. separate 'MS1MZ' and '=' and '0.1' tokens) are split or grouped according to the grammar specification; test with multi-clause queries (AND chains with nested parentheses).52- Unrecognized input (typos, invalid operators, malformed numeric literals) triggers a lexical error with source location (line, column), enabling user diagnosis.53- Token stream is sufficient to reconstruct a recognizable approximation of the original query (round-trip test): serialize tokens back to text and compare against normalized input.5455## Limitations5657- Lexical analysis does not validate semantic correctness (e.g., whether MS1MZ tolerance is physically reasonable or whether a referenced filter exists). Semantic checks are deferred to the parser and AST validator.58- Error recovery is minimal: the lexer typically stops at the first unrecognized token. Batch error collection (reporting all lexical issues in one pass) would require more sophisticated tokenization.59- The lexer assumes well-formed input encoding (UTF-8 or ASCII); binary or mixed-encoding query strings may cause silent misclassification.60- No support for comments or query metadata (e.g., /* comment */ syntax) unless explicitly defined in the grammar.6162## Evidence6364- [other] Implement a lexer to tokenize the input MassQL query string, identifying SQL keywords, MS-specific operators, and numeric literals.: "Implement a lexer to tokenize the input MassQL query string, identifying SQL keywords, MS-specific operators, and numeric literals."65- [readme] MassQL is inspired by SQL, but it attempts to bake in assumptions of mass spectrometry to make querying much more natural for mass spectrometry users.: "MassQL is inspired by SQL, but it attempts to bake in assumptions of mass spectrometry to make querying much more natural for mass spectrometry users."66- [other] Define the MassQL grammar as an extended SQL syntax incorporating mass-spectrometry-specific constructs (e.g., spectrum filters, m/z ranges, intensity thresholds, fragmentation patterns): "Define the MassQL grammar as an extended SQL syntax incorporating mass-spectrometry-specific constructs (e.g., spectrum filters, m/z ranges, intensity thresholds, fragmentation patterns)"67- [other] Serialize the AST to JSON or canonical text format and report parsing success/failure with diagnostic error messages.: "Serialize the AST to JSON or canonical text format and report parsing success/failure with diagnostic error messages."
HolobiomicsLab