Fxview Parquet Consumer

FXView Tick Parquet Consumer

Read FXView tick data from Parquet files produced by the MT5 tick collection system.

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

When to Use This Skill

Use this skill when:

• Reading or querying FXView tick Parquet files
• Building adapters or pipelines for FXView tick data
• Writing DuckDB queries against tick data
• Integrating tick data into Python or Rust pipelines
• Checking tick data schema or file layout

Authoritative Source

https://github.com/terrylica/mql5/blob/05f7c82/docs/tick_research/SCHEMA_VERIFIED.md

This skill summarizes the verified schema. For full verification details, flags distributions, and row group internals, see the source document.

Schema

All 6 columns are non-nullable. Schema is identical between EURUSD and XAUUSD files.

Parquet Footer Metadata

Key-value pairs embedded in every file footer:

File Layout

Directory pattern: {base_path}/FXVIEW_{SYMBOL}/{YYYY}/{SYMBOL}_{YYYYMMDD}.parquet

Example: ~/.cache/opendeviationbar/ticks/FXVIEW_EURUSD/2026/EURUSD_20260323.parquet

IMPORTANT asymmetry: The directory uses the FXVIEW_ prefix (broker-qualified), but the filename and metadata use the bare symbol name. Do not mix these up.

Crash recovery files: {SYMBOL}_{YYYYMMDD}_1.parquet, _2.parquet, etc. Created when the EA restarts and finds an existing file for today. All segments are valid, complete Parquet files with footers.

XAUUSD Differences

XAUUSD has ~2.2x higher tick density than EURUSD.

DuckDB Consumption Examples

Read a single day

SELECT * FROM read_parquet('~/.cache/opendeviationbar/ticks/FXVIEW_EURUSD/2026/EURUSD_20260323.parquet');

Read all files for a symbol (glob)

SELECT * FROM read_parquet('~/.cache/opendeviationbar/ticks/FXVIEW_EURUSD/**/*.parquet');

Convert timestamp to human-readable

SELECT make_timestamp(time_msc * 1000) AS ts, bid, ask, ask - bid AS spread
FROM read_parquet('~/.cache/opendeviationbar/ticks/FXVIEW_EURUSD/2026/EURUSD_20260323.parquet');

Spread statistics

SELECT avg(ask - bid) AS avg_spread, max(ask - bid) AS max_spread
FROM read_parquet('~/.cache/opendeviationbar/ticks/FXVIEW_EURUSD/2026/EURUSD_20260323.parquet');

Flags distribution

SELECT flags, count(*) AS cnt, round(count(*) * 100.0 / sum(count(*)) OVER (), 2) AS pct
FROM read_parquet('...')
GROUP BY flags
ORDER BY cnt DESC;

Anti-Patterns

• Do NOT use last or volume columns for FXView forex (always zero)
• Do NOT assume FXVIEW_ prefix in filenames -- it is ONLY in the directory name
• Do NOT use f64 for prices in Rust production code (use i64 fixed-point via digits metadata)
• Do NOT expect time_msc in seconds -- it is MILLIseconds
• Do NOT hardcode machine paths -- use pattern {base_path}/FXVIEW_{SYMBOL}/{YYYY}/{SYMBOL}_{YYYYMMDD}.parquet

Parquet Internals

• Compression: ZSTD level 3
• time_msc encoding: DELTA_BINARY_PACKED (optimal for monotonic timestamps, dictionary disabled)
• flags encoding: Dictionary encoding (low cardinality -- 3 distinct values)
• bid, ask, last, volume: Default (PLAIN + ZSTD)
• Row groups: ~65,536 rows each (last group has remainder)
• File size: ~1.35 MB for ~416K EURUSD ticks/day

Flags Bitmask Reference

Troubleshooting

Post-Execution Reflection

After this skill completes, check before closing:

1. Did the command succeed? — If not, fix the instruction or error table that caused the failure.
2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match.
3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.