Data Science Analysis

Data Science Analysis

A single, domain-agnostic workflow for answering quantitative questions from local datasets. Data files may live in any directory the task provides (e.g., input/, data/, the current working directory, or an explicit path in the prompt); first identify that data directory, then apply the workflow to it. The steps below apply to any domain (archeology, biomedical, environmental, legal, wildfire, astronomy/heliophysics/space weather, and beyond) and to any file type (tabular CSV/Excel or scientific array/binary formats like .npz, .fits, .h5). Domain-specific gotchas are flagged inline and detailed in references/domain-notes.md.

Core Principles

Workflow

1. Discover & Profile

• Identify the data directory the task provides (e.g., input/, data/, the current working directory, or an explicit path given in the prompt), then glob all files under it (e.g., <data_dir>/**/*) — data files, data dictionaries, metadata, format specs (.fmt, .text, README), helper scripts. Avoid overly restrictive filename filters that might exclude the correct data. Restrict data access to this directory.
• Read data dictionaries / README / format specs first. Map every prompt term (e.g., "damaged", "generally unsafe", "tumor") to its exact column or category — never guess. Cross-reference discovered schemas against format specs to confirm column mappings and units.
• Inspect raw structure before loading: read the first (and last) 10–20 rows raw (header=None) to locate the true header row, metadata/summary/footer rows, delimiters, timestamp formats, and encoding quirks (BOM → utf-8-sig).
• For Excel, list sheet_names via pandas.ExcelFile; prioritize README/Legend/Metadata sheets, then preview each data sheet.
• For scientific/binary files (.npz, .fits, .h5), print all keys and array shapes to identify the correct arrays before use.
• Print exact column names and sample values; use names—not indices—for all downstream selection.

Run scripts/explore_input.py [data_dir] first to profile every file (pass the data directory; defaults to input/). See references/data-ingestion-and-cleaning.md for encoding, header-detection, Excel, and scientific/binary-format patterns.

2. Ingest Robustly

Write a single Python script. Apply these in the first execution to avoid retries:

• Match the tool to the task complexity: use lightweight numpy/csv for simple whitespace-delimited files and direct extraction; reserve pandas for tabular joins and richer processing. Avoid over-engineering (e.g., direct pairwise comparisons rather than full time-series propagation when they suffice).
• Handle encoding robustly: try utf-8-sig (BOM), fall back to latin-1/cp1252; strip non-breaking spaces.
• Strip whitespace from headers and string columns.
• Sanitize numeric strings (commas, currency symbols, quotes) before casting; coerce with pd.to_numeric(..., errors='coerce').
• Cast through float before int (int(float(x))) to handle decimal strings like "42.0".
• For large files, use memory-efficient loading (usecols, dtype, chunksize; Excel read_only=True).

3. Clean & Sanitize

• Missing/malformed placeholders & sentinel fills: map '-', 'NA', 'N/A', blanks, and sentinels (-999, -9999, and space-physics fills like 9.99E32, 999999.9, 99999.9, 9999, 999.9) to NaN before computing any statistic, correlation, or model—including them severely skews results.
• Censored values: handle detection-limit markers ('<0.1', '>100') explicitly before numeric conversion (see domain notes).
• Duplicates & scaled integers: drop duplicate timestamps, exclude non-data/metadata columns, and decode scaled-integer encodings before use.
• String normalization: .str.strip().str.lower() for categorical filters; include na=False in .str.contains()/.str.match() to avoid errors on mixed-type columns.
• Profile unique values in categorical columns before filtering to catch case/whitespace/typo variations.
• Verify data scales: confirm whether values are log-transformed before applying exponentiation; inverse-transform only when raw metrics are required.
• Prefer NaN-ignoring aggregations over dropping rows to avoid silent data loss.

Caution: Do not normalize typos or split concatenated strings unless the task asks for it—preserve exact literal counts for raw-unique-entry queries. See references/data-ingestion-and-cleaning.md.

4. Integrate & Join

• Identify linking keys (sample IDs, gene symbols, entity names, station IDs) across tables.
• Normalize join keys on both sides: strip whitespace, standardize case, zero-pad ID strings (never numeric-coerce IDs with leading zeros).
• Align granularities before merging (e.g., county → state, daily → monthly).
• Verify key overlap empirically (len(inner) vs len(left)) and log unmatched keys to detect silent drops.
• Apply exclusion criteria first (e.g., remove samples "not in the study") before extracting features.
• Prevent data leakage: when comparing a target entity to a background set, filter the target out of the background.

See references/analysis-patterns.md § Joins and references/domain-notes.md for entity normalization (MSA/County suffixes) and biomedical linking.

5. Filter

• Apply all spatial, temporal, and categorical filters before computing any aggregations.
• Translate qualitative constraints ('summer', 'preceding days', 'generally unsafe') into precise numeric/datetime bounds via the data dictionary.
• Group by stable identifiers (conflict ID, entity ID) rather than ambiguous names to prevent double-counting multi-row entries.
• Print intermediate row counts after each filter to verify the subset matches the prompt's scope. Ensure denominators reflect only the filtered subset.

6. Analyze

• Extremum / closest-value queries: sort by primary metric, then secondary tie-breaker; isolate all ties matching the primary condition before applying the tie-breaker. Print tie counts.
• Rates & ratios: align numerator and denominator to the exact same categorical/temporal/geographic scope. Distinguish pooled overall rate (grand totals) from average of periodic rates (mean of per-period rates)—compute exactly what the prompt asks. Guard against division by zero.
• Time-series: sort chronologically before any lag/rolling/diff. Filter to the exact prompt range (don't rely on global extrema); when merging multi-source series, resample to a common cadence and inner-join on timestamp. Verify temporal coverage from the actual data, not filenames. Handle gaps and interpolate missing periods; exclude partial-year data from temporal averages.
• Signal processing: detect peaks/troughs with scipy.signal.find_peaks rather than manual thresholding; rank top-N extrema by magnitude (not raw data); cluster events within a tolerance window to avoid overcounting. See references/analysis-patterns.md § Signal Processing.
• Spatial & orbital: validate coordinate ranges (detect swapped lat/lon); filter by attributes before computing distances; count distinct entities to prevent double-counting. For orbital/coordinate work, use domain libraries (astropy, sgp4/skyfield, pymap3d), normalize longitude conventions, and track units. See references/geospatial.md.
• Forecasting: use strict chronological train/test splits (never random); align lagged features and distinguish forecast-issuance from target dates; evaluate on date-aligned observed vs. predicted values; respect task-specified windows exactly. See references/analysis-patterns.md § Forecasting & Evaluation.
• Statistics: choose robust methods—Spearman for skewed/outlier data, log-transforms for right-skewed outcomes, non-parametric tests for distribution shifts. For causal questions, map variables to roles (treatment/outcome/confounder) and diagnose selection bias.
• Domain scales: align units before analysis (e.g., paleo BP counts backwards—"later" = minimum BP). See references/domain-notes.md.

See references/analysis-patterns.md for tie-breaking, rate, time-series, signal-processing, forecasting, and statistical patterns.

7. Validate

Print intermediate results at every stage:

• Frequency distributions of parsed values and pre/post-filter row counts.
• Matched subsets (city pairs, filtered rows) and tie-breaking subsets to confirm correct records.
• Boundary-straddling edge cases in range filters; pre/post-merge overlap sizes.
• Sanity checks (e.g., warn if numerator exceeds denominator).

8. Output

Save the final answer under the current artifact directory supplied by the QwenPaw Data runtime:

Direct chat:        artifacts/<session_id>/answer.json
TaskGraph node:     artifacts/<session_id>/<graph_id>/<node_id>/answer.json

{"answer": <value>}

• Always use the current session_id. Append graph_id/node_id only when the runtime explicitly supplies an active TaskGraph node; never invent IDs.
• Do not write answer files directly under the shared artifacts/ root.
• Ensure the current artifact directory exists before writing.
• Use the write_file tool (not shell echo/redirection) for proper JSON formatting.
• Match exact requested precision; omit units, thousands separators, markdown, and conversational filler.
• Obey negative constraints (e.g., "No explanation needed")—output only the direct answer value.
• In the text response, cite source files/sheets and summarize filtering/exclusion logic; list actual entities when counting or identifying specific items.

Environment Setup

pip install pandas openpyxl numpy scipy statsmodels

Add shapely (and optionally geopandas) for geospatial tasks; beautifulsoup4 for HTML tables; h5py/astropy for .h5/.fits files; astropy, sgp4/skyfield, pymap3d for orbital mechanics and coordinate conversions. If a specialized package fails to install, fall back to core numpy/scipy. Rely solely on local files—no web access.

Additional Resources