lisonevf Data Cleaning
Prepares financial time series data for analysis by handling common data quality issues.
Real Code Reference
tradinglearn/utils/data_fetcher.py—fetch_stock_data()normalizes TDX raw data to clean DataFrametradinglearn/utils/simple_pytdx2.py— mock data generator with controllable noisetradinglearn/pytdx2/client/quotationClient.py— raw data source (prices need scaling adjustments)
Capabilities
- Missing values: forward fill (ffill), linear interpolation, drop, or flag
- Outlier detection: z-score, IQR, moving average deviation
- Corporate actions: adjust prices for splits and dividends using adjustment factors
- Date alignment: merge multiple series on common trading dates (not calendar dates)
- Data quality: volume anomalies, price gaps > N%, stale data, duplicate rows
Typical Workflow
- Load raw data →
pd.DataFramewith columns: date, open, high, low, close, volume - Check nulls →
df.isnull().sum(), decide fill strategy - Detect outliers → flag bars where
abs(zscore(returns)) > 3 - Adjust for splits → apply adjustment factor series
- Align dates → reindex to intersection of all trading calendars
- Validate → no NaNs, no suspicious jumps, no future dates
Edge Cases
- Non-trading days: use trading calendar, not calendar days
- IPO dates: trim to actual listing date (data before is invalid)
- Suspended stocks: forward-fill last price or mark as NaN
- Pre/after-hours: decide whether to include or filter