vamseeachanta Multi-Format CSV Detection and Deduplication
When ingesting CSVs from sources that evolve formats over time (e.g., Fidelity exports with old vs. new column orders), detect format from header structure rather than filename. The same year/period can exist in both formats with identical transactions. Use column position and presence of distinguishing headers (e.g., 'Account Number' only in new format) as discriminators. Build a unified parser that normalizes both formats to a canonical schema, then deduplicate by transaction fingerprint (date, ticker, quantity, price) across all input files.