vamseeachanta Parallel File Processor
Quick Start
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
import pandas as pd
def process_csv(file_path: Path) -> dict:
"""Process a single CSV file."""
df = pd.read_csv(file_path)
return {'file': file_path.name, 'rows': len(df), 'columns': len(df.columns)}
# Get all CSV files
files = list(Path('data/raw/').glob('*.csv'))
# Process in parallel
results = []
with ThreadPoolExecutor(max_workers=8) as executor:
futures = {executor.submit(process_csv, f): f for f in files}
for future in as_completed(futures):
results.append(future.result())
print(f"Processed {len(results)} files")
When to Use
- Processing large numbers of files (100+ files)
- Batch operations on directory contents
- Extracting data from multiple ZIP archives
- Aggregating results from parallel operations
- CPU-bound file transformations
- IO-bound file operations with proper concurrency
Related Skills
- data-pipeline-processor - Data transformation
- yaml-workflow-executor - Workflow automation
- engineering-report-generator - Report generation
Version History
- 1.1.0 (2026-01-02): Upgraded to SKILL_TEMPLATE_v2 format with Quick Start, Error Handling, Metrics, Execution Checklist, additional examples
- 1.0.0 (2024-10-15): Initial release with FileScanner, ParallelProcessor, progress tracking, result aggregation
Sub-Skills
- Example 1: Process CSV Files (+3)
- Do (+1)
Sub-Skills
- Error Handling
- Execution Checklist
- Metrics
Sub-Skills
- Core Pattern
- Core Components (+5)
- Basic Configuration
- Mode Selection (+2)