Results for “bulk-processing”

Designs and implements big data architectures, processes large-scale datasets with distributed systems, and optimizes data pipelines for throughput using Hadoop, Spark, and cloud platforms.

Compresses long K-line (candlestick) data into a fixed-length sequence using OHLC aggregation rules and applies min-max normalization.

Implements Bulk API 2.0 REST calls for ingest and query jobs, covering CSV format requirements, locator pagination, and v1 vs v2 selection.

Diagnoses and resolves slow Qdrant indexing and data ingestion by optimizing batching, sharding, HNSW parameters, and payload indexing strategies.

Guides when to use Salesforce Bulk API 2.0 and provides commands for importing, updating, upserting, deleting, and exporting records, along with CSV format requirements, limits, and error handling.

Orchestrates a complete bulk RNA-seq differential-expression study from raw FASTQ reads through QC, alignment, quantification, differential expression, pathway enrichment, and publication figures.

Process large-scale ML datasets with distributed streaming execution across CPU/GPU, supporting Parquet, CSV, JSON, images, and integration with PyTorch, TensorFlow, and Ray Train.

Scale pandas and NumPy workflows to larger-than-memory datasets using parallel and distributed computing.

Designs Elasticsearch indexes and mappings, tunes queries, sizes clusters, and handles operations like shard/replica strategy, ILM, monitoring, troubleshooting, and safe reindexing or upgrades.

Scalable data processing for ML workloads. Streaming execution across CPU/GPU, supports Parquet/CSV/JSON/images. Integrates with Ray Train, PyTorch, TensorFlow. Scales from single machine to 100s of nodes. Use for batch inference, data preprocessing, multi-modal data loading, or distributed ETL pipelines.

Evaluates a containerized framework for deploying distributed big data workloads, measuring execution time and cloud cost scaling from four to eight nodes.

Perform differential gene expression analysis for bulk RNA-seq data using PyDESeq2, supporting formulaic designs, Wald tests, FDR correction, LFC shrinkage, and result visualization.

Generates OMG-compliant BPMN 2.0 XML and SVG diagrams from natural language process descriptions, with validation, automatic layout, and optional process optimization advisories.

Use this skill for processing and analyzing large tabular datasets (billions of rows) that exceed available RAM. Vaex excels at out-of-core DataFrame operations, lazy evaluation, fast aggregations, efficient visualization of big data, and machine learning on large datasets. Apply when users need to work with large CSV/HDF5/Arrow/Parquet files, perform fast statistics on massive datasets, create visualizations of big data, or build ML pipelines that don't fit in memory.

Distributed computing for larger-than-RAM pandas/NumPy workflows. Use when you need to scale existing pandas/NumPy code beyond memory or across clusters. Best for parallel file processing, distributed ML, integration with existing pandas code. For out-of-core analytics on single machine use vaex; for in-memory speed use polars.

Guides analysis of Neuropixels recordings from raw data to curated units, covering preprocessing, motion correction, spike sorting, quality metrics, and export.

Document end-to-end business processes in BPMN-style notation, measure cycle times by stage, surface where work spends most of its time waiting vs. being worked, and quantify the gap between processing time and total elapsed time.

Processes STIX 2.1 threat intelligence bundles from TAXII 2.1 servers, normalizing objects into platform-native schemas and routing them to consuming systems.