Results for “data-pipeline”

Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.

Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.

Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.

Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.

Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.

Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.

Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.

Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.

Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.

Scalable data processing for ML workloads. Streaming execution across CPU/GPU, supports Parquet/CSV/JSON/images. Integrates with Ray Train, PyTorch, TensorFlow. Scales from single machine to 100s of nodes. Use for batch inference, data preprocessing, multi-modal data loading, or distributed ETL pipelines.

Scalable data processing for ML workloads. Streaming execution across CPU/GPU, supports Parquet/CSV/JSON/images. Integrates with Ray Train, PyTorch, TensorFlow. Scales from single machine to 100s of nodes. Use for batch inference, data preprocessing, multi-modal data loading, or distributed ETL pipelines.

Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.

Parse FCS (Flow Cytometry Standard) files v2.0-3.1, extract events as NumPy arrays, read metadata and channels, and convert to CSV or DataFrame for flow cytometry data preprocessing.

Implements production LLM operations: RAG pipelines, embeddings, vector databases, fine-tuning, advanced prompt engineering, cost estimation, quality evals, semantic caching, streaming, and agents.

Provides guidance and code for production AI workflows including RAG pipelines, vector databases, embedding indexing, prompt engineering, cost estimation, semantic caching, and quality evaluation.

Process large ML datasets in parallel across CPU or GPU clusters, with streaming execution, multi-format I/O, and integration with Ray Train, PyTorch, and TensorFlow for batch inference and preprocessing pipelines.

Agent profile for design data pipelines, transformations, imports, exports, warehouse models, validation, and freshness checks. Use when Codex needs a specialist agent perspective for planning, implementation, review, debugging, validation, or handoff in this domain.

Cleans credit risk data and screens variables for pre-loan modeling through an 11-step pipeline covering missing rate calculation, IV/PSI filtering, null importance denoising, and correlation removal.

Comprehensive toolkit for developing with the CocoIndex library. Use when users need to create data transformation pipelines (flows), write custom functions, or operate flows via CLI or API. Covers building ETL workflows for AI data processing, including embedding documents into vector databases, building knowledge graphs, creating search indexes, or processing data streams with incremental updates.

Builds Retrieval-Augmented Generation systems with document chunking, embedding generation, vector storage, and retrieval pipelines, including evaluation and optimization.

Fast in-memory DataFrame analytics with Polars — lazy evaluation, parallel execution, and an Apache Arrow backend for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory, for 1-100GB datasets, ETL pipelines, or a faster pandas replacement. For larger-than-RAM data prefer dask or vaex. Part of the AlterLab Academic Skills suite.

Build and deploy bioinformatics workflows as serverless pipelines on the Latch platform using Python decorators, cloud data management, and GPU support.

Use this skill for processing and analyzing large tabular datasets (billions of rows) that exceed available RAM. Vaex excels at out-of-core DataFrame operations, lazy evaluation, fast aggregations, efficient visualization of big data, and machine learning on large datasets. Apply when users need to work with large CSV/HDF5/Arrow/Parquet files, perform fast statistics on massive datasets, create visualizations of big data, or build ML pipelines that do not fit in memory.

Use this skill for processing and analyzing large tabular datasets (billions of rows) that exceed available RAM. Vaex excels at out-of-core DataFrame operations, lazy evaluation, fast aggregations, efficient visualization of big data, and machine learning on large datasets. Apply when users need to work with large CSV/HDF5/Arrow/Parquet files, perform fast statistics on massive datasets, create visualizations of big data, or build ML pipelines that don't fit in memory.

Process whole slide images for digital pathology: detect tissue, extract tiles, and prepare datasets for deep learning pipelines.

Use this skill for processing and analyzing large tabular datasets (billions of rows) that exceed available RAM. Vaex excels at out-of-core DataFrame operations, lazy evaluation, fast aggregations, efficient visualization of big data, and machine learning on large datasets. Apply when users need to work with large CSV/HDF5/Arrow/Parquet files, perform fast statistics on massive datasets, create visualizations of big data, or build ML pipelines that don't fit in memory.

Use this skill for processing and analyzing large tabular datasets (billions of rows) that exceed available RAM. Vaex excels at out-of-core DataFrame operations, lazy evaluation, fast aggregations, efficient visualization of big data, and machine learning on large datasets. Apply when users need to work with large CSV/HDF5/Arrow/Parquet files, perform fast statistics on massive datasets, create visualizations of big data, or build ML pipelines that don't fit in memory.

Use essa skill para processar e analisar grandes conjuntos de dados tabulares (bilhões de linhas) que excedem a RAM disponível. Vaex excels em operações DataFrame out-of-core, avaliação lazy, agregações rápidas, visualização eficiente de big data e machine learning em datasets grandes. Aplique quando usuários precisarem trabalhar com arquivos CSV/HDF5/Arrow/Parquet grandes, realizar estatísticas rápidas em datasets massivos, criar visualizações de big data ou construir pipelines de ML que não cabem em memória.

Designs and implements production-grade RAG systems by chunking documents, generating embeddings, configuring vector stores, building hybrid search pipelines, applying reranking, and evaluating retrieval quality.

Build, run, and debug Nextflow data pipelines and nf-core workflows end to end, covering processes, channels, operators, configuration, testing, and deployment to HPC or cloud.

Integrate Xquik into apps, scripts, data pipelines, or AI agents for X API tasks like tweet search, user lookup, follower export, media actions, and webhook verification.

Out-of-core tabular analytics with Vaex for billion-row datasets that exceed RAM — lazy evaluation, fast aggregations, big-data visualization, and ML on a single machine. Use when working with large CSV/HDF5/Arrow/Parquet files, computing fast statistics on massive datasets, visualizing big data, or building ML pipelines that do not fit in memory. For distributed clusters prefer dask; for in-memory speed prefer polars. Part of the AlterLab Academic Skills suite.

Design data systems by understanding storage engines, replication, partitioning, transactions, and consistency models. Use when the user mentions "database choice", "replication lag", "partitioning strategy", "consistency vs availability", "stream processing", "ACID transactions", "eventual consistency", or "LSM tree vs B-tree". Also trigger when choosing between SQL and NoSQL, designing data pipelines, or debugging distributed system consistency issues. Covers data models, batch/stream processing, and distributed consensus. For system design, see system-design. For resilience, see release-it.

Build clinical deep-learning pipelines with PyHealth: load EHR, signal, and imaging datasets, define prediction tasks, instantiate models, train with the PyHealth Trainer, and compute clinical metrics.

Build and deploy apps/applets on the DNAnexus cloud genomics platform, manage data objects, run workflows, and use the dxpy Python SDK for genomics pipeline development and execution.

Process and analyze mass spectrometry data: import spectra from MGF, mzML, MSP, and JSON formats; apply 40+ filters for metadata harmonization and peak cleaning; compute spectral similarities (cosine, modified cosine) for compound identification; build reproducible processing pipelines.