ETL & Pipelines Agent Skills

Ingest and process high-throughput event streams using Azure Event Hubs SDK for Python, with support for producers, consumers, and checkpointing.

Send, receive, and manage messages in Azure Queue Storage using Python with sync and async clients.

Send custom logs to Azure Monitor via Data Collection Rules and Data Collection Endpoints using the Java SDK.

Manage SMB file shares, directories, and files in Azure Storage using the Python SDK.

Translate Word, PDF, Excel, PowerPoint, and other documents at scale using Azure AI Document Translation SDK with format preservation.

Extract text, tables, and structured data from documents using Azure AI Document Intelligence SDK for .NET, with support for prebuilt and custom models.

Discover, select, and run Holoscan Sensor Bridge example applications on a connected devkit over SSH, filtering by platform, board type, and sensors.

Build synthetic datasets and data generation pipelines using the Data Designer library.

Build video analytics pipelines using NVIDIA DeepStream SDK 9.0 with Python pyservicemaker API, including GStreamer-based video processing, TensorRT inference integration, object detection/tracking, and Kafka/message broker integration.

Index folders of PDFs and other documents into LanceDB for vector search, then query them with semantic search, page filters, verbatim quotes, and cross-document aggregation.

Read and write large cuPyNumeric arrays to HDF5 files using Legate's parallel, distributed HDF5 I/O.

Search archived video using natural language, ingest video files or RTSP streams, and manage ingested sources.

Install and verify cuPyNumeric for Python using conda or pip, including GPU usage checks.

Configure Jetson UPHY lane allocation on Orin/Thor custom carriers by editing ODMDATA tokens and dispatching per-controller kernel-DT overlay skills.

Build synthetic datasets and data generation pipelines using the Data Designer library.

Coordinate the end-to-end pipeline from CAD or source assets to SimReady USD assets, including conversion, material and physics assignment, validation, conformance, and optional packaging.

Accelerate pandas workflows with GPU DataFrames using cuDF and dask-cuDF for ETL, joins, groupby, and large-scale data processing.

Converts NVIDIA TAO DAFT datasets between supported formats using the `tao-daft convert` CLI.

Load sharded datasets (npy, Parquet, HDF5, raw binary) into distributed cuPyNumeric arrays using manual partitioning and Legate task launches.

Generate an OpenAPI 3.1 specification from a browser-trace capture by analyzing observed HTTP traffic, templating URLs, and inferring JSON schemas from request/response samples.

Ingest, index, search, edit, and generate video and audio content from files, URLs, live streams, or desktop capture.

Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Scrapes on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions.

Provides a fast in-memory DataFrame library for datasets that fit in RAM, with lazy evaluation, parallel execution, and an Apache Arrow backend for ETL pipelines and analytics.

Master modern SQL with cloud-native databases, OLTP/OLAP optimization, and advanced query techniques. Expert in performance tuning, data modeling, and hybrid analytical systems.

Ingest, index, search, and edit video and audio from files, URLs, live streams, or desktop sessions with timestamps, subtitles, overlays, and real-time alerts.

Guides database design, implementation, optimization, migrations, data pipelines, and operations for SQL, NoSQL, and modern data platforms.

Build, deploy, evaluate, debug, and measure latency for a GPU-accelerated FastAPI service that detects whether operators perform assembly-line steps in order via event boundary detection and VLM classification.

Create and validate Earth2Studio data source wrappers (DataSource, ForecastSource, DataFrameSource, ForecastFrameSource) from remote stores like S3, GCS, Azure, HTTP, or HuggingFace.

Merges CSV/TSV data with an InDesign template to generate personalized PDFs like certificates, badges, and catalogs.

Generate XML configuration files and execute SSMA Console commands for Oracle to SQL Server database migration, including schema conversion and data migration.

Create, validate, and improve IP geolocation feeds in CSV format with RFC 8805 compliance and opinionated best practices for network operators.

Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness, returning a structured report with exact patch locations.

Design, implement, and validate Python-based IoT Edge modules for telemetry processing, local inference, protocol translation, and edge-to-cloud integration.

Design and plan end-to-end Azure IoT and Smart City solutions: requirements, architecture, security, operations, cost, and a phased delivery plan with concrete implementation artifacts.

Read and query onchain data using events, The Graph, Dune, Ponder, Multicall3, and WebSocket subscriptions, with guidance on block iteration limitations.

Monitors EVM blockchain addresses and contracts for large transfers, new approvals, contract upgrades, unusual gas spends, MEV interactions, and first-time interactions with new contracts.