DevOps & Infra

ivanshamaev Skill

Data Contracts — datacontract.com YAML specification (schema, quality, SLA, ownership, servers, changelog), Data Contract CLI (init/test/lint/diff/breaking/publish/export), SodaCL-style quality checks embedded in contract, breaking change detection (column removal/type change/constraint tightening), CI/CD GitHub Actions PR enforcement, Kafka/S3/database server definitions, contract-driven development (producer-first), semver versioning and deprecation, DataHub and OpenMetadata catalog publishing

ivanshamaev Skill

Terraform for data infrastructure — S3/MinIO data lake buckets (versioning, lifecycle, SSE-KMS), IAM roles for Spark/Airflow (least-privilege, IRSA on EKS), MSK/Kafka clusters (aws_msk_cluster, encryption, custom broker config), Kubernetes data platform (helm_release Airflow + Spark, namespace resource quotas), module layout (modules/ + envs/), typed variables with validation, S3 remote state + DynamoDB locking, Terragrunt DRY configs, GitHub Actions CI/CD pipeline with plan/apply

ivanshamaev Skill

Data platform on Kubernetes — Spark-on-K8s (spark-submit, pod templates, dynamic allocation, RBAC), Airflow on K8s (Helm chart, KubernetesExecutor, KubernetesPodOperator, git-sync, DAG storage), resource management, namespaces, autoscaling

ivanshamaev Skill

Docker for data engineering environments — multi-stage Dockerfiles for dbt/Spark/Airflow images, BuildKit layer caching (--mount=type=cache), private registries (ghcr.io/Harbor), docker buildx multi-platform builds, Docker Compose local data stacks (Spark+Airflow+Kafka+MinIO+Postgres), BuildKit secrets for private PyPI, security hardening (non-root user, slim base images, read-only FS), CI/CD with GitHub Actions build-push-action

ivanshamaev Skill

Prefect 3.x workflows — flows, tasks, deployments, work pools, event-driven triggers, caching, retries, state hooks, Prefect Cloud/server, Python SDK, Docker/K8s infrastructure

ivanshamaev Skill

Great Expectations (GX) — DataContext setup, Data Sources (Pandas/Spark/SQL/file), Expectation Suites, built-in expectations (null/uniqueness/range/set/regex/table-level/statistical), Validation Definitions, Checkpoints, Data Docs, Airflow integration, custom expectations, dbt integration, CI/CD usage, severity levels (warning/critical)

ivanshamaev Bundle

Use when building, reviewing, debugging, or scaling Apache Airflow DAGs declaratively with dag-factory YAML configs — including project setup, loader configuration, defaults hierarchy, custom/provider operators, dynamic task mapping, datasets/assets, callbacks, TaskFlow decorators, Jinja2 templating, environment variables, YAML DRY patterns, large-scale multi-DAG generation, CI/CD linting, and migration from pre-1.0.

taipt1504 Bundle

Kafka and RabbitMQ patterns for Spring Boot 3.x — producer reliability, consumer groups, exactly-once semantics, dead letter topics/queues, Schema Registry with Avro, Spring Cloud Stream, and reactive messaging. Use when writing @KafkaListener consumers, KafkaTemplate producers, RabbitMQ listeners, configuring DLQ/DLT, implementing event-driven microservices, or setting up message serialization with Avro/JSON schemas.

taipt1504 Bundle

Deployment patterns for Java Spring Boot — Docker, Kubernetes, CI/CD, health probes, graceful shutdown. Use when writing Dockerfiles, K8s manifests (Deployment/Service/HPA/ConfigMap), CI/CD pipelines (GitHub Actions, GitLab CI), configuring health probes (liveness/readiness/startup), setting up graceful shutdown, sizing container resources, or containerizing Spring Boot applications.

ivanshamaev Skill

Azure data platform review — ADLS Gen2 (hierarchical namespace/RBAC/lifecycle), Azure Synapse vs Databricks vs HDInsight trade-offs, Event Hubs (Kafka protocol compatible), ADF vs Azure Databricks for ETL, Azure Managed Airflow, AKS with Workload Identity (AAD Pod Identity), Key Vault for secrets, Azure Monitor/Log Analytics, Purview for data catalog/lineage, Private Endpoints for data services, cost management (reserved instances/spot VMs)

ivanshamaev Skill

Observability stack review — three pillars (metrics/logs/traces), Prometheus+Grafana+Loki+Tempo (LGTM stack), OpenTelemetry Collector as universal agent, cardinality management, log aggregation patterns (Fluent Bit→Loki), distributed tracing (Jaeger/Tempo), alerting pipeline (Alertmanager routing), SLO definition and error budget tracking, data platform observability (pipeline freshness/throughput/error rate), observability as code (Grafana provisioning)

ivanshamaev Skill

AIOps continuous platform optimization agent — autonomous optimization loop (observe→analyze→recommend→apply), Kubernetes resource rightsizing (VPA recommendation reader + apply), Kafka consumer lag auto-scale, Spark job configuration tuning from history, idle resource detection (unused deployments/PVCs/topics), compaction and VACUUM scheduling, auto-ANALYZE stale statistics, cost savings attribution, optimization audit trail, human approval gate for high-risk changes

ivanshamaev Skill

AIOps autonomous incident response agent — LLM-driven diagnosis loop (Claude tool-use agent with kubectl/SQL/Prometheus tools), alert-to-action pipeline (PagerDuty webhook → agent trigger), automated RCA generation (failure taxonomy + log correlation), self-healing action executor with approval gate, incident severity classification, runbook automation (structured YAML runbooks executed by agent), escalation logic, incident timeline auto-generation, Slack integration for human-in-the-loop approvals

ivanshamaev Skill

Streaming pipeline reliability — exactly-once semantics (idempotent producer/transactional API/read_committed/Kafka Streams EOS), at-least-once patterns with idempotent consumers, dead letter queue (DLQ) design with Kafka Connect SMT/custom error handler, poison message handling, reprocessing strategy, watermarks and late data handling, checkpoint/savepoint for Flink/Spark Streaming, consumer group rebalance storm prevention, end-to-end latency SLO monitoring, streaming reliability checklist

ivanshamaev Skill

Kubernetes autoscaling review — HPA (CPU/memory/custom/external metrics), stabilizationWindowSeconds anti-flapping, VPA vs HPA decision, Cluster Autoscaler tuning (scale-down delay/utilization threshold), KEDA event-driven autoscaling (Kafka lag/queue depth), spot node optimization, pod disruption budgets, scaling behavior policies (Percent/Pods/Max/Min selectPolicy)

ivanshamaev Skill

OpenTelemetry instrumentation for data platforms — Python auto-instrumentation (opentelemetry-bootstrap), manual span creation for ETL tasks, trace context propagation across Airflow→Spark→dbt, OTLP exporter configuration, W3C TraceContext headers in Kafka messages, resource attributes (service.name/environment/version), Baggage for partition metadata propagation, Collector pipeline (tail sampling), correlation between traces/logs/metrics, Grafana Tempo integration

ivanshamaev Skill

Use when building or reviewing GitHub Actions CI/CD pipelines for data engineering — covering dbt slim CI (state:modified+, --defer, manifest.json), SQLFluff SQL linting with PR annotations, Airflow DAG integrity tests (pytest/DagBag), Great Expectations and Soda data quality gates, multi-stage Docker image builds for Spark/dbt/Airflow pushed to ghcr.io, secrets management with OIDC for AWS/GCP (no static keys), environment-scoped secrets, reusable workflows (workflow_call), matrix builds, composite actions, and full end-to-end dbt project CI/CD workflows.

ivanshamaev Skill

Use when designing or implementing Medallion (Bronze/Silver/Gold) data lakehouse architecture — layer definitions, DDL for each layer, DML load and update patterns, incremental pipeline design, deduplication strategies (row_number/merge/hash/CDC/watermark), schema evolution, data quality gates, partitioning per layer, late-arriving data, orchestration patterns, and best practices for Iceberg-based lakehouses.

ivanshamaev Skill

RBAC audit for data platforms — Kubernetes RBAC (ClusterRole/Role/Binding audit), AWS IAM permission boundary and policy review (AWS IAM Access Analyzer), GCP IAM audit (principle of least privilege/owner role detection), database RBAC (PostgreSQL/Trino/ClickHouse role hierarchies), Airflow RBAC (role-based menu/DAG access), dbt permissions (warehouse roles), service account proliferation detection, wildcard permission detection, privilege escalation paths

ivanshamaev Skill

Airflow production readiness — idempotent tasks (UPSERT/partition overwrite), no top-level code at DAG parse time (Variable.get inside tasks), KubernetesExecutor configuration, connection/variable management (no hardcoded secrets), SLA callbacks, retry with exponential backoff, pool management, max_active_runs, dag_concurrency, celery vs kubernetes executor trade-offs, metadata DB maintenance (airflow db clean), health check endpoints, structured logging

ivanshamaev Skill

AIOps infrastructure anomaly detection — statistical baseline (z-score/IQR/seasonal decomposition), Prometheus anomaly rules (predict_linear/stddev_over_time/MAD), ML-based detection (Isolation Forest/Prophet/LSTM autoencoder on metrics), metric correlation clustering, automated alert threshold tuning from historical data, Kubernetes resource anomaly detection, data pipeline anomaly signals (row count z-score/freshness drift/throughput drop), Python sklearn/Prophet integration, Grafana anomaly panel

ivanshamaev Skill

Data platform self-service API — FastAPI REST API for platform operations (Kafka topic CRUD/Airflow DAG trigger/dbt job run/Trino query execution), OAuth2+JWT authentication with team-based RBAC, async job tracking with status polling, rate limiting per team, OpenAPI spec generation, Terraform provider for API resources, versioned API (v1/v2), audit logging of all platform operations, SDK generation (Python/TypeScript), Kubernetes deployment with HPA

ivanshamaev Skill

Trino Modern Data Stack end-to-end reference architecture — full docker-compose (Kafka + MinIO + Hive Metastore + Trino + Airflow + dbt + Superset + Prometheus + Grafana), medallion lake DDL and pipeline DAGs, Kafka→Iceberg streaming ingest, dbt project layout with Trino profiles, Superset dataset config, Prometheus alert rules, production Kubernetes Helm values, operational runbook for daily maintenance

ivanshamaev Skill

Agentic control plane for data platform — MCP (Model Context Protocol) server exposing platform tools to AI agents (FastMCP/Python SDK), natural language platform operations (trigger DAG/create topic/run dbt/query Trino via LLM), Claude-based platform assistant with tool use, multi-agent platform governance (audit agent/cost agent/reliability agent), MCP server deployment (Docker/Kubernetes), tool authorization and audit logging, agentic workflow patterns (plan-execute-verify), platform chat interface

ivanshamaev Skill

Internal Developer Platform (IDP) for data engineering — Backstage Software Catalog (catalog-info.yaml for pipelines/datasets/services), Software Templates (scaffold new DAG/dbt project/Kafka topic), TechDocs integration, golden path enforcement, Port.io alternative, self-service data pipeline provisioning, platform scorecard (production readiness checks), team ownership model, API gateway for platform services, paved road vs escape hatch pattern

pangzhenying2025 Bundle

Process guidelines, CI/CD, code review, standards reference. 116 reference documents covering processes, standards, technologies, tools.

ivanshamaev Skill

StarRocks Broker Load — LOAD DATA from S3/HDFS/GCS/MinIO/Azure, CSV/Parquet/ORC format support, column mapping expressions, multi-table atomic load, SHOW LOAD status polling, CANCEL LOAD, S3 credential patterns (access key/IAM role/instance profile), parallelism tuning, Airflow integration, wildcard file paths

ivanshamaev Skill

StarRocks CDC pipeline — Flink CDC (flink-cdc-connectors) to StarRocks Primary Key table, Debezium + Kafka → Routine Load upsert, StarRocks Flink connector (exactly-once, DELETE semantics), schema change propagation, Debezium envelope parsing, multi-table CDC fan-out, dead letter queue for failed CDC events, lag monitoring

ivanshamaev Skill

Full local Trino Lakehouse Docker Compose stack — trinodb/trino coordinator+worker, Hive Metastore 3.x with PostgreSQL backend, MinIO object storage (S3-compatible), Apache Airflow, dbt-trino, Apache Superset BI, Prometheus+Grafana monitoring, healthchecks, startup ordering, persistent volumes, isolated networks, catalog property files, Iceberg catalog config, environment variable injection, Trino CLI access

pangzhenying2025 Bundle

Terraform infrastructure for multi-region vehicle cloud deployment. 19 configuration files.

wooyoohl Skill

Use at the end of the pipeline to turn a residual claim, scoped contribution, evidence plan, and reviewer risks into a coherent paper pitch and execution plan.

wooyoohl Bundle

Use throughout the research pipeline to checkpoint progress into documents, manage project directories, preserve decisions, and prevent context drift caused by long conversations.

pangzhenying2025 Bundle

Helm charts for automotive cloud services. 3 configuration files.

pangzhenying2025 Skill

Expert skill in big data focusing on cloud domain applications. Covers 126 topics across cloud domain. Includes 126 skill files covering ASPICE Level 3, AUTOSAR 4.4, ISO 21434, ISO 26262.

pangzhenying2025 Bundle

Pre-commit, pre-push, post-commit, and post-deploy Git hooks for automotive ASPICE compliance. 24 hook scripts across 5 lifecycle stages.

pangzhenying2025 Bundle

Kubernetes configurations for vehicle backend services. 2 configuration files.