ivanshamaev

@ivanshamaev source repo

159 skills
0 followers
9 hours ago last updated

▌
Starrocks Broker Load · ivanshamaev
StarRocks Broker Load — LOAD DATA from S3/HDFS/GCS/MinIO/Azure, CSV/Parquet/ORC format support, column mapping expressions, multi-table atomic load, SHOW LOAD status polling, CANCEL LOAD, S3 credential patterns (access key/IAM role/instance profile), parallelism tuning, Airflow integration, wildcard file paths

0 installs
▌
Starrocks Stream Load · ivanshamaev
StarRocks Stream Load — HTTP PUT API (curl/Python requests), CSV/JSON format parameters (column_separator/jsonpaths/strip_outer_array), columns mapping, partial_update for Primary Key tables, label idempotency, max_filter_ratio, timeout tuning, merge commit mode (3.4+), response JSON parsing, Python bulk loader pattern, error diagnosis

0 installs
▌
Trino Explain Plan Review · ivanshamaev
Trino EXPLAIN and EXPLAIN ANALYZE plan reading — DISTRIBUTED/LOGICAL/IO/VALIDATE formats, fragment types (SINGLE/HASH/ROUND_ROBIN/BROADCAST/SOURCE), exchange node analysis, stage bottleneck detection, data skew identification via task row-count variance, spill detection, operator-level metrics (ScanFilterProject/HashJoin/Aggregation/LocalExchange), cost estimate vs actual row disparity, slow plan patterns and fixes

0 installs
▌
Starrocks Cdc Pipeline · ivanshamaev
StarRocks CDC pipeline — Flink CDC (flink-cdc-connectors) to StarRocks Primary Key table, Debezium + Kafka → Routine Load upsert, StarRocks Flink connector (exactly-once, DELETE semantics), schema change propagation, Debezium envelope parsing, multi-table CDC fan-out, dead letter queue for failed CDC events, lag monitoring

0 installs
▌
Starrocks Explain Plan · ivanshamaev
StarRocks EXPLAIN plan reading — EXPLAIN/EXPLAIN VERBOSE/EXPLAIN COSTS output structure, operator nodes (OlapScanNode/HashJoinNode/AggregationNode/ExchangeNode/SortNode/ProjectNode), reading fragment instances, identifying bottlenecks (shuffle exchange cost, large scan rows, memory spill), runtime filter nodes, partition pruning in scan, query profile (SHOW PROFILELIST/EXPLAIN ANALYZE)

0 installs
▌
Starrocks Partitioning · ivanshamaev
StarRocks partitioning strategy — RANGE partitioning (manual/dynamic/expression), LIST partitioning, dynamic partition properties (enable/time_unit/start/end/prefix/replication_num), partition pruning verification via EXPLAIN, hot/cold tiering with storage_medium, partition lifecycle automation, SHOW PARTITIONS, ADD/DROP/TRUNCATE PARTITION

0 installs
▌
Starrocks Self Healing · ivanshamaev
StarRocks self-healing automation — auto-restart failed Routine Load jobs (detect PAUSED/CANCELLED + RESUME), rebalance tablet distribution after BE add/remove (REBALANCE command), trigger manual compaction for high-score tablets, auto-ANALYZE stale statistics, partition management (auto-drop old partitions), dead load label cleanup, BE disk space alerts + auto-tiering, scheduled health check + remediation agent

0 installs
▌
Trino Admin Cluster Health · ivanshamaev
Trino cluster health monitoring and administration — coordinator/worker health checks, REST API status endpoints (/v1/info /v1/node /v1/query /v1/cluster), JMX MBean metrics (running queries/failed queries/OOM kills/memory pool), Prometheus JMX exporter config, Grafana dashboards, query queue depth monitoring, worker node stability, memory pressure detection, BLOCKED query diagnosis, Web UI interpretation, log analysis, graceful shutdown

0 installs
▌
Trino Docker Compose Stack · ivanshamaev
Full local Trino Lakehouse Docker Compose stack — trinodb/trino coordinator+worker, Hive Metastore 3.x with PostgreSQL backend, MinIO object storage (S3-compatible), Apache Airflow, dbt-trino, Apache Superset BI, Prometheus+Grafana monitoring, healthchecks, startup ordering, persistent volumes, isolated networks, catalog property files, Iceberg catalog config, environment variable injection, Trino CLI access

0 installs
▌
Starrocks Data Modeling · ivanshamaev
StarRocks data modeling — star schema vs wide table design, fact/dimension DDL patterns, denormalization strategy for OLAP, bitmap indexes, bloom filter indexes, sort key optimization, schema review checklist, SCD type 2 with Primary Key table, aggregation table design, BI acceleration patterns, avoiding over-normalization

0 installs
▌
Infra Azure Data Platform Review · ivanshamaev
Azure data platform review — ADLS Gen2 (hierarchical namespace/RBAC/lifecycle), Azure Synapse vs Databricks vs HDInsight trade-offs, Event Hubs (Kafka protocol compatible), ADF vs Azure Databricks for ETL, Azure Managed Airflow, AKS with Workload Identity (AAD Pod Identity), Key Vault for secrets, Azure Monitor/Log Analytics, Purview for data catalog/lineage, Private Endpoints for data services, cost management (reserved instances/spot VMs)

0 installs
▌
Infra Observability Stack Review · ivanshamaev
Observability stack review — three pillars (metrics/logs/traces), Prometheus+Grafana+Loki+Tempo (LGTM stack), OpenTelemetry Collector as universal agent, cardinality management, log aggregation patterns (Fluent Bit→Loki), distributed tracing (Jaeger/Tempo), alerting pipeline (Alertmanager routing), SLO definition and error budget tracking, data platform observability (pipeline freshness/throughput/error rate), observability as code (Grafana provisioning)

0 installs
▌
Airflow Starrocks Etl Best Practices · ivanshamaev
Airflow + StarRocks ETL best practices — idempotent DAG design (label strategy/partition overwrite), retry with backoff, SLA callbacks, dynamic partition creation, duplicate prevention with MERGE/INSERT OVERWRITE, DAG-level concurrency controls, dependency ordering, catchup safety, data lineage tagging

0 installs
▌
Aiops Platform Optimization Agent · ivanshamaev
AIOps continuous platform optimization agent — autonomous optimization loop (observe→analyze→recommend→apply), Kubernetes resource rightsizing (VPA recommendation reader + apply), Kafka consumer lag auto-scale, Spark job configuration tuning from history, idle resource detection (unused deployments/PVCs/topics), compaction and VACUUM scheduling, auto-ANALYZE stale statistics, cost savings attribution, optimization audit trail, human approval gate for high-risk changes

0 installs
▌
Aiops Autonomous Incident Response · ivanshamaev
AIOps autonomous incident response agent — LLM-driven diagnosis loop (Claude tool-use agent with kubectl/SQL/Prometheus tools), alert-to-action pipeline (PagerDuty webhook → agent trigger), automated RCA generation (failure taxonomy + log correlation), self-healing action executor with approval gate, incident severity classification, runbook automation (structured YAML runbooks executed by agent), escalation logic, incident timeline auto-generation, Slack integration for human-in-the-loop approvals

0 installs
▌
Infra Streaming Reliability Review · ivanshamaev
Streaming pipeline reliability — exactly-once semantics (idempotent producer/transactional API/read_committed/Kafka Streams EOS), at-least-once patterns with idempotent consumers, dead letter queue (DLQ) design with Kafka Connect SMT/custom error handler, poison message handling, reprocessing strategy, watermarks and late data handling, checkpoint/savepoint for Flink/Spark Streaming, consumer group rebalance storm prevention, end-to-end latency SLO monitoring, streaming reliability checklist

0 installs
▌
Infra Kubernetes Autoscaling Review · ivanshamaev
Kubernetes autoscaling review — HPA (CPU/memory/custom/external metrics), stabilizationWindowSeconds anti-flapping, VPA vs HPA decision, Cluster Autoscaler tuning (scale-down delay/utilization threshold), KEDA event-driven autoscaling (Kafka lag/queue depth), spot node optimization, pod disruption budgets, scaling behavior policies (Percent/Pods/Max/Min selectPolicy)

0 installs
▌
Infra Opentelemetry Instrumentation · ivanshamaev
OpenTelemetry instrumentation for data platforms — Python auto-instrumentation (opentelemetry-bootstrap), manual span creation for ETL tasks, trace context propagation across Airflow→Spark→dbt, OTLP exporter configuration, W3C TraceContext headers in Kafka messages, resource attributes (service.name/environment/version), Baggage for partition metadata propagation, Collector pipeline (tail sampling), correlation between traces/logs/metrics, Grafana Tempo integration

0 installs
▌
De Cost Optimization · ivanshamaev
DE cost optimization — query cost analysis (Trino/Spark/ClickHouse/BigQuery), compute right-sizing, storage tier recommendations, spot/preemptible instances, partition pruning, Z-order/clustering, materialized view economics, data lifecycle policies, FinOps tagging, cost attribution, budget alerts

0 installs
▌
Kimball Data Modeling · ivanshamaev
Use when designing or implementing dimensional data models following Kimball methodology — fact tables (transaction/snapshot/accumulating), dimension tables, SCD types (1/2/3/4/6), star/snowflake schemas, surrogate keys, conformed dimensions, role-playing dimensions, junk dimensions, bridge tables, date dimensions, degenerate dimensions, DDL patterns, and DML update strategies for loading and maintaining dimensional tables.

0 installs
▌
Github Actions Dataops · ivanshamaev
Use when building or reviewing GitHub Actions CI/CD pipelines for data engineering — covering dbt slim CI (state:modified+, --defer, manifest.json), SQLFluff SQL linting with PR annotations, Airflow DAG integrity tests (pytest/DagBag), Great Expectations and Soda data quality gates, multi-stage Docker image builds for Spark/dbt/Airflow pushed to ghcr.io, secrets management with OIDC for AWS/GCP (no static keys), environment-scoped secrets, reusable workflows (workflow_call), matrix builds, composite actions, and full end-to-end dbt project CI/CD workflows.

0 installs
▌
Medallion Architecture · ivanshamaev
Use when designing or implementing Medallion (Bronze/Silver/Gold) data lakehouse architecture — layer definitions, DDL for each layer, DML load and update patterns, incremental pipeline design, deduplication strategies (row_number/merge/hash/CDC/watermark), schema evolution, data quality gates, partitioning per layer, late-arriving data, orchestration patterns, and best practices for Iceberg-based lakehouses.

0 installs
▌
De Production Readiness · ivanshamaev
Data engineering production readiness — idempotency patterns (INSERT ON CONFLICT, MERGE, partition overwrite), retry strategies (exponential backoff, Airflow retry config), SLA monitoring (Airflow sla_miss_callback, Prometheus PromQL), data freshness checks (Soda freshness, max event_time), structured logging (JSON logs, OpenTelemetry tracing), data reconciliation (row count, checksum, duplicate detection), graceful degradation (circuit breaker, fallback, skip failed partitions), change management (blue/green table swap, schema migration rollback)

0 installs
▌
Vertica Query Optimization · ivanshamaev
Use when optimizing, diagnosing, or reviewing Vertica 11.x SQL query performance — covering EXPLAIN plan reading, projection design for predicates/joins/GROUP BY/ORDER BY/analytic functions, segmentation strategies, column encoding, RLE, sort elimination, Top-K, INSERT-SELECT tuning, DELETE/UPDATE internals, and Data Collector diagnostics.

0 installs
▌
Starrocks Cbo · ivanshamaev
StarRocks cost-based optimizer — ANALYZE TABLE (full/sample/histogram/predicate columns), AUTO ANALYZE configuration, statistics storage (_statistics_ database), SHOW ANALYZE STATUS, stale statistics detection, cardinality estimation failures, CBO join hints (LEADING/broadcast/shuffle), EXPLAIN COSTS reading, multi-column statistics (3.5+)

0 installs
▌
Trino Dbt Platform · ivanshamaev
Production dbt + Trino platform — dbt-trino adapter profiles.yml (ldap/kerberos/oauth/jwt/certificate auth), all materializations (table/view/incremental/materialized_view/ephemeral), incremental strategies (append/merge/delete+insert), Iceberg table_properties in config blocks (format/partitioning/sorted_by/location), on_schema_change, snapshots with TIMESTAMP(6), slim CI with state:modified+ and --defer, dbt project structure (staging/intermediate/mart), ANALYZE post-hook, session_properties in profiles, multi-thread parallelism

0 installs
▌
Starrocks Bucketing · ivanshamaev
StarRocks bucketing (distribution) — DISTRIBUTED BY HASH vs RANDOM, bucket count selection formula, distribution key selection (high cardinality/join key), skew detection and prevention, colocate join setup (colocate_with property), bucket count modification (ALTER TABLE), auto bucket sizing (StarRocks 3.2+), tablet sizing target (1-10 GB)

0 installs
▌
Trino Cost Optimization · ivanshamaev
Trino warehouse cost optimization — query scan cost analysis (system.runtime.queries/system.runtime.tasks), identifying expensive queries by CPU time and data scanned, scan reduction via partition pruning and file compaction, worker autoscaling patterns (scale-to-zero for batch), spot instance strategies for workers, S3 object storage cost (storage vs request costs), Iceberg compaction economics (fewer splits = fewer S3 GET requests), cost attribution by team/user, materialized view break-even analysis, result caching

0 installs
▌
Infra Rbac Audit · ivanshamaev
RBAC audit for data platforms — Kubernetes RBAC (ClusterRole/Role/Binding audit), AWS IAM permission boundary and policy review (AWS IAM Access Analyzer), GCP IAM audit (principle of least privilege/owner role detection), database RBAC (PostgreSQL/Trino/ClickHouse role hierarchies), Airflow RBAC (role-based menu/DAG access), dbt permissions (warehouse roles), service account proliferation detection, wildcard permission detection, privilege escalation paths

0 installs
▌
Dbt Starrocks Models · ivanshamaev
dbt + StarRocks models — dbt-starrocks adapter setup (profiles.yml), all materializations (table/view/incremental/ephemeral), incremental strategies (append/insert_overwrite/unique_key merge), StarRocks-specific model config (engine/keys/partition_by/distributed_by/properties), Duplicate/Aggregate/Primary Key table DDL from dbt, partition_by with date_trunc, on_schema_change behavior

0 installs
▌
Trino Query Optimization · ivanshamaev
Trino distributed SQL query optimization — predicate/projection/aggregation pushdown, join reordering (AUTOMATIC/ELIMINATE_CROSS_JOINS), broadcast vs partitioned joins, dynamic filtering, CBO with ANALYZE, filter-early patterns, partition pruning, avoiding SELECT *, reducing shuffle, cross-catalog query cost, session property tuning, query hints, anti-patterns for slow Trino queries

0 installs
▌
Dbt Starrocks Testing · ivanshamaev
dbt + StarRocks testing — generic tests (not_null/unique/accepted_values/relationships), singular tests (custom SQL assertions), source freshness tests (loaded_at_field), StarRocks-specific volume and freshness tests, dbt-expectations integration, test severity (warn vs error), store_failures for debugging failed tests, ANALYZE before test runs, partition-scoped test SQL

0 installs
▌
Dataops Airflow Production Readiness · ivanshamaev
Airflow production readiness — idempotent tasks (UPSERT/partition overwrite), no top-level code at DAG parse time (Variable.get inside tasks), KubernetesExecutor configuration, connection/variable management (no hardcoded secrets), SLA callbacks, retry with exponential backoff, pool management, max_active_runs, dag_concurrency, celery vs kubernetes executor trade-offs, metadata DB maintenance (airflow db clean), health check endpoints, structured logging

0 installs
▌
Dataops Workflow Orchestration Review · ivanshamaev
Workflow orchestration comparison and review — Airflow vs Prefect vs Dagster vs Temporal (architecture, execution model, dependency management, observability), when to choose each, migration patterns from Airflow to Prefect/Dagster, orchestrator evaluation scorecard, event-driven vs schedule-based triggering, Software-Defined Assets (Dagster) vs task-based (Airflow), hybrid orchestration patterns

0 installs
▌
Aiops Infrastructure Anomaly Detection · ivanshamaev
AIOps infrastructure anomaly detection — statistical baseline (z-score/IQR/seasonal decomposition), Prometheus anomaly rules (predict_linear/stddev_over_time/MAD), ML-based detection (Isolation Forest/Prophet/LSTM autoencoder on metrics), metric correlation clustering, automated alert threshold tuning from historical data, Kubernetes resource anomaly detection, data pipeline anomaly signals (row count z-score/freshness drift/throughput drop), Python sklearn/Prophet integration, Grafana anomaly panel

0 installs
▌
Platform Engineering Data Platform API · ivanshamaev
Data platform self-service API — FastAPI REST API for platform operations (Kafka topic CRUD/Airflow DAG trigger/dbt job run/Trino query execution), OAuth2+JWT authentication with team-based RBAC, async job tracking with status polling, rate limiting per team, OpenAPI spec generation, Terraform provider for API resources, versioned API (v1/v2), audit logging of all platform operations, SDK generation (Python/TypeScript), Kubernetes deployment with HPA

0 installs
▌
Trino Modern Data Stack Reference Architecture · ivanshamaev
Trino Modern Data Stack end-to-end reference architecture — full docker-compose (Kafka + MinIO + Hive Metastore + Trino + Airflow + dbt + Superset + Prometheus + Grafana), medallion lake DDL and pipeline DAGs, Kafka→Iceberg streaming ingest, dbt project layout with Trino profiles, Superset dataset config, Prometheus alert rules, production Kubernetes Helm values, operational runbook for daily maintenance

0 installs
▌
Platform Engineering Agentic Control Plane · ivanshamaev
Agentic control plane for data platform — MCP (Model Context Protocol) server exposing platform tools to AI agents (FastMCP/Python SDK), natural language platform operations (trigger DAG/create topic/run dbt/query Trino via LLM), Claude-based platform assistant with tool use, multi-agent platform governance (audit agent/cost agent/reliability agent), MCP server deployment (Docker/Kubernetes), tool authorization and audit logging, agentic workflow patterns (plan-execute-verify), platform chat interface

0 installs
▌
Platform Engineering Internal Developer Platform · ivanshamaev
Internal Developer Platform (IDP) for data engineering — Backstage Software Catalog (catalog-info.yaml for pipelines/datasets/services), Software Templates (scaffold new DAG/dbt project/Kafka topic), TechDocs integration, golden path enforcement, Port.io alternative, self-service data pipeline provisioning, platform scorecard (production readiness checks), team ownership model, API gateway for platform services, paved road vs escape hatch pattern

0 installs
▌
Infra Grafana Dashboard Review · ivanshamaev
Grafana dashboard review — panel types (timeseries/stat/gauge/table/heatmap), variable templating (datasource/label_values/query), dashboard linking and drilldown, data platform dashboards (pipeline overview/DAG health/Kafka throughput/Spark performance), alerting from panels, annotation markers for deployments, dashboard-as-code (Grafonnet/Terraform grafana provider), performance optimization (query caching/recording rules), dark/light theme standards

0 installs
▌
Infra Terraform Cost Estimator · ivanshamaev
Terraform cost estimation — Infracost breakdown and diff commands (monthly cost per resource), PR cost comments (before/after change delta), budget thresholds (block PRs over cost limit), usage files for variable quantities (S3 GB/requests), AWS pricing data sources, cost tagging strategy, OpenTofu/Terraform cost attribution, FinOps tagging policy enforcement, multi-environment cost comparison (dev vs prod)

0 installs
▌
Airflow Starrocks Cdc Orchestrator · ivanshamaev
Airflow CDC orchestration for StarRocks — watermark-based incremental sync, Routine Load health check DAG, Flink job submission via REST API, offset lag alerting, dead letter queue reprocessing, multi-table incremental sync with dependency graph, schema change detection and pipeline pause/resume

0 installs
▌
Dbt Starrocks Production Readiness · ivanshamaev
dbt + StarRocks production readiness — slim CI with state:modified+, manifest.json artifact storage and --defer for dev, dbt Cloud vs dbt Core deployment, RBAC for dbt users in StarRocks, secrets management for profiles.yml, dbt run/test exit codes in CI, breaking change detection, scheduled dbt runs via Airflow/dbt Cloud, dbt docs generation and hosting, model versioning

0 installs
▌
Infra Kubernetes Cluster Health · ivanshamaev
Kubernetes cluster health assessment — node status and pressure conditions (disk/memory/PID), pod failure diagnosis (CrashLoopBackOff/OOMKilled/Pending/Evicted), control plane health (API server/etcd/scheduler/controller-manager), resource quota utilization, scheduling failures (taints/affinity/insufficient resources), kubectl diagnostic commands, node eviction policies, kubelet troubleshooting

0 installs
▌
Infra Kubernetes Cost Optimizer · ivanshamaev
Kubernetes cost optimization — idle workload detection (zero-replica deployments, always-pending jobs), overprovisioning analysis (request vs actual CPU/memory), namespace-level cost attribution, node bin-packing via requests tuning, spot instance strategy, LimitRange defaults, right-sizing with VPA recommendations, Goldilocks tool, resource efficiency metrics, namespace resource quotas for chargeback

0 installs
▌
Infra Kubernetes Security Audit · ivanshamaev
Kubernetes security audit — RBAC review (ClusterRoleBindings/RoleBindings/ServiceAccount permissions), Pod Security Standards (privileged/hostPID/hostNetwork/runAsRoot), NetworkPolicy enforcement, Secret management (encryption at rest, external-secrets), image vulnerability scanning (Trivy/Grype), admission controllers (OPA Gatekeeper/Kyverno), audit logging, CIS Kubernetes Benchmark, supply chain security (SBOM/image signing)

0 installs
▌
Infra Kubernetes Storage Review · ivanshamaev
Kubernetes storage review — PVC lifecycle (provisioning/binding/reclaim), StorageClass selection (SSD/HDD/NVMe), dynamic vs static provisioning, StatefulSet volume management, volume performance tuning (ReadWriteMany vs ReadWriteOnce), CSI drivers (AWS EBS/GCP PD/Ceph/Longhorn), PVC resize, snapshot and backup, storage class access modes, orphaned PVs detection, shared storage patterns (NFS/CephFS)

0 installs
▌
Infra Secrets Management Review · ivanshamaev
Secrets management review — HashiCorp Vault (KV v2/dynamic credentials/lease renewal/audit log), External Secrets Operator (Vault/AWS SSM/GCP Secret Manager → K8s secrets), secret rotation strategies (database dynamic credentials/TLS cert-manager), detecting secrets in code (gitleaks/truffleHog/Semgrep), Airflow connections from Vault backend, dbt profiles without hardcoded credentials, CI/CD secrets (GitHub OIDC/GitLab CI variables), no plaintext secrets in logs/configs

0 installs
▌
Dataops Disaster Recovery Review · ivanshamaev
Disaster recovery review for data platforms — RTO/RPO definitions, DR strategy selection (cold/warm/hot standby), Airflow metadata DB backup and restore (pg_dump/WAL), Kafka topic replication (MirrorMaker2), data lake replication (S3 CRR/GCS Transfer), Kubernetes cluster backup (Velero), runbook for DR failover, DR testing (game day exercises), backup validation, cross-region Terraform, recovery time measurement

0 installs
▌
Dataops Github Actions Optimizer · ivanshamaev
GitHub Actions optimization for DataOps — path filters to skip unchanged components, concurrency groups (cancel-in-progress), job parallelization, caching strategies (pip/npm/Maven/Docker layers), self-hosted runners for heavy workloads, matrix strategy for multi-environment tests, reusable workflows to reduce duplication, OIDC for cloud credentials (no static keys), artifact retention, workflow timing analysis

0 installs
▌
Dataops Release Readiness Review · ivanshamaev
Release readiness review for data platforms — pre-release checklist (schema migrations tested, backward compatibility, rollback plan), runbook completeness, monitoring and alerting configured, on-call handoff, load/soak testing results, data reconciliation between versions, feature flag strategy, go/no-go criteria, post-release validation (15-min/1-hour/24-hour checkpoints), communication templates

0 installs
▌
Dagster Assets · ivanshamaev
Dagster Software-Defined Assets (SDA) — @asset/@multi_asset decorators, asset dependencies, partitions (daily/static/dynamic/multi), sensors, IO managers, declarative automation (AutomationCondition), jobs, schedules, resource definitions, deployment

0 installs
▌
Data Contracts · ivanshamaev
Data Contracts — datacontract.com YAML specification (schema, quality, SLA, ownership, servers, changelog), Data Contract CLI (init/test/lint/diff/breaking/publish/export), SodaCL-style quality checks embedded in contract, breaking change detection (column removal/type change/constraint tightening), CI/CD GitHub Actions PR enforcement, Kafka/S3/database server definitions, contract-driven development (producer-first), semver versioning and deprecation, DataHub and OpenMetadata catalog publishing

0 installs
▌
Terraform Data Infrastructure · ivanshamaev
Terraform for data infrastructure — S3/MinIO data lake buckets (versioning, lifecycle, SSE-KMS), IAM roles for Spark/Airflow (least-privilege, IRSA on EKS), MSK/Kafka clusters (aws_msk_cluster, encryption, custom broker config), Kubernetes data platform (helm_release Airflow + Spark, namespace resource quotas), module layout (modules/ + envs/), typed variables with validation, S3 remote state + DynamoDB locking, Terragrunt DRY configs, GitHub Actions CI/CD pipeline with plan/apply

0 installs
▌
Clickhouse Olap · ivanshamaev
ClickHouse OLAP — MergeTree family engines (MergeTree/ReplacingMergeTree/AggregatingMergeTree/SummingMergeTree/CollapsingMergeTree), ORDER BY/PARTITION BY/PRIMARY KEY design, data skipping indexes (minmax/set/bloom_filter), TTL tiered storage, materialized views, projections, LowCardinality/Nullable/AggregateFunction types, INSERT batching, FINAL keyword, query optimization (PREWHERE/parallel_replicas), dictionaries, Kafka engine, Python clickhouse-connect, best practices

0 installs
▌
Datahub Catalog · ivanshamaev
DataHub data catalog — metadata ingestion recipes (PostgreSQL/Hive/Spark/dbt/Airflow/Kafka/S3), Python SDK (DatahubRestEmitter, MetadataChangeProposalWrapper), column-level lineage (FineGrainedLineage), GMS/MCE/MCP architecture, Kubernetes Helm deployment, CLI operations, search/discovery REST API and GraphQL

0 installs
▌
Kubernetes Data Platform · ivanshamaev
Data platform on Kubernetes — Spark-on-K8s (spark-submit, pod templates, dynamic allocation, RBAC), Airflow on K8s (Helm chart, KubernetesExecutor, KubernetesPodOperator, git-sync, DAG storage), resource management, namespaces, autoscaling

0 installs
▌
Docker Data Environments · ivanshamaev
Docker for data engineering environments — multi-stage Dockerfiles for dbt/Spark/Airflow images, BuildKit layer caching (--mount=type=cache), private registries (ghcr.io/Harbor), docker buildx multi-platform builds, Docker Compose local data stacks (Spark+Airflow+Kafka+MinIO+Postgres), BuildKit secrets for private PyPI, security hardening (non-root user, slim base images, read-only FS), CI/CD with GitHub Actions build-push-action

0 installs
▌
Mlflow Data Pipelines · ivanshamaev
MLflow for data engineering — tracking server setup (PostgreSQL backend + S3 artifacts), experiment tracking (params/metrics/artifacts/autolog), ETL job metadata logging (row counts, DQ metrics, lineage tags), Model Registry (register/alias/promote), model serving (pyfunc/REST API/batch scoring), MLproject files, Airflow integration (MLflowClientHook/XCom/model promotion), Spark integration (mlflow.spark.autolog/log_model/Delta metadata)

0 installs
▌
Prefect Workflows · ivanshamaev
Prefect 3.x workflows — flows, tasks, deployments, work pools, event-driven triggers, caching, retries, state hooks, Prefect Cloud/server, Python SDK, Docker/K8s infrastructure

0 installs
▌
Pyspark Structured Streaming · ivanshamaev
PySpark Structured Streaming — streaming sources (Kafka/file/rate), output modes (append/complete/update), triggers (ProcessingTime/AvailableNow/Once/Continuous), watermarks, event-time windows (tumbling/sliding/session), stateful aggregations, deduplication, foreachBatch, stream-stream joins, checkpointing, fault tolerance, RocksDB state store, Kafka source/sink options, production patterns

0 installs
▌
Great Expectations · ivanshamaev
Great Expectations (GX) — DataContext setup, Data Sources (Pandas/Spark/SQL/file), Expectation Suites, built-in expectations (null/uniqueness/range/set/regex/table-level/statistical), Validation Definitions, Checkpoints, Data Docs, Airflow integration, custom expectations, dbt integration, CI/CD usage, severity levels (warning/critical)

0 installs
▌
Airflow Dag Factory · ivanshamaev bundle
Use when building, reviewing, debugging, or scaling Apache Airflow DAGs declaratively with dag-factory YAML configs — including project setup, loader configuration, defaults hierarchy, custom/provider operators, dynamic task mapping, datasets/assets, callbacks, TaskFlow decorators, Jinja2 templating, environment variables, YAML DRY patterns, large-scale multi-DAG generation, CI/CD linting, and migration from pre-1.0.

0 installs
▌
Dataops Airflow Observability · ivanshamaev
Airflow observability — StatsD/OpenTelemetry metrics (scheduler_heartbeat/task_duration/pool_open_slots/dag_processing_total_parse_time), Prometheus scraping, Grafana dashboards (DAG success rate/task duration/slot utilization), structured task logging (JSON formatter), OpenTelemetry traces for task spans, DAG SLA miss alerts, anomaly detection on task duration, Airflow audit log, DagBag parse error monitoring, dead letter queue for failed tasks

0 installs
▌
Dataops Blue Green Deployment · ivanshamaev
Blue/green deployment for data platforms — Kubernetes blue/green with service selector swap, Argo Rollouts blue-green strategy, database schema migration compatibility (backward-compatible DDL), dbt blue/green schema swap (swap_schema macro), Airflow DAG versioning (dag_id with version suffix), traffic cutover and rollback procedure, smoke tests before cutover, Canary vs blue/green decision guide

0 installs
▌
Dataops Jenkins Modernization · ivanshamaev
Jenkins modernization for DataOps — migration from Freestyle to Declarative Pipeline, Jenkinsfile best practices, shared libraries (vars/src structure), parallel stages, Kubernetes agent pods (docker-in-docker vs kaniko), credential management (withCredentials/Jenkins Credential Store), pipeline-as-code with Multibranch Pipeline, Blue Ocean UI, migration path to GitHub Actions, JCasC (Jenkins Configuration as Code)

0 installs
▌
Dataops Self Healing Platform · ivanshamaev
Self-healing data platform — auto-restart failed Airflow DAGs (idempotency required), Kafka consumer group auto-resume after partition rebalance, circuit breaker for flapping pipelines, automatic partition backfill on gap detection, auto-scale Kubernetes resources on OOM detection, stale statistics auto-ANALYZE, DQ gate auto-quarantine pattern, dead letter queue reprocessing, watchdog agents for heartbeat monitoring, self-healing Airflow sensor (RoutineLoadLagSensor pattern)

0 installs
▌
Infra Alert Fatigue Reduction · ivanshamaev
Alert fatigue reduction — SLO-based alerting (burn rate vs threshold), multi-window burn rate alerts (short/long window), AlertManager inhibition and silencing, alert deduplication (group_by), routing by severity (PagerDuty for critical / Slack for warning), alert ownership labels, runbook links in annotations, alert review process (weekly noise audit), flapping detection (for/pending period tuning), absent() vs rate() alerting patterns, ticket-based escalation for warning-level noise

0 installs
▌
Infra Network Security Review · ivanshamaev
Network security review for data platforms — Kubernetes NetworkPolicy (default-deny + allow patterns), VPC/subnet design (private subnets for all data services), security group rules audit (0.0.0.0/0 detection), TLS everywhere (Kafka mTLS/Trino HTTPS/DB SSL), service mesh (Istio mTLS), DNS security (private hosted zones), egress filtering (no unrestricted outbound), VPC peering vs PrivateLink, network flow logs analysis, firewall rules review

0 installs
▌
Infra Prometheus Optimization · ivanshamaev
Prometheus optimization — recording rules (level:metric:operations naming), cardinality explosion diagnosis (tsdb status API), scrape interval tuning, remote write configuration (batch size/queue capacity), Thanos/Mimir for long-term retention, federation vs remote write, AlertManager routing trees (inhibition/silencing), absent() alerts for missing series, relabeling to drop expensive labels, TSDB compaction, ServiceMonitor vs PodMonitor patterns

0 installs
▌
Infra Terraform Security Scan · ivanshamaev
Terraform security scanning — tfsec static analysis (AWS/GCP/Azure misconfigurations), Checkov IaC policy checks (750+ rules, CIS benchmarks), tflint security rules, KICS scanner, S3 encryption/public access/versioning checks, IAM least-privilege patterns, security group rule review (0.0.0.0/0), KMS key rotation, VPC flow logs, CloudTrail enabled, pre-commit hooks, SARIF output for GitHub Security tab, policy-as-code with OPA/Sentinel

0 installs
▌
Dataops Airflow Cost Optimizer · ivanshamaev
Airflow cost optimization — KubernetesPodOperator right-sizing (request vs actual CPU/memory), spot node tolerations for batch tasks, task consolidation (reduce pod-per-task overhead), pool-based concurrency control, idle worker cleanup, log retention policy (S3 lifecycle), metadata DB right-sizing, CeleryExecutor worker autoscaling (KEDA Kafka/queue depth), DAG run frequency analysis (oversceduled DAGs), cloud cost attribution per DAG

0 installs
▌
Infra AWS Data Platform Review · ivanshamaev
AWS data platform review — S3 data lake (lifecycle/replication/encryption/access points), EMR vs Glue ETL trade-offs, MSK Kafka configuration, RDS/Aurora for metadata, Redshift vs Athena for analytics, MWAA (managed Airflow), EKS for containerized pipelines, IAM roles (IRSA for EKS/EMREC2 instance profiles), Lake Formation row/column security, AWS Glue catalog, VPC data platform networking, cost optimization (S3 Intelligent-Tiering/Spot instances)

0 installs
▌
Infra GCP Data Platform Review · ivanshamaev
GCP data platform review — BigQuery (dataset IAM/column-level security/partitioned tables/clustering/reservations), GCS data lake (uniform bucket-level access/lifecycle/CMEK), Dataproc vs Dataflow vs Spark on GKE, Pub/Sub streaming, Cloud Composer (managed Airflow), Workload Identity for GKE, VPC Service Controls, Data Catalog, Dataplex for data governance, BigQuery cost optimization (slot reservations vs on-demand), Cloud Logging/Monitoring

0 installs
▌
Infra Gitops Deployment Review · ivanshamaev
GitOps deployment review — ArgoCD Application/AppProject CRDs, app-of-apps pattern, sync policies (automated+self-heal+prune), drift detection and remediation, FluxCD HelmRelease/Kustomization, git repository structure for GitOps (app code vs config repo separation), image update automation, progressive delivery (Argo Rollouts canary/blue-green), GitOps for data platform components (Airflow/Kafka/Spark)

0 installs
▌
MCP Server · ivanshamaev
MCP (Model Context Protocol) server development — FastMCP/Python SDK, tools/resources/prompts primitives, STDIO and Streamable HTTP transports, Claude Desktop/Claude Code client config, MCP Inspector testing, security best practices (input validation/OAuth2/confused deputy prevention), production Docker/Kubernetes deployment, agentic data platform integration

0 installs
▌
Openlineage · ivanshamaev
OpenLineage data lineage tracking — RunEvent/Job/Dataset/facet spec, Marquez backend setup, Airflow/Spark/dbt integrations, column-level lineage, custom emitters, lineage-based impact analysis

0 installs
▌
Pyspark Etl · ivanshamaev
Use when designing, implementing, reviewing, or optimizing production PySpark ETL/DataFrame pipelines at GB-TB+ scale, including schemas, joins, partitioning, window functions, writes, UDF avoidance, and Spark performance diagnostics.

0 installs
▌
Airflow Dags · ivanshamaev
Apache Airflow DAG authoring — DAG definition, TaskFlow API (@task/@dag decorators), operators (Bash/Python/SQL/HTTP), sensors (poke/reschedule modes), TaskGroups, dynamic task mapping (expand/partial), branching, trigger rules, XComs, Pools, callbacks, cross-DAG pipelines, best practices

0 installs
▌
Apache Flink · ivanshamaev
Apache Flink — streaming architecture (JobManager/TaskManager/slots/parallelism), PyFlink Table API and SQL (Kafka/filesystem DDL, windowing TUMBLE/HOP/SESSION, aggregations), DataStream API (KeyedStream, stateful operations ValueState/MapState, process functions), event-time vs processing-time, watermark strategies, checkpointing (exactly-once, RocksDB state backend), Kafka source/sink (delivery guarantees), backpressure, savepoints, deployment (standalone/YARN/Kubernetes)

0 installs
▌
Apache Kafka · ivanshamaev
Apache Kafka — topics, partitions, consumer groups, producer/consumer configuration (acks, idempotence, compression, commit strategies), exactly-once semantics, Kafka Connect (source/sink connectors, SMTs, DLQ), Schema Registry, consumer lag monitoring, CLI operations, Python confluent-kafka examples, partition strategy, replication, retention

0 installs
▌
Cdc Debezium · ivanshamaev
Change Data Capture with Debezium — PostgreSQL/MySQL/Oracle CDC connectors, change event structure (before/after/op/source), snapshot modes, Kafka Connect deployment, SMT transformations, outbox pattern, Iceberg/Delta sink integration, idempotency guarantees

0 installs
▌
Data Vault 2 · ivanshamaev
Use when designing or implementing Data Vault 2.0 data models — Hubs, Links, Satellites, Reference Tables, Same-As Links, Multi-Active Satellites, Effectivity Satellites, Point-in-Time tables, Bridge tables, Business Vault patterns, hash key generation, DDL for all entity types, insert-only DML load patterns, pipeline architecture, and constructing Information Marts from the vault.

0 installs
▌
RAG Data Pipeline · ivanshamaev
Use when designing, building, or debugging a RAG (Retrieval-Augmented Generation) data pipeline — document ingestion, chunking strategies (fixed/recursive/semantic/structure-aware), embedding models (OpenAI/Cohere/sentence-transformers), vector stores (pgvector/Chroma/Qdrant/Weaviate), incremental refresh, hybrid retrieval (dense + BM25 + RRF), re-ranking with cross-encoders, metadata filtering, and production monitoring.

0 installs
▌
De Postmortem Writer · ivanshamaev
DE blameless postmortem — incident timeline reconstruction, impact quantification, root cause chain (5 Whys), corrective actions (CAPA), postmortem document template, severity classification, SLA breach reporting, communication templates, lessons-learned facilitation, postmortem review checklist

0 installs
▌
Feature Store · ivanshamaev
Feast feature store — feature repo layout, Entity/FeatureView/FeatureService definitions, offline store (BigQuery/Redshift/Snowflake/file), online store (Redis/DynamoDB/SQLite), point-in-time correct joins, feast materialize, feast apply, Python SDK get_online_features/get_historical_features, push sources, streaming feature views, Airflow integration

0 installs
▌
Postgresql Data Engineering · ivanshamaev
PostgreSQL for data engineering — declarative partitioning (RANGE/LIST/HASH/pg_partman), index types (B-Tree/BRIN/GIN/GIST/partial/covering), COPY bulk load, EXPLAIN ANALYZE plan reading, autovacuum tuning, window functions, JSONB, CTEs, LATERAL joins, and bulk-load patterns (UNLOGGED/pg_bulkload)

0 installs
▌
Trino Iceberg · ivanshamaev
Use when writing, optimizing, or maintaining Apache Iceberg tables with Trino — covering table DDL, all ALTER TABLE operations, partition transforms, bucketing, sorted tables, DML (INSERT/UPDATE/DELETE/MERGE), EXPLAIN plan reading, join optimization, statistics with ANALYZE, table maintenance (optimize/expire_snapshots/remove_orphan_files), schema evolution, time travel, and metadata table diagnostics.

0 installs
▌
Aiops Query Cost Analyzer · ivanshamaev
AIOps autonomous query cost analyzer — Trino system.runtime.query_history cost SQL (CPU time/peak memory/bytes scanned), ClickHouse system.query_log top expensive queries, Spark UI cost attribution by user/job/DAG, BigQuery INFORMATION_SCHEMA.JOBS cost analysis, LLM-driven query rewrite recommendations (missing partition filters/broadcast hints/aggregation pushdown), cost anomaly detection (user over daily budget), automated query tagging, chargeback report by team/project

0 installs
▌
Dataops Airflow Ha Review · ivanshamaev
Airflow high availability review — multi-scheduler setup (HA schedulers with PostgreSQL row-level locking), CeleryExecutor worker pool sizing, KubernetesExecutor pod limits, metadata DB HA (PostgreSQL HA with PgBouncer connection pooling), DAG sync strategy (git-sync sidecar/S3/GitDagBundle), log storage (S3/GCS), webserver multi-replica, triggerer HA, rolling upgrades without downtime, Celery Flower monitoring, heartbeat alerts

0 installs
▌
Starrocks Admin Query Monitor · ivanshamaev
StarRocks query monitoring and resource management — slow query log, SHOW PROCESSLIST, information_schema.query_history, resource groups (CREATE RESOURCE GROUP), classifiers, CPU/memory/concurrency limits, query queuing, KILL QUERY, workload isolation, Audit Loader plugin

0 installs
▌
Trino Airflow Lakehouse Pipelines · ivanshamaev
Airflow-orchestrated Iceberg lakehouse ETL pipelines — Bronze/Silver/Gold layer DAGs, Iceberg maintenance jobs (optimize/expire_snapshots/remove_orphan_files) as Airflow tasks, snapshot expiration scheduling, compaction DAG patterns, partition-by-partition backfill with dynamic task mapping, late-arriving data handling, watermark tracking table, post-load data quality gates (row count/freshness/null rate checks via TrinoHook), full medallion pipeline DAG example

0 installs
▌
Trino Production Readiness Review · ivanshamaev
Trino production readiness checklist and review — coordinator HA (active/passive with load balancer), worker autoscaling, TLS/HTTPS configuration, authentication setup, resource groups for multi-tenancy, JVM sizing, spill configuration, fault-tolerant execution for batch workloads, monitoring (Prometheus/alerting), graceful shutdown, backup strategy for Hive Metastore, catalog security, query history retention, log rotation, Kubernetes deployment checklist

0 installs
▌
Infra Compliance Readiness · ivanshamaev
Compliance readiness for data platforms — SOC2 Type II controls mapping (CC6/CC7/CC8), GDPR data processing requirements (PII inventory/right-to-erasure/data residency), PCI-DSS data platform scope reduction, CIS Benchmarks (Kubernetes/AWS/GCP), audit log completeness (Vault/Kubernetes/cloud trail), data retention and deletion policies, encryption at rest/transit evidence, access review processes, compliance-as-code (OPA policies), vulnerability management (CVE SLA)

0 installs
▌
Infra Kafka Cost Optimizer · ivanshamaev
Kafka cost optimization — tiered storage (S3/GCS remote log offload, local.retention.ms vs retention.ms), topic retention right-sizing (per-topic audit SQL), log compaction economics, partition count right-sizing (over-partitioned cluster detection), consumer group idle cleanup, broker instance right-sizing (disk vs compute), Redpanda vs Confluent Cloud cost comparison, MirrorMaker2 cross-region cost, compression savings calculator

0 installs
▌
Airflow Starrocks Data Quality · ivanshamaev
Airflow + StarRocks data quality gates — post-load row count validation, freshness checks (MAX(updated_at) vs expected), null explosion detection, duplicate key audit on Primary Key tables, volume anomaly detection (z-score vs 7-day avg), SODA-style SQL quality checks embedded in DAG, quarantine pattern for bad partitions

0 installs
▌
Starrocks Admin Backup Restore · ivanshamaev
StarRocks backup and restore — CREATE REPOSITORY (S3/HDFS/NFS), BACKUP DATABASE/TABLE, SHOW BACKUP/RESTORE status, RESTORE with partition granularity, cross-cluster restore, incremental backup strategy, snapshot lifecycle, disaster recovery RTO/RPO targets

0 installs
▌
Starrocks Admin Cluster Health · ivanshamaev
StarRocks cluster health monitoring — FE/BE/CN node status, tablet health, replica status, compaction backlog, storage imbalance, SHOW commands, system tables (information_schema/sys), alert thresholds, FE quorum checks, BE disk usage, dead replica detection, Prometheus metrics

0 installs
▌
Trino Federated Query Architecture · ivanshamaev
Trino federated query architecture across heterogeneous sources — cross-catalog JOIN patterns (Iceberg+PostgreSQL+MySQL+Kafka+ClickHouse), pushdown behavior per connector type, minimizing cross-catalog data movement, materializing JDBC data into Iceberg, query routing strategy, connector-specific limitations (JDBC serial fetch, Kafka read-once), performance cost model for federated joins, metadata caching, CREATE TABLE AS SELECT federation patterns, catalog isolation design

0 installs
▌
Trino Lakehouse Platform Architect · ivanshamaev
Trino-based Modern Data Platform architecture design — decoupled storage/compute, Iceberg as open table format, Hive Metastore/Glue catalog, MinIO/S3 object storage, Kafka ingestion layer, dbt transformation, Airflow orchestration, Superset BI — coordinator/worker topology, catalog design, metadata-driven architecture, multi-layer lakehouse (Bronze/Silver/Gold), federated query across heterogeneous sources

0 installs

159 published skills · page 1 of 2