all publishers

ivanshamaev

@ivanshamaev source repo

159 published skills · page 2 of 2

  1. ▌
    Mage AI Pipelines · ivanshamaev
    Mage AI data pipelines — block types (data loader/transformer/exporter/sensor/custom), pipeline YAML, hybrid SQL+Python blocks, triggers (schedule/event/API), streaming pipelines, dbt integration, Spark integration, Docker/Kubernetes deployment, io_config.yaml, pipeline variables, callbacks, backfills
    24 repo stars
  2. ▌
    Sqlmesh · ivanshamaev
    SQLMesh data transformation framework — model kinds (FULL/INCREMENTAL_BY_TIME_RANGE/INCREMENTAL_BY_UNIQUE_KEY/SCD_TYPE_2/VIEW/SEED), plan/apply workflow, virtual environments, state-aware deploys, audits, unit tests, CI/CD, dbt migration
    24 repo stars
  3. ▌
    Vertica · ivanshamaev
    Use when writing, reviewing, debugging, or optimizing SQL for Vertica — covering DDL (CREATE/ALTER/DROP TABLE, columns, projections, segmentation, partitions), DML (INSERT, UPDATE, DELETE, MERGE, TRUNCATE, COPY), CRUD patterns, and Vertica-specific performance guidance including encoding, segmentation keys, partition pruning, and query optimization.
    24 repo stars
  4. ▌
    Dbt Core · ivanshamaev
    dbt Core (adapter-agnostic) — project structure, models (SQL/Python), sources and refs, materializations (table/view/incremental/ephemeral), incremental strategies (append/merge/delete+insert), is_incremental() macro, snapshots (SCD Type 2), seeds, tests (generic/singular/severity), Jinja macros, dbt-utils package, node selection syntax (graph operators), hooks, exposures, metrics (MetricFlow), CI/CD (slim CI with state:modified+), dbt_project.yml, profiles.yml, multi-adapter support (Spark/PostgreSQL/ClickHouse/BigQuery)
    24 repo stars
  5. ▌
    Ray Data · ivanshamaev
    Ray Data distributed data processing — Dataset API, read_parquet/read_csv/read_json/read_delta_sharing, map/filter/flat_map/map_batches, groupby/aggregations, Actors for stateful transforms, Ray remote functions, write_parquet/write_iceberg, streaming execution, GPU batch inference, integration with Spark and Airflow
    24 repo stars
  6. ▌
    Redpanda · ivanshamaev
    Redpanda Kafka-compatible streaming — cluster setup, topic config, rpk CLI, producer/consumer tuning, tiered storage, Schema Registry, Kafka Connect compatibility, monitoring, Docker/Kubernetes deployment
    24 repo stars
  7. ▌
    Sqlfluff · ivanshamaev
    SQLFluff SQL linter — .sqlfluff config, dialect selection (ansi/bigquery/clickhouse/duckdb/hive/postgres/snowflake/sparksql/trino), rule sets, templating (Jinja/dbt), fix command, VS Code integration, pre-commit hook, CI/CD GitHub Actions, custom rules, noqa inline suppression
    24 repo stars
  8. ▌
    Dbt Trino · ivanshamaev
    Use when writing, configuring, or optimizing dbt projects targeting Trino or Starburst — covering profiles.yml setup, all authentication methods, materializations (table/view/incremental/materialized_view/ephemeral), incremental strategies (append/merge/delete+insert), table properties (format/partitioning/sorted_by), on_schema_change, seeds, snapshots, grants, session properties, data modeling patterns (Kimball/staging/intermediate/mart), dbt project structure, tests, and CI/CD.
    24 repo stars
  9. ▌
    Soda Core · ivanshamaev
    Soda Core data quality — SodaCL checks (row_count, missing, invalid, duplicate, freshness, schema, reference, custom SQL), configuration.yml for PostgreSQL/Spark/ClickHouse/BigQuery, soda scan CLI, Airflow integration, dbt integration, alerting
    24 repo stars
  10. ▌
    Spark SQL · ivanshamaev
    Use when writing, reviewing, debugging, or optimizing production Spark SQL for Hive/lakehouse/HDFS tables, including CTE-heavy queries, joins, windows, partition pruning, Hive Metastore operations, insert/overwrite safety, query hints, statistics, EXPLAIN plans, AQE, skew, materialization, and SQL performance diagnostics.
    24 repo stars
  11. ▌
    Dbt Macros · ivanshamaev
    dbt Jinja macros — macro authoring fundamentals, Jinja syntax (blocks/filters/tests), context variables (this/target/adapter/execute), run_query, adapter.dispatch for cross-database macros, generate_schema_name, hooks, custom materializations, dbt-utils patterns
    24 repo stars
  12. ▌
    Delta Lake · ivanshamaev
    Delta Lake — table DDL (CREATE/PARTITIONED BY/LOCATION/TBLPROPERTIES), DML (INSERT/UPDATE/DELETE/MERGE), MERGE patterns (upsert/SCD2/CDC), schema evolution (mergeSchema/overwriteSchema/ALTER TABLE), OPTIMIZE compaction, Z-ORDER BY data skipping, VACUUM, Time Travel (VERSION AS OF/TIMESTAMP AS OF), RESTORE, DESCRIBE HISTORY, shallow/deep clone, Change Data Feed, streaming read/write, deletion vectors, table properties
    24 repo stars
  13. ▌
    Aiops Observability Copilot · ivanshamaev
    AIOps observability copilot — natural language to PromQL/LogQL translation, alert explanation in plain English (what fired/why/impact), Grafana dashboard auto-generation from service topology, anomaly narrative generation from metric patterns, on-call context enrichment (related alerts/recent deploys/runbook links), log pattern clustering for noise reduction, SLO status explanation, automated weekly observability health digest
    24 repo stars
  14. ▌
    Dataops Root Cause Analysis · ivanshamaev
    Root cause analysis for DataOps incidents — 5-Why analysis for pipeline failures, failure taxonomy (infrastructure/data/logic/dependency/config/concurrency), Airflow diagnosis (task state SQL/scheduler heartbeat/DagBag errors), Spark diagnosis (OOM/skew/FetchFailed/serialization), Kafka consumer lag spike RCA, data quality anomaly investigation (volume/freshness/distribution shift), log correlation across components, timeline reconstruction, impact quantification SQL
    24 repo stars
  15. ▌
    Infra Docker Best Practices · ivanshamaev
    Docker best practices — multi-stage builds (builder/runtime separation), layer cache optimization (dependency install before source copy), minimal base images (distroless/alpine/slim), security hardening (non-root USER, read-only FS, no SUID binaries), .dockerignore patterns, BuildKit secrets for private registries, image vulnerability scanning (Trivy), COPY vs ADD, CMD vs ENTRYPOINT patterns, data engineering Dockerfiles (dbt/Spark/Python ETL)
    24 repo stars
  16. ▌
    Infra Kafka Platform Review · ivanshamaev
    Kafka production platform review — broker configuration (replication factor/min.insync.replicas/rack awareness/KRaft mode), topic design (partition count formula/compaction/retention), consumer group management (lag monitoring/rebalance tuning/cooperative sticky), producer tuning (acks/idempotence/compression/batching), security (SASL_SSL/ACLs/mTLS), JMX metrics to Prometheus (kafka-exporter/JMX exporter), alert rules (under-replicated partitions/ISR shrink/consumer lag/disk), capacity planning, Strimzi Kubernetes operator
    24 repo stars
  17. ▌
    Starrocks Aggregation Optimizer · ivanshamaev
    StarRocks aggregation optimization — one-phase vs two-phase aggregation, pre-aggregation pushdown to scan, Aggregate Key table automatic merge, GROUP BY on distribution key (local aggregation), COUNT DISTINCT optimization (multi-phase), HLL/BITMAP approximate aggregation, aggregation spill tuning, EXPLAIN aggregation nodes
    24 repo stars
  18. ▌
    Starrocks Concurrency Optimizer · ivanshamaev
    StarRocks high-concurrency optimization — pipeline_dop tuning for BI workloads, resource groups for workload isolation (short_query type), query queue configuration, FE connection pool, tablet scan parallelism, point query optimization (Primary Key table direct lookup), query cache for repeated dashboard queries, BE thread pool tuning
    24 repo stars
  19. ▌
    Starrocks Data Quality Guardian · ivanshamaev
    StarRocks data quality — freshness queries (MAX(updated_at) vs expected SLA), duplicate detection on Primary Key tables, null anomaly SQL (column-level null rate vs baseline), volume drift detection (day-over-day ratio), cross-table referential integrity checks, data completeness for partitions, StarRocks-specific quality checks (tablet health, compaction lag, replication factor), Python DQ scan class
    24 repo stars
  20. ▌
    Starrocks Lakehouse Integration · ivanshamaev
    StarRocks lakehouse integration — Iceberg/Hive/Delta external catalogs (HMS/Glue/REST), cross-catalog INSERT INTO SELECT, partition filter pushdown verification, external table statistics (ANALYZE on Iceberg), writing back to Iceberg from StarRocks (3.1+), Delta Lake catalog setup, Unity Catalog (3.2+), query federation across StarRocks + Iceberg + Hive in single SQL, cache invalidation (REFRESH EXTERNAL TABLE)
    24 repo stars
  21. ▌
    Dataops Cicd Pipeline Review · ivanshamaev
    DataOps CI/CD pipeline review — pipeline stages for data projects (lint/test/build/scan/deploy), trunk-based vs GitFlow branching, environment promotion (dev→staging→prod), artifact versioning (Docker image tags/dbt manifest), deployment gating (DQ checks/smoke tests/manual approval), rollback strategy, pipeline observability (DORA metrics), reusable workflow patterns, monorepo vs polyrepo CI strategy
    24 repo stars
  22. ▌
    Dataops Postmortem Generator · ivanshamaev
    DataOps blameless postmortem generation — severity matrix (SEV1-4), postmortem Markdown template with timeline/impact/root cause/action items, 3 filled-in examples (Kafka rebalance/dbt incremental data loss/ClickHouse schema mutation), CAPA framework (Corrective/Action/Preventive/Action), impact quantification SQL, stakeholder communication templates, 60-min facilitation guide, postmortem review checklist, Git-based postmortem repository management
    24 repo stars
  23. ▌
    Infra Multi Cloud Governance · ivanshamaev
    Multi-cloud governance — cloud-agnostic data platform patterns, federated identity (OIDC/SAML between AWS/GCP/Azure), Terraform multi-cloud modules, cross-cloud data replication (S3↔GCS/Azure), unified cost management (FinOps Foundation framework), cloud-agnostic observability (OpenTelemetry), policy enforcement (OPA Gatekeeper across clouds), disaster recovery cross-cloud, vendor lock-in avoidance (open formats Iceberg/Parquet), centralized secrets management (HashiCorp Vault)
    24 repo stars
  24. ▌
    Starrocks Admin Storage Balancer · ivanshamaev
    StarRocks storage balancing — tablet distribution across BEs, disk skew detection (SHOW BACKENDS disk usage), cluster_balance analysis (SHOW PROC '/cluster_balance'), balance_load_score_threshold tuning, tablet migration monitoring, BE decommission, adding new BE nodes, storage capacity forecasting, imbalance alerts
    24 repo stars
  25. ▌
    Starrocks Medallion Architecture · ivanshamaev
    StarRocks Medallion (Bronze/Silver/Gold) architecture — Bronze raw ingestion (Duplicate Key), Silver cleansed layer (Primary Key upsert/dedup), Gold aggregated layer (Aggregate Key or MV), incremental transform patterns (INSERT OVERWRITE partition, watermark-based), layer DDL patterns, CDC Bronze landing, dbt-compatible layer design, inter-layer INSERT SELECT pipelines
    24 repo stars
  26. ▌
    Aiops Capacity Planning Agent · ivanshamaev
    AIOps capacity planning agent — predictive resource forecasting (Prophet/linear regression on Prometheus metrics), Kubernetes HPA/VPA right-sizing recommendations, node capacity headroom calculator, Kafka partition and broker capacity model, Spark executor sizing from job history, data lake storage growth forecast (S3/GCS), cluster autoscaler simulation, automatic HPA threshold tuning, cost-per-job attribution, capacity planning report generator
    24 repo stars
  27. ▌
    Dataops Sla Monitoring · ivanshamaev
    Data platform SLA monitoring — freshness SLAs (data available by X:XX), completeness SLAs (row count variance < N%), latency SLAs (pipeline completes within N minutes), SLA breach detection (Prometheus/Soda/SQL), SLA burn rate alerting, SLA reporting dashboards, SLA miss root cause linking, cascading SLA dependencies (upstream delay propagation), SLA definition process, error budget tracking, consumer notification on SLA breach
    24 repo stars
  28. ▌
    Infra Terraform Review · ivanshamaev
    Terraform code review — module structure (main/variables/outputs/modules layout), variable validation blocks, type constraints, sensitive variables, locals vs variables, resource naming conventions, provider version pinning, data sources vs resources, module composition patterns, state management (remote backend S3/GCS), workspace strategy, dependency management (depends_on vs implicit), terraform fmt/validate/tflint checks, DRY with Terragrunt
    24 repo stars
  29. ▌
    Airflow Starrocks Backfill · ivanshamaev
    Airflow + StarRocks backfill patterns — historical partition reprocessing, date range loops with INSERT OVERWRITE, Broker Load replay for S3 partitions, safe backfill with max_active_runs=1, partial backfill resume from checkpoint, backfill idempotency via label strategy, clearing downstream tasks before rerun, parallel partition backfill with concurrency limits
    24 repo stars
  30. ▌
    Airflow Starrocks Pipeline · ivanshamaev
    Airflow + StarRocks pipeline orchestration — MySqlHook for DDL/DML, Broker Load trigger + poll pattern, Stream Load via HTTP operator, Routine Load job lifecycle (pause/resume/stop), sensor for load completion, ANALYZE TABLE post-load, partition-aware DAG templating, Airflow variables for StarRocks connection config, XCom label passing
    24 repo stars
  31. ▌
    Starrocks Admin Compaction · ivanshamaev
    StarRocks compaction tuning — base compaction vs cumulative compaction, compaction score analysis, write amplification reduction, BE config parameters (compaction_threads/min_cumulative_compaction_num_singleton_deltas/max_compaction_candidate_num), SHOW PROC '/compactions', manual compaction trigger, Primary Key table compaction, rowset management
    24 repo stars
  32. ▌
    Starrocks Schema Evolution · ivanshamaev
    StarRocks schema evolution — ALTER TABLE ADD/DROP/MODIFY/REORDER COLUMN, column type widening rules, adding columns to Aggregate/Unique/Primary Key tables, schema change types (direct vs linked vs sorted), SHOW ALTER TABLE COLUMN progress, rollback safety, backward-compatible patterns, dbt model evolution, zero-downtime migration checklist
    24 repo stars
  33. ▌
    Trino File Layout Optimization · ivanshamaev
    Trino data file layout optimization for Iceberg — Parquet vs ORC file format selection, target file size tuning (iceberg.target-max-file-size), row group size, Parquet/ORC column encoding choices, Bloom filter indexes, sorted_by for min/max skipping, small file detection via $files metadata table, OPTIMIZE compaction strategies, partition design impact on file count, Z-order equivalent via sorted_by, split sizing and parallelism (iceberg.minimum-assigned-split-weight), write parallelism tuning
    24 repo stars
  34. ▌
    Starrocks Join Optimization · ivanshamaev
    StarRocks join optimization — broadcast join (small dimension to all BEs), shuffle/hash join (repartition both sides), colocate join (no network transfer, same distribution key), bucket shuffle join, nested loop join (cross join/non-equi), join strategy selection rules, runtime filter effect on joins, join hints, EXPLAIN join strategy verification, skew join handling
    24 repo stars
  35. ▌
    Starrocks Realtime Modeling · ivanshamaev
    StarRocks real-time data modeling — Primary Key table streaming upserts (Flink/Kafka Routine Load), mutable dimension tables (partial column UPDATE), real-time pre-aggregation with Aggregate Key + streaming inserts, Routine Load for Kafka CDC, real-time materialized view refresh, low-latency BI patterns, latency vs consistency trade-offs
    24 repo stars
  36. ▌
    Trino Resource Group Governance · ivanshamaev
    Trino resource group workload governance — resource-groups.properties configuration (maxQueued/hardConcurrencyLimit/softMemoryLimit/hardCpuLimit/schedulingPolicy/schedulingWeight), hierarchical group trees, selector rules (user/source/queryType/clientTags/regex), multi-tenant tenant isolation patterns, per-user dynamic sub-groups (${USER} template), scheduling policies (fair/weighted_fair/weighted/query_priority), CPU quota periods, database-backed configuration (MySQL/PostgreSQL), JMX monitoring of group utilization
    24 repo stars
  37. ▌
    Dbt Starrocks Semantic Layer · ivanshamaev
    dbt + StarRocks semantic layer — dbt Semantic Layer (MetricFlow) on StarRocks, metric definitions (simple/ratio/cumulative/derived), dimension definitions, saved queries, semantic model DDL, exposures for BI tool documentation, dbt-metricflow query CLI, linking metrics to StarRocks materialized views for performance, business KPI governance patterns
    24 repo stars
  38. ▌
    Starrocks AI Query Autotuner · ivanshamaev
    StarRocks AI query autotuner — autonomous SQL optimization agent workflow (EXPLAIN COSTS → analyze → recommend), materialized view recommendation from slow query log, index recommendation (bitmap/bloom filter), partition pruning diagnosis, join order hints generation, statistics staleness detection and auto-ANALYZE trigger, slow query pattern classification, query rewrite suggestions
    24 repo stars
  39. ▌
    Starrocks Materialized Views · ivanshamaev
    StarRocks materialized views — synchronous MV (rollup/aggregate acceleration), asynchronous MV (CREATE MATERIALIZED VIEW REFRESH ASYNC/MANUAL/SCHEDULED, partition-aware refresh, query rewrite, nested MVs, MV on Iceberg/Hive external tables), SHOW MATERIALIZED VIEWS, ALTER/REFRESH/DROP MV, transparent query rewrite via EXPLAIN, use_mv hint
    24 repo stars
  40. ▌
    Starrocks Realtime Analytics · ivanshamaev
    StarRocks real-time analytics — Kafka → Routine Load → Primary Key table for sub-second freshness, low-latency BI query patterns, real-time dashboard design (materialized view for pre-aggregation), metric store patterns, window-based freshness for streaming dashboards, colocate join for real-time multi-table queries, resource group isolation for OLAP vs ingestion
    24 repo stars
  41. ▌
    Starrocks Routine Load Kafka · ivanshamaev
    StarRocks Routine Load for Kafka — CREATE ROUTINE LOAD DDL (all PROPERTIES/KAFKA clause parameters), JSON/CSV/Avro format config, desired_concurrent_number tuning, SHOW ROUTINE LOAD columns, PAUSE/RESUME/ALTER/STOP, consumer lag monitoring, error log analysis, exactly-once semantics, Schema Registry for Avro, Kafka SASL/SSL config, idempotent upsert to Primary Key table
    24 repo stars
  42. ▌
    De Architecture Decision · ivanshamaev
    DE architecture decision records (ADR) — ADR template, trade-off analysis framework, technology evaluation criteria, common DE architecture decisions (storage format, orchestrator, streaming vs batch, lakehouse vs warehouse), decision reversibility, stakeholder communication
    24 repo stars
  43. ▌
    De Root Cause Analysis · ivanshamaev
    DE pipeline root cause analysis — failure taxonomy, lineage tracing, upstream/downstream impact, log analysis patterns, Airflow/Spark/dbt failure diagnosis, data quality anomaly RCA, 5-Whys template, incident timeline reconstruction, runbook-style investigation checklists
    24 repo stars
  44. ▌
    Duckdb · ivanshamaev
    DuckDB in-process OLAP analytics — reading Parquet/CSV/JSON/Iceberg/Delta from S3/local, SQL (window functions, PIVOT, ASOF join, QUALIFY), Python API (duckdb.connect, fetchdf, register, UDFs), extensions (httpfs/iceberg/delta/postgres), performance tuning, COPY TO, persistent DB
    24 repo stars
  45. ▌
    Airbyte · ivanshamaev
    Airbyte ELT — source/destination connectors, sync modes (Full Refresh Overwrite/Append, Incremental Append/Deduped), cursor fields, primary keys, catalog/streams, deployment (abctl, Kubernetes Helm, Airbyte Cloud), Connector Builder, Python CDK (HttpStream, IncrementalMixin), normalization (dbt-based, _airbyte_raw_ tables), Airbyte API, Terraform provider, schema evolution, monitoring, Airflow AirbyteTriggerSyncOperator integration
    24 repo stars
  46. ▌
    Starrocks Memory Tuning · ivanshamaev
    StarRocks memory tuning — BE memory architecture (mem_limit, query pool, load pool, compaction pool, metadata cache), query memory limits (query_mem_limit session var), memory spill to disk (spill_mode), OOM analysis (be.out log), mem_tracker hierarchy, FE heap tuning (JAVA_OPTS), Primary Key table persistent index memory, jemalloc tuning
    24 repo stars
  47. ▌
    Trino Airflow Orchestration · ivanshamaev
    Airflow orchestration for Trino pipelines — TrinoOperator and TrinoHook usage, trino_conn_id connection setup, idempotent DAG patterns (INSERT INTO / MERGE with deterministic run_ids), partition-aware scheduling with Airflow logical_date, sensor patterns (TrinoCheckOperator), SLA monitoring, retry configuration for Trino queries, incremental load orchestration, dbt + Trino Airflow integration (BashOperator/DbtRunOperator), metadata-driven DAG generation
    24 repo stars
  48. ▌
    Trino Dbt Query Performance · ivanshamaev
    Optimizing dbt-generated SQL performance on Trino — ephemeral vs view vs table materialization trade-offs, CTE explosion patterns, partition-aware incremental filters (bounded watermarks), MERGE vs delete+insert strategy selection, avoiding full-refresh anti-patterns, query hints in dbt SQL (BROADCAST hint), dbt thread tuning, session_properties for dbt runs, ANALYZE post-hooks, incremental model design patterns for Iceberg, avoiding small file proliferation from frequent incremental runs
    24 repo stars
  49. ▌
    Trino Self Healing Platform · ivanshamaev
    Autonomous self-healing Trino platform — Python watchdog agents for automatic worker restart detection, hung query killer, OOM-killed query resubmission, Iceberg compaction auto-trigger on small-file detection, stale statistics auto-ANALYZE, cluster memory pressure relief (kill low-priority queries), anomaly detection on query latency (Z-score), Claude-based RCA generation for incidents, Prometheus AlertManager webhook integration, Airflow self-healing sensor
    24 repo stars
  50. ▌
    Starrocks Admin Security · ivanshamaev
    StarRocks security and RBAC — CREATE USER/ROLE, GRANT/REVOKE privileges (catalog/database/table/view/MV/function/resource group level), built-in roles (cluster_admin/db_admin/user_admin/public), LDAP/LDAP group auth, row-level security policies, column masking, SSL/TLS, audit log, privilege inheritance, security best practices
    24 repo stars
  51. ▌
    Trino Iceberg Best Practices · ivanshamaev
    Production Apache Iceberg best practices with Trino — hidden partitioning (day/month/year/hour/bucket/truncate transforms), partition evolution, sorted tables, file compaction (optimize/expire_snapshots/remove_orphan_files), snapshot management, metadata tables ($snapshots/$files/$partitions/$history), schema evolution, time travel queries, MERGE/UPDATE/DELETE DML, ANALYZE for CBO, Iceberg catalog types (HMS/Glue/REST/Nessie), small file prevention, Bloom filters
    24 repo stars
  52. ▌
    Trino Observability Platform · ivanshamaev
    Trino observability and monitoring platform — JMX Prometheus exporter configuration (running queries/failed queries/OOM kills/execution latency P50/P90/P99/memory pool metrics), Grafana dashboard panels, OpenTelemetry trace propagation, query-level event listener for structured logging, Prometheus alert rules (worker loss/queue depth/OOM/failure rate/p99 latency), log aggregation patterns, query history analysis via REST API, slow query detection SQL
    24 repo stars
  53. ▌
    Dbt Starrocks Performance · ivanshamaev
    dbt + StarRocks performance tuning — model DAG optimization (fan-in/fan-out), incremental merge window sizing, partition-aware incremental filters, avoiding full table rebuilds, query plan hints in dbt SQL (LEADING/JOIN hints), materialized view as dbt model target, concurrent model execution with threads, pre/post hooks for ANALYZE TABLE, dbt run selectors to minimize rebuilt models
    24 repo stars
  54. ▌
    Starrocks AI Incident Rca · ivanshamaev
    StarRocks AI incident RCA — autonomous root cause analysis agent for FE/BE failures, memory spike diagnosis (BE heap/process memory), Routine Load pause RCA (ErrorLogUrls + error patterns), Kafka consumer lag spike analysis, compaction backlog detection, tablet replication failures, OOM kill patterns, query timeout clusters, FE GC pressure detection, structured incident report generation
    24 repo stars
  55. ▌
    Starrocks Ddl Table Types · ivanshamaev
    StarRocks table types — Duplicate Key (append-only log/event), Aggregate Key (pre-aggregation metrics), Unique Key (upsert dedup), Primary Key (full DML upsert/delete, MVCC), CREATE TABLE DDL, key column selection, aggregate functions (SUM/MAX/MIN/REPLACE/HLL_UNION/BITMAP_UNION), when to use each type, storage engine differences
    24 repo stars
  56. ▌
    Starrocks Files Ingestion · ivanshamaev
    StarRocks file ingestion — FILES() table function (SELECT/INSERT from S3/HDFS Parquet/ORC/CSV without CREATE TABLE), Iceberg external catalog (HMS/Glue/REST), CREATE EXTERNAL CATALOG, cross-catalog INSERT INTO SELECT, schema auto-detection, partition filter pushdown on external tables, SHOW CREATE CATALOG, external table DDL patterns
    24 repo stars
  57. ▌
    Starrocks Query Optimizer · ivanshamaev
    StarRocks query optimization — vectorized execution engine, pipeline execution model, runtime filters (bloom/in/min-max), predicate pushdown, partition pruning, sort merge optimization, query hints (SET_VAR/JOIN hints), session variables for optimizer control, CBO statistics (ANALYZE TABLE/AUTO ANALYZE), query cache, pipeline parallelism tuning
    24 repo stars
  58. ▌
    Trino Memory And Spill Tuning · ivanshamaev
    Trino memory management and spill-to-disk tuning — query.max-memory/query.max-total-memory/query.max-memory-per-node properties, memory.heap-headroom-per-node, JVM heap sizing (80% of RAM), spill-enabled/spiller-spill-path/spill-compression-codec configuration, exchange buffer tuning, OOM kill diagnosis, memory pool sizing, fault-tolerant execution exchange manager (S3/filesystem), session-level memory overrides, memory-intensive operator patterns (HashJoin/Sort/Window/GroupBy)
    24 repo stars
  59. ▌
    Trino Security And Governance · ivanshamaev
    Trino security hardening and data governance — TLS/HTTPS setup, internal-communication shared secret, authentication (LDAP/OAuth2/JWT), file-based access control (catalog/schema/table/column rules, column masking, row-level filtering, query kill permissions), OPA integration, Ranger for dynamic row-filter and column-mask, catalog isolation by domain, user group mapping, impersonation rules, audit logging via event listener, password file authentication for development
    24 repo stars