Eventstreams

Eventstreams

When to Use

• Ingesting real-time event data into Microsoft Fabric from Event Hubs, IoT Hub, or Kafka
• Building streaming pipelines with in-flight transformations (filter, aggregate, window)
• Routing streaming data to Eventhouse (KQL database) for real-time analytics
• Routing streaming data to Lakehouse for near-real-time Delta streaming
• Deciding between Eventstreams and Event Hubs for a streaming use case
• Preparing for Microsoft Fabric Data Engineer Associate (DP-700) exam

Core Jobs

1. Eventstream Architecture

• Eventstream = Fabric-native no-code streaming service; visual canvas authoring
• Data flow: Source → (optional Transformations) → Destinations
• Eventstream runs continuously; no separate job to start/stop
• Fully managed; no Spark Structured Streaming setup required

2. Event Sources

• Custom App source: connect using Event Hubs-compatible endpoint and SAS key or Managed Identity

3. Stream Destinations

• Eventhouse path: lowest latency; query with KQL; best for operational dashboards
• Lakehouse path: higher latency (Delta micro-batch); best for analytical workloads that also need Spark

4. Stream Transformations

Transformations applied in-flight before data reaches destination:

• Time windows for aggregation:
  • Tumbling — fixed, non-overlapping (e.g., 1-minute buckets)
  • Sliding — overlapping windows (e.g., last 5 minutes, updated every 1 minute)
  • Session — dynamic window based on activity gaps

5. Eventhouse (KQL) Basics

• Eventhouse = KQL database optimized for real-time, time-series data in Fabric
• Auto-ingest from Eventstream: data lands in KQL table continuously
• Query with KQL (Kusto Query Language):

  // Count events per minute
  Events
  | where Timestamp > ago(1h)
  | summarize count() by bin(Timestamp, 1m)
  | render timechart
  
  // Filter and project
  Events
  | where EventType == "click"
  | project UserId, Timestamp, Page
  

• KQL tables also queryable with T-SQL (limited subset)
• Real-time dashboards in Fabric connect directly to Eventhouse

6. Eventstreams vs Event Hubs

• Use Eventstreams when your destination is Fabric (Lakehouse or Eventhouse)
• Use Event Hubs directly when distributing events to multiple non-Fabric consumers

Key Concepts

• Eventhouse — KQL database in Fabric; optimized for real-time time-series; auto-ingest from Eventstreams
• KQL (Kusto Query Language) — query language for Eventhouse; | where, | summarize, | project, | render
• Tumbling window — fixed non-overlapping time intervals for aggregation
• CDC (Change Data Capture) — stream database row changes (INSERT/UPDATE/DELETE) as events
• Derived stream — create multiple downstream branches from one source stream
• Custom App source — use Event Hubs-compatible SDK to push events to Eventstream

Checklist

• Eventhouse (KQL) chosen for millisecond-latency real-time queries?
• Lakehouse chosen when near-real-time is acceptable and Spark access needed later?
• In-flight transformations (filter, aggregate) configured before destination to reduce write volume?
• Time window type (tumbling/sliding/session) chosen based on aggregation requirement?
• Consumer group configured for Event Hubs/Kafka source (avoid sharing with other consumers)?
• Sample data source used for testing before connecting production Event Hub?
• Reflex configured for event-driven alerts on anomalous streaming data?

Output Format

• 🔴 Critical — routing high-frequency events directly to Lakehouse without aggregation (may cause write bottleneck)
• 🟡 Warning — using Lakehouse destination when millisecond latency is required (use Eventhouse instead)
• 🟡 Warning — no consumer group specified for Kafka/Event Hubs source (may conflict with other consumers)
• 🟢 Suggestion — add filter transformation to reduce event volume before Eventhouse ingestion

Exam Tips

• Eventhouse = KQL database — queryable with KQL (Kusto Query Language); also supports limited T-SQL
• Eventstream → Eventhouse = real-time analytics path — lowest latency (milliseconds); best for operational dashboards
• Eventstream → Lakehouse = near-real-time — Delta streaming; slightly higher latency; use when Spark access needed
• KQL summarize count() by bin(Timestamp, 1m) — standard pattern for 1-minute aggregation on time-series data
• Custom App source = Event Hubs-compatible endpoint — use Fabric SDK or Event Hubs SDK; send events using SAS or Managed Identity
• Eventstream transformations happen in-flight — no separate Spark Structured Streaming job needed; transformations run inside Eventstream before data hits destination