Fabric Integration

Microsoft Fabric Integration

Overview

Microsoft Fabric is the unified analytics platform that includes Power BI, Data Factory, Data Engineering, Data Science, Real-Time Intelligence, and Data Warehouse. Power BI is deeply integrated as the visualization and semantic modeling layer of Fabric.

Direct Lake Mode

Direct Lake is a storage mode exclusive to Fabric that reads data directly from delta tables in OneLake without importing or sending DirectQuery requests.

How Direct Lake Works

1. Framing: On refresh, Direct Lake copies only metadata (Parquet file references) from delta tables -- takes seconds
2. On-demand loading: When a query hits the model, data is loaded from Parquet files directly into the VertiPaq engine
3. No data duplication: Unlike Import, no copy of data is stored in the semantic model
4. Near-import performance: Once loaded into memory, queries run at VertiPaq speed

Direct Lake vs Import vs DirectQuery

Direct Lake Variants (2025-2026 GA)

Creating a Direct Lake Semantic Model

In Power BI Desktop (2025+ preview):

1. Get Data > OneLake data hub
2. Select Fabric lakehouse or warehouse
3. Choose tables (loaded as Direct Lake automatically)
4. Build measures and relationships in Desktop
5. Publish to Fabric workspace

Via Fabric Service:

1. Open lakehouse/warehouse in Fabric
2. Click "New semantic model"
3. Select tables to include
4. Open model in web to add measures and relationships

Programmatically via TOM:

var database = new Database() { Name = "DirectLakeModel" };
var model = new Model() { Name = "DirectLakeModel" };
database.Model = model;

// Direct Lake partition source
var table = new Table() { Name = "Sales" };
table.Partitions.Add(new Partition() {
    Name = "Sales-DL",
    Mode = ModeType.DirectLake,
    Source = new EntityPartitionSource() {
        EntityName = "Sales",
        SchemaName = "dbo",
        ExpressionSource = new ExpressionSource() {
            Expression = "DatabaseQuery"
        }
    }
});
model.Tables.Add(table);

Critical distinction: DL/OL does NOT fall back to DirectQuery. If data cannot be served from memory, the query fails. This means DL/OL models must be carefully sized within capacity guardrails.

Direct Lake Guardrails by Capacity

Max Memory is a soft limit for paging -- exceeding it causes performance degradation but not failure.

Max model size on disk/OneLake is a hard guardrail -- exceeding causes DQ fallback (DL/SQL) or query failure (DL/OL).

Direct Lake Fallback Configuration

Monitor fallback in Fabric Capacity Metrics app -- frequent fallback indicates model design issues.

Common fallback triggers (DL/SQL):

• Columns not loaded into memory due to capacity limits
• Calculated columns on Direct Lake tables (may trigger DQ fallback)
• Certain DAX patterns that require full table scan
• Stale framing (delta tables changed but model not re-framed)
• File/row-group count exceeding capacity guardrails

Power BI Embedded with Direct Lake (GA March 2025)

Direct Lake mode is now fully supported for embedded analytics, backed by Microsoft SLA. Generate embed tokens for Direct Lake semantic models using the same embed token API as Import/DirectQuery models.

Framing (Refresh)

# Trigger framing via REST API
POST https://api.powerbi.com/v1.0/myorg/groups/{workspaceId}/datasets/{datasetId}/refreshes
{
  "type": "Automatic"
}

Framing is extremely fast (seconds) compared to Import refresh (minutes/hours). Schedule frequent framing for near-real-time data.

OneLake

OneLake is Fabric's unified data lake -- a single store for all analytics data, built on Azure Data Lake Storage Gen2 with delta format.

OneLake Shortcuts

Connect to external data without copying:

OneLake File API

Access OneLake data programmatically:

# Using Azure Storage SDK (OneLake supports ADLS Gen2 API)
from azure.storage.filedatalake import DataLakeServiceClient

service_client = DataLakeServiceClient(
    account_url="https://onelake.dfs.fabric.microsoft.com",
    credential=token_credential
)

file_system_client = service_client.get_file_system_client(workspace_id)
directory_client = file_system_client.get_directory_client(f"{lakehouse_name}.Lakehouse/Tables")

Fabric Lakehouse

A lakehouse combines data lake flexibility with warehouse SQL capabilities:

Power BI connectivity:

• SQL Analytics Endpoint: Read-only SQL endpoint for DirectQuery or Direct Lake
• Delta tables: Native format for Direct Lake
• Notebooks: Write data from Spark notebooks, read in Power BI

Lakehouse to Power BI Flow

[Data Sources] --> [Fabric Notebooks/Pipelines] --> [Lakehouse Delta Tables]
     |                                                       |
     v                                                       v
[Power Query Dataflows Gen2]                    [Direct Lake Semantic Model]
                                                             |
                                                             v
                                                     [Power BI Reports]

Fabric Warehouse

Fully managed SQL warehouse in Fabric:

• T-SQL support: Full DML (INSERT, UPDATE, DELETE, MERGE)
• Auto-distributed storage: No index tuning needed
• Direct Lake compatible: Tables accessible as Direct Lake sources
• Cross-database queries: Query across warehouses and lakehouses

Dataflow Gen2

Cloud-based ETL in Fabric, evolution of Power BI Dataflows:

Gen1 deprecation: Microsoft has announced Gen1 is legacy. Migrate to Gen2 for all new development. Gen2 supports all Gen1 connectors plus Fabric-native destinations.

Dataflow Gen2 Output Destinations (2025-2026)

Dataflow Gen2 to Direct Lake Pipeline

1. Create Dataflow Gen2 in Fabric workspace
2. Connect to source (any Power Query connector)
3. Set destination to Lakehouse (creates delta tables)
4. Create semantic model on top of lakehouse tables (Direct Lake)
5. Build reports on the semantic model

Real-Time Intelligence

Fabric Real-Time Intelligence items now have GA lifecycle management (2025):

Eventstream to Power BI Pipeline

[Event Hubs/Kafka/IoT Hub] --> [Eventstream] --> [KQL Database] --> [Power BI DirectQuery]
                                     |
                                     +--> [Lakehouse] --> [Power BI Direct Lake]

Notebooks for Data Prep

Fabric notebooks (PySpark/Spark SQL) write data that Power BI consumes:

# Write DataFrame to lakehouse delta table
df.write.format("delta").mode("overwrite").saveAsTable("Sales")

# Optimized write with partitioning
df.write.format("delta") \
    .partitionBy("Year", "Month") \
    .mode("overwrite") \
    .option("overwriteSchema", "true") \
    .saveAsTable("Sales")

# V-Order optimization (improves Direct Lake read performance)
spark.conf.set("spark.sql.parquet.vorder.enabled", "true")
df.write.format("delta").mode("overwrite").saveAsTable("Sales")

V-Order: A write-time optimization that orders Parquet data for faster Direct Lake reads. Enable in notebook or pipeline configuration.

Fabric Items in Power BI Context

Semantic Link (sempy)

Python library for Power BI semantic model interaction in Fabric notebooks:

import sempy.fabric as fabric

# List datasets in workspace
datasets = fabric.list_datasets()

# Read data from semantic model using DAX
df = fabric.evaluate_dax(
    dataset="SalesModel",
    dax_string="EVALUATE SUMMARIZECOLUMNS('Date'[Year], 'Product'[Category], \"Sales\", [Total Sales])"
)

# Read model metadata
tables = fabric.list_tables(dataset="SalesModel")
measures = fabric.list_measures(dataset="SalesModel")

# Refresh dataset
fabric.refresh_dataset(dataset="SalesModel")

Composite Models with Direct Lake (2025 Preview)

Mix Direct Lake tables with Import tables in a single semantic model:

Key consideration: Composite model queries involving both DL and Import tables may have different performance characteristics. Test with production data volumes.

Additional Resources

Reference Files

• references/fabric-architecture-patterns.md -- Medallion architecture, data mesh patterns, and Fabric workspace design strategies