Databricks Isv Java Sdk

Databricks SDK for Java (ISV)

Use this skill when implementing or testing Databricks SDK for Java (databricks-sdk-java) integrations for PWAF-compliant workspace management, Unity Catalog, Jobs, and REST-style API access.

PWAF Documentation Links

Requirements

• SDK: com.databricks:databricks-sdk-java:0.54.0 (or later) from Maven Central
• Java: 11+ required
• Install: Add to Maven pom.xml (see dependencies section below)

Authentication Decision Guide

Which authentication method to use?

Production / automated workloads?
  → OAuth M2M (client credentials)  ✅ RECOMMENDED

User-interactive flows?
  → U2M External Browser (custom OAuth app)  ✅ SUPPORTED

Already have an OAuth access token?
  → U2M Token-Env  ✅ SUPPORTED

Local development/testing only?
  → PAT (Personal Access Token)  ⚠️ LIMITED

Authentication Comparison

Note: These examples do NOT use the SDK's built-in databricks-cli OAuth app. ISV/partner applications should register custom OAuth apps in App connections for proper branding, audit trails, and scoped permissions.

Auth Mapping (DatabricksConfig)

CLIENT_ID Distinction (CRITICAL)

Using the wrong client_id causes: "OAuth application with client_id not available in Databricks account"

SDK vs REST APIs

• No SQL warehouse needed: The SDK uses REST APIs (e.g., UC Tables API at /api/2.1/unity-catalog/tables/). No httpPath or DATABRICKS_HTTP_PATH required.
• Use cases: Workspace management, Unity Catalog metadata, Jobs orchestration, Clusters, DBFS, etc.
• For SQL queries: Use the Databricks JDBC Driver (databricks-jdbc) with a SQL warehouse httpPath.

User-Agent (Required per PWAF)

Call once before creating WorkspaceClient:

import com.databricks.sdk.core.UserAgent;

UserAgent.withProduct("YourCompany_YourProduct", "1.0.0");
UserAgent.withPartner("YourCompany");

These are static/global registrations applied to all SDK HTTP requests in the JVM.

Environment Variables Reference

Important:

• Use APP_AUTH_TYPE (not DATABRICKS_AUTH_TYPE) because the SDK reads DATABRICKS_AUTH_TYPE internally.
• Do not mix M2M and U2M environment variables. Use env -i for clean test environments.

Complete Examples

PAT Authentication (Testing Only)

import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.core.DatabricksConfig;
import com.databricks.sdk.core.UserAgent;
import com.databricks.sdk.service.catalog.TableInfo;

public class SdkPatExample {
    public static void main(String[] args) {
        UserAgent.withProduct("YourCompany_YourProduct", "1.0.0");
        UserAgent.withPartner("YourCompany");

        String host = System.getenv("DATABRICKS_HOST");
        String token = System.getenv("DATABRICKS_TOKEN");

        DatabricksConfig config = new DatabricksConfig()
                .setHost(ensureScheme(host))
                .setToken(token);

        WorkspaceClient client = new WorkspaceClient(config);

        TableInfo table = client.tables().get("samples.nyctaxi.trips");
        System.out.println("PAT OK: " + table.getName() 
            + " (" + table.getColumns().size() + " columns)");
    }

    private static String ensureScheme(String host) {
        if (!host.startsWith("https://") && !host.startsWith("http://")) {
            return "https://" + host;
        }
        return host;
    }
}

Env vars: DATABRICKS_HOST, DATABRICKS_TOKEN

OAuth M2M Authentication (Production Recommended)

import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.core.DatabricksConfig;
import com.databricks.sdk.core.UserAgent;
import com.databricks.sdk.service.catalog.TableInfo;

public class SdkOAuthM2MExample {
    public static void main(String[] args) {
        UserAgent.withProduct("YourCompany_YourProduct", "1.0.0");
        UserAgent.withPartner("YourCompany");

        String host = System.getenv("DATABRICKS_HOST");
        String clientId = System.getenv("DATABRICKS_CLIENT_ID");
        String clientSecret = System.getenv("DATABRICKS_CLIENT_SECRET");

        DatabricksConfig config = new DatabricksConfig()
                .setHost(ensureScheme(host))
                .setClientId(clientId)
                .setClientSecret(clientSecret);

        WorkspaceClient client = new WorkspaceClient(config);

        TableInfo table = client.tables().get("samples.nyctaxi.trips");
        System.out.println("OAuth M2M OK: " + table.getName() 
            + " (" + table.getColumns().size() + " columns)");
    }

    private static String ensureScheme(String host) {
        if (!host.startsWith("https://") && !host.startsWith("http://")) {
            return "https://" + host;
        }
        return host;
    }
}

Env vars: DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET

How it works:

1. SDK POSTs to https://<host>/oidc/v1/token with grant_type=client_credentials
2. Receives access_token and expiry
3. Attaches token as Authorization: Bearer <token> on every REST call
4. Refreshes automatically before expiry

Setup: Create service principal in Account Console → Settings → Service principals. Generate OAuth secret.

U2M Custom OAuth App

Uses a custom OAuth app registered in App connections:

import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.core.DatabricksConfig;
import com.databricks.sdk.core.UserAgent;
import com.databricks.sdk.service.catalog.TableInfo;
import java.util.Arrays;

public class SdkU2MCustomOAuthAppExample {
    public static void main(String[] args) {
        UserAgent.withProduct("YourCompany_YourProduct", "1.0.0");
        UserAgent.withPartner("YourCompany");

        String host = System.getenv("DATABRICKS_HOST");
        String clientId = System.getenv("DATABRICKS_U2M_CLIENT_ID");
        String clientSecret = System.getenv("DATABRICKS_U2M_CLIENT_SECRET"); // required by Java SDK
        String redirectUri = System.getenv("DATABRICKS_REDIRECT_URI");

        DatabricksConfig config = new DatabricksConfig()
                .setHost(ensureScheme(host))
                .setAuthType("external-browser")
                .setClientId(clientId)
                .setScopes(Arrays.asList("all-apis"));

        if (clientSecret != null && !clientSecret.isBlank()) {
            config.setClientSecret(clientSecret);
        }
        if (redirectUri != null && !redirectUri.isBlank()) {
            config.setOAuthRedirectUrl(redirectUri);
        }

        config.resolve();

        WorkspaceClient client = new WorkspaceClient(config);

        TableInfo table = client.tables().get("samples.nyctaxi.trips");
        System.out.println("U2M Custom OAuth App OK: " + table.getName());
    }

    private static String ensureScheme(String host) {
        if (!host.startsWith("https://") && !host.startsWith("http://")) {
            return "https://" + host;
        }
        return host;
    }
}

Env vars: DATABRICKS_HOST, DATABRICKS_U2M_CLIENT_ID

Recommended: DATABRICKS_U2M_CLIENT_SECRET (required by Java SDK for proper auth configuration)

Optional: DATABRICKS_REDIRECT_URI (must match App connections registration)

CRITICAL: Use DATABRICKS_U2M_CLIENT_ID for custom OAuth apps — NOT DATABRICKS_CLIENT_ID (that's for M2M).

Note: Testing showed the Java SDK requires DATABRICKS_U2M_CLIENT_SECRET for custom OAuth apps to properly configure external-browser auth. Without it, you may get "cannot configure default credentials" errors.

U2M Token-Env (Pre-obtained Token)

import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.core.DatabricksConfig;
import com.databricks.sdk.core.UserAgent;
import com.databricks.sdk.service.catalog.TableInfo;

public class SdkU2MTokenEnvExample {
    public static void main(String[] args) {
        UserAgent.withProduct("YourCompany_YourProduct", "1.0.0");
        UserAgent.withPartner("YourCompany");

        String host = System.getenv("DATABRICKS_HOST");
        String accessToken = System.getenv("DATABRICKS_ACCESS_TOKEN");
        if (accessToken == null || accessToken.isBlank()) {
            accessToken = System.getenv("DATABRICKS_TOKEN");
        }

        DatabricksConfig config = new DatabricksConfig()
                .setHost(ensureScheme(host))
                .setToken(accessToken);

        WorkspaceClient client = new WorkspaceClient(config);

        TableInfo table = client.tables().get("samples.nyctaxi.trips");
        System.out.println("U2M Token-Env OK: " + table.getName());
    }

    private static String ensureScheme(String host) {
        if (!host.startsWith("https://") && !host.startsWith("http://")) {
            return "https://" + host;
        }
        return host;
    }
}

Env vars: DATABRICKS_HOST, DATABRICKS_ACCESS_TOKEN (or DATABRICKS_TOKEN)

Token Refresh for U2M Flows

U2M access tokens expire after approximately 1 hour. For long-running applications, you need to handle token refresh.

Token Lifetime Summary

SDK Token Refresh Pattern (Recommended)

The SDK's DatabricksConfig.authenticate() method automatically handles token refresh when using persistent configuration. For single-shot scripts, re-run the browser flow when tokens expire:

public class TokenRefreshExample {
    private DatabricksConfig config;
    private WorkspaceClient client;
    private long tokenExpiresAt;
    private static final long REFRESH_BUFFER_MS = 5 * 60 * 1000; // 5 minutes

    public TokenRefreshExample() {
        this.config = createConfig();
        this.client = new WorkspaceClient(config);
        this.tokenExpiresAt = System.currentTimeMillis() + (55 * 60 * 1000); // ~55 min
    }

    private DatabricksConfig createConfig() {
        String host = System.getenv("DATABRICKS_HOST");
        String clientId = System.getenv("DATABRICKS_U2M_CLIENT_ID");
        String clientSecret = System.getenv("DATABRICKS_U2M_CLIENT_SECRET");

        DatabricksConfig cfg = new DatabricksConfig()
                .setHost(host.startsWith("https://") ? host : "https://" + host)
                .setAuthType("external-browser")
                .setClientId(clientId)
                .setClientSecret(clientSecret)
                .setScopes(Arrays.asList("all-apis", "offline_access")); // offline_access for refresh_token

        cfg.resolve();
        return cfg;
    }

    public WorkspaceClient getClient() {
        if (System.currentTimeMillis() > (tokenExpiresAt - REFRESH_BUFFER_MS)) {
            // Token nearing expiration, re-authenticate
            this.config = createConfig();
            this.client = new WorkspaceClient(config);
            this.tokenExpiresAt = System.currentTimeMillis() + (55 * 60 * 1000);
        }
        return client;
    }
}

Manual Refresh Token Flow (Advanced)

For applications that need to refresh without user interaction, request offline_access scope to receive a refresh_token:

// When obtaining initial token, include offline_access scope
config.setScopes(Arrays.asList("all-apis", "offline_access"));

// The SDK caches the refresh_token in ~/.databricks/token-cache.json
// On subsequent runs, the SDK will automatically use the refresh_token
// if the access_token is expired but refresh_token is still valid

Important: The offline_access scope must be enabled for your OAuth app in App connections.

Best Practices

1. Request offline_access: Include this scope for long-running applications
2. Check expiration before operations: Avoid mid-operation failures
3. Handle refresh failures gracefully: Fall back to re-authentication
4. Don't cache tokens in code: Use SDK token cache or secure storage

Environment Variables Reference

Required Imports / Maven Dependencies

<!-- Databricks SDK for Java -->
<dependency>
    <groupId>com.databricks</groupId>
    <artifactId>databricks-sdk-java</artifactId>
    <version>0.54.0</version>
</dependency>

<!-- Suppress SDK logging (optional) -->
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-nop</artifactId>
    <version>2.0.16</version>
</dependency>

Pin to specific minor version (SDK is in Beta). Java 11+ required.

Multi-Auth Pattern

Support multiple auth types in a single binary using a custom selector (e.g., APP_AUTH_TYPE):

String authType = System.getenv("APP_AUTH_TYPE");
if (authType == null || authType.isBlank()) authType = "oauth_m2m";

DatabricksConfig config = new DatabricksConfig()
        .setHost(ensureScheme(host));

switch (authType.toLowerCase()) {
    case "pat":
        config.setToken(System.getenv("DATABRICKS_TOKEN"));
        break;
    case "oauth_m2m":
        config.setClientId(System.getenv("DATABRICKS_CLIENT_ID"));
        config.setClientSecret(System.getenv("DATABRICKS_CLIENT_SECRET"));
        break;
    case "u2m_custom_oauth_app":
        config.setAuthType("external-browser");
        config.setClientId(System.getenv("DATABRICKS_U2M_CLIENT_ID"));
        config.setClientSecret(System.getenv("DATABRICKS_U2M_CLIENT_SECRET"));
        config.setScopes(Arrays.asList("all-apis"));
        String redirect = System.getenv("DATABRICKS_REDIRECT_URI");
        if (redirect != null && !redirect.isBlank()) config.setOAuthRedirectUrl(redirect);
        config.resolve();
        break;
    case "u2m_token_env":
        String token = System.getenv("DATABRICKS_ACCESS_TOKEN");
        if (token == null || token.isBlank()) token = System.getenv("DATABRICKS_TOKEN");
        config.setToken(token);
        break;
}

WorkspaceClient client = new WorkspaceClient(config);

CRITICAL: Use APP_AUTH_TYPE (not DATABRICKS_AUTH_TYPE). The SDK reads DATABRICKS_AUTH_TYPE internally.

Implementation Learnings (Java SDK v0.54.0)

Key findings from testing that differ from documentation or other SDKs:

1. U2M Custom OAuth App Requires Client Secret

Unlike some other SDKs, the Java SDK requires DATABRICKS_U2M_CLIENT_SECRET even for public OAuth apps:

// REQUIRED: Both client ID and secret must be set
config.setClientId(System.getenv("DATABRICKS_U2M_CLIENT_ID"));
config.setClientSecret(System.getenv("DATABRICKS_U2M_CLIENT_SECRET")); // Required!

Without client secret, you get: "cannot configure default credentials"

2. SDK Reads DATABRICKS_AUTH_TYPE Internally

The SDK uses DATABRICKS_AUTH_TYPE internally. If you set this environment variable, it may conflict with programmatic config. Use a custom variable like APP_AUTH_TYPE for your own auth type selection.

3. config.resolve() Must Be Called

For external-browser auth, config.resolve() must be called after configuration but before creating WorkspaceClient:

DatabricksConfig config = new DatabricksConfig()
        .setHost(host)
        .setAuthType("external-browser")
        .setClientId(clientId)
        .setClientSecret(clientSecret)
        .setScopes(Arrays.asList("all-apis"));

config.resolve();  // REQUIRED: initializes HTTP client

WorkspaceClient client = new WorkspaceClient(config);

SDK Workarounds Summary

⚠️ WorkspaceClient resource cleanup (SDK 0.54) WorkspaceClient does not implement Closeable or AutoCloseable in SDK 0.54. Do not call workspaceClient.close() — it causes a compile error: cannot find symbol: method close(). Correct pattern for a wrapper's close() method:

@Override
public void close() {
    workspaceClient = null;  // GC handles cleanup; no explicit close needed
}

Redirect URI Reference

Validation Query

Verify User-Agent telemetry is being recorded correctly:

SELECT event_time, user_agent, action_name, request_params
FROM system.access.audit
WHERE event_time > current_timestamp() - INTERVAL 1 HOUR
  AND lower(user_agent) LIKE '%yourcompany%'
ORDER BY event_time DESC
LIMIT 10;

U2M Browser Environment Requirements

When using env -i for U2M browser flows, include these variables for Desktop.browse() to work:

# Essential for browser launch
HOME="$HOME"       # Required: token cache location
PATH="$PATH"       # Required: find java, browser binaries
JAVA_HOME="$JAVA_HOME"  # Required: find java

# For macOS (Desktop.browse() typically works)
# For Linux, may also need:
DISPLAY="$DISPLAY"   # For X11-based browser launch

Auth Isolation

Run tests with a clean environment using env -i:

# PAT test
env -i HOME="$HOME" PATH="$PATH" JAVA_HOME="$JAVA_HOME" \
  DATABRICKS_HOST="$DATABRICKS_HOST" \
  DATABRICKS_TOKEN="$DATABRICKS_TOKEN" \
  APP_AUTH_TYPE=pat \
  mvn exec:java -Dexec.mainClass="com.example.SdkPatExample"

# OAuth M2M test
env -i HOME="$HOME" PATH="$PATH" JAVA_HOME="$JAVA_HOME" \
  DATABRICKS_HOST="$DATABRICKS_HOST" \
  DATABRICKS_CLIENT_ID="$DATABRICKS_CLIENT_ID" \
  DATABRICKS_CLIENT_SECRET="$DATABRICKS_CLIENT_SECRET" \
  APP_AUTH_TYPE=oauth_m2m \
  mvn exec:java -Dexec.mainClass="com.example.SdkOAuthM2MExample"

# U2M Custom OAuth App test (include client secret per Java SDK requirements)
env -i HOME="$HOME" PATH="$PATH" JAVA_HOME="$JAVA_HOME" \
  DATABRICKS_HOST="$DATABRICKS_HOST" \
  DATABRICKS_U2M_CLIENT_ID="$DATABRICKS_U2M_CLIENT_ID" \
  DATABRICKS_U2M_CLIENT_SECRET="$DATABRICKS_U2M_CLIENT_SECRET" \
  DATABRICKS_REDIRECT_URI="$DATABRICKS_REDIRECT_URI" \
  APP_AUTH_TYPE=u2m_custom_oauth_app \
  mvn exec:java -Dexec.mainClass="com.example.SdkU2MCustomOAuthAppExample"

# U2M Token-Env test
env -i HOME="$HOME" PATH="$PATH" JAVA_HOME="$JAVA_HOME" \
  DATABRICKS_HOST="$DATABRICKS_HOST" \
  DATABRICKS_ACCESS_TOKEN="$DATABRICKS_ACCESS_TOKEN" \
  APP_AUTH_TYPE=u2m_token_env \
  mvn exec:java -Dexec.mainClass="com.example.SdkU2MTokenEnvExample"

Error Handling Patterns

Production applications should handle authentication and API errors gracefully.

Basic Error Handling

import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.core.DatabricksConfig;
import com.databricks.sdk.core.DatabricksException;
import com.databricks.sdk.service.catalog.TableInfo;

public class SdkErrorHandlingExample {
    public static void main(String[] args) {
        try {
            WorkspaceClient client = createClient();
            TableInfo table = client.tables().get("samples.nyctaxi.trips");
            System.out.println("Success: " + table.getName());
        } catch (DatabricksException e) {
            handleDatabricksError(e);
        } catch (Exception e) {
            handleGeneralError(e);
        }
    }

    private static WorkspaceClient createClient() {
        String host = System.getenv("DATABRICKS_HOST");
        String clientId = System.getenv("DATABRICKS_CLIENT_ID");
        String clientSecret = System.getenv("DATABRICKS_CLIENT_SECRET");

        if (host == null || host.isBlank()) {
            throw new IllegalStateException("DATABRICKS_HOST is required");
        }
        if (clientId == null || clientId.isBlank()) {
            throw new IllegalStateException("DATABRICKS_CLIENT_ID is required");
        }
        if (clientSecret == null || clientSecret.isBlank()) {
            throw new IllegalStateException("DATABRICKS_CLIENT_SECRET is required");
        }

        DatabricksConfig config = new DatabricksConfig()
                .setHost(host.startsWith("https://") ? host : "https://" + host)
                .setClientId(clientId)
                .setClientSecret(clientSecret);

        return new WorkspaceClient(config);
    }

    private static void handleDatabricksError(DatabricksException e) {
        String message = e.getMessage();

        if (message.contains("INVALID_PARAMETER_VALUE") || message.contains("invalid_client")) {
            System.err.println("Authentication failed: Invalid credentials");
            System.err.println("  - Verify DATABRICKS_CLIENT_ID is correct");
            System.err.println("  - Verify DATABRICKS_CLIENT_SECRET is correct");
        } else if (message.contains("PERMISSION_DENIED") || message.contains("403")) {
            System.err.println("Authorization failed: Insufficient permissions");
            System.err.println("  - Verify service principal has required permissions");
        } else if (message.contains("NOT_FOUND") || message.contains("404")) {
            System.err.println("Resource not found");
        } else if (message.contains("TEMPORARILY_UNAVAILABLE") || message.contains("503")) {
            System.err.println("Service temporarily unavailable - retry with backoff");
        } else {
            System.err.println("Databricks API error: " + message);
        }
    }

    private static void handleGeneralError(Exception e) {
        String message = e.getMessage();

        if (e instanceof java.net.UnknownHostException) {
            System.err.println("Network error: Cannot resolve host");
            System.err.println("  - Verify DATABRICKS_HOST is correct");
            System.err.println("  - Check network connectivity");
        } else if (e instanceof java.net.SocketTimeoutException) {
            System.err.println("Network timeout - retry with backoff");
        } else if (message != null && message.contains("cannot configure default credentials")) {
            System.err.println("Configuration error: Missing or invalid auth configuration");
            System.err.println("  - For M2M: Set DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET");
            System.err.println("  - For U2M: Set DATABRICKS_U2M_CLIENT_ID and DATABRICKS_U2M_CLIENT_SECRET");
        } else {
            System.err.println("Unexpected error: " + message);
            e.printStackTrace();
        }
    }
}

Retry Pattern with Exponential Backoff

public class RetryHelper {
    private static final int MAX_RETRIES = 3;
    private static final long INITIAL_BACKOFF_MS = 1000;

    public static <T> T withRetry(java.util.function.Supplier<T> operation) {
        int attempt = 0;
        while (true) {
            try {
                return operation.get();
            } catch (DatabricksException e) {
                if (!isRetryable(e) || attempt >= MAX_RETRIES) {
                    throw e;
                }
                long backoff = INITIAL_BACKOFF_MS * (long) Math.pow(2, attempt);
                System.err.println("Retry " + (attempt + 1) + "/" + MAX_RETRIES 
                    + " after " + backoff + "ms");
                try {
                    Thread.sleep(backoff);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new RuntimeException("Interrupted during retry", ie);
                }
                attempt++;
            }
        }
    }

    private static boolean isRetryable(DatabricksException e) {
        String msg = e.getMessage();
        return msg.contains("TEMPORARILY_UNAVAILABLE") 
            || msg.contains("503")
            || msg.contains("429")  // Rate limited
            || msg.contains("RESOURCE_EXHAUSTED");
    }
}

// Usage:
// TableInfo table = RetryHelper.withRetry(() -> client.tables().get("catalog.schema.table"));

Error Classification Reference

HTTP Status Codes

DatabricksException Error Codes

Retryable Error Detection

private static boolean isRetryable(DatabricksException e) {
    String msg = e.getMessage();
    return msg.contains("TEMPORARILY_UNAVAILABLE") 
        || msg.contains("503")
        || msg.contains("429")
        || msg.contains("RESOURCE_EXHAUSTED")
        || msg.contains("DEADLINE_EXCEEDED")
        || msg.contains("Connection reset")
        || msg.contains("Read timed out");
}

Troubleshooting

Common Errors

OAuth-Specific Errors

SDK Errors

U2M Custom OAuth App Errors

Logging

Host Normalization

The SDK accepts both https://myworkspace.cloud.databricks.com and myworkspace.cloud.databricks.com. Being explicit with https:// is recommended:

private static String ensureScheme(String host) {
    if (!host.startsWith("https://") && !host.startsWith("http://")) {
        return "https://" + host;
    }
    return host;
}

Key Differences from Databricks JDBC Driver