Keystore Credstore

Keystore Service

Migrate keystore and password storage to SAP Credential Store.

Purpose

Replace Neo's KeyStoreService and PasswordStorage JNDI resources with SAP Credential Store service using mTLS authentication and REST API.

Detection

This skill applies if any of these patterns are found:

In web.xml

<resource-ref>
    <res-ref-name>KeyStoreService</res-ref-name>
    <res-type>com.sap.cloud.crypto.keystore.api.KeyStoreService</res-type>
</resource-ref>

<!-- OR -->
<resource-ref>
    <res-ref-name>PasswordStorage</res-ref-name>
    <res-type>com.sap.cloud.security.password.PasswordStorage</res-type>
</resource-ref>

In Java source files

import com.sap.cloud.crypto.keystore.api.KeyStoreService;
import com.sap.cloud.security.password.PasswordStorage;

@Resource(name = "KeyStoreService")
private KeyStoreService keyStoreService;

@Resource(name = "PasswordStorage")
private PasswordStorage passwordStorage;

Prerequisites

Working directory: This skill must run inside the -cf-migration copy of your app, created by jakarta-java25-migration or neo-to-cf-migration-orchestrator. If your current directory does not end in -cf-migration, switch to it before proceeding.

Before invoking this skill, ensure you have invoked:

1. sdk-replacement - Use the sdk-replacement skill
   • Sets up SAP Cloud SDK
   • REQUIRED before this skill

Also required:

• Credential Store entitlement in subaccount

Transformation Steps

Step 1: Remove Resource References from web.xml

Remove these from web.xml:

<resource-ref>
    <res-ref-name>KeyStoreService</res-ref-name>
    <res-type>com.sap.cloud.crypto.keystore.api.KeyStoreService</res-type>
</resource-ref>

<resource-ref>
    <res-ref-name>PasswordStorage</res-ref-name>
    <res-type>com.sap.cloud.security.password.PasswordStorage</res-type>
</resource-ref>

Step 2: Copy Client Classes

Copy the following helper classes to your project from assets/. All assets declare package com.example.document;, so place them flat under src/main/java/com/example/document/:

Note on the package: com.example.document is a placeholder. Rename to your project's package (e.g. com.acme.security) when you copy the assets in, and update the imports below to match.

These classes handle:

• Reading service credentials from VCAP_SERVICES via DefaultServiceBindingAccessor
• Parsing PEM certificates (X.509) and PKCS#1 private keys (BouncyCastle PEMParser)
• Creating an SSLContext for mTLS (PKCS12 keystore in memory)
• REST API calls to Credential Store using java.net.http.HttpClient

The classes have no Lombok dependency — getters are written by hand so they build cleanly under JDK 25.

Required Maven dependencies

Add these to your application's pom.xml. The cf-tomcat-bom + sdk-modules-bom BOMs already manage versions for scp-cf (which transitively brings in service-binding-api, BouncyCastle, Apache HttpClient, Jackson, and SLF4J), so you only need the scp-cf dependency itself plus jakarta.servlet-api for the servlet:

<properties>
    <!-- sap_java_buildpack_jakarta supports SapMachine 17, 21, and 25.
         Always pin the runtime JRE explicitly via JBP_CONFIG_SAP_MACHINE_JRE in
         mtad.yaml / manifest.yml, and make sure this compile target matches that
         major version — a higher class file version yields
         java.lang.UnsupportedClassVersionError at servlet load time (HTTP 500). -->
    <maven.compiler.source>25</maven.compiler.source>
    <maven.compiler.target>25</maven.compiler.target>

    <!-- Resolve via sdk-replacement Step 3. -->
    <cf-tomcat-bom-version>RESOLVED_CF_TOMCAT_BOM_VERSION</cf-tomcat-bom-version>
    <sdk-modules-bom-version>RESOLVED_SDK_MODULES_BOM_VERSION</sdk-modules-bom-version>
    <jakarta.servlet-api.version>6.1.0</jakarta.servlet-api.version>
</properties>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.sap.cloud.sjb.cf</groupId>
            <artifactId>cf-tomcat-bom</artifactId>
            <version>${cf-tomcat-bom-version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
        <dependency>
            <groupId>com.sap.cloud.sdk</groupId>
            <artifactId>sdk-modules-bom</artifactId>
            <version>${sdk-modules-bom-version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>com.sap.cloud.sdk.cloudplatform</groupId>
        <artifactId>scp-cf</artifactId>
    </dependency>
    <dependency>
        <groupId>jakarta.servlet</groupId>
        <artifactId>jakarta.servlet-api</artifactId>
        <version>${jakarta.servlet-api.version}</version>
        <scope>provided</scope>
    </dependency>
</dependencies>

Pin the runtime JRE explicitly. sap_java_buildpack_jakarta supports SapMachine 17, 21, and 25. The buildpack's implicit JRE choice changes over time, so don't rely on it — always pin the version in mtad.yaml (or manifest.yml) and keep the compile target in pom.xml aligned with it. A class file newer than the runtime JRE causes Tomcat to fail loading the servlet with java.lang.UnsupportedClassVersionError: ... class file version N.0, this version of the Java Runtime only recognizes class file versions up to M.0 and every request returns HTTP 500.

For Java 25 (the recommended target for new migrations) add this to your module properties: block in mtad.yaml:

JBP_CONFIG_COMPONENTS: "jres: ['com.sap.xs.java.buildpack.jre.SAPMachineJRE']"
JBP_CONFIG_SAP_MACHINE_JRE: '{ version: 25.+ }'

Same form for any other supported major version — substitute 17.+ or 21.+. Build locally with a matching JDK (e.g. SapMachine 25 from https://sapmachine.io/). The Lombok-free helper classes in assets/ compile cleanly on any supported version.

Step 3: Update Application Code

The HTTP URL contract changes — do NOT preserve the old Neo query parameters.

The Neo KeyStoreServlet was driven by params shaped around its JNDI API: ?method=getKeyStore&keyStoreName=<file>&password=<password>. None of those map onto SAP Credential Store, which is namespace-scoped and addresses each credential by alias. The migrated servlet MUST accept the new contract:

URL	Behaviour
GET /keystore?namespace=<ns>	List all credentials in <ns> (calls CredStoreClient.retrieveCredentials(ns))
GET /keystore?namespace=<ns>&alias=<name>	Retrieve a single credential by alias (calls CredStoreClient.retrieveCredential(alias, ns))
GET /keystore (no namespace)	Reject with 400 Bad Request, body Namespace is required!

A Neo password-storage app likewise drops to: GET /?namespace=<ns>&alias=<name> calling CredStoreClient.retrievePassword(alias, ns).

Why this matters: do NOT carry over the method=getKeyStore / keyStoreName= / password= validation block from the original Neo servlet. Doing so makes the migrated app keep returning 400 for the actual contract a credstore-aware caller will use, and silently breaks integration tests. Replace the old guard with the namespace check shown below.

Before (Neo KeyStoreServlet — JNDI lookup, method=…&keyStoreName=…&password=…):

import com.sap.cloud.crypto.keystore.api.KeyStoreService;
import java.security.KeyStore;
import javax.naming.InitialContext;
import javax.servlet.http.*;

public class KeyStoreServlet extends HttpServlet {
    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        String method = request.getParameter("method");
        if (method == null || !"getKeyStore".equals(method)) {
            // Neo-shaped guard. DELETE THIS BLOCK in the migrated version —
            // the new contract does not use a "method" parameter.
            response.sendError(HttpServletResponse.SC_BAD_REQUEST, "method invalid");
            return;
        }
        KeyStoreService svc = (KeyStoreService) new InitialContext().lookup("java:comp/env/KeyStoreService");
        KeyStore keyStore = svc.getKeyStore(
            request.getParameter("keyStoreName"),
            request.getParameter("password").toCharArray());
        // ... iterate keyStore.aliases() ...
    }
}

After (Cloud Foundry — ?namespace=<ns>[&alias=<name>]):

package com.example.document;

// CredStoreClient and CredStoreResponse live in the same package — no import needed.
import jakarta.servlet.http.HttpServlet;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;

import java.io.IOException;
import java.io.PrintWriter;
import java.security.GeneralSecurityException;

public class KeyStoreServlet extends HttpServlet {

    private static final String PARAM_ALIAS = "alias";
    private static final String PARAM_NAMESPACE = "namespace";

    private final CredStoreClient credStoreClient;

    public KeyStoreServlet() throws GeneralSecurityException, IOException {
        this.credStoreClient = new CredStoreClient();
    }

    @Override
    public void destroy() {
        // Release the mTLS private key when the servlet is taken out of service.
        // CredStoreClient holds the key for its whole lifetime (the SSLContext
        // reuses it across requests), so it must only be destroyed here — never
        // per request — otherwise subsequent mTLS handshakes fail.
        credStoreClient.close();
    }

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
        String alias = request.getParameter(PARAM_ALIAS);
        String namespace = request.getParameter(PARAM_NAMESPACE);

        if (isBlank(namespace)) {
            sendError(response, "Namespace is required!", HttpServletResponse.SC_BAD_REQUEST);
            return;
        }

        CredStoreResponse credstoreResponse = isBlank(alias)
                ? credStoreClient.retrieveCredentials(namespace)               // list all keys
                : credStoreClient.retrieveCredential(alias, namespace);        // single key

        if (credstoreResponse.isSuccess()) {
            sendOk(response, isBlank(alias)
                    ? "Successfully retrieved credentials."
                    : "Successfully retrieved credential.");
        } else {
            sendError(response, credstoreResponse.getMessage(), HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
        }
    }

    private static boolean isBlank(String s) { return s == null || s.isBlank(); }

    private static void sendOk(HttpServletResponse response, String body) throws IOException {
        try (PrintWriter out = response.getWriter()) { out.println(body); }
    }

    private static void sendError(HttpServletResponse response, String body, int status) throws IOException {
        response.setStatus(status);
        try (PrintWriter out = response.getWriter()) { out.println(body); }
    }
}

The corresponding web.xml must drop the <resource-ref> (Step 1) AND map the servlet at /keystore (or / for password apps) so the URL above resolves. Any Neo ErrorUtil helper that imported javax.servlet.* must be ported to jakarta.servlet.* or inlined as private helpers like above.

Before (Neo Password Storage — JNDI lookup, no namespace param):

import com.sap.cloud.security.password.PasswordStorage;

@Resource(name = "PasswordStorage")
private PasswordStorage passwordStorage;

public void usePassword(HttpServletRequest request) throws Exception {
    String alias = request.getParameter("alias");                     // alias only
    char[] password = passwordStorage.getPassword(alias);
    // ... use password ...
}

After (Cloud Foundry — ?namespace=<ns>&alias=<name>):

@Override
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
    String alias = request.getParameter("alias");
    String namespace = request.getParameter("namespace");      // NEW required param

    if (isBlank(alias) || isBlank(namespace)) {
        response.getWriter().println("Alias and namespace must be provided as query parameters.");
        return;
    }

    CredStoreResponse r = credStoreClient.retrievePassword(alias, namespace);
    response.getWriter().println(r.isSuccess()
            ? "Password retrieved successfully."
            : "Failed to retrieve password.");
}

Step 4: Update MTA Descriptor

🛑 HARD RULE — the credstore resource MUST be org.cloudfoundry.existing-service. Do not emit type: org.cloudfoundry.managed-service for credstore, and do not add a parameters.service: credstore / service-plan: block. Getting this wrong is the single most common failure of this migration:

If you emit…	What happens at cf deploy
managed-service (service: credstore)	Broker rejects it — Service broker credstore failed with: Illegal parameters or arguments used — because the mTLS config is set out-of-band, not in the MTA. Deploy fails.
existing-service ✅	Binds to the operator-provisioned instance. Correct.

The credstore instance is created and mTLS-configured outside the MTA (by the space operator, or by CI before deploy). The MTA only binds to it.

modules:
  - name: ${app-name}
    type: java.tomcat
    path: target/<artifactId>.war
    parameters:
      buildpack: sap_java_buildpack_jakarta
      disk-quota: 1024M
      memory: 1024M
    properties:
      ENABLE_SECURITY_JAVA_API_V2: true
      SET_LOGGING_LEVEL: 'ROOT: INFO'
      # Pin SapMachine JRE 25 explicitly. The WAR is compiled to Java 25 (class file
      # version 69); without these env vars the buildpack runs whichever JRE is the
      # implicit default at deploy time (currently 21, class file 65), and Tomcat
      # rejects the servlet with java.lang.UnsupportedClassVersionError. The compile
      # target in pom.xml MUST equal the major version pinned here. Both env vars
      # are required — the first activates SapMachineJRE, the second selects 25.
      JBP_CONFIG_COMPONENTS: "jres: ['com.sap.xs.java.buildpack.jre.SAPMachineJRE']"
      JBP_CONFIG_SAP_MACHINE_JRE: '{ version: 25.+ }'
    requires:
      - name: credstore-service

resources:
  # The credstore service instance and its stored credentials are MANUALLY
  # provisioned and shared across applications in the space — do NOT declare
  # this as a managed-service. SAP BTP caps credstore standard at 1 instance
  # per space, and any namespaces/aliases (the actual secret material) are
  # operator-owned. An MTA that creates the credstore would be deleted by
  # `cf undeploy --delete-services`, taking every other app's credentials
  # with it.
  #
  # Bind to it with `org.cloudfoundry.existing-service`. The resource `name:`
  # below MUST match the actual service-instance name in the target space —
  # by convention `credstore-service`, but verify with `cf services | grep
  # credstore` if you're unsure.
  - name: credstore-service
    type: org.cloudfoundry.existing-service

What this skill does NOT do: create the credstore instance, define its mTLS configuration, or seed namespaces/aliases. Those are one-time per-space setup steps performed by the space operator.

Before provisioning, ask the user:

"Is this a development/testing environment or a production environment?

• For development/testing: use the free plan (10 credentials, 0.1 MB — sufficient for testing)
• For production: use the standard plan (100,000 credentials, 100 MB)"

Then provision accordingly:

# Development/testing:
cf create-service credstore free credstore-service -c '{"authentication":{"type":"mtls"}}'

# Production:
cf create-service credstore standard credstore-service -c '{"authentication":{"type":"mtls"}}'

The migrated app only consumes credentials that are already there.

Plan guidance: The free plan requires the Credential Store entitlement to be configured in the subaccount — add it via BTP Cockpit → Global Account → Entitlements → Add Service Plans → Credential Store → free.

Step 5: Create Credentials in Credential Store

Using BTP Cockpit (Service Marketplace → Credential Store → Manage Instance) or the Credential Store API:

1. Create the namespace. Pick a name that matches what your servlet expects.
2. Add credentials inside that namespace — each addressed by an alias. Add a Password (name + value) for password-storage migrations, or a Key (name + PEM certificate + PEM private key) for keystore-API migrations.

For the integration tests in this repo to pass, use the fixtures the tests expect:

Migration scenario	Namespace	Alias
storing-passwords	pass-storage-app	password named test (any value)
keystore-api	keystore-app	key named keystore-app-key (any cert/key pair)

The validate-migration prompt and KeystoreIntegrationTest / PassStoreIntegrationTest both hit these names verbatim. If your namespace or alias differs, the deployed app will return 404 credential_not_found from the credstore broker (servlet status 200 with body Failed to retrieve credential) and the tests will fail — even though the migration itself is correct.

Configuration Files

No new configuration files required. Credentials are accessed via service binding.

CF Services

Verification

1. Compile Check

mvn clean install

2. Verify Service Binding

cf env ${app-name} | grep -A 20 "credstore"
# Should show url, certificate, and key under VCAP_SERVICES.credstore[0].credentials

3. Verify the Container Picked Up JRE 25

cf ssh ${app-name} -c 'cat app/META-INF/.sap_java_buildpack/sap_machine_jre/release | head -1'
# Expected: JAVA_VERSION="25.0.x"
# If you see 21.x or 17.x, the JBP_CONFIG_SAP_MACHINE_JRE env var from Step 4
# wasn't set or wasn't picked up — the servlet will fail to load with
# UnsupportedClassVersionError on the first request.

4. Test Credential Access

The migrated servlet uses the new namespace-scoped contract from Step 3. The URL must include ?namespace=… — without it the servlet correctly returns 400.

# List all keys in a namespace (alias omitted)
curl -i "https://${app-url}/keystore?namespace=my-app-namespace"
#   → HTTP 200, body: "Successfully retrieved credentials."

# Retrieve a specific key by alias
curl -i "https://${app-url}/keystore?namespace=my-app-namespace&alias=my-certificate"
#   → HTTP 200, body: "Successfully retrieved credential."

# Missing namespace — sanity-check the validation
curl -i "https://${app-url}/keystore"
#   → HTTP 400, body: "Namespace is required!"

For password-storage apps: the URL is ?namespace=<ns>&alias=<password-name> returning Password retrieved successfully. on 200.

Common Issues

These are real failure modes we have hit on this codebase. The cures are listed in roughly the order you would try them.

Common runtime errors → root cause

java.lang.UnsupportedClassVersionError: ... class file version 69.0, this version of the Java Runtime only recognizes class file versions up to 65.0

Symptom: App deploys and starts, but every request hits HTTP 500 (first request) followed by 404 ("servlet marked unavailable"). The stack trace appears in cf logs <app-name> --recent.

Cause: The WAR is compiled to Java 25 (class file version 69) but the deployed container runs SapMachine 21 (class file 65). The buildpack's implicit JRE choice does not match <maven.compiler.target>.

Fix: Add the two JBP_CONFIG_* env vars from Step 4 to the module's properties: block in mtad.yaml. The compile target in pom.xml MUST equal the major version in JBP_CONFIG_SAP_MACHINE_JRE. Re-deploy.

Confirm at runtime with cf ssh <app-name> -c 'cat app/META-INF/.sap_java_buildpack/sap_machine_jre/release | head -1' — it should say JAVA_VERSION="25.0.x".

Service broker error ... Quota is not sufficient for this request, up to 1 standard instance/s for space

Symptom: Deploy fails during Processing service "credstore-service"... with the quota broker error above.

Cause: The mtad declares the credstore as org.cloudfoundry.managed-service, which asks the broker to create a fresh instance. SAP BTP caps credstore standard at 1 instance per space, so the broker rejects the create.

Fix: Change the resource type to org.cloudfoundry.existing-service and remove the parameters: block — the credstore is operator-managed and shared, not owned by this MTA. The result should look exactly like:

  - name: credstore-service
    type: org.cloudfoundry.existing-service

See Step 4 for the full mtad context. If credstore-service doesn't exist in the space at all, ask the user whether this is a development/testing or production environment, then provision accordingly:

# Development/testing:
cf create-service credstore free credstore-service -c '{"authentication":{"type":"mtls"}}'

# Production:
cf create-service credstore standard credstore-service -c '{"authentication":{"type":"mtls"}}'

This skill does NOT create credstore instances or credentials.

Controller operation failed: 404 Not Found: Service instance credstore-service not found

Symptom: Deploy fails almost immediately with the 404 above (or a similar 404 naming whatever value is in the resource's name:), repeated 4× then Process failed.

Cause: Mtad declares org.cloudfoundry.existing-service (correctly), but no instance with that exact name: exists in the target space. Either the operator hasn't provisioned the shared credstore yet, or the name in mtad.yaml doesn't match what's actually in the space.

Fix: Run cf services | grep credstore in the target space. If you see an instance under a different name, update the resource's name: (and every matching requires.name: in the module) to that exact string. If you see no credstore at all, that's an operator setup gap — ask whoever owns the space to provision it as described in Step 5 of this skill. The migration does NOT auto-create credstore instances; the credentials inside are operator-seeded.

Error collecting system parameters: A higher version of your MTA is already deployed

Symptom: Deploy fails immediately after Detected new MTA version.

Cause: Someone deployed the same MTA ID at a higher version: previously (often during local debugging). The MTA deployer rejects downgrades by default.

Fix (preferred): Bump the version: in mtad.yaml above whatever's currently deployed (e.g. 0.0.1 → 1.0.0).

Alternative: Pass --version-rule ALL to cf deploy, but that's a one-off escape hatch — fix the version field for permanence.

Error detaching services from MTA ... CF-ServiceInstanceNotFound: <name>-credstore (4 retries, then Process failed)

Symptom: App and bind both succeed; the deploy script then tries to detach an old service from the previous MTA manifest, retries 4×, and exits 1.

Cause: A prior version of the MTA listed a service the current mtad.yaml no longer references (e.g. pass-store-credstore, keystore-credstore). The MTA's persisted manifest still remembers it and tries to "detach" it on every deploy. The orphan service is in create failed state, so the cleanup hangs.

Fix: Detach the orphan service from the MTA manifest WITHOUT deleting it. The shared credstore-service is operator-managed and may hold credentials that other apps depend on — --delete-services would destroy it.

Detach the specific orphan only:

cf v3-unbind-service <orphan-name> <app-name> 2>/dev/null || true
cf delete-service <orphan-name> -f    # ONLY if you confirmed the orphan is a
                                       # leftover *owned* by this MTA (e.g.
                                       # `pass-store-credstore`), NOT the shared
                                       # `credstore-service`.

For the persisted MTA manifest, use the targeted form:

cf undeploy <mta-id> --delete-service-keys --delete-service-brokers -f

Note the absence of --delete-services — keep the operator-owned credstore-service intact. The next deploy will re-bind to it via existing-service from the descriptor.

SSL handshake failure during CredStoreClient init

Cause: mTLS certificate or key parsing failed.

Fix:

1. Confirm the service binding actually has both certificate and key keys (cf env <app-name> | grep -A 20 credstore).
2. The key must be PEM-encoded PKCS#1 (begins -----BEGIN RSA PRIVATE KEY-----) — the KeyParser in assets/ only accepts PKCS#1. PKCS#8 (-----BEGIN PRIVATE KEY-----) would need a converter; if you see this, regenerate the binding with authentication.type: mtls (see Step 4) — the broker emits PKCS#1 by default.

cannot find symbol: method getUrl()/getKey()/getCertificate()/getMessage()/isSuccess()

Cause: Lombok was added back to a class that was previously Lombok-free. Lombok versions older than ~1.18.36 produce no bytecode under JDK 25, so @Getter-annotated fields silently lose their accessor methods.

Fix: Don't reintroduce Lombok in this skill's helper classes. The shipped versions in assets/ write getters by hand for exactly this reason. If you must use Lombok elsewhere in the project, pin >= 1.18.40.

Security Best Practices

1. Namespace Isolation: Use separate namespaces for different environments
2. Key Rotation: Regularly rotate keys and certificates
3. Least Privilege: Only request credentials that are needed
4. Audit Logging: Enable audit logging in Credential Store
5. Memory Clearing: Clear sensitive data from memory when done

Next Steps

After completing this skill, proceed to other applicable skills:

• ../monitoring-logging/SKILL.md - Monitor credential access