# Salesforce Multicloudj Multicloudj > MultiCloudJ End-to-End Feature Development - Skill: `tomevault-io/salesforce-multicloudj-multicloudj` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/salesforce-multicloudj-multicloudj` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/salesforce-multicloudj-multicloudj/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/salesforce-multicloudj-multicloudj --- # MultiCloudJ End-to-End Feature Development ## Overview Implements a new feature across the multicloudj SDK through structured phases: requirements gathering via user interview, cross-cloud research for semantic uniformity, client/driver API design, provider implementations, unit tests, and conformance tests with WireMock record/replay. ## Provider Isolation Principle **CRITICAL:** Each provider implementation (AWS, GCP, Alibaba) is completely independent and must NEVER reference another provider in any way. ### The Rule Providers are isolated islands that: - Each independently implements the abstract driver contract (e.g., `AbstractBlobStore`, `AbstractDocstore`) - Have NO knowledge of other provider implementations - Have NO dependencies on other provider modules - Have NO comparisons or references to other providers in code OR comments ### What This Means in Practice **NEVER:** - Import classes from another provider module (e.g., `AwsBlobStore` importing from `blob-gcp`) - Add dependencies between provider modules in `pom.xml` - Write comments like "Unlike GCP, AWS does X" or "Similar to Alibaba's approach" - Write comments like "GCP handles this differently" or "AWS uses a different pattern" - Copy code between providers with comments referencing the source - Compare implementation approaches in javadoc or inline comments **DO:** - Each provider implements the driver contract independently - Document WHY a provider does something based on that provider's SDK behavior - Focus comments on the cloud provider's native SDK, not other multicloudj providers - Let conformance tests verify behavioral parity, not implementation similarity ### Example: WRONG ❌ ```java // AWS implementation @Override protected PutObjectResponse doPutObject(PutObjectRequest request) { // Unlike GCP which uses resumable uploads, AWS uses multipart // Similar to how AliOss handles large objects return s3Client.putObject(...); } ``` ### Example: CORRECT ✅ ```java // AWS implementation @Override protected PutObjectResponse doPutObject(PutObjectRequest request) { // S3 automatically handles objects up to 5GB in a single PUT operation return s3Client.putObject(...); } ``` ### Why Provider Isolation Matters 1. **Maintainability:** Each provider evolves independently based on its cloud SDK changes 2. **Testability:** Provider tests don't break when other providers change 3. **Clarity:** Implementation decisions are based on the provider's SDK, not other providers 4. **Extensibility:** New providers can be added without touching existing ones 5. **Semantic Uniformity:** The driver contract (abstract class) defines the unified behavior, not cross-provider comparisons The driver contract and conformance tests ensure all providers behave the same for the end user. Providers achieve this independently, not by copying or comparing with each other. ## When to Use - Adding a new operation or capability to an existing service (blob, docstore, pubsub, sts) - Adding a new service module end-to-end - User says "add feature", "implement X operation", "support Y across providers" ## When NOT to Use - Bug fixes to existing implementations (use systematic-debugging) - Refactoring without behavior change - Documentation-only changes (use docs-guides) ## Process Flow ```mermaid flowchart TD A([User request received]) --> B[Phase 1: Interview] B --> C[Phase 2: Cross-cloud research] C --> D{Semantic differences?} D -->|yes| E[Interview user on resolution] D -->|no| F[Phase 3: API design] E --> F F --> G{User approves API?} G -->|yes| H[Phase 4: Provider implementations] G -->|revise| F H --> I[Phase 5: Unit tests] I --> J[Phase 6: Conformance tests] J --> K{Run record mode?} K -->|yes| L[Collect credentials and record] K -->|later| M[Write abstract IT only] L --> N[Phase 7: Create PR] M --> N N --> O([Done]) ``` ## Phase 1: User Interview Gather requirements before any code. Ask the user: 1. **What service?** (blob, docstore, pubsub, sts, or new) 2. **What operation?** (e.g., "object versioning", "batch delete with filters", "topic dead-letter queue") 3. **What's the user-facing behavior?** (what should the end-user SDK call look like?) 4. **Which providers?** (AWS/GCP/Ali - default is all three) 5. **Are there known provider differences?** (the user may already know gotchas) 6. **Any constraints?** (backwards compatibility, performance, specific SDK versions) Do NOT proceed until you have clear answers to at least questions 1-3. ## Phase 2: Cross-Cloud Research Research how each cloud provider implements this feature natively. For each provider (AWS, GCP, Alibaba): 1. **Read the cloud SDK documentation** - understand the native API surface 2. **Identify the SDK classes/methods** that implement this feature 3. **Note semantic differences** between providers: - Different parameter names or types - Different behavior for edge cases (nulls, empty lists, limits) - Features supported by some providers but not others - Different error codes/conditions for the same failure **Present findings to user as a comparison table:** | Aspect | AWS | GCP | Alibaba | |--------|-----|-----|---------| | API method | `s3.putObjectRetention()` | `storage.objects().update()` | `oss.setBucketLifecycle()` | | Supports X | Yes | Partial | No | | Error on Y | 403 | 404 | 400 | ### Resolving Semantic Differences If providers differ, present the user with options: - **Option A: Lowest common denominator** - only expose what all providers support - **Option B: Best-effort with capability flags** - expose all, throw `UnsupportedOperationException` where unavailable - **Option C: Semantic translation** - map different provider behaviors to a unified model Interview the user on which approach to take. The goal is **semantic uniformity for the end user** - they should not need to know which cloud they're running on. ## Phase 3: API Design Design the cloud-agnostic API. This involves changes to the `-client` module: ### 3a. Request/Response Objects Create in the client module's appropriate package: ``` {service}/{service}-client/src/main/java/com/salesforce/multicloudj/{service}/driver/ ``` Follow existing patterns: - Request objects use Builder pattern with `Builder` inner class - Response objects are immutable with getters - Use `@Getter` from Lombok where the module already uses it ### 3b. Abstract Driver Method Add the abstract `do*` method to the abstract class (e.g., `AbstractBlobStore`): ```java protected abstract {ResponseType} do{OperationName}({RequestType} request); ``` ### 3c. Public Client Method Add the public method to the client class (e.g., `BucketClient`): ```java public {ResponseType} {operationName}({RequestType} request) { return multiCloudJLogger.traceOperation( "{operationName}", () -> blobStore.{operationName}(request) ); } ``` ### 3d. Validation Add validation rules to the validator class (e.g., `BlobStoreValidator`) if inputs need validation. **Present the API design to the user for approval before implementing providers.** ## Phase 4: Provider Implementations For each provider (aws, gcp, ali): ### 4a. Implement the `do*` Method In the provider's main class (e.g., `AwsBlobStore`): ```java @Override protected {ResponseType} do{OperationName}({RequestType} request) { // 1. Transform multicloudj request → provider SDK request // 2. Call provider SDK // 3. Transform provider SDK response → multicloudj response // 4. Handle provider-specific exceptions via getException() } ``` ### 4b. Add Transformer Methods In the provider's transformer class (e.g., `AwsTransformer`): - `to{ProviderRequest}({MulticloudRequest})` - converts outgoing request - `to{MulticloudResponse}({ProviderResponse})` - converts incoming response ### 4c. Exception Mapping Update `ErrorCodeMapping` (or equivalent) if the new operation introduces new error conditions. ### 4d. Build and Verify ```bash mvn clean install -DskipTests mvn test -pl {service}/{service}-{provider} ``` ## Phase 5: Unit Tests Write unit tests that verify real behavior, not fake coverage. ### What Makes a Real Unit Test - **Tests behavior, not implementation** - verify the output given an input - **Mocks the cloud SDK client** (S3Client, Storage, etc.), NOT internal classes - **Covers error paths** - exception mapping, validation failures, edge cases - **Tests transformer logic** - correct field mapping between types ### What Is Fake Coverage (DO NOT WRITE) - Tests that only verify a method was called without checking correctness - Tests that pass null/empty and assert no exception (unless that's the contract) - Tests that duplicate the implementation logic in assertions - Tests with no meaningful assertions ### Run Unit Tests ```bash mvn clean install -DskipTests mvn test -pl {service}/{service}-aws -Dtest={TestClassName} mvn test -pl {service}/{service}-gcp -Dtest={TestClassName} ``` ## Phase 6: Conformance Tests Conformance tests ensure all providers behave identically for the same operation. ### 6a. Abstract Conformance Test Add test methods to the existing abstract IT class in the client module (e.g., `AbstractBlobStoreIT`): ``` {service}/{service}-client/src/test/java/.../Abstract{Service}StoreIT.java ``` Every conformance test must include both **positive** and **negative** scenarios: - **Positive scenario**: Happy path — the operation succeeds with valid inputs and the result is correct - **Negative scenario**: Error path — the operation fails gracefully with invalid inputs (e.g., nonexistent key, invalid parameters) and throws the expected exception This ensures all providers handle both success and failure consistently. ### 6b. Run in Record Mode (Local with Credentials) Record mode captures real HTTP interactions with the cloud provider. **Ask the user for credentials:** - **AWS:** `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` - **GCP:** `GOOGLE_APPLICATION_CREDENTIALS` (path to service account JSON) - **Alibaba:** Handled separately on special machines - skip record mode for Ali and override in the Ali{Service}IT and add @Disabled so they are not executed. **Only run the specific tests impacted by your change** - never record the entire IT suite. This keeps the PR minimal and avoids regenerating unrelated mapping files. ```bash # AWS record - only the new/changed test method export AWS_ACCESS_KEY_ID="{from_user}" export AWS_SECRET_ACCESS_KEY="{from_user}" export AWS_SESSION_TOKEN="{from_user}" mvn test -pl {service}/{service}-aws -Dtest="{Provider}{Service}StoreIT#{testMethodName}" -Drecord # GCP record - only the new/changed test method export GOOGLE_APPLICATION_CREDENTIALS="{path_from_user}" mvn test -pl {service}/{service}-gcp -Dtest="{Provider}{Service}StoreIT#{testMethodName}" -Drecord ``` After recording: - Verify new mapping files appear in `src/test/resources/mappings/` - Verify no sensitive credentials leaked into mapping files - **NEVER manually edit or create mapping files** - all mappings must be generated by WireMock recording. If stale files exist from a failed run, delete them and re-record. - Commit the mapping files (they enable CI replay mode) ### 6c. Run in Replay Mode (CI - Default) ```bash # No credentials needed - uses recorded WireMock stubs mvn test -pl {service}/{service}-aws -Dtest="{Provider}{Service}StoreIT#{testMethodName}" mvn test -pl {service}/{service}-gcp -Dtest="{Provider}{Service}StoreIT#{testMethodName}" ``` ### 6d. Alibaba Conformance Tests Write the abstract conformance test (it runs for all providers). The Ali-specific IT class extends it just like AWS/GCP, but recording happens on dedicated machines. The IT harness class already exists (e.g., `AliBlobStoreIT`) — do not attempt to record locally for Ali. ## Phase 7: Create Pull Request After implementation and testing are complete, create a PR following MultiCloudJ conventions. ### 7a. Commit Changes ```bash # Stage all changes git add -A # Create commit with descriptive message git commit -m "{service}: {brief description of feature} {Detailed description of what was changed and why} - Added X to Y - Updated Z for compatibility - Tests: {description of tests added} " ``` ### 7b. Push Branch ```bash # Push to origin git push origin {branch-name} ``` ### 7c. Create PR with Proper Title **PR Title Format:** `{service}: {concise description}` Examples: - `sts: onboard proxy configurations for sts interface` - `blob: add object versioning support` - `docstore: implement batch delete with filters` - `pubsub: add dead-letter queue configuration` **PR Description Template:** ```markdown ## Summary {1-2 sentence overview of the feature} ## Changes - Added `{MethodName}` to `{ClassName}` - Implemented {feature} for AWS, GCP, and Alibaba Cloud - Added unit tests for {components} - {If applicable} Added conformance tests with WireMock recording ## API Example ```java // Show how users will call the new API {ServiceClient} client = {ServiceClient}.builder("aws") .withRegion("us-west-2") .{newMethod}({parameters}) .build(); ``` ## Testing - Unit tests: `mvn test -pl {service}/{service}-{provider}` - {If applicable} Conformance tests: recorded for AWS/GCP, replay mode verified ## Cross-Cloud Compatibility - AWS: {describe implementation or limitations} - GCP: {describe implementation or limitations} - Alibaba: {describe implementation or limitations} ``` ### 7d. Verify PR Checklist Before submitting, verify: - [ ] PR title follows `{service}: {description}` format - [ ] All unit tests pass locally - [ ] Conformance tests pass (if applicable) - [ ] Checkstyle passes (Google Java Style) - [ ] No provider-specific code in `-client` modules - [ ] Documentation/examples added (if needed) - [ ] WireMock mappings committed (if recorded) - [ ] No credentials leaked in mapping files ## File Checklist For a feature added to an existing service (e.g., blob), you will typically touch: | Layer | File | Change | |-------|------|--------| | Request/Response | `{service}-client/.../driver/{Request}.java` | New file | | Request/Response | `{service}-client/.../driver/{Response}.java` | New file | | Abstract driver | `{service}-client/.../driver/Abstract{Store}.java` | Add `do*` method | | Public API | `{service}-client/.../client/{Client}.java` | Add public method | | Validator | `{service}-client/.../driver/{Validator}.java` | Add validation (if needed) | | AWS impl | `{service}-aws/.../Aws{Store}.java` | Implement `do*` | | AWS transformer | `{service}-aws/.../AwsTransformer.java` | Add conversion methods | | GCP impl | `{service}-gcp/.../Gcp{Store}.java` | Implement `do*` | | GCP transformer | `{service}-gcp/.../GcpTransformer.java` | Add conversion methods | | Ali impl | `{service}-ali/.../Ali{Store}.java` | Implement `do*` | | Ali transformer | `{service}-ali/.../AliTransformer.java` | Add conversion methods | | AWS unit test | `{service}-aws/.../{FeatureOrComponent}Test.java` | New or updated | | GCP unit test | `{service}-gcp/.../{FeatureOrComponent}Test.java` | New or updated | | Ali unit test | `{service}-ali/.../{FeatureOrComponent}Test.java` | New or updated | | Conformance test | `{service}-client/.../Abstract{Service}StoreIT.java` | Add test methods | | AWS IT harness | `{service}-aws/.../Aws{Service}StoreIT.java` | Already exists | | GCP IT harness | `{service}-gcp/.../Gcp{Service}StoreIT.java` | Already exists | | Ali IT harness | `{service}-ali/.../Ali{Service}StoreIT.java` | Already exists | | AWS mappings | `{service}-aws/src/test/resources/mappings/` | Recorded stubs | | GCP mappings | `{service}-gcp/src/test/resources/mappings/` | Recorded stubs | ## Common Mistakes - **Adding provider-specific types in client module** - the client module must never import AWS/GCP/Ali SDKs - **Skipping the user interview** - assumptions about semantics lead to rework - **Implementing before researching all three clouds** - discovering a provider can't support the feature after implementation is costly - **Writing conformance tests that are provider-specific** - the abstract IT must work for ALL providers - **Leaking credentials in WireMock recordings** - always verify mapping files before committing - **Writing unit tests that only test happy path** - error mapping and edge cases are where bugs hide - **Not running `mvn clean install -DskipTests` before running individual module tests** - inter-module dependencies need to be built first - **Manually editing or creating WireMock mapping files** - all mappings must be produced by WireMock recording only. If a recording is stale or corrupt, delete it and re-record. - **Recording the entire IT suite instead of just the new test** - only run the specific impacted test method in record mode to keep the PR diff minimal ## Red Flags - STOP and Reconsider - You're implementing a provider without having researched how the other providers handle it - You're adding a method to the abstract class without a corresponding public client method - You're writing a conformance test that uses provider-specific setup - You're importing `software.amazon.awssdk` or `com.google.cloud` in a `-client` module - You're writing a unit test with no assertions about returned values or thrown exceptions - You're running record mode without asking the user for credentials first - You're about to manually edit a WireMock mapping JSON file - STOP: delete and re-record instead - You're running the full IT class in record mode when only one test method is new/changed - use `#testMethodName` filter - **You're writing a comment in a provider that references another provider** (e.g., "Unlike GCP", "Similar to AWS") - STOP: providers must be isolated - **You're importing classes from another provider module** (e.g., `blob-aws` importing from `blob-gcp`) - STOP: providers cannot depend on each other --- > Source: [salesforce/multicloudj](https://github.com/salesforce/multicloudj) — distributed by [TomeVault](https://tomevault.io).