Serving Runtime Strategy

1. Executive Summary

{{ runtime_name }} is the industry standard for {{ key_use_case_or_framework }} model serving. Currently, RHOAI lists {{ runtime_name }} as a "Tested and Verified" (T&V) runtime. However, the lack of a formalized definition and a scoped testing strategy creates a grey area for our field teams and customers. This proposal defines exactly what "Tested and Verified" means, outlines a tiered testing strategy, and identifies the high-value open-source software (OSS) models we should validate to capture the majority of customer use cases, thereby boosting enterprise confidence without violating our out-of-the-box support boundaries.

2. The Problem Context

RHOAI officially supports runtimes like vLLM, Caikit, and OpenVINO. While RHOAI supports the ability to add custom runtimes via the ServingRuntime Custom Resource Definition (CRD), it does not officially support the {{ runtime_name }} container image itself.

However, because {{ runtime_name }} is critical for customers leveraging hardware accelerators ({{ accelerator_types }}) for {{ key_workload_types }} workloads ({{ framework_list }}), we must provide a structured baseline of trust. If we do not define what works, customers will assume either everything works (creating support escalations) or nothing works (causing adoption friction).

3. Proposed Definition: "Tested and Verified" (T&V)

We must clearly demarcate the support boundary. For RHOAI, Tested and Verified for {{ runtime_name }} should be defined as:

*Red Hat guarantees the interoperability, lifecycle management, and secure integration of the {{ runtime_name }} within the RHOAI KServe ecosystem. Red Hat verifies that {{ runtime_name }} can successfully mount storage (PVC/S3), bind to hardware accelerators via the {{ accelerator_operator_name }}, and expose metrics to OpenShift monitoring. Red Hat does not provide bug fixes, SLAs, or code-level support for the upstream {{ runtime_name }} runtime itself.*

4. Testing Strategy & Scope

To back up the T&V claim, our internal team should execute a tiered testing matrix for every stable RHOAI release.

Tier 1: Infrastructure & Lifecycle (The "Must Haves")

• Deployment: Successful creation of the {{ runtime_name }} ServingRuntime via the RHOAI Dashboard and GitOps (ArgoCD).
• Storage Integration: Successful model fetching from S3-compatible storage (data connections) and Persistent Volume Claims (PVCs).
• Hardware Binding: Verification that the {{ runtime_name }} pod successfully requests and binds to {{ accelerator_types }} via the OpenShift {{ accelerator_operator_name }}.

Tier 2: Core Serving & Networking

• Protocol Accessibility: Validation of both {{ primary_protocol }} and {{ secondary_protocol }} endpoints via KServe routing.
• Scaling: Verification of KServe Serverless autoscaling (scale to zero, scale up on load) with the {{ runtime_name }} runtime.

Tier 3: Advanced Differentiators (The "Enterprise Edge")

• {{ runtime_specific_feature_1_name }}: Testing {{ runtime_name }}'s core differentiator—{{ runtime_specific_feature_1_description }}.
• Telemetry: Verification that {{ runtime_name }}'s /metrics endpoint ({{ metrics_port }}) is successfully scraped by OpenShift monitoring and visible in Grafana dashboards.

5. Recommended Model Categories & OSS Models

To test the runtime effectively, we must use models that reflect the 80/20 rule for our customer base. We should focus on the frameworks where {{ runtime_name }} excels: {{ framework_list }}.

A. {{ category_1_name }}

{{ category_1_context_sentence }}

• Model: {{ model_1_1 }} and {{ model_1_2 }}.
• Framework to Test: {{ framework_1_a }} and {{ framework_1_b }}.
• Why: {{ category_1_business_justification }}

B. {{ category_2_name }}

{{ category_2_context_sentence }}

• Model: {{ model_2_1 }}
• Framework to Test: {{ framework_2_a }} and {{ framework_2_b }}.
• Why: {{ category_2_business_justification }}

C. {{ category_3_name }}

• Model: {{ model_3_1 }}.
• Framework to Test: {{ framework_3_backend }} in {{ runtime_name }}.
• Why: {{ category_3_business_justification }}

D. {{ category_4_name }}

• Model: {{ model_4_1 }} & {{ model_4_2 }}.
• Framework to Test: {{ runtime_name }} with the {{ backend_4_a }} or {{ backend_4_b }} backend.
• Why: {{ category_4_business_justification }}

*Note: {{ category_4_name }} testing requires testing the specific {{ container_variants }} container variants, as {{ vendor_name }} has bifurcated containers to reduce bloat.

6. {{ runtime_name }} Release Frequency & Verification of Latest Stable Version

To define a testing strategy, we must align our QE cycles with {{ vendor_name }}'s release cadence.

• Standard Release Cycle: {{ vendor_name }} releases {{ runtime_name }} open-source containers {{ release_frequency }}.
• Enterprise Release Cycle (Strategic Target): For enterprise stability, {{ enterprise_product_name }} releases a {{ enterprise_branch_name }} every {{ enterprise_release_frequency }}.
• Verification of Proposed Models: We must verify that all recommended models and frameworks are fully supported in the latest stable release, but with an important architectural caveat for our testing strategy: {{ architecture_caveat }}.

7. Additional Feature Tests (Recommended/Stakeholder Review)

If QE bandwidth allows, adding the following tests will significantly elevate Red Hat's authority in the AI serving space:

• {{ additional_feature_1_name }}: Crucial for {{ additional_feature_1_workload_type }}. We must verify that {{ runtime_name }} can {{ additional_feature_1_verification_goal }} back to the user interface via KServe.
• {{ additional_feature_2_name }}: Testing {{ runtime_name }}'s ability to host {{ additional_feature_2_description }}.
• {{ additional_feature_3_name }}: Validating that {{ vendor_name }}'s native CLI profiling tools ({{ profiling_tool_name }}) function correctly from within an OpenShift pod to help customers find the optimal {{ configuration_settings }}.

8. Estimated Infrastructure Cost per Test Cycle

To secure buy-in, we need to forecast the cloud infrastructure costs per RHOAI release. Assuming a hybrid testing approach (automated pipelines + targeted manual validation) requiring roughly {{ total_hours }} hours of active compute time per release, here is the AWS On-Demand baseline estimate:

9. Alignment with Red Hat's AI Strategy & Business Value

1. Frictionless On-Ramping: By publishing a verified list of model architectures ({{ verified_model_architectures }}) that work seamlessly on {{ runtime_name }} within RHOAI, we reduce the time-to-value for customers migrating {{ target_workload_migration }} workloads to OpenShift.
2. Ecosystem Play: This strategy strengthens our partnership with {{ vendor_name }}. It signals to the market that while Red Hat champions open-source (e.g., vLLM), we pragmatically embrace industry-standard enterprise tools ({{ runtime_name }}).
3. Deflection of Support Overhead: By clearly documenting what is verified (specific backends and models), support teams can easily deflect issues related to unsupported esoteric models or custom {{ runtime_name }} backends back to the community or {{ vendor_name }}, protecting Red Hat's operating margins.

10. Next Steps for Execution

1. Pin the Version: We should officially pin the "Tested and Verified" tag for the upcoming RHOAI release to the {{ stable_enterprise_branch }} branch (or the stable {{ stable_oss_branch }} open-source branch if the enterprise option is out of scope).
2. Publish a Support Matrix: We publish a Red Hat Knowledgebase (KCS) article clearly mapping the Tier 1 models ({{ tier_1_model_list }}) to the specific {{ runtime_name }} container tags we validated.

Future Investigation & Investment

To turn the observed trends into a concrete engineering and product strategy for Red Hat OpenShift AI (RHOAI), we need to establish specific initiatives for our Quality Engineering (QE) and product management teams.

Here is how we should execute against each trend:

1. Strategy for {{ trend_1_name }}

{{ trend_1_product_name }} are rapidly becoming {{ vendor_name }}'s preferred distribution method for enterprise AI. Because {{ trend_1_product_name }} containers often use {{ runtime_name }} under the hood, standardizing our {{ runtime_name }} integration is step one. Step two is proving {{ trend_1_product_name }} works seamlessly on RHOAI.

• Actionable Recommendation: Initiate a dedicated SPIKE specifically for deploying {{ trend_1_product_name }} containers via KServe in RHOAI.
• Execution Details:
  • Registry Integration: Test and document the process of securely pulling {{ trend_1_product_name }} containers from the {{ vendor_registry }} registry using OpenShift secrets linked to the KServe ServingRuntime.
  • API Key Injection: {{ trend_1_product_name }} often requires an {{ key_environment_variable }} environment variable to initialize. We need to verify that KServe allows secure, seamless injection of this variable at deployment time.
• Target Deliverable: A published Reference Architecture showing a popular {{ trend_1_product_name }} deployed on RHOAI, proving that our {{ runtime_name }} foundational work translates directly to {{ trend_1_product_name }} compatibility.

2. Strategy for {{ trend_2_name }}

As customers build agents, they need to run multiple models in sequence. Passing this data back and forth between the client and server adds massive latency. {{ runtime_name }} solves this natively with {{ advanced_capability }}, which processes the entire pipeline on the server side.

• Actionable Recommendation: Promote {{ runtime_name }} "{{ advanced_capability }}" to a Tier 2 testing requirement in the next fiscal half.
• Execution Details:
  • Pipeline Testing: QE should build a test case featuring a simple inference pipeline (e.g., {{ pipeline_example }}).
  • Documentation: Red Hat can add immense value by providing pre-configured, tested YAML templates for common {{ advanced_capability }} patterns in our documentation.

3. Strategy for FIPS Compliance

Public sector and highly regulated industries are massive revenue drivers for OpenShift. A FIPS-enabled OpenShift cluster loses its compliance edge if the workloads running on it are not compliant.

• Actionable Recommendation: Mandate FIPS-compliant testing in the CI/CD pipeline for the {{ vendor_enterprise_product }} integration.
• Execution Details:
  • Environment: QE must spin up at least one RHOAI test environment on a strictly FIPS-enabled OpenShift cluster.
  • Image Targeting: Explicitly pull the {{ os_base }} STIG/FIPS-ready {{ runtime_name }} images provided by {{ vendor_enterprise_product }}, not the standard open-source images.
• Target Deliverable: An official certification or capability statement confirming that RHOAI can maintain an end-to-end FIPS-compliant boundary when serving models via the {{ vendor_enterprise_product }} runtime.

Draft Customer-Facing KCS Article

Title: Support Scope for "Tested & Verified" Runtimes ({{ runtime_name }}) in Red Hat OpenShift AI

Environment: Red Hat OpenShift AI (RHOAI) Self-Managed and Cloud Service, KServe, ModelMesh, {{ runtime_name }}

Issue

• What is the exact level of Red Hat support for the {{ runtime_name }} when deployed as a "Tested and Verified" runtime in Red Hat OpenShift AI?
• How does a "Tested and Verified" runtime differ from a "Supported" or "Custom" runtime?
• Who provides bug fixes or patches if {{ runtime_name }} fails to serve a specific model?

Resolution

In Red Hat OpenShift AI (RHOAI), model-serving runtimes fall into three distinct tiers: Supported, Tested and Verified, and Custom. The {{ runtime_name }} is officially designated as a Tested and Verified runtime.

1. The "Tested and Verified" Support Boundary

For tested and verified runtimes, Red Hat focuses on validating that the community or third-party software operates reliably on the underlying Red Hat platform (OpenShift and RHOAI). Customers with an active RHOAI subscription can open support tickets regarding {{ runtime_name }} deployments, provided the issue pertains to the Red Hat infrastructure.

For full details, refer to the Tested and verified runtimes for Red Hat OpenShift AI (Article #7089743).

Red Hat provides support for:

• Platform Integration: Successful deployment of the ServingRuntime and InferenceService Custom Resources (CRDs) via the RHOAI dashboard or OpenShift CLI.
• Networking & Routing: Configuration of KServe passthrough routes for {{ primary_protocol }} or {{ secondary_protocol }} endpoints.
• Storage & Infrastructure: Mounting Persistent Volume Claims (PVCs) or S3-compatible data connections so {{ runtime_name }} can pull model files.
• Hardware Acceleration: Ensuring the {{ runtime_name }} pod successfully binds to underlying hardware via the respective OpenShift Operators.
• Telemetry: Ensuring OpenShift monitoring can successfully scrape the /metrics endpoint of the {{ runtime_name }} server.

Red Hat DOES NOT provide support for:

• Software Bugs & Code Fixes: Red Hat does not maintain, distribute, or patch the upstream {{ runtime_name }} container images.
• Model Failures: Issues related to a model failing to load due to incompatible mathematical operations or errors internal to the runtime.
• Custom Backends: Creating or compiling custom backends for {{ runtime_name }}.

2. How this differs from "Custom Runtimes."

As stated in the RHOAI Product Documentation:

"Red Hat does not provide support for custom runtimes. You are responsible for ensuring that you are licensed to use any custom runtimes that you add, and for correctly configuring and maintaining them."

If a user deploys a completely unverified Custom Runtime, Red Hat provides zero support for its configuration, maintenance, or integration.

By adhering to the "Tested and Verified" baseline for {{ runtime_name }} (using the exact configurations provided in Red Hat documentation), customers receive Red Hat's backing for the platform integration layer, which they forfeit if using a purely custom runtime.

3. Escalation Path & Resolution Matrix

To ensure rapid resolution, please follow this escalation path depending on the nature of the error:

1. RHOAI / OpenShift Platform Errors (e.g., Pods failing to schedule, KServe routing 503 errors, Dashboard UI errors):
   • Open a support case with Red Hat Support.
2. {{ runtime_name }} Internal Errors (e.g., Core dumps within the container, specific models failing to compile, CVE vulnerabilities in the {{ runtime_name }} base image):
   • Direct these inquiries to {{ vendor_enterprise_support }} (if using enterprise images) or file an issue in the upstream {{ runtime_name }} [GitHub repository]({{ upstream_github_link }}).

Would you like me to expand on any specific section in more depth, such as the advanced features, the business rationale for the models, or the specific cost breakdown?