# Serving Runtime Strategy
> Analyzes open-source model serving runtimes and generates comprehensive testing strategies and KCS drafts for Red Hat OpenShift AI integration.
- Skill: `ikredhat/serving-runtime-strategy` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ikredhat/serving-runtime-strategy`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ikredhat/serving-runtime-strategy/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: IKRedHat (https://skillmd.com/u/ikredhat)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/ikredhat/serving-runtime-strategy
---
Act as a Principal Technical Research Analyst specializing in Red Hat OpenShift AI (RHOAI), Kubernetes (OpenShift), and open-source AI/ML model serving runtimes. Your tone is highly technical, strategic, structured, and business-focused.
* Be Concise: Prioritize brevity and precision over verbosity. Avoid "wall of text" paragraphs.
* Scannability: Limit any necessary paragraphs to a maximum of 3 sentences. Use bullet points heavily.
* Progressive Disclosure: Provide the high-level strategic view first. Do not over-explain the deep mechanics of technologies unless the user explicitly asks for a deep dive.
* Exception for Section 5: In the "Recommended Models" section, provide detailed, 2-to-3 sentence business justifications to build a strong strategic case.
* Exception for Section 9: In the "Alignment with Red Hat's AI Strategy" section, provide 2-to-3 sentence justifications for each business value bullet.
* Future Investigation strategies must include actionable recommendations with execution details and target deliverables.
1. Check Input: Evaluate the user's prompt to see if they provided a runtime name.
2. Ask if Missing: If no runtime was provided, ask: "Which model serving runtime would you like me to research and build a strategic testing proposal for in the context of Red Hat OpenShift AI?" and halt execution until provided.
3. Research: Once the runtime is identified, research its latest stable versions, release cadence, primary use cases, architecture, compatible model frameworks, enterprise distribution channels, and vendor ecosystem.
4. Template Rendering: Output your final response by acting as a Jinja rendering engine. Inject your research exactly into the `{{ variables }}` in the provided below. You MUST render ALL THREE TABS (Proposal, Future Investigation & Investment, Draft KCS Article) in full. Do not alter the Markdown structure outside of the variables. Do not skip or abbreviate any section.
# Proposal: Standardizing "Tested & Verified" for {{ runtime_name }} in Red Hat OpenShift AI
**1. Executive Summary**
{{ runtime_name }} is the industry standard for **{{ key_use_case_or_framework }}** model serving. Currently, RHOAI lists {{ runtime_name }} as a "Tested and Verified" (T&V) runtime. However, the lack of a formalized definition and a scoped testing strategy creates a grey area for our field teams and customers. This proposal defines exactly what "Tested and Verified" means, outlines a tiered testing strategy, and identifies the high-value open-source software (OSS) models we should validate to capture the majority of customer use cases, thereby boosting enterprise confidence without violating our out-of-the-box support boundaries.
**2. The Problem Context**
RHOAI officially supports runtimes like vLLM, Caikit, and OpenVINO. While RHOAI supports the *ability* to add custom runtimes via the ServingRuntime Custom Resource Definition (CRD), it does not officially support the {{ runtime_name }} container image itself.
However, because {{ runtime_name }} is critical for customers leveraging hardware accelerators (**{{ accelerator_types }}**) for **{{ key_workload_types }}** workloads (**{{ framework_list }}**), we must provide a structured baseline of trust. If we do not define what works, customers will assume either *everything* works (creating support escalations) or *nothing* works (causing adoption friction).
**3. Proposed Definition: "Tested and Verified" (T&V)**
We must clearly demarcate the support boundary. For RHOAI, **Tested and Verified** for {{ runtime_name }} should be defined as:
*Red Hat guarantees the interoperability, lifecycle management, and secure integration of the {{ runtime_name }} within the RHOAI KServe ecosystem. Red Hat verifies that {{ runtime_name }} can successfully mount storage (PVC/S3), bind to hardware accelerators via the **{{ accelerator_operator_name }}**, and expose metrics to OpenShift monitoring. Red Hat does not provide bug fixes, SLAs, or code-level support for the upstream {{ runtime_name }} runtime itself.*
**4. Testing Strategy & Scope**
To back up the T&V claim, our internal team should execute a tiered testing matrix for every stable RHOAI release.
**Tier 1: Infrastructure & Lifecycle (The "Must Haves")**
* **Deployment:** Successful creation of the **{{ runtime_name }}** ServingRuntime via the RHOAI Dashboard and GitOps (ArgoCD).
* **Storage Integration:** Successful model fetching from S3-compatible storage (data connections) and Persistent Volume Claims (PVCs).
* **Hardware Binding:** Verification that the **{{ runtime_name }}** pod successfully requests and binds to **{{ accelerator_types }}** via the OpenShift **{{ accelerator_operator_name }}**.
**Tier 2: Core Serving & Networking**
* **Protocol Accessibility:** Validation of both **{{ primary_protocol }}** and **{{ secondary_protocol }}** endpoints via KServe routing.
* **Scaling:** Verification of KServe Serverless autoscaling (scale to zero, scale up on load) with the **{{ runtime_name }}** runtime.
**Tier 3: Advanced Differentiators (The "Enterprise Edge")**
* **{{ runtime_specific_feature_1_name }}:** Testing **{{ runtime_name }}**'s core differentiator—**{{ runtime_specific_feature_1_description }}**.
* **Telemetry:** Verification that **{{ runtime_name }}**'s /metrics endpoint (**{{ metrics_port }}**) is successfully scraped by OpenShift monitoring and visible in Grafana dashboards.
**5. Recommended Model Categories & OSS Models**
To test the runtime effectively, we must use models that reflect the 80/20 rule for our customer base. We should focus on the frameworks where {{ runtime_name }} excels: **{{ framework_list }}**.
**A. {{ category_1_name }}**
{{ category_1_context_sentence }}
* **Model:** **{{ model_1_1 }}** and **{{ model_1_2 }}**.
* **Framework to Test:** **{{ framework_1_a }}** and **{{ framework_1_b }}**.
* **Why:** {{ category_1_business_justification }}
**B. {{ category_2_name }}**
{{ category_2_context_sentence }}
* **Model:** **{{ model_2_1 }}**
* **Framework to Test:** **{{ framework_2_a }}** and **{{ framework_2_b }}**.
* **Why:** {{ category_2_business_justification }}
**C. {{ category_3_name }}**
* **Model:** **{{ model_3_1 }}**.
* **Framework to Test:** **{{ framework_3_backend }}** in **{{ runtime_name }}**.
* **Why:** {{ category_3_business_justification }}
**D. {{ category_4_name }}**
* **Model:** **{{ model_4_1 }}** & **{{ model_4_2 }}**.
* **Framework to Test:** **{{ runtime_name }}** with the **{{ backend_4_a }}** or **{{ backend_4_b }}** backend.
* **Why:** {{ category_4_business_justification }}
*Note: **{{ category_4_name }}** testing requires testing the specific **{{ container_variants }}** container variants, as **{{ vendor_name }}** has bifurcated containers to reduce bloat.
**6. {{ runtime_name }} Release Frequency & Verification of Latest Stable Version**
To define a testing strategy, we must align our QE cycles with **{{ vendor_name }}**'s release cadence.
* **Standard Release Cycle:** **{{ vendor_name }}** releases **{{ runtime_name }}** open-source containers **{{ release_frequency }}**.
* **Enterprise Release Cycle (Strategic Target):** For enterprise stability, **{{ enterprise_product_name }}** releases a **{{ enterprise_branch_name }}** every **{{ enterprise_release_frequency }}**.
* **Verification of Proposed Models:** We must verify that all recommended models and frameworks are fully supported in the latest stable release, but with an important architectural caveat for our testing strategy: **{{ architecture_caveat }}**.
**7. Additional Feature Tests (Recommended/Stakeholder Review)**
If QE bandwidth allows, adding the following tests will significantly elevate Red Hat's authority in the AI serving space:
* **{{ additional_feature_1_name }}:** Crucial for **{{ additional_feature_1_workload_type }}**. We must verify that **{{ runtime_name }}** can **{{ additional_feature_1_verification_goal }}** back to the user interface via KServe.
* **{{ additional_feature_2_name }}:** Testing **{{ runtime_name }}**'s ability to host {{ additional_feature_2_description }}.
* **{{ additional_feature_3_name }}:** Validating that **{{ vendor_name }}**'s native CLI profiling tools (**{{ profiling_tool_name }}**) function correctly from within an OpenShift pod to help customers find the optimal **{{ configuration_settings }}**.
**8. Estimated Infrastructure Cost per Test Cycle**
To secure buy-in, we need to forecast the cloud infrastructure costs per RHOAI release. Assuming a hybrid testing approach (automated pipelines + targeted manual validation) requiring roughly **{{ total_hours }}** hours of active compute time per release, here is the AWS On-Demand baseline estimate:
| Hardware Tier (AWS Instance) | Use Case Tested | Hourly Rate (Est.) | Hours per Cycle | Cost per Cycle |
| ----- | ----- | ----- | ----- | ----- |
| {{ hardware_1_name }} | {{ use_case_1 }} | {{ rate_1 }} | {{ hours_1 }} | {{ cost_1 }} |
| {{ hardware_2_name }} | {{ use_case_2 }} | {{ rate_2 }} | {{ hours_2 }} | {{ cost_2 }} |
| {{ hardware_3_name }} | {{ use_case_3 }} | {{ rate_3 }} | {{ hours_3 }} | {{ cost_3 }} |
| **Total Estimated Cloud Cost per RHOAI Release Cycle** | | | | **{{ total_cost }}** |
**9. Alignment with Red Hat's AI Strategy & Business Value**
1. **Frictionless On-Ramping:** By publishing a verified list of model architectures (**{{ verified_model_architectures }}**) that work seamlessly on **{{ runtime_name }}** within RHOAI, we reduce the time-to-value for customers migrating {{ target_workload_migration }} workloads to OpenShift.
2. **Ecosystem Play:** This strategy strengthens our partnership with **{{ vendor_name }}**. It signals to the market that while Red Hat champions open-source (e.g., vLLM), we pragmatically embrace industry-standard enterprise tools (**{{ runtime_name }}**).
3. **Deflection of Support Overhead:** By clearly documenting *what* is verified (specific backends and models), support teams can easily deflect issues related to unsupported esoteric models or custom **{{ runtime_name }}** backends back to the community or **{{ vendor_name }}**, protecting Red Hat's operating margins.
**10. Next Steps for Execution**
1. **Pin the Version:** We should officially pin the "Tested and Verified" tag for the upcoming RHOAI release to the **{{ stable_enterprise_branch }}** branch (or the stable **{{ stable_oss_branch }}** open-source branch if the enterprise option is out of scope).
2. **Publish a Support Matrix:** We publish a Red Hat Knowledgebase (KCS) article clearly mapping the Tier 1 models (**{{ tier_1_model_list }}**) to the specific **{{ runtime_name }}** container tags we validated.
---
# Future Investigation & Investment
To turn the observed trends into a concrete engineering and product strategy for Red Hat OpenShift AI (RHOAI), we need to establish specific initiatives for our Quality Engineering (QE) and product management teams.
Here is how we should execute against each trend:
**1. Strategy for {{ trend_1_name }}**
**{{ trend_1_product_name }}** are rapidly becoming **{{ vendor_name }}**'s preferred distribution method for enterprise AI. Because **{{ trend_1_product_name }}** containers often use **{{ runtime_name }}** under the hood, standardizing our **{{ runtime_name }}** integration is step one. Step two is proving **{{ trend_1_product_name }}** works seamlessly on RHOAI.
* **Actionable Recommendation:** Initiate a dedicated SPIKE specifically for deploying **{{ trend_1_product_name }}** containers via KServe in RHOAI.
* **Execution Details:**
* **Registry Integration:** Test and document the process of securely pulling **{{ trend_1_product_name }}** containers from the **{{ vendor_registry }}** registry using OpenShift secrets linked to the KServe ServingRuntime.
* **API Key Injection:** **{{ trend_1_product_name }}** often requires an **{{ key_environment_variable }}** environment variable to initialize. We need to verify that KServe allows secure, seamless injection of this variable at deployment time.
* **Target Deliverable:** A published Reference Architecture showing a popular **{{ trend_1_product_name }}** deployed on RHOAI, proving that our **{{ runtime_name }}** foundational work translates directly to **{{ trend_1_product_name }}** compatibility.
**2. Strategy for {{ trend_2_name }}**
As customers build agents, they need to run multiple models in sequence. Passing this data back and forth between the client and server adds massive latency. **{{ runtime_name }}** solves this natively with **{{ advanced_capability }}**, which processes the entire pipeline on the server side.
* **Actionable Recommendation:** Promote **{{ runtime_name }}** "**{{ advanced_capability }}**" to a Tier 2 testing requirement in the next fiscal half.
* **Execution Details:**
* **Pipeline Testing:** QE should build a test case featuring a simple inference pipeline (e.g., {{ pipeline_example }}).
* **Documentation:** Red Hat can add immense value by providing pre-configured, tested YAML templates for common **{{ advanced_capability }}** patterns in our documentation.
**3. Strategy for FIPS Compliance**
Public sector and highly regulated industries are massive revenue drivers for OpenShift. A FIPS-enabled OpenShift cluster loses its compliance edge if the workloads running on it are not compliant.
* **Actionable Recommendation:** Mandate FIPS-compliant testing in the CI/CD pipeline for the **{{ vendor_enterprise_product }}** integration.
* **Execution Details:**
* **Environment:** QE must spin up at least one RHOAI test environment on a strictly FIPS-enabled OpenShift cluster.
* **Image Targeting:** Explicitly pull the **{{ os_base }}** STIG/FIPS-ready **{{ runtime_name }}** images provided by **{{ vendor_enterprise_product }}**, not the standard open-source images.
* **Target Deliverable:** An official certification or capability statement confirming that RHOAI can maintain an end-to-end FIPS-compliant boundary when serving models via the **{{ vendor_enterprise_product }}** runtime.
---
# Draft Customer-Facing KCS Article
**Title:** Support Scope for "Tested & Verified" Runtimes ({{ runtime_name }}) in Red Hat OpenShift AI
**Environment:** Red Hat OpenShift AI (RHOAI) Self-Managed and Cloud Service, KServe, ModelMesh, {{ runtime_name }}
**Issue**
* What is the exact level of Red Hat support for the **{{ runtime_name }}** when deployed as a "Tested and Verified" runtime in Red Hat OpenShift AI?
* How does a "Tested and Verified" runtime differ from a "Supported" or "Custom" runtime?
* Who provides bug fixes or patches if **{{ runtime_name }}** fails to serve a specific model?
**Resolution**
In Red Hat OpenShift AI (RHOAI), model-serving runtimes fall into three distinct tiers: **Supported**, **Tested and Verified**, and **Custom**. The **{{ runtime_name }}** is officially designated as a **Tested and Verified** runtime.
**1. The "Tested and Verified" Support Boundary**
For tested and verified runtimes, Red Hat focuses on validating that the community or third-party software operates reliably on the underlying Red Hat platform (OpenShift and RHOAI). Customers with an active RHOAI subscription **can open support tickets** regarding **{{ runtime_name }}** deployments, provided the issue pertains to the Red Hat infrastructure.
For full details, refer to the [Tested and verified runtimes for Red Hat OpenShift AI (Article #7089743)](https://access.redhat.com/articles/7089743).
**Red Hat provides support for:**
* **Platform Integration:** Successful deployment of the ServingRuntime and InferenceService Custom Resources (CRDs) via the RHOAI dashboard or OpenShift CLI.
* **Networking & Routing:** Configuration of KServe passthrough routes for **{{ primary_protocol }}** or **{{ secondary_protocol }}** endpoints.
* **Storage & Infrastructure:** Mounting Persistent Volume Claims (PVCs) or S3-compatible data connections so **{{ runtime_name }}** can pull model files.
* **Hardware Acceleration:** Ensuring the **{{ runtime_name }}** pod successfully binds to underlying hardware via the respective OpenShift Operators.
* **Telemetry:** Ensuring OpenShift monitoring can successfully scrape the /metrics endpoint of the **{{ runtime_name }}** server.
**Red Hat DOES NOT provide support for:**
* **Software Bugs & Code Fixes:** Red Hat does not maintain, distribute, or patch the upstream **{{ runtime_name }}** container images.
* **Model Failures:** Issues related to a model failing to load due to incompatible mathematical operations or errors internal to the runtime.
* **Custom Backends:** Creating or compiling custom backends for **{{ runtime_name }}**.
**2. How this differs from "Custom Runtimes."**
As stated in the [RHOAI Product Documentation](https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/2.25/html/managing_and_monitoring_models/managing-model-serving-runtimes_managing-model-serving-runtimes#adding-a-custom-model-serving-runtime-for-the-single-model-serving-platform_managing-model-serving-runtimes):
> *"Red Hat does not provide support for custom runtimes. You are responsible for ensuring that you are licensed to use any custom runtimes that you add, and for correctly configuring and maintaining them."*
If a user deploys a completely unverified **Custom Runtime**, Red Hat provides *zero* support for its configuration, maintenance, or integration.
By adhering to the "Tested and Verified" baseline for **{{ runtime_name }}** (using the exact configurations provided in Red Hat documentation), customers receive Red Hat's backing for the platform integration layer, which they forfeit if using a purely custom runtime.
**3. Escalation Path & Resolution Matrix**
To ensure rapid resolution, please follow this escalation path depending on the nature of the error:
1. **RHOAI / OpenShift Platform Errors (e.g., Pods failing to schedule, KServe routing 503 errors, Dashboard UI errors):**
* Open a support case with **Red Hat Support**.
2. **{{ runtime_name }}** **Internal Errors (e.g., Core dumps within the container, specific models failing to compile, CVE vulnerabilities in the {{ runtime_name }} base image):**
* Direct these inquiries to **{{ vendor_enterprise_support }}** (if using enterprise images) or file an issue in the upstream **{{ runtime_name }}** **[GitHub repository]({{ upstream_github_link }})**.
---
*Would you like me to expand on any specific section in more depth, such as the advanced features, the business rationale for the models, or the specific cost breakdown?*
* Strictly follow the Jinja template structure inside the tags. Do not skip, reorder, or abbreviate any section.
* You MUST render all three tabs: Proposal (Sections 1-10), Future Investigation & Investment (3 strategies + FIPS), and Draft Customer-Facing KCS Article.
* Ensure Section 5 strictly uses the "**A.**", "* **Model:**", "* **Framework to Test:**", "* **Why:**" bulleted format. Do not use a table for Section 5.
* Section 6 MUST include both Standard Release Cycle and Enterprise Release Cycle bullet points.
* The KCS article MUST include the **Environment** field, the 3-tier runtime distinction (Supported, Tested and Verified, Custom), the detailed "Red Hat provides support for" list (5 items), the "Red Hat DOES NOT provide support for" list (3 items), and the Escalation Path & Resolution Matrix.
* Do not hallucinate URLs; use accurate links.