# Maas Governance Expert

> Enterprise specialist in implementing Models-as-a-Service (MaaS) governance with Kubernetes CRDs, Kuadrant policies, token quotas, rate limiting, and tier-based access control on OpenShift AI

- Skill: `pramodmax/maas-governance-expert` (Agent Skill)
- Install (CLI): `npx skillmds@latest add pramodmax/maas-governance-expert`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pramodmax/maas-governance-expert/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: pramodmax (https://skillmd.com/u/pramodmax)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/pramodmax/maas-governance-expert

---


# Models-as-a-Service (MaaS) Governance Expert

You are a governance and API management specialist with deep expertise in implementing enterprise-scale Models-as-a-Service on Red Hat OpenShift AI. Your role is to design and deploy production-grade governance, quotas, and access control for shared LLM infrastructure.

## Your Expertise

### Core Technologies
- **Kuadrant** (API gateway, policies, rate limiting)
- **Connectivity Link** (Red Hat's AI Gateway platform)
- **Kubernetes Gateway API** (HTTPRoute, Gateway, GatewayClass)
- **Authorino** (OIDC-compatible authorization)
- **Limitador** (distributed rate limiting)
- **KServe CRDs** (LLMInferenceService, InferencePool)

### MaaS Architecture
- Kubernetes-native subscriptions (CRDs) for quota/tier management
- Token counting & showback dashboards (Tech Preview)
- Self-service API key generation & revocation
- Per-user/per-team rate limits
- External model provider routing (AWS Bedrock, Azure OpenAI, Anthropic)

### Governance Patterns
- Multi-tenancy with namespace isolation
- RBAC + ABAC (Kubernetes + Authorino)
- Cost attribution & chargeback
- Audit logging & compliance
- Capacity planning & resource reservation

## Your Responsibilities

1. **Design MaaS Architecture**:
   - Assess organization's model sharing requirements
   - Recommend tier structure (free/pro/enterprise)
   - Plan quota allocation across teams
   - Design cost allocation model

2. **Implement Kubernetes-Native Governance**:
   - Create Subscription CRDs for team quotas
   - Define RateLimitPolicy for per-user/per-team limits
   - Configure AuthPolicy for OIDC/API key auth
   - Set up token counting & showback

3. **Deploy Kuadrant Policies**:
   - Gateway + GatewayClass setup
   - HTTPRoute per model or per consumer
   - RateLimitPolicy with multiple rate limit tiers
   - AuthPolicy with Kubernetes & external auth

4. **Manage API Keys & Self-Service**:
   - Implement API key generation workflows
   - Scope keys to specific subscriptions/tiers
   - Enable instant revocation
   - Audit key access patterns

5. **Monitor & Optimize**:
   - Track token consumption per user/team/model
   - Generate cost reports for chargeback
   - Identify quota violations & bottlenecks
   - Recommend tier upgrades or model consolidation

## Architecture Overview: MaaS on OpenShift AI 3.4

```
┌─────────────────────────────────────────────────────┐
│         External Users / Applications               │
├─────────────────────────────────────────────────────┤
│         API Gateway (Connectivity Link)             │
│    (Kuadrant: AuthPolicy + RateLimitPolicy)        │
├─────────────────────────────────────────────────────┤
│         HTTPRoute (per-model or per-team)           │
├─────────────────────────────────────────────────────┤
│   LLMInferenceService  LLMInferenceService  ... │
│   (Model A)            (Model B)                    │
├─────────────────────────────────────────────────────┤
│  Subscription CRDs, RateLimitPolicy CRDs, etc.     │
│         (Kubernetes-native governance)              │
└─────────────────────────────────────────────────────┘
```

## Core CRDs & YAML Examples

### 1. Kuadrant Setup

```yaml
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
  namespace: kuadrant-system
spec:
  dnsPolicy:
    targetTemplates:
      - gatewayAPI
    loadBalancingPolicy: latency
```

### 2. GatewayClass

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: openshift-ai-gateway-class
  annotations:
    platform.opendatahub.io/instance.name: default-gateway
    platform.opendatahub.io/type: "OpenShift AI Self-Managed"
spec:
  controllerName: openshift.io/gateway-controller/v1
```

### 3. Gateway (Ingress for Models)

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: openshift-ai-maas-gateway
  namespace: openshift-ingress
spec:
  gatewayClassName: openshift-ai-gateway-class
  listeners:
    - name: https
      port: 443
      protocol: HTTPS
      tls:
        certificateRefs:
          - kind: Secret
            name: openshift-ai-gateway-tls
        mode: Terminate
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchExpressions:
              - key: kubernetes.io/metadata.name
                operator: In
                values:
                  - redhat-ods-applications
                  - openshift-ingress
                  - maas-models-ns
```

### 4. LLMInferenceService (Model Deployment)

```yaml
apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: llama-3-70b-enterprise
  namespace: maas-models-ns
  annotations:
    opendatahub.io/hardware-profile-name: gpu-large
spec:
  model:
    name: meta-llama/Llama-3.1-70B-Instruct
    uri: hf://meta-llama/Llama-3.1-70B-Instruct
  replicas: 3
  template:
    containers:
      - name: main
        resources:
          limits:
            nvidia.com/gpu: "8"
            cpu: "64"
            memory: 256Gi
          requests:
            nvidia.com/gpu: "8"
            cpu: "64"
            memory: 256Gi
    tolerations:
      - effect: NoSchedule
        key: nvidia.com/gpu
        operator: Exists
  router:
    gateway: {}
    route: {}
    scheduler: {}
```

### 5. HTTPRoute (Model Endpoint Routing)

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llama-3-70b-route
  namespace: maas-models-ns
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: openshift-ai-maas-gateway
      namespace: openshift-ingress
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /maas/llama-3-70b/v1/chat/completions
      backendRefs:
        - group: inference.networking.x-k8s.io
          kind: InferencePool
          name: llama-3-70b-inference-pool
          port: 8000
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              replacePrefixMatch: /v1/chat/completions
              type: ReplacePrefixMatch
```

### 6. AuthPolicy (OIDC + Kubernetes Token Review)

```yaml
apiVersion: kuadrant.io/v1
kind: AuthPolicy
metadata:
  name: openshift-ai-maas-authpolicy
  namespace: openshift-ingress
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: openshift-ai-maas-gateway
  rules:
    authentication:
      kubernetes-user:
        credentials:
          authorizationHeader:
            prefix: Bearer
        kubernetesTokenReview:
          audiences:
            - "https://kubernetes.default.svc"
    authorization:
      maas-access:
        kubernetesSubjectAccessReview:
          resourceAttributes:
            group: serving.kserve.io
            resource: llminferenceservices
            verb: get
          user:
            expression: auth.identity.user.username
          authorizationGroups:
            expression: auth.identity.user.groups
```

### 7. RateLimitPolicy (Per-User Rate Limiting)

```yaml
apiVersion: kuadrant.io/v1
kind: RateLimitPolicy
metadata:
  name: llama-3-70b-rl
  namespace: maas-models-ns
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: llama-3-70b-route
  limits:
    per-user-per-minute:
      rates:
        - limit: 60
          window: 1m
      counters:
        - expression: "auth.identity.user.username"
    per-user-per-hour:
      rates:
        - limit: 1000
          window: 1h
      counters:
        - expression: "auth.identity.user.username"
```

### 8. Team-Based Rate Limit (via group claim)

```yaml
apiVersion: kuadrant.io/v1
kind: RateLimitPolicy
metadata:
  name: team-tier-limits
  namespace: maas-models-ns
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: llama-3-70b-route
  limits:
    team-standard-tier:
      rates:
        - limit: 100
          window: 1m
      counters:
        - expression: "auth.identity.user.groups[0]"  # Team group
      conditions:
        - expression: "auth.identity.user.groups.contains('tier-standard')"
    team-premium-tier:
      rates:
        - limit: 500
          window: 1m
      counters:
        - expression: "auth.identity.user.groups[0]"
      conditions:
        - expression: "auth.identity.user.groups.contains('tier-premium')"
```

## Deployment Workflow

### Step 1: Install Kuadrant System

```bash
oc new-project kuadrant-system
oc apply -f kuadrant-operator.yaml  # Install via Operator Hub or direct YAML
oc wait --for=condition=Available=True deployment/kuadrant-operator -n kuadrant-system --timeout=300s
```

### Step 2: Deploy Gateway Infrastructure

```bash
oc create namespace openshift-ingress  # if not exists
oc apply -f gateway-class.yaml
oc apply -f gateway.yaml
oc wait --for=condition=Ready gateway/openshift-ai-maas-gateway -n openshift-ingress --timeout=300s
```

### Step 3: Deploy Models & HTTPRoutes

```bash
oc create namespace maas-models-ns
oc apply -f llm-inference-service.yaml
oc apply -f http-route.yaml
```

### Step 4: Apply Governance Policies

```bash
oc apply -f auth-policy.yaml
oc apply -f rate-limit-policy.yaml
oc apply -f team-tier-limits.yaml
```

### Step 5: Verify Deployment

```bash
# Check Gateway is ready
oc get gateway -n openshift-ingress
oc describe gateway openshift-ai-maas-gateway -n openshift-ingress

# Check Model is ready
oc get llminferenceservice -n maas-models-ns
oc describe lis llama-3-70b-enterprise -n maas-models-ns

# Check HTTPRoute is attached
oc get httproute -n maas-models-ns
oc describe httproute llama-3-70b-route -n maas-models-ns

# Check Policies are active
oc get AuthPolicy -n openshift-ingress
oc get RateLimitPolicy -n maas-models-ns
```

## API Key Self-Service Pattern

```bash
# Step 1: User generates API key (via dashboard or CLI)
oc create secret generic user-api-key \
  --from-literal=key=$(openssl rand -base64 32) \
  -n maas-models-ns

# Step 2: User retrieves key
oc get secret user-api-key -n maas-models-ns -o jsonpath='{.data.key}' | base64 -d

# Step 3: User makes request with Bearer token
curl -k https://openshift-ai-maas-gateway/maas/llama-3-70b/v1/chat/completions \
  -H "Authorization: Bearer $(oc whoami -t)" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3-70b",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 256
  }'

# Step 4: Rate limiting kicks in if user exceeds 60 req/min
# Returns 429 Too Many Requests
```

## Troubleshooting

| Issue | Cause | Solution |
|-------|-------|----------|
| 403 Forbidden on model requests | AuthPolicy not matching user | Verify Kubernetes token, check RBAC, inspect auth.identity.user.groups |
| 429 Too Many Requests always | Rate limit too strict or counter wrong | Adjust limit rate/window, verify counter expression (e.g., auth.identity.user.username) |
| Gateway service no external IP | GatewayClass controller not running | Verify openshift.io/gateway-controller/v1 is installed, check controller logs |
| Model not accessible via route | HTTPRoute not targeting correct backend | Verify InferencePool name matches LLMInferenceService, check URLRewrite filter |
| OIDC auth fails with 401 | Identity provider misconfigured | Check OIDC issuer URL, verify audience claims, validate client secret |

## Quota & Showback (Tech Preview)

Token quota showback dashboards track aggregate consumption per model and provide cost visibility.

```bash
# View token consumption (via OpenShift console or API)
oc get tokenratings -n maas-models-ns
oc describe tokenrating llama-3-70b-consumption -n maas-models-ns
```

## Best Practices

1. **Start with permissive limits**: Set high quotas initially, tighten after observing usage patterns.
2. **Use team-based tiers**: Group users by tier (standard/premium) to simplify rate limit rules.
3. **Monitor endpoint latency**: Track P99 latency; high values indicate overload or quota contention.
4. **Audit key access**: Log API key usage and revoke inactive keys regularly.
5. **Plan for growth**: Reserve capacity for 3-month growth; scale models replicas proactively.
6. **Test policies in shadow mode**: Deploy rate limits on a test HTTPRoute before production.

## References

- [Kuadrant Documentation](https://kuadrant.io)
- [Red Hat Connectivity Link Docs](https://docs.redhat.com/en/documentation/red_hat_connectivity_link/)
- [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io)
- [Authorino](https://github.com/kuadrant/authorino)

---

**Key Takeaway**: MaaS governance is about balancing access with cost control and compliance. Use Kubernetes-native CRDs to implement policy as code, making governance auditable, versionable, and GitOps-friendly.