# Oracle Dba Rac High Availability

> USE THIS SKILL WHENEVER the user mentions RAC, Real Application Clusters, Grid Infrastructure / GI, Clusterware, CRS, OHASD, crsctl / srvctl / olsnodes / cluvfy, OCR (Oracle Cluster Registry) or voting disk, SCAN / SCAN listener, node VIP, cluster interconnect / private network / Cache Fusion / GCS / GES, node eviction or split-brain or reboot/fence, instance start/stop on a cluster, cluster-managed database SERVICES, TAF / Transparent Application Failover, Application Continuity (AC) / FAN / ONS / FCF, connection or runtime load balancing (CLB / RLB), policy-managed vs admin-managed databases, or ANY of the codes CRS-xxxx, ORA-29740 (eviction/reconfiguration), ORA-481/ORA-29702 (CSS/clusterware), ORA-12545/ORA-12516 (SCAN/listener), or "gc blocks lost" / high "gc cr/current block receive time" — EVEN IF they do not name a specific task or tool. Covers read-only cluster/interconnect/service/eviction diagnostics, OCR manual backup, RAC instance and service start/stop/relocate, VIP & SCAN runtime ops, SCAN port

- Skill: `shreyas70773/oracle-dba-rac-high-availability` (Agent Skill, multi-file: 29 files)
- Install (CLI): `npx skillmds@latest add shreyas70773/oracle-dba-rac-high-availability`
- Raw SKILL.md: https://api.skillmd.com/api/skills/shreyas70773/oracle-dba-rac-high-availability/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: Shreyas70773 (https://skillmd.com/u/shreyas70773)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/shreyas70773/oracle-dba-rac-high-availability

---

# Oracle DBA - RAC / High Availability

In-DB RAC diagnostics (cluster/interconnect/service/eviction reports via SQLcl MCP
`dba_ai_conn`) plus host-level Grid Infrastructure operations (`crsctl` / `srvctl` /
`ocrconfig` / `olsnodes`) for Oracle 19c and 23ai (EE; single-instance, RAC and
Data Guard topologies; on-prem). Everything self-executing is **T1 (observe)** or
**T2 (reversible, dry-run-default)**. Every **structural / recovery / fencing**
action (OCR or voting-disk restore, node add/remove, SCAN identity change, the full
AC architecture rollout, eviction remediation) is **T3** and lives only in
`references/runbooks.md` — it prints commands for a human and never self-executes.

> The GI stack (`crsctl`, `srvctl`, `ocrconfig`, `ocrcheck`, `olsnodes`, `cluvfy`)
> is a **host-level** toolset. SQLcl MCP runs in-DB SQL only (restrict level 4 = no
> host commands), so GI operations ship as guarded `.sh` / `.ps1` (mechanism
> `script:`); in-DB visibility ships as `sql:` run through `dba_ai_conn`. A future
> guarded `oracle-dba-ops` MCP would front exactly these host scripts.

## Scope & risk map

| Task | Level | Tier | Mechanism (sqlcl-mcp:sql/… \| script:scripts/… \| runbook:references/runbooks.md#…) | Idempotent? |
|------|-------|------|-------------------------------------------------------------------------------------|-------------|
| RAC cluster + instance topology report | L1 | T1 | `sqlcl-mcp:sql/rac_topology.sql` | yes — read-only |
| In-DB cluster awareness / instance-roster / service status | L1 | T1 | `sqlcl-mcp:sql/cluster_status.sql` | yes — read-only |
| Host GI / Clusterware status snapshot (crsctl/srvctl/olsnodes/ocrcheck/cluvfy) | L1 | T1 | `script:scripts/clusterware_diag.sh` / `.ps1` | yes — read-only (status verbs only; fresh timestamped report) |
| Interconnect + Cache Fusion (GCS/GES) health report | L2 | T1 | `sqlcl-mcp:sql/interconnect_health.sql` | yes — read-only |
| Database service inventory (DBA_SERVICES + active-services + session spread) | L1 | T1 | `sqlcl-mcp:sql/service_inventory.sql` | yes — read-only |
| Services / TAF / Application-Continuity in-DB attribute report | L1 | T1 | `sqlcl-mcp:sql/services_taf_status.sql` | yes — read-only |
| Connection/runtime load-balancing posture check (remote_listener, CLB/GOAL/FAN) | L2 | T1 | `sqlcl-mcp:sql/load_balancing_check.sql` | yes — read-only |
| Node-eviction / split-brain in-DB forensics (alert-log + GES reconfig timeline) | L2 | T1 | `sqlcl-mcp:sql/eviction_forensics.sql` | yes — read-only |
| Node-eviction / split-brain OS+GI evidence bundle (ocssd/crsd/cssd logs, OS messages) | L2 | T1 | `script:scripts/eviction_evidence.sh` / `.ps1` | yes — read-only (copies log tails; fresh timestamped bundle) |
| RAC instance status | L1 | T1 | `script:scripts/rac_instance_control.sh status` / `.ps1 status` | yes — read-only |
| RAC instance start / stop (srvctl) | L1 | T2 | `script:scripts/rac_instance_control.sh start\|stop` / `.ps1` | yes — running→start noop; down→stop noop |
| Cluster-managed service status | L1 | T1 | `script:scripts/rac_service.sh status` / `.ps1 status` | yes — read-only |
| Cluster-managed service add / start / stop / modify / relocate (srvctl) | L2 | T2 | `script:scripts/rac_service.sh add\|start\|stop\|modify\|relocate` / `.ps1` | yes — existing/running/stopped → noop |
| Cluster-managed service remove (srvctl) | L2 | T2 | `script:scripts/rac_service.sh remove` / `.ps1` (needs `--execute` + `--confirm-remove`) | yes — absent → noop |
| Service failover-attribute show (srvctl config + in-DB) | L1 | T1 | `script:scripts/configure_service_ac.sh show` / `.ps1 show` | yes — read-only |
| Service Application-Continuity (TRANSACTION replay) attribute set | L2 | T2 | `script:scripts/configure_service_ac.sh set-ac` / `.ps1` | yes — already TRANSACTION → noop |
| Service classic SELECT-TAF attribute set | L2 | T2 | `script:scripts/configure_service_ac.sh set-taf` / `.ps1` | yes — already SELECT → noop |
| Service CLB_GOAL / GOAL(RLB) / FAN tuning (in-DB, DBMS_SERVICE) | L2 | T2 | `script:scripts/lb_config.sh set` / `.ps1` (gated wrapper; runs `sql/service_set_lb_attrs.sql` only on `--execute`) | yes — already matching → NOOP |
| Connection/runtime load-balancing posture check (via wrapper) | L2 | T1 | `script:scripts/lb_config.sh show` / `.ps1 show` (runs `sql/load_balancing_check.sql`) | yes — read-only |
| VIP / SCAN / SCAN-listener status | L1 | T1 | `script:scripts/vip_scan_config.sh status` / `.ps1 status` | yes — read-only |
| Node VIP stop / start (srvctl) | L2 | T2 | `script:scripts/vip_scan_config.sh vip-stop\|vip-start` / `.ps1` | yes — already up/down → noop |
| SCAN listeners stop / start (srvctl) | L2 | T2 | `script:scripts/vip_scan_config.sh scan-stop\|scan-start` / `.ps1` | yes — already up/down → noop |
| SCAN listener PORT change (srvctl, reversible) | L2 | T2 | `script:scripts/vip_scan_config.sh scan-modify-port` / `.ps1` | yes — already on port → noop |
| OCR backup list / integrity show (ocrconfig/ocrcheck) | L2 | T1 | `script:scripts/ocr_backup.sh list\|show` / `.ps1` | yes — read-only |
| OCR manual backup (ocrconfig -manualbackup; write-only, root) | L2 | T2 | `script:scripts/ocr_backup.sh manualbackup` / `.ps1` | yes — recent manual backup within `--window-hours` → noop |
| RAC cluster installation & configuration (Grid Infrastructure install + DB/cluster config) | L3 | **T3** | `runbook:references/runbooks.md#cluster-install` | no — fresh GI/RAC install (gridSetup/root scripts) |
| OCR / voting-disk restore, replace, repair, or move | structural | **T3** | `runbook:references/runbooks.md#ocr-voting` | no — replaces cluster registry / quorum |
| Node eviction / split-brain remediation (interconnect/fence/reboot recovery) | recovery | **T3** | `runbook:references/runbooks.md#eviction-remediation` | no — operates on a fractured/fenced cluster |
| Add / delete a cluster node (addnode / deletenode / GI reconfig) | structural | **T3** | `runbook:references/runbooks.md#node-add-remove` | no — changes cluster membership |
| Add a NEW node VIP / change SCAN NAME or SCAN VIP set | structural | **T3** | `runbook:references/runbooks.md#vip-scan-structural` | no — changes network identity / DNS-GNS |
| Application Continuity end-to-end architecture rollout (service create + client/UCP/JDBC-replay + drain for planned switch) | architecture | **T3** | `runbook:references/runbooks.md#app-continuity` | partial — attribute set (above) is T2; full rollout + client wiring is T3 |
| Interconnect / Cache Fusion remediation (change private NIC, HAIP, jumbo frames) | structural | **T3** | `runbook:references/runbooks.md#interconnect-tuning` | no — touches OS network / GI private interconnect config |
| Rolling GI / RU patch, GI restart/relocate of all resources | structural | **T3** | `runbook:references/runbooks.md#gi-rolling-restart` | no — cluster-wide; coordinate with patching skill |

## Preconditions

- **SQLcl MCP connection `dba_ai_conn`** is reachable and wallet-backed. Every
  `sql/*.sql` report and every in-DB precheck/postcheck/idempotency probe runs
  through it via `connect_db` / `Connect-Db` (e.g.
  `connect_db rac rac-topology "@sql/rac_topology.sql"`). The connection is
  least-privilege and needs only SELECT on the referenced `GV$/V$/DBA_*` views and
  (for `service_set_lb_attrs.sql`) EXECUTE on `DBMS_SERVICE`, granted out of band.
  Never SYS/SYSTEM — except inside a T3 runbook that explicitly states `AS SYSDBA`,
  supplied interactively by the operator.
- **Grid Infrastructure host tools use OS / Grid-owner (or root) authentication, not
  DB credentials.** `crsctl` / `srvctl` / `olsnodes` / `ocrcheck` run as the GI
  owner; `ocrconfig -manualbackup` and full `ocrcheck` corruption detail require
  **root**. The scripts locate binaries under `--grid-home`/`$ORACLE_HOME/bin` or
  `$PATH` and **never** accept DB credentials on argv.
- **Secrets resolve from the Oracle Wallet / external password store. No plaintext
  anywhere.** The host scripts pass no secrets; SQLcl connects via the wallet alias
  `dba_ai_conn`. Output is run through `_odb_redact`.
- **Helpers are sourced from `_common`** (`scripts/lib.sh` / `lib.ps1`): `connect_db`,
  `log_event`, `emit_metric`, `require_dry_run`, `guard_blocked_dry_run`,
  `require_approval_token`, `precheck`, `postcheck`, `print_banner`, the `EX_*` exit
  codes, and `_odb_redact`. Not reinvented.
- **Logging**: structured `key=value` lines to the canonical paths
  `/var/log/oracle-dba/rac/rac.log` (Linux) /
  `C:/ProgramData/oracle-dba/logs/rac/rac.log` (Windows); skill segment `rac`.
- **T2 dry-run default**: every T2 subcommand previews and changes nothing unless
  given `--execute`. `remove` additionally requires `--confirm-remove`; OCR
  `manualbackup` additionally requires running as root on the `--execute` path.

## Procedures

Host scripts ship as behaviour-equivalent `.sh` (Linux) / `.ps1` (Windows) pairs and
default to `--dry-run`; pass `--execute` to act (see each script's `--help`). SQL
reports run read-only through `dba_ai_conn`.

### Cluster / topology / service / load-balancing reports (T1, in-DB)
- **Trigger**: "is this RAC?", roster/thread check, service inventory, LB posture, or a first-look health pass.
- **Precheck**: none (read-only); `dba_ai_conn` reachable.
- **Action**: `connect_db rac <task> "@sql/rac_topology.sql"` (also `cluster_status.sql`, `service_inventory.sql`, `services_taf_status.sql`, `load_balancing_check.sql`).
- **Postcheck**: n/a — reports print sections plus machine-parseable `KEYVAL` lines.
- **Rollback note**: none — read-only.

### Host Clusterware status snapshot (T1)
- **Trigger**: GI/CRS troubleshooting; "is the stack up?", resource/SCAN/ASM/votedisk state.
- **Precheck**: `crsctl` present (else `EX_PRECHECK` — not a GI host / wrong `--grid-home`).
- **Action**: `scripts/clusterware_diag.sh [--grid-home PATH] [--out DIR]` / `.ps1`.
- **Postcheck**: n/a — writes a fresh timestamped report; a missing command is recorded and the run continues.
- **Rollback note**: none — every command is a status/query verb; nothing mutates.

### Interconnect / Cache Fusion health (T1, in-DB)
- **Trigger**: slow RAC, "gc" waits, suspected interconnect/NIC/MTU problem.
- **Precheck**: none (read-only).
- **Action**: `connect_db rac interconnect-health "@sql/interconnect_health.sql"`.
- **Postcheck**: read the `KEYVAL avg_gc_cr_recv_ms` / `gc_blocks_lost_total` lines; `>15 ms` or rising lost blocks → escalate to `#interconnect-tuning`.
- **Rollback note**: none — read-only. Acting on private NIC/HAIP is T3.

### Node-eviction / split-brain forensics (T1)
- **Trigger**: a node was evicted / rebooted / fenced; ORA-29740; reconfiguration in the alert log.
- **Precheck**: none (read-only).
- **Action**: in-DB `connect_db rac eviction-forensics "@sql/eviction_forensics.sql"` **and** OS+GI bundle `scripts/eviction_evidence.sh [--grid-base PATH] [--hours N]` / `.ps1`.
- **Postcheck**: correlate timestamps — ocssd.log "network/disk heartbeat" / "missed checkin" is **authoritative** for the eviction decision; the DB only reacts.
- **Rollback note**: none — copies log tails only. Remediation is `#eviction-remediation`.

### RAC instance start / stop (T1 status / T2 act)
- **Trigger**: bring an instance up/down for maintenance; confirm cluster runtime state.
- **Precheck**: `srvctl` present; for start/stop a valid `-i <instance>`; idempotency probe via `srvctl status database`.
- **Action**: `scripts/rac_instance_control.sh status -d <db>` (T1); `… start|stop -d <db> -i <inst> [-o immediate] --execute` (T2) / `.ps1`.
- **Postcheck**: instance reports running (start) / not running (stop).
- **Rollback note**: reversible by the mirror verb — `stop` undoes `start` and vice-versa; the cluster keeps the resource definition.

### Cluster-managed service lifecycle (T1 status / T2 add/start/stop/modify/relocate/remove)
- **Trigger**: define/start/stop/move a service, change preferred/available placement, or retire a service.
- **Precheck**: `srvctl` present; valid `-d`/`-s`; `service_exists` / `service_running` probes; `modify` needs ≥1 of `--preferred/--available/--cardinality`; `relocate` needs `--old`/`--new`.
- **Action**: `scripts/rac_service.sh <add|start|stop|modify|relocate|remove> -d <db> -s <svc> [opts] --execute` / `.ps1`. `remove` also requires `--confirm-remove` (clients lose the service).
- **Postcheck**: per verb — service defined / running / stopped / removed.
- **Rollback note**: each verb has a mirror (add↔remove, start↔stop, relocate back with swapped `--old`/`--new`); `remove` is undone by re-adding the same placement, but in-flight sessions are dropped.

### Service failover-attribute (AC / TAF) tuning (T1 show / T2 set)
- **Trigger**: enable Application Continuity (transactional replay) or classic SELECT TAF on an existing service.
- **Precheck**: `srvctl` present; `show` is read-only; `set-*` confirms the service exists (`config service`) and probes current `-failovertype` for idempotency.
- **Action**: `scripts/configure_service_ac.sh show -d <db> -s <svc>` (T1); `… set-ac|set-taf -d <db> -s <svc> [--replay-timeout N] [--failover-retry N] --execute` (T2) / `.ps1`.
- **Postcheck**: `-failovertype` is now TRANSACTION (AC) / SELECT (TAF); cross-check with `sql/services_taf_status.sql`.
- **Rollback note**: reversible — re-run `set-taf`/`set-ac` (or `srvctl modify service -failovertype NONE` per `#app-continuity`). NOTE: AC/TAF only protects FAN/ONS-aware, replay-capable clients; full client wiring is `#app-continuity` (T3).

### Connection / runtime load-balancing tuning (T1 check / T2 set)
- **Trigger**: skewed sessions, missing server-side CLB, enable RLB/FAN for an existing service.
- **Precheck**: `scripts/lb_config.sh show -s <svc>` (T1) runs `sql/load_balancing_check.sql` to confirm `remote_listener`→SCAN and current `CLB_GOAL`/`GOAL`/`FAN`; the `set` path additionally allow-lists CLB/RLB/FAN and confirms the service exists in `DBA_SERVICES` BEFORE any DB call.
- **Action**: `scripts/lb_config.sh set -s <svc> -clb LONG|SHORT -rlb NONE|SERVICE_TIME|THROUGHPUT -fan TRUE|FALSE [--execute]` / `.ps1`. This is the **gated T2 wrapper**: it previews by default (`--dry-run`) and only on `--execute` invokes `sql/service_set_lb_attrs.sql` via `dba_ai_conn`. The SQL no-ops (prints `NOOP`) if attributes already match; otherwise `DBMS_SERVICE.MODIFY_SERVICE` applies them and re-displays. Do **not** run `service_set_lb_attrs.sql` directly (ungated mutation).
- **Postcheck**: the wrapper re-queries `DBA_SERVICES` to confirm resulting `CLB_GOAL/GOAL/FAN`; re-run `lb_config.sh show` to confirm.
- **Rollback note**: reversible — re-run `set` with the prior values. For **cluster-managed** services the cluster (OCR) is the source of truth; the host `srvctl modify service -clbgoal/-rlbgoal/-notification` path is the preferred mechanism on RAC and is documented in `#app-continuity` (this SQL path is the single-instance / non-CRS fallback and verification helper).

### VIP / SCAN runtime operations (T1 status / T2 act)
- **Trigger**: drain a node VIP, bounce SCAN listeners, change the SCAN listener port.
- **Precheck**: `srvctl` present; valid `-n <node>` (VIP) or `-p <port>` 1–65535; `vip_running` / `scan_running` / `current_scan_port` probes for idempotency.
- **Action**: `scripts/vip_scan_config.sh status` (T1); `… vip-stop|vip-start -n <node>` / `scan-stop|scan-start` / `scan-modify-port -p <port>` `--execute` (T2) / `.ps1`. The port change auto-bounces the SCAN listener.
- **Postcheck**: VIP/SCAN running or stopped as intended; SCAN listener on the requested port.
- **Rollback note**: reversible by the mirror verb / re-running with the old port. Adding a NEW node VIP or changing the SCAN **name/VIP set** is structural → `#vip-scan-structural` (T3).

### OCR backup (T1 list/show / T2 manual backup)
- **Trigger**: verify OCR integrity, list existing backups, or take a manual OCR backup before a risky GI change.
- **Precheck**: `ocrconfig`/`ocrcheck` present; idempotency = a manual backup younger than `--window-hours` (default 24) → noop; `--execute` requires **root** and a healthy `ocrcheck`.
- **Action**: `scripts/ocr_backup.sh list|show` (T1); `scripts/ocr_backup.sh manualbackup [--window-hours N] --execute` (T2, root) / `.ps1`.
- **Postcheck**: a manual OCR backup now exists within the window.
- **Rollback note**: a manual backup is write-only — it never alters the live OCR, so there is nothing to undo. **Restoring/replacing** OCR is T3 → `#ocr-voting`.

## Tier-3 runbooks

All **install, structural, recovery, fencing and architecture** work is Tier 3 and
lives in **`references/runbooks.md`**: `#cluster-install`, `#ocr-voting`,
`#eviction-remediation`, `#node-add-remove`, `#vip-scan-structural`,
`#app-continuity`, `#interconnect-tuning`, `#gi-rolling-restart`. A fresh Grid
Infrastructure / RAC install (`#cluster-install`) is root/SYSDBA-interactive and
**never self-executes** — like every other T3 it is plan-only and gated by
`require_approval_token`.

These runbooks **PRINT exact crsctl / srvctl / ocrconfig / SQL commands for a human
operator and NEVER self-execute.** They are gated by `require_approval_token` /
`Require-ApprovalToken`: an automated agent's only sanctioned action is to print the
relevant section and stop with `ERR_APPROVAL` (6) until a human supplies a real
change ticket/token (`--token <TICKET>` or `ODB_APPROVAL_TOKEN`; placeholders are
rejected). The T2 scripts deliberately refuse the structural escape hatches (OCR
restore, new VIP, SCAN identity change, node add/remove) and point to the runbook
instead. SYSDBA / root are T3-only, interactively-supplied exceptions. When in
doubt, runbook.

See `references/version-notes.md` for 19c-vs-23ai behavioural deltas.