Oracle Dba Rac High Availability

Oracle DBA - RAC / High Availability

In-DB RAC diagnostics (cluster/interconnect/service/eviction reports via SQLcl MCP dba_ai_conn) plus host-level Grid Infrastructure operations (crsctl / srvctl / ocrconfig / olsnodes) for Oracle 19c and 23ai (EE; single-instance, RAC and Data Guard topologies; on-prem). Everything self-executing is T1 (observe) or T2 (reversible, dry-run-default). Every structural / recovery / fencing action (OCR or voting-disk restore, node add/remove, SCAN identity change, the full AC architecture rollout, eviction remediation) is T3 and lives only in references/runbooks.md — it prints commands for a human and never self-executes.

The GI stack (crsctl, srvctl, ocrconfig, ocrcheck, olsnodes, cluvfy) is a host-level toolset. SQLcl MCP runs in-DB SQL only (restrict level 4 = no host commands), so GI operations ship as guarded .sh / .ps1 (mechanism script:); in-DB visibility ships as sql: run through dba_ai_conn. A future guarded oracle-dba-ops MCP would front exactly these host scripts.

Scope & risk map

Preconditions

• SQLcl MCP connection dba_ai_conn is reachable and wallet-backed. Every sql/*.sql report and every in-DB precheck/postcheck/idempotency probe runs through it via connect_db / Connect-Db (e.g. connect_db rac rac-topology "@sql/rac_topology.sql"). The connection is least-privilege and needs only SELECT on the referenced GV$/V$/DBA_* views and (for service_set_lb_attrs.sql) EXECUTE on DBMS_SERVICE, granted out of band. Never SYS/SYSTEM — except inside a T3 runbook that explicitly states AS SYSDBA, supplied interactively by the operator.
• Grid Infrastructure host tools use OS / Grid-owner (or root) authentication, not DB credentials. crsctl / srvctl / olsnodes / ocrcheck run as the GI owner; ocrconfig -manualbackup and full ocrcheck corruption detail require root. The scripts locate binaries under --grid-home/$ORACLE_HOME/bin or $PATH and never accept DB credentials on argv.
• Secrets resolve from the Oracle Wallet / external password store. No plaintext anywhere. The host scripts pass no secrets; SQLcl connects via the wallet alias dba_ai_conn. Output is run through _odb_redact.
• Helpers are sourced from _common (scripts/lib.sh / lib.ps1): connect_db, log_event, emit_metric, require_dry_run, guard_blocked_dry_run, require_approval_token, precheck, postcheck, print_banner, the EX_* exit codes, and _odb_redact. Not reinvented.
• Logging: structured key=value lines to the canonical paths /var/log/oracle-dba/rac/rac.log (Linux) / C:/ProgramData/oracle-dba/logs/rac/rac.log (Windows); skill segment rac.
• T2 dry-run default: every T2 subcommand previews and changes nothing unless given --execute. remove additionally requires --confirm-remove; OCR manualbackup additionally requires running as root on the --execute path.

Procedures

Host scripts ship as behaviour-equivalent .sh (Linux) / .ps1 (Windows) pairs and default to --dry-run; pass --execute to act (see each script's --help). SQL reports run read-only through dba_ai_conn.

Cluster / topology / service / load-balancing reports (T1, in-DB)

• Trigger: "is this RAC?", roster/thread check, service inventory, LB posture, or a first-look health pass.
• Precheck: none (read-only); dba_ai_conn reachable.
• Action: connect_db rac <task> "@sql/rac_topology.sql" (also cluster_status.sql, service_inventory.sql, services_taf_status.sql, load_balancing_check.sql).
• Postcheck: n/a — reports print sections plus machine-parseable KEYVAL lines.
• Rollback note: none — read-only.

Host Clusterware status snapshot (T1)

• Trigger: GI/CRS troubleshooting; "is the stack up?", resource/SCAN/ASM/votedisk state.
• Precheck: crsctl present (else EX_PRECHECK — not a GI host / wrong --grid-home).
• Action: scripts/clusterware_diag.sh [--grid-home PATH] [--out DIR] / .ps1.
• Postcheck: n/a — writes a fresh timestamped report; a missing command is recorded and the run continues.
• Rollback note: none — every command is a status/query verb; nothing mutates.

Interconnect / Cache Fusion health (T1, in-DB)

• Trigger: slow RAC, "gc" waits, suspected interconnect/NIC/MTU problem.
• Precheck: none (read-only).
• Action: connect_db rac interconnect-health "@sql/interconnect_health.sql".
• Postcheck: read the KEYVAL avg_gc_cr_recv_ms / gc_blocks_lost_total lines; >15 ms or rising lost blocks → escalate to #interconnect-tuning.
• Rollback note: none — read-only. Acting on private NIC/HAIP is T3.

Node-eviction / split-brain forensics (T1)

• Trigger: a node was evicted / rebooted / fenced; ORA-29740; reconfiguration in the alert log.
• Precheck: none (read-only).
• Action: in-DB connect_db rac eviction-forensics "@sql/eviction_forensics.sql" and OS+GI bundle scripts/eviction_evidence.sh [--grid-base PATH] [--hours N] / .ps1.
• Postcheck: correlate timestamps — ocssd.log "network/disk heartbeat" / "missed checkin" is authoritative for the eviction decision; the DB only reacts.
• Rollback note: none — copies log tails only. Remediation is #eviction-remediation.

RAC instance start / stop (T1 status / T2 act)

• Trigger: bring an instance up/down for maintenance; confirm cluster runtime state.
• Precheck: srvctl present; for start/stop a valid -i <instance>; idempotency probe via srvctl status database.
• Action: scripts/rac_instance_control.sh status -d <db> (T1); … start|stop -d <db> -i <inst> [-o immediate] --execute (T2) / .ps1.
• Postcheck: instance reports running (start) / not running (stop).
• Rollback note: reversible by the mirror verb — stop undoes start and vice-versa; the cluster keeps the resource definition.

Cluster-managed service lifecycle (T1 status / T2 add/start/stop/modify/relocate/remove)

• Trigger: define/start/stop/move a service, change preferred/available placement, or retire a service.
• Precheck: srvctl present; valid -d/-s; service_exists / service_running probes; modify needs ≥1 of --preferred/--available/--cardinality; relocate needs --old/--new.
• Action: scripts/rac_service.sh <add|start|stop|modify|relocate|remove> -d <db> -s <svc> [opts] --execute / .ps1. remove also requires --confirm-remove (clients lose the service).
• Postcheck: per verb — service defined / running / stopped / removed.
• Rollback note: each verb has a mirror (add↔remove, start↔stop, relocate back with swapped --old/--new); remove is undone by re-adding the same placement, but in-flight sessions are dropped.

Service failover-attribute (AC / TAF) tuning (T1 show / T2 set)

• Trigger: enable Application Continuity (transactional replay) or classic SELECT TAF on an existing service.
• Precheck: srvctl present; show is read-only; set-* confirms the service exists (config service) and probes current -failovertype for idempotency.
• Action: scripts/configure_service_ac.sh show -d <db> -s <svc> (T1); … set-ac|set-taf -d <db> -s <svc> [--replay-timeout N] [--failover-retry N] --execute (T2) / .ps1.
• Postcheck: -failovertype is now TRANSACTION (AC) / SELECT (TAF); cross-check with sql/services_taf_status.sql.
• Rollback note: reversible — re-run set-taf/set-ac (or srvctl modify service -failovertype NONE per #app-continuity). NOTE: AC/TAF only protects FAN/ONS-aware, replay-capable clients; full client wiring is #app-continuity (T3).

Connection / runtime load-balancing tuning (T1 check / T2 set)

• Trigger: skewed sessions, missing server-side CLB, enable RLB/FAN for an existing service.
• Precheck: scripts/lb_config.sh show -s <svc> (T1) runs sql/load_balancing_check.sql to confirm remote_listener→SCAN and current CLB_GOAL/GOAL/FAN; the set path additionally allow-lists CLB/RLB/FAN and confirms the service exists in DBA_SERVICES BEFORE any DB call.
• Action: scripts/lb_config.sh set -s <svc> -clb LONG|SHORT -rlb NONE|SERVICE_TIME|THROUGHPUT -fan TRUE|FALSE [--execute] / .ps1. This is the gated T2 wrapper: it previews by default (--dry-run) and only on --execute invokes sql/service_set_lb_attrs.sql via dba_ai_conn. The SQL no-ops (prints NOOP) if attributes already match; otherwise DBMS_SERVICE.MODIFY_SERVICE applies them and re-displays. Do not run service_set_lb_attrs.sql directly (ungated mutation).
• Postcheck: the wrapper re-queries DBA_SERVICES to confirm resulting CLB_GOAL/GOAL/FAN; re-run lb_config.sh show to confirm.
• Rollback note: reversible — re-run set with the prior values. For cluster-managed services the cluster (OCR) is the source of truth; the host srvctl modify service -clbgoal/-rlbgoal/-notification path is the preferred mechanism on RAC and is documented in #app-continuity (this SQL path is the single-instance / non-CRS fallback and verification helper).

VIP / SCAN runtime operations (T1 status / T2 act)

• Trigger: drain a node VIP, bounce SCAN listeners, change the SCAN listener port.
• Precheck: srvctl present; valid -n <node> (VIP) or -p <port> 1–65535; vip_running / scan_running / current_scan_port probes for idempotency.
• Action: scripts/vip_scan_config.sh status (T1); … vip-stop|vip-start -n <node> / scan-stop|scan-start / scan-modify-port -p <port> --execute (T2) / .ps1. The port change auto-bounces the SCAN listener.
• Postcheck: VIP/SCAN running or stopped as intended; SCAN listener on the requested port.
• Rollback note: reversible by the mirror verb / re-running with the old port. Adding a NEW node VIP or changing the SCAN name/VIP set is structural → #vip-scan-structural (T3).

OCR backup (T1 list/show / T2 manual backup)

• Trigger: verify OCR integrity, list existing backups, or take a manual OCR backup before a risky GI change.
• Precheck: ocrconfig/ocrcheck present; idempotency = a manual backup younger than --window-hours (default 24) → noop; --execute requires root and a healthy ocrcheck.
• Action: scripts/ocr_backup.sh list|show (T1); scripts/ocr_backup.sh manualbackup [--window-hours N] --execute (T2, root) / .ps1.
• Postcheck: a manual OCR backup now exists within the window.
• Rollback note: a manual backup is write-only — it never alters the live OCR, so there is nothing to undo. Restoring/replacing OCR is T3 → #ocr-voting.

Tier-3 runbooks

All install, structural, recovery, fencing and architecture work is Tier 3 and lives in references/runbooks.md: #cluster-install, #ocr-voting, #eviction-remediation, #node-add-remove, #vip-scan-structural, #app-continuity, #interconnect-tuning, #gi-rolling-restart. A fresh Grid Infrastructure / RAC install (#cluster-install) is root/SYSDBA-interactive and never self-executes — like every other T3 it is plan-only and gated by require_approval_token.

These runbooks PRINT exact crsctl / srvctl / ocrconfig / SQL commands for a human operator and NEVER self-execute. They are gated by require_approval_token / Require-ApprovalToken: an automated agent's only sanctioned action is to print the relevant section and stop with ERR_APPROVAL (6) until a human supplies a real change ticket/token (--token <TICKET> or ODB_APPROVAL_TOKEN; placeholders are rejected). The T2 scripts deliberately refuse the structural escape hatches (OCR restore, new VIP, SCAN identity change, node add/remove) and point to the runbook instead. SYSDBA / root are T3-only, interactively-supplied exceptions. When in doubt, runbook.

See references/version-notes.md for 19c-vs-23ai behavioural deltas.